Tony Russell-Rose is director of UXLabs, a UX research and design consultancy specializing in complex search and information access applications. Previously Tony has led R&D teams at Canon, Reuters, HP Labs and BT Labs, and seems happy to work for pretty much any organization that has 'Labs' in the title. He has a PhD in Artificial Intelligence and is author of Designing the Search Experience (Morgan Kaufmann, 2012). Tony is a DZone MVB and is not an employee of DZone and has posted 28 posts at DZone. You can read more from them at their website. View Full User Profile

Ask DZ: How do you compare two text classfiers?

05.06.2012
| 4354 views |
  • submit to reddit

I need to compare two text classifiers – one human, one machine. They are assigning multiple tags from an ontology. We have an initial corpus of ~700 records tagged by both classifiers. The goal is to measure the ‘value added’ by the human. However, we don’t yet have any ground truth data (i.e. agreed annotations).

Any ideas on how best to approach this problem in a commercial environment (i.e. quickly, simply, with minimum fuss), or indeed what’s possible?

I thought of measuring the absolute delta between the two profiles (regardless of polarity) to give a ceiling on the value added, and/or comparing the profile of tags added by each human coder against the centroid to give a crude measure of inter-coder agreement (and hence difficulty of the task). But neither really measures the ‘value added’ that I’m looking for, so I’m sure there must better solutions.

Suggestions, anyone? Or is this as far as we can go without ground truth data?

Published at DZone with permission of Tony Russell-rose, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)