Arbitrary Taxonomies and The Illusion of Precision (Pt 2)

27 04 2010

In my first post on this subject, I made a fuss about how we fool ourselves into thinking we’ve made imprecise things precise by assigning decimal scores without numerical data and attempting to rank in order more objects than we can even think about at once.  The rant want was inspired by some insights I received from Getting Things Done and its approach to filing, but before returning to that book I wish to discuss another one.

One of my favorite books about living in reality (much more favorite than Getting Things Done) is The Thinker’s Toolkit, by former CIA analyst Morgan Jones.  The Toolkit is a compendium of 14 techniques for adding precision to the imprecise task of decision making.  A number of these techniques, like decision matrices, should not be foreign to most technical people and many non technical people may be familiar with them as well.  There are additional techniques with which I was not familiar, but they all share a common theme: find a way to structure your thoughts before they get the best of you.

One of the simplest tools in the toolkit was Jones’ pair-wise ranking system.  This is another moment when I had a revelation about something I should’ve already known.  Not only is pair-wise ranking simple and logical, it is the first sorting technique you learn in your kindergarten programming class.  The idea is that you cannot take ten object and rank them one to ten in a single pass.  That is, you cannot look at the entire group at once and assign rankings.  You need to simplify the problem by examining items two at a time.  Compare item A to item B against a single attribute, then compare item A to item C against the same attribute.  Continue until you’ve done all pair-wise combinations.  If you need to rank the same items against multiple attributes, you can use the same sorting against each attribute and then sort the attributes themselves to obtain weights.  It can all be rather labor intensive, but it may be the only way to achieve some kind of precision when you’re performing subjective evaluations.  It’s not appropriate for all situations, but it’s the only appropriate method I know of for situations that demand that kind of rigor.

I’m on a bit of a tangent here, but I wanted to remark on how this scheme becomes counter-intuitive when we apply is to people.  It all makes sense when the items in question are, say, make/buy alternatives.  However, when ranking people it contradicts our instincts for judging each person on their own merits.  Everyone from Olympic judges to creative writing teachers attempts to divide artistic performances into technical components and then use a combination of experience and recommend practices within the industry to apply scores to individuals.  They are attempting to arrives each person’s score in a way that is not biases by the performance of others in the same event or class.  This is important because it is difficult not to be influence by the order in which you grade individuals.  I think, however, that comparing individuals is precisely what you need to do in these situations.  Despite the number of technical categories in which you divide an artistic work, there is no way of avoiding grading on a scale.  So one might as well use pair-wise sorting to remove the bias of order.  It seems unfair and undemocratic, but it’s probably more likely to yield an accurate ranking than methods currently in use.

I know I promised to wrap up this topic in the current post, but life events (the birth of a little one) have prevented that.  So I will have to delay the conclusion (and the return to my main topic) for a third post.  At that time I will bring us back to the practice of software engineering and, of course, alphabetic sorting.




Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: