Arbitrary Taxonomies and The Illusion of Precision (Pt 2)

27 04 2010

In my first post on this subject, I made a fuss about how we fool ourselves into thinking we’ve made imprecise things precise by assigning decimal scores without numerical data and attempting to rank in order more objects than we can even think about at once.  The rant want was inspired by some insights I received from Getting Things Done and its approach to filing, but before returning to that book I wish to discuss another one.

One of my favorite books about living in reality (much more favorite than Getting Things Done) is The Thinker’s Toolkit, by former CIA analyst Morgan Jones.  The Toolkit is a compendium of 14 techniques for adding precision to the imprecise task of decision making.  A number of these techniques, like decision matrices, should not be foreign to most technical people and many non technical people may be familiar with them as well.  There are additional techniques with which I was not familiar, but they all share a common theme: find a way to structure your thoughts before they get the best of you.

One of the simplest tools in the toolkit was Jones’ pair-wise ranking system.  This is another moment when I had a revelation about something I should’ve already known.  Not only is pair-wise ranking simple and logical, it is the first sorting technique you learn in your kindergarten programming class.  The idea is that you cannot take ten object and rank them one to ten in a single pass.  That is, you cannot look at the entire group at once and assign rankings.  You need to simplify the problem by examining items two at a time.  Compare item A to item B against a single attribute, then compare item A to item C against the same attribute.  Continue until you’ve done all pair-wise combinations.  If you need to rank the same items against multiple attributes, you can use the same sorting against each attribute and then sort the attributes themselves to obtain weights.  It can all be rather labor intensive, but it may be the only way to achieve some kind of precision when you’re performing subjective evaluations.  It’s not appropriate for all situations, but it’s the only appropriate method I know of for situations that demand that kind of rigor.

I’m on a bit of a tangent here, but I wanted to remark on how this scheme becomes counter-intuitive when we apply is to people.  It all makes sense when the items in question are, say, make/buy alternatives.  However, when ranking people it contradicts our instincts for judging each person on their own merits.  Everyone from Olympic judges to creative writing teachers attempts to divide artistic performances into technical components and then use a combination of experience and recommend practices within the industry to apply scores to individuals.  They are attempting to arrives each person’s score in a way that is not biases by the performance of others in the same event or class.  This is important because it is difficult not to be influence by the order in which you grade individuals.  I think, however, that comparing individuals is precisely what you need to do in these situations.  Despite the number of technical categories in which you divide an artistic work, there is no way of avoiding grading on a scale.  So one might as well use pair-wise sorting to remove the bias of order.  It seems unfair and undemocratic, but it’s probably more likely to yield an accurate ranking than methods currently in use.

I know I promised to wrap up this topic in the current post, but life events (the birth of a little one) have prevented that.  So I will have to delay the conclusion (and the return to my main topic) for a third post.  At that time I will bring us back to the practice of software engineering and, of course, alphabetic sorting.

Arbitrary Taxonomies and The Illusion of Precision (Pt 1)

18 04 2010

I’m often amazed at how the insights I mine from books suffer from an embarrassing lack of sophistication (my own, not the books’).  I recently completed Getting Things Done, a popular productivity self-help book and reading assignment for a management class I took.  The best thing about most business books is that their titles don’t beat around the bush.  They aren’t written for impressing people with clever turns of phrase; they’re written for busy people who walk into a book store and ask for “that book that will help me get stuff done” or “that book that will teach me the secrets of highly effective individuals.”

Since the management crowd is very sophisticated and wordly sort, their literature is supposed to posses what politicians call gravitas.  That leaves me in the awkward position of admitting that yes, I read Getting Things Done, but the the best thing I learned from it is that I should store all my reference material using a single alphabetic storage system.  Is owning up to learning seventh grade level organizational skills from one of the most popular executive handbooks worse than just saying that I never read the thing at all?  After all, in the latter case I can perpetually complain that I am so busy (because I’m so important) that I haven’t gotten around to it.  In the former case I am revealing myself as the dog who just drank the single malt you’ve been saving.  Then again, maybe the corner office set just isn’t as polished as they appeared from down the hall.

I can help it.  I love alphabetic filing.  And I love that Getting Things Done reminded me that I love alphabetic filing.  And I also love that alphabetic filing  has made me feel that I may just be wiser than lots of people who are smarter than me, from programming language designers to Olympic judges.  And I always love a book that makes me think I’m smart.

The modern world urges us all toward obsessive compulsiveness.  I know you think you’re the easy going guy or girl who doesn’t care if the pictures hang straight, but our collective corporate culture may have you so confused that you don’t even recognize the other OCD gremlins who might be barking up your tree.  Let me give you an example.  Rate the movie “Avatar” on a scale of 1-5 stars, with 5 being the best rank.  Was there any part of you that was compelled to saying something like “3.5 stars” (or “1.5 stars”)?  If not, can you think of anyone you know that you’re sure would respond with a number that includes a decimal point?  I’m sure that if you didn’t put it in there, you can think of more than a few people who just might.

Why is that? I asked you to rank something using stars, which are objects that come in whole numbers.  What’s 3.5 stars supposed to be?  Were you not satisfied with the choices I gave you?  Even giving you 5 stars was too much.  The idea that the human mind can rank anything as subjective as movie quality with anything that even approaches the need for fractional increments is absurd – and a little silly too.  If you require more than 4 or 5 choices to produce a ranking that you cannot base on quantifiable numbers, you are deluded into believing your judgments posses more precision than is biologically feasible and should therefore forfeit your signed copy of Good To Great.

I’m teasing you, but I hope you can see my point.  We know that we cannot hold more and five or six things at once in our short term memory.  And that’s just for simple memorization.  We probably need to reduce that number if we’re expected to process these entities.  Assigning rankings, considering alternatives and all the other tasks that go along with making decisions with our knowledge can only be processed in small chunks and then synthesized into a whole.  Each step of the way, we need to identify the appropriate way to process each chunk.  If the chunk can be evaluated objectively, then hooray.  If not, then we need a means of simplifying the problem and limit the unavoidable effects of subjectivity.

In my next post, I’ll talk about how I learned to do that, and what all of this has to do with alphabetic filing.

Defect Tracking Tools Sometimes Inhibit the Scientific Method

15 04 2010
Most defect tracking tools are not made for investigating defects.  They are essential for cataloguing the solution to a problem and  track its integration into the code base.  This helps managers review proposed changes, track defect density by various categories, monitor employee workload, and report to their keepers what all of these bugs are going to cost.  This is only the small list of all the useful statistics a well stocked defect tracker can provide to your manager.
However, I have found these tools a burden for the developer.  For example, can they record symptoms without a known cause?  Frequently, the same defect may cause multiple symptoms or multiple defects may result in the same symptom.  None of this is known at the time the symptom is observed, but defect trackers want to bind a single set of symptoms to a single cause.  Their capabilities for merging or linking multiple entries are crude at best.  When such capabilities are provided, they are again oriented more for the management viewpoint than the engineering viewpoint.
The same is true for a defect tracker’s search capabilities.  If I want statistics, then there is no end to the ways a sophisticated tracking tool will allow me to cut and sort data.  Do you want to know how many defects the code base acquire between March 11 and April 17 of last year that cost more than 40 hours to fix and resulted from requirements modifications?  I can get you that in a second.  Did you just encounter a symptom that vaguely resembles a something you saw a year ago?  Do you want me to determine whether or not the conditions under which that symptom was originally observed are the same as the conditions under which you are now operating?  You’re out of luck.  Defect trackers are, after all, mostly a webby veneer stretched over a bunch of RDB tables that are hidden from the user. All that information about the actual test case was shoved into the text of the defect description and we have no way of codifying test scenarios the way we would if both managers and tools developers treated us like real scientists.  We aren’t, of course, but that’s no reason not to aim high.

Management is Engineering

10 04 2010

I recently completed a software management course. I am a technical person who likes being a technical person, but as an architect I have enough responsibility to justify learning how the other half lives. Almost everyone managing engineering projects at any level below that of a VP (and many VP’s too) has a technical background. Many of them, however, have let their degrees expire while they’ve pursued other interests. There’s nothing wrong with this, but I find it curious that many in upper management make the same silly assumption that many individual contributors make: that management and engineering are somehow mutually exclusive.
Most of my classmates were not technical leads. Most were product development leads and control account managers. Many were on the track to become program managers, directors and the like. I may have been the only technical lead. I’m not sure. However, in the class exit interview I was asked about my decision to take the class, considering my role. My interviewer didn’t think it was inappropriate for me; indeed she was happy I enrolled. What she wanted to know was the secret to getting more technical people involved in these sorts of classes. It seems that managers sometimes view true-to-the-bone engineers – the sort who would never considering entering management – a bit like parents see their teenage children. What makes them tick? How do you put a finger on those ineffable qualities that make something or someone “cool”? Why are they so sullen all the time? How do we keep their attention and get them to respect us? And why do they sleep all day?
I consider myself one of those true blue technical folks. I’ve always loved math and science. I was never undeclared in college, never unsure of my general direction in life. I have seen most episodes of Star Trek several times. Basically, I’m a rather boring stereotypical geek. And I swore to myself I would never become Dilbert’s pointy-haired boss. But these things creep up on you, like the dawning realization that what you think of as the “new” Perl Jam record came out ten years ago. Do you want to know my answer to why more technical people don’t take management classes? Managers, like parents, have to realize that it is not their job to be cool. They should revel in their dorky middle-agedness. What engineers have to realize is that they’ve been looking at management all wrong.
Isn’t management just another form of engineering? Besides the human resources issues, engineering management is about solving technical problems using data. When done right, they involve no fewer neurons than more typical engineering problems. And they are no less essential just because the output is a schedule or a budget. Often the output is a work flow of sort sort, which even makes the exercise close to programming. We do have a name for this, after all, and we call it software engineering. Programming is something you can do by yourself. Software engineering requires an organization that is smart about process, and every process needs its champions.
The experience that first changed my attitude was my first attempt at an integrated schedule and budget. My initial approach to the task was bravado. I said to myself, “I have an advanced degree; if I can survive non linear dynamical systems I can fill out some dates and dollars.” We’re the engineers; we’re the gifted wunderkinds who carry the company on our backs. However, my first pass through the numbers reminded me of a Rubik cube, or one of those puzzles where you have to slide the tiles around to make a pictures. My frustration came from all the effort I put in to creating tidy patches of sense, only to watch them melt away when I tried the make the whole story hang together. Have you ever seen some with a half-solve Rubik cube that he proudly left on a shelf, never to be touched again? He managed to massage one side into a solid color and settled for half a victory rather than risk total failure by reaching for the complete solution. That was me.
And it was my own fault. All I did at first was feed numbers into the beast, willy-nilly. I tried one thing, and if that didn’t work I tried something else. Is that how my momma taught me to solve problems? I realized that somewhere along the way, I lost my discipline. Although engineers might experiment from time to time, they don’t usually come across coherent designs by just trying stuff until it works. Nobody has enough monkeys or enough typewriters. After years of practice at creating preliminary designs, writing test cases and refining my code bases, I through away all my experience the moment I encountered a problem that didn’t strike me as “an engineering problem.” I should’ve known better, considering that I did take operations research once upon a time. Not only did my efforts frustrate me, they’d bored me. I told myself, this is proof that you’re not cut out for this.
But then I saw how it could be solved by analyzing the pieces and working toward a solution using a plan. And wouldn’t you know, I wasn’t the first person to put together a budget and schedule scheme? There were techniques, and those techniques bore an uncanny resemblance to my old friend, engineering. I look upon my former self with embarrassment. I can barely bring myself to admit my naivete. Really. It was, however, an enlightening mistake. It was not an example of stupidity, foolishness or even inexperience. It was an example of how the categories with which we use to organize the world prevent us from seeing how alike we may be to people we regard as our opposites