How About Defect Tracking With Expert Systems?

20 07 2010

In an earlier post I discussed an architecture for defect tracking tools which would separate the concepts of symptoms and causes, thus allowing the tool to more closely resemble how problems manifest in real software and how engineering typically approach those problems. I mentioned the need for such a tool to link to a scientific database, or electronic lab notebook. This is necessary because the way we organize test data during the investigation of a macro problem is different then way we track those problems. I also mentioned that the architecture would make it easier to role the results of defect tracking into a larger knowledge base, allowing engineers to research issues they’re having by referring to a collection of previously observed symptoms. It occurs to me now that what I was describing was the beginning of a merger between defect tracking tools and expert systems.
Is this done anywhere? There are lots of expert systems used for diagnostics and there are lots of bug trackers, but are there systems that combine features of the two? The more I think about it, the more it seems like this is the only reasonable way to practice software engineering.
One way to approach the issue to integrate a defect tracker and an expert system database as two distinct but closely coupled tools. If I were using the architecture I described earlier, I would start by going to my expert system first. I would investigate my symptoms and determine whether or not there’s already a solution for my problem. That is, maybe my symptoms do not represent a software defect at all. Rather, diagnostics reveal a fault in hardware or installation parameters. If I don’t find a known solution, then the process of exercising the expert system has got me started on the path of breaking down my problem into well defined pieces. This will help me to design tests for further investigation and it will provide the basis for a new addition to the expert knowledge base once I’ve diagnosed my bug the hard way.
Upon exiting the expert system without a solution, I would turn to the bug tracker to document what I know. Maybe when I started I thought I had a single symptom, but my tour though the knowledgebase gave me a new perspective and I create multiple symptoms. This will make it easier to migrate the information back into the knowledgebase when I’ve fixed the bug, and I can still link all of these symptoms to a single task. Remember, the primary goal of a defect tracker is to allow me to manage my work.
To make life easier, maybe my defect tracker has a means of storing symptom data as expert system entities. That I can move back and forth between the two tools in native format rather than, say, cutting and pasting text between them. That doesn’t mean I should shy away from free form text. My defect tracker should have room for this too, since in the early stages of problem investigation I might want to brain dump a lot of random thoughts and observations. The same can be said for the objects storing the causes.
What I’ve constructed here is a trio of inter-related tools: a defect tracker, an expert system knowledgebase and an electronic lab notebook. All three share a two-way relationship with the other two. Clearly, the tracker is closely related to the knowledgebase, but test results in the notebook may be merged into the knowledge base two without necessary going through the tracker first. After all, you may stumble upon some properties of the system that don’t contribute much to your particular investigation but which are not currently represented in the knowledgebase and might be of use in the future. Likewise, the knowledge base could aid in interpreting test results and iterating over test designs throughout the investigation process.
It occurs to me to ask the question: are these really three separate tools, or three views on the same model? I think ideally they are the latter. Practical concerns may causes us to deploy our environment with separate tools because we might be able to cobble them together from existing software. But I could imagine a single vendor building a defect tracker and lab notebook interface on top of an expert system which actually manages everything.




One response

22 07 2010

I think I should revise my last statement about building this entire contraption as a series of views for a single data model managed by the expert system. After further thought, I realized this doesn’t make any sense. My original, off-the-cuff, thought was that the relationship between bug tracking, experimentation and expert diagnostics is so close that they’re the same thing. In theory this might be true, but only if we make the concepts so abstract that they’re useless. In practice, a failure to separate the defect tracking, ELN and expert knowledgebase into separate data models would lead to a messy garbage-in, garbage-out phenomenon.

For example, suppose I’m trying to solve a program crash. I’d start by hitting up an expert system for a diagnosis. I create an initial query and through the Q&A process I get a more refined query. This either leads to a specific solution or to a better idea of what tests I need to run or what other steps I should pursue in my investigation. So far so good. I know have a refined query that I can use to define my symptoms somewhat formally. But I don’t want to hang up my developers with formal definitions. They may like the extra clarity because it might help them rule out a lot of dead ends, but they’re also going to want a place to hold more free-form thoughts and observations.

Now I start to run tests. The tests are designed to fill in the missing information in the knowledge base. Results from those tests might add new facts to that knowledge base, whether or not those facts are relevant to my current investigation. But now I run into a number of other problems. Is every test result a fact worthy of posterity? How long are they going to remain relevant given the changing code base? Many are probably just too specific to be useful. And where do I store my test results? The data structure of an electronic lab notebook is fundamentally different from that of an expert system.

When I finally solve my problem, I may find out that it was a typo in a single line of code. Thanks to some amount of rigor which my expert system and the scientific process have forced on me, I have no connected a well defined set of symptoms to a well defined root cause. So what? I fix the problem and it goes away, but how useful is this information to future diagnostics? If it is useful, it is in a very abstract form. Perhaps I learned something about a better way to investigate a particular kind of problem or about some of the consequences of the way the system was designed, but I need to give it some careful thought before I start entering data into my knowledge base. Just dumping the raw results of tests and fixed will not do me or anyone esle any good.

Even so, that doesn’t mean all the data I collected from this investigation goes away. I will still want to review changes made and revisit test cases when I encounter a similar problem. This leads me to believe that my notion of building the entire integration environment with an expert system is just another example of an engineer who found a hammer and went looking for nails. Defect trackers, ELNs and expert systems are three tools that are designed for specific tasks and should be kept separate though related.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: