The general idea behind automated debugging is simple: a tool does some analysis on a piece of code, crunches some numbers, and finds the most likely location of a bug.
Chris Parnin (Ph.D. Student, College of Computing, Georgia Tech) recently published a paper entitled “Are Automated Debugging Techniques Actually Helping Programmers?” in ISSTA’11. Over 10,000 papers have been published on the subject of automated debugging. Most of these papers describe techniques for finding potential areas that a bug might exist in. Prior to this paper, only five have evaluated automated debugging techniques with living programmers. Many find no benefit, and are tested on very short programs.
In his new paper, Parnin examines the common assumption about how programmers use these tools. The common conceptual model has programmers linearly inspecting each of the results found by the tool. In his user studies, only those users that were unable to fix any of the bugs used the tool in this way. The other developers jumped around on the list.
Parnin also examines whether or not the tool makes debugging faster, and found no statistical difference across the entire group. However, a statistical difference was found if the test was limited only to the highest performing developers in the study. The benefit of this type of tool may be primarily in helping developers to navigate an unfamiliar code-base, as it only provides suggestions of where to look for the bug and developers experienced with a code-base may have an idea of where to look already.
Parnin’s paper provides a good example of the surprising results that you can get when testing with actual users. A lot of software is developed based on assumptions or speculation about how users will use the software, and developer intuition on usability is often wrong. In the case of automated debugging tools, a lot of work has gone into developing tools based on assumptions about user behavior that now appear to be incorrect, and on tools that are useful in a narrower set of cases than was expected.