[ Help Index ]

Painless Bug Tracking 

by Joel Spolsky, founder, Fog Creek Software

I was born with only two bug-storing-slots in my brain. At any given time, I can only remember two bugs. If you ask me to remember three, one of them will fall on the floor and get swept under the bed with the dust bunnies, who will eat it.

Keeping a database of bugs is one of the hallmarks of a good software team. I never cease to be amazed at how few teams are actually doing this. One of the biggest incorrect facts that programmers consistently seem to believe is that they can remember all their bugs or keep them on post-it notes.

If I can bend your ear a moment, I’d like to explain a pretty painless way to do bug tracking. First of all, you need a real database. On teams of two people writing a little bit of code over the course of a long weekend, it’s probably OK to use a text file as the database. Anything larger, and you’re going to need a real bug tracking database. That’s what FogBUGZ is for.

Let’s follow a bug around, for the purpose of illustration, from the moment it’s born until someone finally puts it out of its misery. We’ll follow the famous Bug 1203. Here’s what FogBUGZ shows for that bug:

Here’s what happened.

Mikey the Programmer is hacking away on the new FTP client feature of his groovy Macintosh software. At some point, because he’s feeling frisky, he writes his own string-copy function. That’ll teach them pesky reusability police! Bwa ha ha!

Bad things happen when you don’t reuse code, Mikey. And today, the bad thing that happened was that Mikey forgot to null-terminate the copied string. But he never noticed the problem because most of the time he happened to be copying strings into pre-zeroed memory.

Later that week, Jill the Very, Very Good Tester is banging away at the code, rolling her forehead back and forth on the keyboard or some equally cruel test.  (Incidentally, most good testers are named Jill, or some variation like Gillian.) Suddenly something very strange happens: the ftp daemon she was testing against crashed. Yes, I know it’s a Linux machine and Linux machines never crash (no snorting sounds from the slashdot crowd, please) but this dang thing crashed. And she wasn’t even touching the server, she was just FTPing files to it using Mikey’s Mac code.

Now, Jill is a very, very good tester, so she’s kept a careful log of what she was doing (the precise pitch and yaw of her head as she rolled it on the keyboard is in her lab book, for example). She reboots everything, starts with a clean machine, repeats the steps, and — Lo and Behold — it happens again! The Linux ftp daemon crashed again! That’s twice in one day, now!

Jill squints at the list of repro steps in her notebook. There are about 20 steps. Some of them don’t seem related. After a bit of experimentation, Jill is able to whittle the problem down to four steps that always cause the same behavior. Now she’s ready to file a bug.

Jill enters the new bug in the bug tracking database. Now, the very act of entering a bug requires some discipline: there are good bug reports and bad bug reports.

Three Parts To Every Good Bug Report

And the Lord spake, saying, “First shalt thou take out the Holy Pin. Then, shalt thou count to three, no more, no less. Three shall be the number thou shalt count, and the number of the counting shalt be three. Four shalt thou not count, nor either count thou two, excepting that thou then proceed to three. Five is right out. Once the number three, being the third number, be reached, then lobbest thou thy Holy Hand Grenade of Antioch towards thou foe, who being naughty in my sight, shall snuff it.”

— Monty Python and the Holy Grail

It’s pretty easy to remember the rule for a good bug report. Every good bug report needs exactly three things.

Seems easy, right? Maybe not. As a programmer, people regularly assign me bugs where they left out one piece or another.

If you don’t tell me how to repro the bug, I probably will have no idea what you are talking about. “The program crashed and left a smelly object on the desk.” That’s nice, honey. I can’t do anything about it unless you tell me what you were doing. In fact it’s good practice to list every step from starting the application to seeing the bug. It’s always a nice touch to see a tester listing steps starting with “launch the app.”

Now, I admit that there are two cases where it’s hard to get exact steps to repro. Sometimes you just don’t remember, or you’re just transcribing a bug from “the field.” (By the way, why do they call it “the field?” Is it, like, a field of rye or something? Anyway...) The other time it’s OK not to have repro steps is when the bug happens sometimes but not all the time, but you should still provide repro steps, with a little annotation that says that it doesn’t happen too often. In these cases, it’s going to be really hard to find the bug, but we can try.

If you don’t specify what you expected to see, the programmer may not understand why this is a bug. “The menus drop down when you move the mouse over them.” So what? What did you expect? Ah, you say that it is a GUI standard that the menus only drop down when you click the mouse! Now I understand why you consider this a bug.

Part three. What you saw instead. If you don’t tell me this, I don’t know what the bug is. That one is kind of obvious. But how often do you hear people saying, “I tried to run the foozblatter and it didn’t work!” Oh. Really. What does this mean, “it didn’t work?” Did it display an error message? What error message? Did it look like it was foozblatting but the fooz was not blat afterwords? Did it crash the program? These are all very important distinctions. People in my company have learned that if they come up to me using the exact words “it didn't work” they are going to get screamed at.

FogBUGZ won’t force you to enter all three parts of a bug report. That’s a part of the philosophy of FogBUGZ: every field is optional, because it’s better to get something captured in the database even if it’s not complete, so you don’t forget the bug, which is even worse. But you should always go back later and provide the missing details.

Back To Our Story

Anyhoo. Jill enters the bug. In FogBUGZ it gets automatically assigned to the lead developer for that project. And therein lies the second concept: every bug needs to be assigned to exactly one person at all times, until it is closed. A bug is like a hot potato: when it’s assigned to you, you are responsible to resolve it, somehow, or assign it to someone else.

Willie, the lead developer, looks at the bug, decides it’s probably something to do with the ftp server, and resolves it as “won’t fix.” After all, they didn’t write the ftp server.

In FogBUGZ, there's a very important distinction between a resolved bug and a closed bug. A resolved bug is still open, and somebody still needs to confirm that the resolution is acceptable before it can be finally closed. That person is usually the person who opened the bug in the first place — only they can be sure that the fix is a real fix for the bug they actually saw. So by default, FogBUGZ assigns resolved bugs back to the person who opened them.

This is a crucial point. Bugs do not go away just because a programmer thinks they should. Under most circumstances, only the person who opened the bug can close the bug. The programmer can resolve the bug, meaning, “hey, I think this is done,” but to actually close the bug and get it off the books, the original person who opened it needs to confirm that it was actually fixed or agree that it shouldn’t be fixed for some reason.

Jill gets an email telling her that the bug is back in her court. She looks at it and reads Willie the Lead Developer’s comments. Something doesn’t sound right. People have been using this ftp server for years and it doesn’t crash. It only crashes when you use Mikey’s code. So Jill reactivates the bug explaining her position, and the bug goes back to Willie. This time Willie assigns the bug to Mikey to fix.

Mikey studies the bug, thinks long and hard, and completely misdiagnoses the bug. He fixes some altogether different bug, and then resolves the one Jill opened.

The bug is back to Jill, this time marked “Resolved (Fixed).” Jill tries her repro steps with the latest build, and, lo and behold, the Linux server crashes. This is why it’s so important to let the original person who opened the bug decide whether to close it or not: in the real world, it’s not unusual to find that fully half of the initial resolutions are incorrect for some reason or don't really fix the reported bug.

So Jill reactivates the bug again and assigns it straight back to Mikey.

Mikey is perplexed, but he finally tracks down the source of the bug. He fixes it, tests it, and — Eureka! The repro case no longer crashes the ftp server. Once again, he resolves it as “Fixed.” Jill also tries the repro steps, discovers that the bug is good ‘n’ fixed, and closes it out. Only now will the bug disappear from most filters, which only include open bugs.

If you are developing code, even on a team of one, without an organized database listing all known bugs in the code, you are simply going to ship low quality code. On good software teams, not only is the bug database used universally, but people get into the habit of using the bug database to make their own “to-do” lists, they set their default page in their web browser to the list of bugs assigned to them, and they start wishing that they could assign bugs to the office manager to stock more Mountain Dew.

FogBUGZ, and bug tracking in general, is quite easy once you understand the life cycle of a bug. Anybody can report it, it always belongs to exactly one person, anybody can resolve it, then only the person who opened it should close it. To make your bug tracking a little bit more successful we’ve written up the Top Ten Tips for Successful Bug Tracking. Good luck, and happy extermination!

  FogBUGZ Home Page

  Email: fogbugz@fogcreek.com