One of our customers wanted to use the FogBugz API from C#, but the API was only available via a REST interface.  So they wrote a C# wrapper around the FogBugz API to let them do what they wanted, and was gracious enough to share it with us.  You can download it from this link, which also includes a description of how it works.

One of our customers (John Fuex) wanted a dynamically generated table of contents feature for his FogBugz wiki.  Rather than wait for us to code it, he wrote it up himself using the FogBugz wiki template and some ASP code.  You can download the files from the FogBugz Ecosystem wiki and try it out yourself!

Before coming to Fog Creek Software, I never had to think about who my customers were. When I was working as a consultant, my customer was the client that was paying the bills; if they wanted a feature and were willing to pay for the development, then that's what I did. When I was developing prototype software for a research group, my customers were the scientists; if they needed software to acquire data, analyze data, or control equipment, then I would build it for them. There was never any question about who I should be listening to, because they were real people that I interacted with daily.

Things are different in the software product world. FogBugz has thousands of existing customers and many more potential customers. And each of those customers uses FogBugz in a slightly different way. Some customers use FogBugz as:

• A simple bug tracker
• A help desk to manage customer support
• Project management software
• Support forum software
• All of the above and then some, as we do here

Because of the multitude of ways in which FogBugz is used by our customers, defining what the "customer" wants can be extremely difficult. When I was a consultant, I could just ask the client to find out what feature I should implement next. Because there was a single voice to listen to, my decisions were easy.

When there are thousands of voices, feature decisions become much harder to make.

• Do we ask for a majority vote on each feature?
• Do we pick features that will make many of our customers a little happier?
• Do we pick features that will make a minority of our customers a lot happier?
• Do we trust our own design judgment on which features will work well together?

In the end, we do what we think will make our customers in aggregate happiest and most productive. Unfortunately, because we have limited resources, we can't do everything. When we don't implement a customer's feature request in a given release, they are disappointed, as if we were a consultant that failed to deliver what was promised.

To some extent, that's a consequence of our customer service ideals. When you call us, a capable person picks up the phone and is able to help you do what you want. We respond to all emails personally within one business day, and regularly respond on our discussion boards. That level of personal interaction raises the expectations of our customers that we are serving them directly. So when our software fails to deliver what they personally need, we are failing to live up to our own lofty standards.

What our customers can't see is that we live up to those standards internally. When we are doing triage for a release, we have customer advocates arguing for each of the features, and we have long discussions to balance all of the factors discussed above. Unfortunately, some features will always have to get postponed (but still kept in FogBugz to be considered for the next release). 

Making a software product is hard. I had it much easier when I worked as a consultant with only one voice to satisfy, but the flip side was that I could only make one company better with the software I wrote. FogBugz makes thousands of companies better at once, which makes it worth sorting through all of those voices to decide what should be implemented next.

Things have been hopping here in the office (and correspondingly quiet on the blog) for the past couple months while we've been beta testing FogBugz 6. I have found it interesting to see how the beta test process has evolved. 

Early in the beta testing period, our beta customers were finding dozens of bugs, from major ones like not being able to install in certain configurations to minor ones like fixing the placement of buttons in the German version. After our intrepid developers tracked down those bugs and fixed them, we tested the fixes internally and sent out a new beta release with the collected fixes. 

As the beta testing period has continued, the problems being found are getting more obscure. A new feature doesn't have quite the functionality one customer desires. Another customer finds an error that we can't reproduce internally despite our best efforts. 

We still want to fix these problems, but every code change requires more testing to ensure that nothing else has been broken by the change. So throughout the beta process, we have had to become more selective as to what kinds of problems we will fix. At this point in the beta process, we are only fixing bugs that cause FogBugz to crash. There are minor bugs that we have decided to not fix for the FogBugz 6 release, and that decision kills us. We want to send out an absolutely perfect release with no known problems. But if we tried to fix every problem, we'd be changing code and re-testing for many more months, if not years. 

We think that FogBugz 6 is a big step forward for FogBugz with many exciting new features, and we're eager to get it into the hands of customers. So at some point, we have to make the choice that releasing FogBugz 6 is more important than fixing the remaining bugs, even though that means that we will already have dozens of open cases to be fixed in a future 6.1 release. We'll get back to closing those cases one by one after the release. 

In my last post, I discussed the qualities that made for a good tester.   And Joel explains why a company should hire testers.  But what motivates a good tester? 

When I asked "Jill the Very, Very Good Tester" about how she approached her job, she stated that her two main responsibilities were to the end-user and to the company.  Let's explore what she means.

End-User Responsibility

Look around a typical software company.  You'll see that it's a technology-centric environment.  Everybody there can configure their own computer, has a complicated cell phone, and plays video games with interfaces so complex that only a teenage boy could learn them.  When they interact with technology, they know what the designer was trying to do because technologists have similar mindsets.  Since they only interact with other technologists, it's easy for them to forget that not everybody thinks like they do.  For instance, an ex-coworker once uttered with a straight face, "That's not a bug!  We have a 15 step workaround for that, so we don't have to fix it!"

The tester's job in such situations is to be the user advocate.  They need to explain how the rest of the world will react to the technology by constructing reproduction cases with the structure of:

• 1,2,3) This is what I did. 
• Expected: This is the result I expected. 
• Actual: This is the result I got.

Such cases demonstrate for the developers what the expectations of the rest of the world will be.  A good tester will anticipate the reactions of the non-technologist end users and catch unexpected behaviors before they are released to paying customers.

Company Responsibility

The tester also has a responsibility to the company.  They manage the risk for the company by providing clear feedback on the current status of the software and its readiness for release.  We all know that when a developer says they're 90% done, they're about halfway there.  But we need testers to inform the developers and their managers of the true status.

A tester will document what works and what doesn't in the software.  They have test plans written, with a method for exercising every feature described in the specification.  They deliver the current status to make it abundantly clear how far the project is from being complete.  One of Jill's suggestions for reporting is in the form of a two-dimensional matrix, where each feature is tested on each of the different deployment platforms; once developers see all the empty spaces, they will be less convinced they're almost done.

There will be pressure from management to give convenient answers, but it's the tester's responsibility to deliver accurate information.  Managers can not make the right decision on whether the software is ready to release without the truth.  They may decide to release it anyway, but then they are making a fully informed decision and are less likely to be surprised later if their customers are outraged.

Sounds awesome, doesn't it?  The tester as the representative for the Common Man, and the standard-bearer for Truth within the organization.  Things rarely work out so nobly for the poor tester, but I thought Jill's perspective may explain why she's a "Very, Very Good Tester".

I'm currently in the Quality Assurance (QA) phase of the Software Management Training Program here at Fog Creek Software.  That's why I'm currently running the beta test of FogBugz 6 among other things.  To prepare for my stint in QA, I talked to "Jill the Very, Very Good Tester", immortalized in The Basics of Bug Tracking, for her thoughts on testing, and reflected on the qualities of the good testers I worked with in my previous career as a software developer. 

Good testers see the world differently than software developers.  Or from most people, really.  When they're in testing mode, they try things that no "sane" person would ever try.  They see the flaws, the interstices where systems grind against each other, and figure out ways to exploit them.

• What happens if I start doing one task, interrupt to do a second, go back to the first task, and then skip to a third task?
• What happens if I paste a ridiculously long piece of text including all sorts of non-standard international characters into every possible text field, including the zip code?
• What happens if I intentionally try to inject SQL code into the database?
• What happens if I don't understand any of the instructions and just start double-clicking randomly on everything I can see? 
• What happens if I just start hitting random keys on the keyboard?

I used to work with a guy named Henry who was incredibly good at this.  I'd spend hours testing my code, clicking on every button, trying every possible combination of inputs I could think of.  I'd give him the software, and he'd be back at my desk in five minutes, saying "Eric, I broke it!"  I'd ask him what he did, and he'd explain how he was doing something that nobody had ever requested or even mentioned, and because the software wasn't designed to do that, it broke.  Fortunately, Henry would also have figured out a way around the bug so that he could keep working while I was trying to fix it, cursing all the way.

At the same time as they are coming up with all these crazy ways to break the software, good testers also have to be incredibly methodical.  They map out every aspect of the software before starting, create a test plan to make sure they exercise all of the desired functionality from the specification, and systematically go through every corner of the software.  These are the kind of people you want running your FDA certification process because they will follow the process to the letter and nothing will fall through the cracks (a former coworker of mine is now a consultant in that field).

When they do find a place where the software doesn't work as expected, they reproduce the bug in a minimal number of steps and file a bug.  Joel describes the importance of this careful reproduction process in the The Basics of Bug Tracking, both in helping to track down the bug and making sure that it's well and truly fixed.   

Somebody who's good at the methodical part may lack the craziness necessary to invent truly dastardly ways of breaking things.  Somebody else who has the completely off-the-wall perspective to try absurd things may not have the patience to figure out exactly what they did that finally broke the software.  So finding testers who have both skills is hard. 

I'm not particularly good at testing yet.  I still think like a programmer, so I know what a feature is supposed to do and tend to do what the developer expected when using that feature.  I'm still working on developing that outsider's perspective, of trying things no programmer would ever try because it shouldn't work. 

I'm also learning to test in a more methodical fashion.  I'm improving at my coverage when doing a full test of a new feature.  And when I come across odd behavior in FogBugz, I can track down a decent set of reproduction steps, and often even isolate it to a section of code that's causing the problem. 

Being a tester has encouraged me to think outside the perspective of a software developer, and to develop aspects of myself that didn't get used as a programmer.  I'll be glad to have that different perspective as I move forward into a career in technology management. 

As I noted previously, the beta test process has been started for FogBugz 6.  Thanks to those of you who have already signed up! 

You may be wondering why you have not already been contacted, so I thought I'd go into the beta testing process in more detail.

We have been using FogBugz 6 internally for many months now in the well-known process of dogfooding.  If anything, we're too familiar with it because we have adapted to the way that it works.  Beta testers let us get fresh eyes on FogBugz 6 again, to remind us how it looks to a newcomer.  Beta testers give us feedback on mouse sequences we've long since internalized, and graphic design issues that we don't even see any more.

Beta testing also helps us with technical issues.  FogBugz 6 has been well tested on our installation of Windows 2003 Server with a SQL Server 2005 backend, but our customers have a variety of configurations that FogBugz has to work on.  We have several virtual machines (VMs) set up with alternate configurations for testing, but since we don't spend all day in FogBugz on those configurations, they aren't as well tested as our main FogBugz installation.  Beta testers give us the opportunity to have people use FogBugz 6 and let us know what doesn't work on their particular system.

So why don't we release the beta to everybody who signed up at once?  As Eric Raymond quipped, "Given enough eyeballs, all bugs are shallow", so the more eyes we get looking at this, the better, right? 

The problem with that approach is that most of the beta testers at any given time will find the same bugs over and over again.  Jakob Nielsen, guru of usability testing, has shown that the return on adding more users diminishes quickly after 5 users because the added users are just reporting the same problems.  If we started the beta test with everybody, we would get dozens of reports of obvious bugs, and then beta testers would stop looking further because they've already found a bug in our product. 

So we stage the beta. 

1. We release it to a few people, get their feedback, and fix all the bugs that we can. 
2. We release it to a few more, who let us know if we didn't fix those original bugs, and uncover new bugs that the first beta testers missed because they were blocked by the original bugs. 
3. We fix those new bugs and any original bugs that need re-fixing and repeat step 2.
4. And so it goes until we're confident enough to build a release candidate of FogBugz 6 and let more people try it. 

So if your name hasn't come up in the beta lottery yet, don't fret.  We've got plenty more testing to do, and we'll be needing more beta testers right up until we're ready to release. 

In my last essay, I compared FogBugz to a pitching machine, lobbing balls towards the sweet spot of software developers. However, an unattended pitching machine eventually runs out of balls, so somebody needs to collect balls and refill the machine regularly. 

Similarly, FogBugz needs a project manager who can keep feeding tasks to developers as fast as they can do them. The project manager needs to:

• Talk to customers, both internal or external, to figure out what tasks need to be done,
• Enter those tasks into FogBugz. 
• Decide when those tasks need to be done, assigning them to releases, and setting due dates and priorities.
• Assign those cases to the appropriate developers, balancing workload and expertise. 
• Keep track of the progress of the developers on those tasks.

Of course, a developer could do all of those things, but wouldn't you rather have your developers doing what they're best at? FogBugz lets the project manager take care of everything but the software itself, creating the development abstraction layer.

Imagine that you are managing a new release of your company's product. You talked to your customers, figured out what needs to go into this release, and opened dozens of different features as FogBugz cases. After creating each case with due dates and priorities, you assigned them to appropriate developers. Let's also say that your developers have gone through the features assigned to them and estimated the time necessary to accomplish those tasks.

Now you can get the status of your project by doing a filter of the open cases in the new release. At the bottom of the page, you'll see a summary that looks like:
Cases on this page
Total  30
Cases without estimate  1
Total estimated time remaining  9 days 2.94 hours
Total elapsed time  2 days 3.79 hours

You can sort the list of open cases by assignee so you can tell which developer has the highest workload, and assign some of their cases to somebody with fewer cases. If the estimated time remaining is going to make you miss your ship date, you can re-sort the features by priority and move the lowest priority cases to a future release until the estimated time remaining is consistent with your deadline. 

FogBugz makes it easy for a project manager to see and manage the current state of the project. By letting the project manager take care of refilling the task lists, FogBugz allows developers to stay in their zone, crunching through coding tasks and delivering quality software.

P.S. If you'd like to learn more about how to use FogBugz for project management, Mike Gunderloy has written the definitive guide to FogBugz: Painless Project Management with FogBugz. It covers FogBugz in depth and provides a lot of helpful tips for managing successful projects with FogBugz.

As we've been hinting at for weeks, we're closing in on the beta release of FogBugz 6.  We're really excited about all of the new functionality in this version.  We've been using it in-house for months, but things always turn up in the field that we miss.  That's where you come in. 

Please sign up for the beta test.

We'll get you a pre-release version of FogBugz 6 that you can try out, and then we'll wait for your feedback.  Priority will be given to existing FogBugz customers if we get too many applicants, but the more testers we have, the better.  I can't commit to a specific date for the beta, but we're going to get it out there as soon as we can.

Update: More details on the beta testing process.

Imagine that you're at the ballpark taking batting practice.  The pitching machine is lobbing balls through the strike zone.  You're taking nice and easy swings, driving balls to all fields.

Suddenly, a baseball hits you in the back of the head.

You whirl around to yell at the thrower, and get pegged again by another ball from somebody else.  Soon balls are flying in from all directions, and you're pummelled by the onslaught.

Welcome to the life of a software developer.  When life is going well, they are in the zone, marching through their bugs and features and writing good software.

But then a manager interrupts them to do "just this one thing", knocking them out of the zone.  While they're dealing with that, another coworker needs emergency help finishing up a task.  More last-minute items get dumped on the poor developer.  Eventually they're just trying to survive the week, the zone a distant memory.

It's happened to all of us.  And it ain't no fun.  So how does FogBugz help with that?

FogBugz is like the pitching machine - it feeds tasks to developers in an orderly fashion.  The developer settles down in front of their computer, reads their email, checks Reddit, opens FogBugz and says "Show me the open cases that are part of the next release that are assigned to me, sorted by priority".  Boom.  There's the list, in order, of what the developer needs to work on.

When new tasks arise, they are not thrown straight at the developer, jolting them out of the zone.  Instead, they are fed into FogBugz, where the project manager takes care of prioritizing the task, placing it in the proper release, and assigning it to the appropriate developer.  The task shows up on the list in FogBugz, but the developer hasn't been disturbed from the zone.

By using FogBugz to manage the tasks confronting a software development team, you get more productive developers.  FogBugz lets them hit balls out of the park.

P.S. Thanks to Brett, one of our developers, for the baseball analogy.

If you frequent the Joel on Software discussion forums, you know that you now have the ability to register for an account.  The immediate benefit is obvious - getting the little green check next to your name in your posts is pretty sweet.  But what else can you do with an account?

Joel mentions the ability to star posts in his introduction.  If there's a discussion thread you're following closely, you can star it by clicking the empty star next to the title.  Once you do, it's easily accessible via the Starred menu in the upper right corner.  You'll notice that the Starred menu also includes a Recent section which lists the posts that you've been reading, which is more convenient than the browser history since it persists even if you switch computers.

Another benefit of being able to log in is using the expanded search capabilities.  There's a big search box at the top of the page when you're logged in.  Go ahead and hit the Search button without entering any text in the box.  Click on the "Advanced search features for discussion topics" link to open up the list of all the "axes" on which you can search.  These axes add significantly to the search capabilities of the forum.

For instance, I often want to know if there's been a response to threads I posted on.  Now it's easy for me to check - I search for
editedby:eric -lasteditedby:eric
which does a search for threads where I posted, but wasn't the last poster.  To keep it to recent threads, I can search for
edited:"this week" editedby:eric -lasteditedby:eric
or
edited:"this month" editedby:eric -lasteditedby:eric

If I'm feeling nostalgic and want to see the threads I posted on in March, I can use
editedby:eric edited:"March 2007"

Of course, I can combine all of these search axes with keywords to restrict the search further.

Hopefully, this sneak peek of what Joel called "our in-house, under-development, unannounced version of FogBugz 6" will whet your appetite for other search improvements in FogBugz coming your way.  Stay tuned...

You're going to see increased traffic on this blog starting now, as Joel and Michael have asked the members of the Software Management Training Program (SMTP) team to talk about our experiences at Fog Creek Software, our products, and the people that work here.  If you've had any interaction with Fog Creek over the past year, you've probably talked to one of us, as we have been handling customer service as well as a variety of other responsibilities around the office, but this will give us an opportunity to talk more broadly about what's going on here in the office.

By way of introduction, we are:
Sumana Harihareswara, handling marketing for Fog Creek
Eric Nehrlich, developing the quality assurance department
Dan Ostlund, leading the sales efforts
Jason Rosoff, managing technical support

If you have any questions about Fog Creek Software, its products, or the SMTP program, drop us a line at customer-service@fogcreek.com and we'll be happy to answer you!