This is a call for beta testers interested in trying out our new version control product: Kiln (http://www.kilnhg.com/). Mercurial is the version control system under the hood, so if you've got FogBugz and wanted to try out a DVCS, this is a great opportunity. If you just want to experience the awesomeness of version control, code review, and bug tracking all working together seamlessly, this is a great opportunity.

I have to mention that this is a beta. There will be some bugs during installation, and possibly after. The good news is that Kiln has been running for several months and hosting production code for a bunch of people in the On Demand environment. We think the risk is pretty low, but as software developers you know there is no way for us to guarantee that there won't be bugs.

We are specifically looking for licensed (install it on your server) testers, although you can use the same form if you're a FogBugz on Demand customer (the more the merrier!):

https://shop.fogcreek.com/beta-kiln/

Licensed Kiln will only run on Windows 2003/2008 Server with SQL Server 2005/2008/2008 Express. Folks running other operating systems or database servers are out of luck for the time being.

ONE MORE THING: This release also introduces a new version of FogBugz. There have been a few UI changes to support having FogBugz integrate with other applications. If you don't want to test Kiln, but would be willing to test the new version of FogBugz, send me an email to jason (at) fogcreek (dot) com.

In the early part of this year, I read Joel's 7 Steps to remarkable customer service. The piece resonated with me, and it was one of the reasons that I applied to the SMTP program. Customer service in the world of technology is, generally speaking, an embarrassing and dehumanizing process. Here was a person who believed that there was a better way to do it.

Shortly after joining the company, I asked the question "Why do we give this information away freely? Aren't we concerned that our competitors are bound to read it, see the wisdom of it, and emulate it?" At that point, I had about 4 weeks of customer service duty under my belt and was confident that this model could be scaled and replicated. Now, after 6 months and lots of conversations with customers, I understand why it is to our advantage to let people know that this is the way we do things.

It is hard work.

Real customer service is emotional labor. You have to dance between being truly empathetic to a frustrated customer's problem and self preserving as a customer, pushed beyond the limits of human endurance, takes out their frustrations on you directly. The former is the norm and the latter the exception, but both efforts take a lot of energy. Add to that the fact that the problems I am asked to solve are frequently subtly complex technical issues, and I am spent at the end of the day. The payback comes in the form of a tremendous sense of satisfaction from seeing Joel's suggestions work to make not just satisfied but fanatical customers.

The problem is, I honestly don't think most of the people that apply for customer service jobs could do it. Either they don't have the people skills, but are excellent technically, or they don't have the technical skills. This is not to say that I am a phenom, by any means. It is proof positive of the draw of the promised career advancement once the customer service stint is over.

I am willing and motivated to work hard because these are the same customers that I hope to be serving as I move on to QA and Marketing and, eventually, to management. Understanding them, and their needs, gives me the information I need to make decisions and trades-offs about the products on which they rely. If you ask most customer service people what they do, I bet they would say "I answer email and telephone calls". Customer advocacy is my job. It is the best way to keep customers happy and continue to increase our revenue

You have to get the environment right.

To get people motivated and give them the tools they need to do the job of customer management takes real effort. While Joel's steps are conveniently broken up to make them more understandable, combined they create the environment necessary to make my daily efforts possible. The entire company is oriented toward the customer. We are constantly evaluating how what we do and what we say will affect our customers.

This is hard to believe for some customers because they have been asking for this one feature (or that one) for 2 years, and we haven't implemented it. Although it is easy to imagine us in our ivory tower laughing with scorn at your feature request, I can honestly say that we agonize over our decisions as to which features to implement when. We honestly feel badly about not getting to your suggestion, but we can't cloister ourselves for 10 years writing the next version, so we need to choose.

Our incredibly lenient return policy allows me the freedom to say to someone "Try it, if you don't like it you get your money back immediately, no questions asked" and "I can't recommend that you buy FogBugz because it isn't going to do everything you want" and "I am really sorry that we couldn't get you connected, here is your money back and a free pass for your next connection". All of this is intended to show that the second reason we aren't at risk is that it is almost impossible to replicate this environment without tremendous effort, and true belief that it is worth it.

By telling customers that this is how we do business, and actually sticking to it, we make our customers confident in our ability to make them happy. We get more risk takers and advocates that push our products into markets where they might otherwise go unnoticed. And my life is better because when customers call or email, they (mostly) treat me like a human being that wants to help them, instead of their worst enemy.

We aren't at risk at all; we have a distinct advantage, a differentiator. While other service organizations understand that these are good ideas, they aren't able to implement them (the environment isn't right) or they won't implement them (don't want to do the hard work). In the rare cases that I come across one that is able and willing, dealing with the company is such a pleasant departure that I find myself liking them, even if they are one of our competitors.

First, thanks to all of the folks who wrote in with suggestions for Under the Hood topics.  Keep them coming!  One of those suggestions, and something that we get frequent questions about in technical support, was a request for a guided tour through source control integration into FogBugz. 

The goal of this installment of Under the Hood is to give you a foundation in the interaction between any source control system and FogBugz.  You will learn:

1. What information FogBugz needs to be given with each commit
2. How FogBugz processes that information
3. How FogBugz displays that information and links back to your source control system.

In order to make this less abstract, we will use FogCreek's own configuration, with Subversion, as an example.

The end goal of the integration process is to get FogBugz to display all of the commits related to a particular case in the bug interface.

By looking at this, we know that FogBugz displays specific information about each commit associated with this bug.  It shows us the revision number from the source control system and the filename, including the path.  Just beneath the surface it is also tracking the previous revision number so that it can link back to your source control system to show you what has changed since the last time this file was modified (also known as a diff).  From this, we can deduce that the four key pieces of information are file name, current revision, previous revision, and BugID.  The question is, how do we get this information to FogBugz?

In order for any integration to work, you need to tell your source control system which case number is associated with a particular checkin.  In our example, this information is included in the comment required by each checkin.  We use the key word "BugzId" right before each case number so that we can look for it later when Subversion executes our post-commit hook. 

Note: A post-commit hook is a mechanism built in to Subversion which allows you to create a shell script that will be executed immediately after a commit is completed (most source control systems provide something similar). 

In this case, our post-commit hook causes Subversion to create a log file of the commit with information about each file committed, the revision numbers, and to which bug that checkin is related.  Then, we parse the log file and scrape out the key information that we identified above. 

Note: The post-commit script creates the log file, and passes it to the logBugDataSVN script which parses it and processes the web requests.  You can find both of these files in your FogBugz\accessories\SourceControl\Subversion directory

For each of the BugzIds that we find in the log file, we build a programmatic request to the cvsSubmit.asp page in our FogBugz installation.  When completed, the request for the first file in our example looks like this:

http://your.fogbugz.com/fogbugz/cvsSubmit.asp?ixBug=19&sFile=trunk/website/default.asp&sPrev=0&sNew=5

Note: Subversion doesn't require the previous version number to display the diffs, so we pass it a 0 for sPrev.  Also, this URL works just as well typed in the address bar of your browser as it does being created and called programmatically.

Assuming we get a valid response from the FogBugz web server, we move on to the next file name or BugzId in the log file.  The remainder of the processing is done by cvsSubmit.  It takes each of the parameters we pass to it, and inserts it into the appropriate fields in the CVS table of the FogBugz database.  If you look at the schema, you will see that there is a field that corresponds by name to each of the parameters from the URL above.

The last thing that FogBugz needs to do is display this information in a useful way when you view a bug that has checkins associated with it.  You will notice that in the screen shot, both the file name and the revision number are active links.  For this to work, the source control system with which you are integrating must have a web interface.  The first link, from the filename, sends you to the log of activity associated with that particular file. The second, from the revision number, sends you to a diff between that revision number and the previous one stored in your source control system.  To generate these links correctly, FogBugz needs to be told how to format them.  This configuration is controlled by the "Source Control URL for logs" and "Source Control URL for diffs" properties on the Site configuration page within FogBugz.

Below each property, you will see examples of URLs for all of the supported source control systems. In our example, we are using WebSVN, so the example URLs are:

• Logs - http://.../subversion/filedetails.php?rep=0&path=^FILE&rev=0&sc=1
• Diffs - http://.../subversion/diff.php?rep=0&path=^FILE&rev=^R2&sc=1

Each of the URLs contains variables that FogBugz will replace with the correct value from the CVS table when it renders the bug information.  For the first file in our example,

^FILE = trunk/website/default.asp
^R1 = 0
^R2 = 5

The URLs for this checkin are:

• Logs - http://our.subversion.com/subversion/filedetails.php?rep=0&path=trunk/website/default.asp&rev=0&sc=1
• Diffs - http://our.subversion.com/subversion/diff.php?rep=0&path=trunk/website/default.asp&rev=5&sc=1

If you had a source control system that required a different format for these URLs, you would replace them on the Site page and use the ^FILE, ^R1, and ^R2 variables where needed.  Finally, FogBugz repeats this process for each of the files it finds in the CVS table associated with this bug, and displays them as the list under the heading of "Checkins". 

With this information, it should be possible to integrate nearly any source control system into FogBugz.  If you want to see more detail about Subversion integration, take a look at ourKnowledge Base article on Subversion Integration.  As always, if you have any questions or comments, please don't hesitate to contact us.  And keep those suggestions for future articles coming! 

We have knowledge base articles about how to fix things in FogBugz when they go wrong.  Many of these articles also have troubleshooting steps to help you navigate a tricky problem.  These guides have been tested here many times, and they work. But people occasionally complain that although they worked, they were a bit confusing. 

This confusion is due, at least in part, to the issue that they don't know, and we don't always tell them, about the circumstances that are responsible for the trouble. This is the right approach for a knowledge base article whose purpose is to identify and solve your problem as quickly as possible, but for those analytical minds out there (like mine) a little context goes a long way to helping you solve and prevent problems.  In that spirit, I am going to do a series of articles showing what's "Under the Hood" of FogBugz so that our more technically inclined customers can see what is going on.

In this post, I'll talk about exactly how FogBugz email works, and, in particular, what the mysterious FogBugz Maintenance Service (FBMS) does. 

First, let's review how email works in FogBugz.  FogBugz does not send or receive mail as a result of any direct user action.  Because we want FogBugz to continue receiving mail even if nobody is around, we have a service, the FogBugz Maintenance Service on Windows and the fogbugzmaintd daemon on Unix/Mac, that regularly runs and makes sure that mail is sent and received (as well as doing a couple other routine tasks).

After its first start, the FBMS waits 15 seconds and then it hits heartbeat.asp (or php).  It does this by creating an HTTP connection to the maintenance URL that you have configured in your Site administration tool inside of FogBugz.

Note: This is why we ask you to check this URL in our KB article on troubleshooting Mailbox/Notification problems.

Heartbeat calls singleHeartbeat which is essentially a checklist of activities. The first one that is mail related is retrieving email from the POP3 accounts that you have configured in the Mailboxes administration tool.

To do this, the FBMS creates an instance of CMailboxList (.asp/php).  It then loops through all of the mailboxes that you have configured.  From each mailbox, it retrieves the oldest message by creating a POP3 connection to the mail server and requesting the message.  Take note, that it only retrieves one mail message from each mailbox on every pass.  You can simulate this communication by going to step 7 of our built-in diagnostics which is linked at the bottom of the Site page. 

Note: Diagnostics are our solution for giving you an easy way to validate your mail settings are correct.

If it encounters any errors, it will display them in FogBugz just beneath the navigation bar (Warning: Some errors have occurred in your FogBugz installation.).  If no errors are encountered, it moves on to its next task. 

Next, the FBMS deals with mail in the FogBugz mail queue.  Mail in the mail queue consists of both notifications and email responses to inquiries in FogBugz.  To accomplish this, the FBMS creates an instance of CMailQueue.  This will load into memory all of the records that appear in the MailQueue table of the FogBugz database.  The FBMS then takes N records from that queue and creates an SMTP connection to the server you configured in the Site administration tool and sends them out.  In this case, N is controlled by the value of cMaxMailMessages which is set in heartbeat.asp.  By default, cMaxMailMessages is set to 1. This communication can be simulated in step 6 of the included diagnostics.  Just like with your POP mailboxes, if there is an error here, it will display as a warning inside of FogBugz.

After the FBMS finishes its checklist, it records the amount of time it took to do its work.  Then, to prevent it from becoming a resource hog, it uses a set of rules to decide how long to wait before its next pass.  The rules are as follows:

• If there is still more work to do (e.g. more messages waiting to be sent or received)
  • Wait the lesser of two minutes or the amount of time it took to complete the work of the last pass
• If there is no more work to do
  • Wait 15 seconds

Once one of these rules is satisfied, the process starts all over again.

I hope that this first look under the hood has been helpful.  If you have any questions, or suggestions for future "Under the Hood" topics, please contact us.

Creating and supporting software is a dynamic process.  There are influences from customers, programmers, managers, sales and support staff, etc. that drive requirements and necessarily influence an application's development.  In the world of shrink wrapped software destined for customer's servers, you add an additional dimension of complexity: the platform. 

Developing for deployment on Windows is a bit like working on aSherman tank.  It is complex, but there is a litany of documentation and diagrams that let you KNOW where everything is.   This means that once you successfully make a change on one tank, you can confidently replicate that change across your entire fleet because they were manufactured on the same assembly line.  In software terms, when we add features to or fix bugs on Windows, we can quickly validate that they work and distribute them with confidence.

Developing for Linux, on the other hand, is more like working to transplant a specific type of tree, which requires a specific ecosystem, in one hundred different forests.  To be certain, there are many similarities between forests, but occasionally the differences can be devastatingly complex.   Linux, like a forest, has quite literally grown over time.  While all of the flavors share common roots, the variety of contributors and the flexibility given at the time of setup means that each system is a forest unto itself.   The result is that, in contrast to Windows, when we make a change and test it, there really is no guarantee that it will work across the wide variety of distributions.

FogBugz is regularly installed on every major distribution of Linux.  As a result, we are tasked with the challenge of being able to deploy and support FogBugz across these varied environments.  This is a challenge which we have accepted as a necessity in our efforts to serve our customers.  To manage it, we have developed a set of tools that help us in both pre and post deployment support to navigate the ecology that is Posix based operating systems. 

Among the foremost of the tools in our toolbox is virtual machine software.  We use VMWare, but there are a host of free options out there.  Virtual machines allow us to simulate a customer's installation environment from our desktops.  On a single machine, we have VMs of fully functioning FogBugz installs on:

• RedHat
• CentOS
• Debian
• Ubuntu
• FedoraCore
• FreeBSD

When a customer contacts us with a combination for which we don't have a VM created, we can quickly create one.  The time it takes to create it is an investment in our ability to provide installation support not only for that customer but also for every other customer that uses that platform.

Many of our customers know that we have a compiled binary object which depends on several pieces of the operating system including the version of the GCC and PHP.  This means that over time, as the ecosystem of Linux changes, we need to rebuild this binary object on the "new" versions as they come to market.  It can be quite a daunting task, but the reusability of virtual machines, and the ever improving ease of upgrading Linux distributions, it is easier to keep up. 

Because of the way we currently distribute FogBugz, many customers have run into the issue of not having the object they need in the installer.  We have developed quite a library of objects that aren't in the latest release but are available through support to our customers thanks to the VMs that we have created.  Looking ahead, we are working hard on a better distribution mechanism for these binary objects in a future version of FogBugz.

The great news is that there is an additional benefit to developing these virtual machines other than release testing.  Post-deployment technical support is made more efficient with the use of VMs as well.  Recently, I successfully reproduced a very nasty, and elusive, bug with mail importing on Ubuntu Linux.  Working with a customer, I recreated their environment almost exactly using a copy of an existing Ubuntu VM.  As a result, I was able to give our developers an instance of FogBugz that exhibited this bug so that they can fix it.  In this case, repro steps might have helpful but not terribly informative, and the creation of the VM is a real boon to my ability to support the customer by ensuring the developers have exactly what they need to find a solution. What is so great about this particular discovery was that the bug appears to be in a piece of common code (PEAR), meaning that the fix will help other customers that have been bitten by this bug. 

Working in customer service, I am very interested in what people are saying about our products and services. In order to research what is being said along with who is saying it, I started to collect a list of sites where people commonly post about products in our category and search for posts there etc. Every time someone linked to an external blog I would bookmark that so that I could check on it periodically. Finally, I used Google to search for a few key terms, including the company name, and would browse through the results. The issue with all of this was that there was no way to immediately tell if I had visited a resource or not. Nor could I tell that I had forgotten to check one of my sources. I spent an inordinate amount of time reviewing old information only to realize 100 words in, that I had read it already.

Enter Google Alerts.

One day, I was clicking around in my Gmail account and I ran across the my services link. Right at the top of the list was the mysterious Google Alerts feature. Google describes them as:

… email updates of the latest relevant Google results (web, news, etc.) based on your choice of query or topic.

The configuration only requires you to plug in a search term and select how often you want Google to tell you about new things that it has found that match. It literally saved me a couple of hours a month. Not a lot of time in abstract, but keep in mind that those hours were previously completely wasted and totally annoying. Now I use it to keep track of a bunch of companies and people who I like to keep tabs on.

Every once in a while, it really helps me at work by letting me know about something really good ( or really bad ) that is going on with our customers. By getting and staying informed, I am armed in the battle for excellent customer service with the weapon that is timely relevant information.

For example, I have a Google Alert configured to tell me once a day about anything new that references Fog Creek Software. Recently, it informed me of a blog post (one that I would very likely have never seen otherwise) which detailed the poor experience of one of our Copilot users. I used this information to reach out to a customer who rightfully complained about a bad experience that was the result of a bug in our software. He not only responded positively to my effort to apologize but also made himself available to troubleshoot the problem he experienced. I helped him by recognizing the issue and compensating him for the trouble. He is helping us by providing us with a source of information, should we need it, about a bug in our software. In addition, I would hope that it may lead to some repeat business for us in the future. In my opinion, it represents a small but important victory in an effort to ensure a truly enjoyable customer experience.