Creating and supporting software is a dynamic process.  There are influences from customers, programmers, managers, sales and support staff, etc. that drive requirements and necessarily influence an application's development.  In the world of shrink wrapped software destined for customer's servers, you add an additional dimension of complexity: the platform. 

Developing for deployment on Windows is a bit like working on aSherman tank.  It is complex, but there is a litany of documentation and diagrams that let you KNOW where everything is.   This means that once you successfully make a change on one tank, you can confidently replicate that change across your entire fleet because they were manufactured on the same assembly line.  In software terms, when we add features to or fix bugs on Windows, we can quickly validate that they work and distribute them with confidence.

Developing for Linux, on the other hand, is more like working to transplant a specific type of tree, which requires a specific ecosystem, in one hundred different forests.  To be certain, there are many similarities between forests, but occasionally the differences can be devastatingly complex.   Linux, like a forest, has quite literally grown over time.  While all of the flavors share common roots, the variety of contributors and the flexibility given at the time of setup means that each system is a forest unto itself.   The result is that, in contrast to Windows, when we make a change and test it, there really is no guarantee that it will work across the wide variety of distributions.

FogBugz is regularly installed on every major distribution of Linux.  As a result, we are tasked with the challenge of being able to deploy and support FogBugz across these varied environments.  This is a challenge which we have accepted as a necessity in our efforts to serve our customers.  To manage it, we have developed a set of tools that help us in both pre and post deployment support to navigate the ecology that is Posix based operating systems. 

Among the foremost of the tools in our toolbox is virtual machine software.  We use VMWare, but there are a host of free options out there.  Virtual machines allow us to simulate a customer's installation environment from our desktops.  On a single machine, we have VMs of fully functioning FogBugz installs on:

• RedHat
• CentOS
• Debian
• Ubuntu
• FedoraCore
• FreeBSD

When a customer contacts us with a combination for which we don't have a VM created, we can quickly create one.  The time it takes to create it is an investment in our ability to provide installation support not only for that customer but also for every other customer that uses that platform.

Many of our customers know that we have a compiled binary object which depends on several pieces of the operating system including the version of the GCC and PHP.  This means that over time, as the ecosystem of Linux changes, we need to rebuild this binary object on the "new" versions as they come to market.  It can be quite a daunting task, but the reusability of virtual machines, and the ever improving ease of upgrading Linux distributions, it is easier to keep up. 

Because of the way we currently distribute FogBugz, many customers have run into the issue of not having the object they need in the installer.  We have developed quite a library of objects that aren't in the latest release but are available through support to our customers thanks to the VMs that we have created.  Looking ahead, we are working hard on a better distribution mechanism for these binary objects in a future version of FogBugz.

The great news is that there is an additional benefit to developing these virtual machines other than release testing.  Post-deployment technical support is made more efficient with the use of VMs as well.  Recently, I successfully reproduced a very nasty, and elusive, bug with mail importing on Ubuntu Linux.  Working with a customer, I recreated their environment almost exactly using a copy of an existing Ubuntu VM.  As a result, I was able to give our developers an instance of FogBugz that exhibited this bug so that they can fix it.  In this case, repro steps might have helpful but not terribly informative, and the creation of the VM is a real boon to my ability to support the customer by ensuring the developers have exactly what they need to find a solution. What is so great about this particular discovery was that the bug appears to be in a piece of common code (PEAR), meaning that the fix will help other customers that have been bitten by this bug.