Joel Spolsky's talk on shipping bug-free code and the latest features in FogBugz 8 and Kiln 2
Want to try FogBugz or Kiln? Get started with a free, 45-day trial. There's no need to enter a credit card and signing up takes only a minute.
Thanks for coming out, Seattle. Happy Duallie - excuse me for a second. I have - what do you call them, slides, but before I start with that, bear with me a second. I'm just going to report that crash because I've seen this before and it's driving me crazy. Count down - I'm crashing - full screen - at the end - ASAP, and I'm going to set a due date on it, too, and get this fixed by tomorrow, please. All right, good.
Okay. So, I've got a bunch of slides. Where are they? Here we are, FogBugz. Does anybody - before I get started, does anybody use FogBugz in this audience? Wow, okay. So that's like a little less than half. Does anybody use some other kind of thing that - that tracks bugs, manages projects, or whatever? Does anybody not track bugs? Okay, we'll talk afterwards.
For those of you that are new to FogBugz, it's - although its origin is - is in bug tracking, at this point, it really handles the entire life cycle of the whole circular application life cycle of - of software - everything from gathering ideas from customers by email, by a web interface, to writing down your requirements, and managing your requirements, breaking down requirements into features that you might want to implement, breaking down features into tasks, breaking down tasks into subtasks, and so on and so forth, assigning those to individual developers to work on, and then they can estimate if you care about schedules. Some people don't stack overflow. But you can put estimates in, and if you put estimates, then you can track your time and you can see when you're going to ship and we'll run this thing called evidence-based scheduling, which does a whole Monte Carlo simulation to decide with - when you're going to ship and with what probability. And then what do you do? You write code and then the QA team finds bugs, and your customers find bugs, and they feed it all back to you through FogBugz, and you track them and get them fixed. So, that's kind of what the FogBugz side of the equation does.
The companion project, Kiln, is for your source code. It's built on top of a mercurial, which is an open-source distributed version control system. We'll talk about distributed a lot. And it adds a whole bunch of features that are the kind of things that commercial software developers usually are looking for - people that want support essentially and a bunch of nifty things like code reviews, repository, and organizer. All right, we're going to talk about all that.
We've just shipped the new versions, FogBugz 8 and Kiln 2.0, and they've got a bunch of nifty new features there; should be a part of - oh, look. Case resolved. So, excuse me, let's just see what he says. "Yeah, QA caught that bug, too. It should be fixed in the current release." All right. So this is the classic - you are hallucinating, moron because I don't think it's fixed, right. Am I - Open link in new tab? Let's see.
All right, I can't - I don't have much evidence here. This is some kind of random starting up problem, probably a different problem altogether. There's some check-ins here, so I can see how it was fixed. It was fixed with this one check-in. And the check-in is in the wrong repository. So, let me explain that a little bit.
With standard version control - if you're used to using subversion or CVS or, or Visual Source Safe or one of those previous what's called them legacy version controls - there's some Microsofties in the room, I can tell. If you - if you're used to using those systems, the idea was that you had a big old centralized repository living on the magical server and that held every past version of everything and the complete history of what everybody did and all the meta data and all that kind of stuff. And what you had on your local machine was just a current copy of that stuff, which you edited, and then, you know, you sent your changes back to the server. So that's centralized version control and you usually thought in terms of having one repository.
When you're using distributed version control systems, you always have loads of repositories. You have lots and lots of repositories, and you can create them very quickly. That's very cheap to have complete copies of your repository, and they actually make your life better; not worse. And we'll talk about that some more, too, about all the different kinds of repositories you might have, and the reasons you might have different repositories.
Essentially, we've got this dinky little project that we did to make that countdown app that you saw at the beginning there, and it's already got two repositories: the main repository and then there's this testing repository where we throw things that we want to work on, and we test them. And if they work, then, we take those changes and push them up from the testing repository to the main repository where everybody gets them. So, that's just a very simple workflow and you're going to see much more complicated ones as the day winds on.
But the point I wanted to make is that Galen here says that the problem is fixed, but I can see that it's only fixed in the testing repository so that doesn't help me. Let's look at the fix just in case it really is the right fix. Okay, so here's the check-in and there's only one file, and there's only one change, and the change is changing the version number stamp, so this is unlikely to be to really fix the problem. So luckily, FogBugz has a reactivate button here, which you can say, "no, you moron, that is not the problem that fix is just a version number." Okay, I now feel bad. Got to I remember that I'm not in New York. I'm going to have to disagree with your assessment there.
Okay. Now, what I just inadvertently downloaded is that FogBugz 8 has editing and we - the reason we didn't use to have this is because we had a lot of customers that wanted, you know, accountability, and they wanted a complete track record of everything that ever happened. So, we do technically have the ability, when you edit something, you can see the revisions. It's kind of like a Wiki where you can see all the changes that have been made. Hopefully, Galen will not notice this, that that has been revised.
All right. So let's go on to my demo here. I didn't make these slides; apologize for the bouncing. This is - this slide is to let you know that FogBugz now has a marketing department that is in charge of making slides, so that's awesome. No, good. I don't - I don't really know what to say. Have you ever been white water rafting? This is Seattle. People been white water rafting, sort of? There's no teamwork involved. The paddles are just given to you to give you something to hold on to because you're going down and you're going to hit every rock. That's how it works. Let me see here. Let's see what the response was. I think he may have looked at it pretty soon. Okay, oh, you're totally right. I know - I'm guessing it has to do with the draw a clock-face function, but I honestly don't have time to fix this right now. What is he - what is he saying? That he doesn't have time to - excuse me? Doesn't have time to fix this. I still have a lot of cases I need to finish for Kiln 2.0, yeah, but I'm doing a live demo right now.
All right. Let's see if this is true. I'm trying to figure out what Galen is allegedly working on here, all his cases. Let's just do this here. List of cases. All right. These are my cases. If you're new to FogBugz, this is the main screen where everybody spends all their time. I'm looking at a list of cases, and you can filter it anyway you want, so there's this sort of filtering section up here that you use to define which cases you're looking to - you're looking at. I'm only looking at my cases right now. You know, I can narrow it down to only see Bugz. I could - I could just look at Galen's cases. And then, you can, you know, you can rearrange the columns anyway you want. You can - you can change which columns are showing. You can do all kinds of beautiful stuff here to create the exact filter that you want to look at. And when you have it the way you want to look at it, you can save it with a name. You can share it Galen's bugs, and that way we'll be able to come back to that same exact filter later. So, these are all his cases. He said he has a lot of cases in Kiln - Project Kiln. So, he does have a bunch. Looks like I opened the most of them. So, I'll take some responsibility for that. But I still can't really tell is he - is he - does this project really depend on him? Are these big cases? Are they small cases? Some of them have some remaining time. They probably have some estimates, kind of hard to see here.
There are some other ways of looking at this. I have a filter that we're using on the team for - the Kiln project, just showing how many cases they assign to each user. So one thing you're noticing here is this is still the filter screen. We just have this option now to see the filter screen graphically as a, you know, PacMan chart, or a bar chart, or a column chart. And so, indeed, Galen does have an awful lot of cases here in Kiln compared to the other people on the team. What that one is? So, but he - they may all be small and other people may - may have bigger ones. And this doesn't really say anything about the schedule. This just says he's got a lot of cases. So, I'm going to take some time and go off and look at the Kiln schedule.
And here what we're looking at is a feature in FogBugz called Evidence-based Scheduling. I want to look at the completion of probability, completion date over time. Evidence-based Scheduling tells you when you're going to ship. But rather than just giving you a date, which would be fictitious, it gives you a range of possible dates combined with probabilities. So, what we can see here is, this is on the "X" axis dates, and we are always in the left-hand column, I'll mark that origin. And the green line is when we think we're going to ship, or we've - or it's the official date, which is January 16th. And then, there's a whole bunch of - and then this - these little dots show the probability that we will actually ship for any given date.
So for example, we only really have a one percent chance of even making it by February 21st, and this looks like there's a very high probability that we're kind of coming around here. So, we have, let's say, an 80 percent chance of March 7th, but that's still a couple of months late. Once again, this tells me that the project is probably in trouble. Because that curve is very steep, it means that we have pretty reliable data we can count on here. If the curve was shallow, that would mean that we don't have such good historical data.
I don't - I still don't know really who - who to blame, but I can do - I can do these things, this thing called the Per User Timeline and that's going to show me what each user on the team is doing, or what each developer - sorry - on the team is doing. So, we've go three major developers that have a bunch of features assigned to them.
Allen here has very little code. He's working right now, but there's a chance he's going to be finished this week, and he must be a pretty good estimator because there's a narrow range of possible dates during which he might finish.
Anita is new on the team. We don't have any evidence for her yet. She has not yet completed any features. And so FogBugz doesn't really trust her estimates yet, and so, when she said, "I need to do these 16 things" FogBugz - every time it did a different Monte Carlo simulation, it kind of assigned a different potential time to that thing that she was going to actually do. And so what we're looking at here is a wide range of possible dates in which Anita might finish. I mean, she might finish really soon. We just don't know, and it might take her a long time. But she's almost certain. She was 85 percent probability she will be done by the official date. So, we don't have to worry about Anita. Even though we don't know if she's a good estimator or not, she's not critical path, so to speak.
Now, Galen here is a very good estimator and I know that because there is a narrow range of possible dates at which he might complete his work. And however, there is no chance that he will do that until February 21st. So, I can immediately see here that he is actually the problem with Kilning. And what I should do is, take some of those features I assigned to him and assign them to Allen; but I'll do that later.
In the meantime, let me show you - I just want to show you a little bit more stuff around here while I'm demoing the screen, which is kind of cool. This is the milestone you're looking at. You can have as many milestones as you want, and you can see what's going on with that milestone. And you can decide, hey, will we ship any sooner if we only did priority one through four, instead of priority one through five? No, because in our system, everything is priority one, okay.
Let me just show you one more schedule for a much more complicated project. We've been working on a game called Frogger for the Amiga. We think it's going to make a big comeback sometime next summer. And we've got the whole team working on it, and this is the final release. It's going to be multiplayer. It's awesome. The final release is going to happen the middle of next year probably. And what you're seeing here is a much more complicated schedule. First of all, there's all this red stuff. The red stuff indicates a probability that somebody gets blocked. And so looking at Galileo for February 3rd, there's only a ten percent chance that he's actually working on anything. There's a 90 percent chance he's waiting for the servers in Brazil. We're going to set up two data centers, one in Iceland, which will handle Amiga users in the Shetland Islands; and one in Brazil, which will handle, you know, Falkland Islands, Easter Isles, and that kind of stuff. Not a lot of Amigas left actually.
These servers are being - being installed by Marie. Notice that she's got some stuff in orange. Orange means that she is working on something else at this time, on a different project altogether. So, FogBugz is really good at letting you say these developers over here on these projects - and you can even say, "I'm 50 percent on this and 50 percent on that." And then you can say this project is dependent on that project, or this milestone has got to wait for that other milestone, and maybe even some other project. And - and FogBugz will use all that to make a schedule.
And what you can see here is that right now, most of the team is working happily. So, they're all in blue because they are actually working on some - some task here. But fairly soon now, a bunch of them are going to finish the work that they have to do and get stuck waiting for their servers and it's a nice thing to find out about that earlier rather than later.
Now what is going on, you may have been wondering with Isaac Newton here who is planning to finish his features only in the middle of 2019. Well, this is your - have you ever had a developer where you assigned them a task that they don't want to do, and so they estimate it as ten years? That is probably what happened here. There is probably something he doesn't want to do that is going to take ten years allegedly. I hope so. Maybe recreate the Amiga.
All right. Where was I? So we were - we were looking to see what Galen had to do. There's the - there's the Kiln Project. We can actually - I have another filter here, which is the same filter here, which is kind of interesting. There's another new thing that we do in FogBugz 8, which is we'll show you how a filter has changed over time. So, this is 30 days of summarizing the current filter that I'm looking at. This is really useful if you're just looking at, you know all the bugs that I have to fix.
The - the scheduling stuff I just showed you is great for scheduling features where you know in advance how many features you're going to implement and what they are. And you can try to estimate them and at least - at least the problem is tractable. But it doesn't work so well for bugs because sometimes you just don't know how many bugs you're going to find, or how many bugs you're going to come in especially if you're doing beta tasks and stuff like that, it's a little bit harder to schedule that stuff.
So, the way that QA teams traditionally look at a product to decide whether it's ready, is to see the rate of incoming bugs versus the rate of bug fixes and look to see when those lines are going to cross. And so, one of the things that you always want to know is do I have less and less bugs or more and more bugs, and how is the number of bugs changing. And that's what this screen does. So, it lets you take any filter you can imagine and see how it has changed over time.
One of the things I can see here is that specifically Galen's cases in Kiln, I could actually just narrow it down to him assigned to Galen. If I look specifically to Galen's cases, I can see that that number is going up and going down over time, and that means that he is actually working here, right. He's getting cases assigned to him. He's resolving some of them, so I can tell, at least, that he's not sleeping in which case that would probably either be a growing line or if nobody was working on it, it would just be flat. So, I'm going to take his word for it.
Where were we? We were at the - the story up to today is that - oops that's where I am - the story up to today is that Case 537 - he's not going to fix that, so I will. I was just telling somebody a minute ago that I still program. All right. I don't even know where this code is. I certainly don't know where that function is. Luckily Kiln has search. So, I'll just search for that function, and it found it for me, and it's in a file called Countdown.cpp, and I can click on that to see the code, and there is the code. And if there's anyone know programming - Windows programmers in the audience you can help me find the bug. And I - I could check who wrote it actually if I want. That's kind of a cool feature here. I can click on this little annotate button and that will show me who wrote each line of code. And there's a check-in right here by Galen. Does that say add build script? This is going to be a disaster. That's nice. I can see all the - I can see all the commits right there for each line of code. So basically, as I look through code, I can see - in this case, Galen did everything. But I can see who wrote what code, and - and the entire history that kind of all in one easy scrubbable place.
And I think that this defined interlocked increment. What that does is that it's replacing all of the atomically safe operations that are thread safe with a non-thread safe increment; thus making it unsafe. So, I'm going to delete that anyway just for kicks. What the hell? It doesn't look like it makes any sense. I don't even really know where the code lives, but again, Kiln does, and it lives in a repository and that's the URL for the repository. So, I'm going to copy that, and I - I'm going to use the famous DOS prompt to get the code. Now there are GUI tools for all this, but I'm just a - I'm a typer.
Let me explain this command line that I just typed here. Hg runs Mercurial. Mercurial is the version control system at the heart of Kiln. So, get it, Hg is the atomic symbol for Mercury. It's like one of those programmer jokes. Hg clone. Clone says that entire repository, clone it. Make a complete copy of the repository, everything, including all the old versions, and all the history of everything, and all that kind of stuff and then I gave it the URL of the repository, and I didn't tell it where to clone it to. So, it put it in a subdirectory with the same name. So, there is that entire clone of the repository. You can see a bunch of icons there. These are the files. This is the source code. They all have little green checkies, and that means that I have not touched those files since I last checked them out.
And there's a subdirectory here called .hg, which contains don't go in there. I went in there. Don't delete anything in there. That's very dangerous. It contains the entire history of its repository. It's all there; all these old versions and stuff, and all the - in fact, let me show it to you. I can get a little local log here. 'Hgtk' is the graphical version because 'tk' means graphics.
This stuff comes from UNIX. Here's a - here are all the changes. Notice, I'm scrolling through every single change - Galen wrote most this code two weeks ago. Scrolling through all these changes and for each change, you can see on the left-hand side which files changed, and you can click on a file and you can see the comment and the exact change that was made in that commit. And notice how fast I'm scrolling here. Remember that's because all this stuff is local. It's on my local hard drive on, well, SSD, on this laptop. And that - I don't have to go to the server to find the history of anything, to find comments, and that actually is going to make operations a little bit faster.
Sort of two questions that I always get at this point is: Do you always have to check out the entire repository with all the old versions every time you want to just fix one line of code? And the answer is yes, it's not as bad as it seems. It's - believe it or not, first, it actually makes things faster because there's a lot of operations can be done locally. You can quickly locally determine what you've changed and once you've got the check out, the only communication between you and the server are change sets, and so those are very tiny. And that actually makes Mercurial practical to use on large distributed teams over the Internet.
There - there client/server version control systems, who I won't name, that literally you have to buy a third-party product if you want them to even work remotely well over the Internet, which is pretty fast. So, this is a much more efficient way basically to sip bandwidth instead of guzzling it by having the local repository.
A lot of people use Subversion. That's sort of the big number-one competitor for Mercurial and Git these days. If you ever do a checkout with Subversion, you're getting two copies of the code. You're actually getting the current version and you're getting a copy of the current version. So, if the current version is "X" bites, you've got "2X" disk space used. And the reason is, because Subversion wants to be able to do fast reverts and fast diffs so you can see what you've changed quickly without having to go to the server to try to negotiate and decide what you've changed.
And that's kind of reasonable, but if you actually look at a Mercurial checkout, it is less than "2X" because all you have is the current version and then diffs going back in time, and those diffs are compressed, and very efficient, and stuff like that, and you actually wind up with a much smaller - much smaller checkout surprisingly.
So, whoo - that is - here's my checkout and I now have it on my local machine and there it all is, and I am going to countdown dot cpp. I'm going to edit that file and I hope I have visual studio on this laptop.
All right, visual notepad. So, it will be. And the bug was on line 63, oh my, look at that. Go to line works in notepad. It's like the only feature it has what's - it's the one feature that they - all right. I know. It has some kind of Unicode thing for upside down script. I'm deleting those lines of code and saving that. And now, you see a little red exclamation point because I saved that and I'm just going to check it in commit, fixed Bug 537, I hope. Commit. And now it doesn't have a little red checkmark because it's committed and I could commit something else now if I wanted to. That app doesn't close itself automatically. And now you're thinking wait a minute. You didn't even check that. You didn't even test to see if it built. Why are you committing that? And if you're thinking that, it's because you are stuck in an old mindset. The old mindset of version control systems was that there were two operations, which were inextricably bound because you only had one central repository.
The two operations are commit and inflict. By - and by inflict, I mean inflict your crazy source code change on all those other people on your team. And you could not commit without inflicting, or let's just say share - commit and share. And in fact, it never even occurs to people that those are separate operations. In DVCS, those are separate operations. You do them separately because my commit happened locally. It only happened on the local copy of the repository that I have right here and didn't go off to the server.
Why is that good? Well, that's good because now I can use version control. If you look at people on teams - on Subversion teams, or on traditional version control teams, or old-time Microsofties. These were - remember slime. Those things, those tools - what would happen is, that people would be afraid to commit because they didn't want to - you know, they had some incomplete half-baked feature that was probably going to break a million things. And they didn't want to check in until they had finished all that stuff, until they had cleaned that up and made it nice, and they were pretty sure that they weren't going to break the build for everybody and then they would sneak in the middle of the night and try to check it in and do a build, and you know.
And essentially what was happening is that they were going for weeks and weeks at a time without using version control at all. And so they couldn't - they weren't annotating everything that they did. They weren't writing down reasons to remind themselves why code was written because they were only checking in. They only had one commit message per month. They weren't getting the benefits of version control; that stuff was all stuck on their local hard drive; that, you know, it's Friday afternoon. It's 5:00. You want to go home because it's Duallie tomorrow or whatever, Thursday afternoon. And the - you've got a bunch of changes and they're completely half-baked, and you don't want to inflict them on people, but you don't want to check them in, so you're missing all this great version control stuff. You can't do experiments where you just try something random and then revert if it doesn't work because you're not using version control. So, that's one of the great things about DVCS is, you commit all the time and it doesn't really matter, and this commit, once again, it only happened on the local machine so you're probably running - worrying about what will happen if my hard drive goes caput.
So what I'll do is, I'll just go ahead and push that change up to the server, and you can see that I've pushed the change up to the server. Interesting observation, I still have not inflicted my code on anybody else. All I've done is, I put it on one of the servers on the repository, on the - sorry - one of the repositories on the server, which is that testing repository. So - so now, if I look at the testing repository, there's my change. But if I look at the main repository over here, it's not there yet.
All right. Now the reason I pushed it to the server, there's a couple of reasons I pushed it to the server. One is that now I can kind of optionally share it with people, like other people can find it there because it's somewhere on the server where other people can get it. It's just not in the main repository. But also, it's backed up. So, the reason I pushed it to the server is because I want Galen to look at what I just did and tell me what he thinks about it.
And I'm going to start a code review here, using the Kiln Code Review feature. And this is what a code review looks like. You've got some code on the right-hand side and a conversation on the left-hand side. And the conversation on the left-hand side works just like FogBugz. You assign it to people, and they assign it back to you, and you can close it when you're done. So, I'm going to take these lines of code and say, "Why would you do this? Wouldn't that be thread unsafe?" And send it to him. Now, he's got, like, a little link to the code that I was pointing to right there. And when I send this to them - kind of a neat feature here, which is this is all tightly integrated between Kiln and FogBugz.
So, when I look at what cases are assigned to Galen that one is going to be right here at the top. In other words, everything that we want Galen to be working on is in one place. Code reviews, bugs that he needs to fix, features he needs to work on, even emails that came in from customers that he's responsible for replying to, and they can all be prioritized together and he really only has to go to one place to find all the stuff that he's supposed to be working on.
So, this is a code review case. I can see here that it has updates, which means that he's sitting there right now actually responding. And he says, "I can't believe I wrote that. Thanks for fixing this. Cool." So, I close out that code - code review. I still haven't tested this; haven't even built it. Oh, yeah. If you type it three times, sometimes it works. Now, CL is unfound. Locate CL. All right. I don't even have locate.
I'm not going to be able to build this. But - but we do have a build server, and the build server is watching for check-ins, and it's looking for any check-in that has a tag. This is the check-in I just made, the change that I just made. It's looking for anything that has a tag that starts with the word build, and if it finds it, it - it pulls out that - that version and makes a build, then throws it on a shared drive. Build test Case 537. So, I'm going to kick off a build that way, and you can see that that test server is - that build server is now building my codes and there's the tag that I just added here.
And while that builds, let's go back to my talk here. All right, so marketing team again. FogBugz and Kiln allow you to synergize your full functional multi - all right. You guys read that. I'm going to check if the build is ready. Yes, that's my build. That's block clock. Let's see if that actually fixes it. Hopefully it will. Moment of suspense. It actually looks smoother, and there was no minus one, and there was no crash. I'm going to call it a fix. Cool.
So, remember I've got the fix here in the testing repository. It's not in the main repository, so nobody else is going to get the benefit of this fix unless they know to look in the testing repository. When I want to take that change that I made in the testing repository, give it to everybody, I'll go to the In/Out tab and. In the In/Out tab, basically every repository has a little cue of changes waiting to go out and changes waiting to come in. So, this is the outgoing cue and there's - there's my fix right there, and there's the tag that I added. Adding a tag is actually a change too. And this is that stupid thing that Galen did a while ago, that was adding a version number, which didn't really fix anything. And when I click push, Kiln is going to take those three changes and push them up to the main countdown repository. And you can see now in the main countdown repository, those three changes are now there.
So, one thing which you do all the time with distributed version control is you take changes and you make them in some little local private repository. You can even make your own - like I could even make my own little repository here. I could just say, "Make Joel test - testing repository," and just make a dinky little repository. And this repository already, if I go in there - already has everything. It's got the current history of where it came from, and that was done in an efficient and super fast way so that was low cost.
And now I could put my changes in there, and they would be on the server, and they wouldn't be bothering anybody, and I could check them in on Friday at 5:00 when I wanted to go home. And - and once they were all done - or once they were ready, when I wanted to share them with people, I could move them to a different repository.
Okay. So, I fixed a bunch of cases. I'm kind of running low on time here and I want to take questions. Let me - let me just close out that case. It was - it was in the countdown project, and it's assigned to me. It's not assigned to Galen. And, that one? No, that's a different one. No, that says blue screen. So there's a bunch of blue screen crashes here. Oh, there's another one. Okay. We've got a bunch here that I'm - that I fixed. So, I checked off those three and I'm resolving all three of these at once. Fixed with Case 537.
And one - one thing you'll notice here. Since I'm working on multiple cases at the same time, there's a little preview window here so I can sort of scroll left and right and see those three cases, and that's just a quick way to work with both bugs. If you're confident that it's really all the same bug, you just close them. But if you actually want to preview them a little bit, you can do that. So that's closed.
So, let's look at that case, that 537 case, once again. We have here the complete history. We have here a link to the code review, and we have a link to the check-ins in Kiln that actually resolved that bug and not only do we have that, but we can see a complete list of all the repositories that have the fix.
Now this is really important. When you have lots of repositories, you might have a QA repository or repositories for different customers that want slightly different versions. It's really nifty to be able to see which repositories have that fix so that you can be certain. I mean, this is the usual. This is - this is the standard bug flow, right. QA person finds a bug, reports it; developer fixes the bug; sends the bug back to the QA person saying, "All right. It's fixed." Resolves it. QA person tries the code. Of course, it's not fixed. And then, they need to go - stand up, go down the hall, ask the developer where the code is that allegedly fixes this bug. Is it - do I have it even yet? Have you checked it in yet? Is it in - what is it checked into? Do I have it in my repository yet? So, we just answer all those questions right away for you, and we solve sort of kind of the number-one problem that QA people have that's been ignored over all these years. So, I have all that information all kind of tied up in one place, which is really useful, and it's all cross-linked from place to place. If I go back into the code, I can see where the bug was. And if I go into the bug, I can see where the code was. All right. So, that's it.
We have a - I just want to update one last thing. In the week since I fixed this, we have a documentation page in the Wiki. The Wiki has been vastly improved in FogBugz 8. It's got nice navigation features that it didn't use to have, and it's got a completely better editor. So, I just want - I just want to edit this little comment, "Joel finally fixed this," and save it. And I don't need to comment. Done.
Okay. So, right, back to the slide show. How many slides do I have? I have 89 slides and I want to take some questions. So, I'll just go fast. Okay, FogBugz has issue tracking. You can - there's a case list. All right. We saw this, cool. Skip that one. You can report bugs. You can track things, yeah. There's a case history. You saw that. Advanced search, okay you can search. You can create outlines of tasks and hierarchies. You know, actually most of this material - we have a new marketing team, but they're also lazy and they appear to have cut and pasted this from our website. So, if I'm going a little too fast for you, then, homework, www.fogcreek.com, you'll find all this stuff there.
You can create outlines, hierarchical outlines of tasks if you're an MS project fan. You can enter estimates for things, and check milestones. And then we have evidence-based scheduling. It tells you when you're going to ship, and there are developer timelines, and burn-down charts for those of you using Scrum, and the history of the developer, and time sheets, and customer emails. You can get email from customers and we automatically sort it out, send it to the right person. It's magic. Editor, knowledge bases, organization, all this stuff, and branch and merge.
Okay, yeah. Hosting. We host this for you. FogBugz says - FogCreek has a professional hosting, professional servers. It's optional. You can run it on your own server if you are masochistic. No, it's not so bad. That's not bad either. It's pretty easy to set up. Oh, large binary file support. The one thing that people always ask me is, if you're making a complete copy of the entire repository, that's all - every time you want to make a change to the code, that's all good and fine for source code like text files. But, you know, we have a website, and website has a bunch of movies on it. There's always a different version of the movies, and gigantic flash files, and huge JPEGs. And all those old versions are - make it impossible to checkout an entire old repository to clone an old repository because there are so much of them. So, we added an extension to Mercurial, called the Kiln big files extension. And what that does is take large files that in the history --not the current version, but the ones in the history - and it kind of stashes them somewhere on the server. So, you don't have them locally on your hard drive if it's, you know, larger than a certain size that you configure, but you do have the current version. So, if you ever need to go back in time and history, you have to, at that point, go talk to the server.
Code Reviews, we saw. Code reviews are really a big deal at FogCreek. It's kind of astonishing how many people, all of a sudden, start doing code reviews because it's so easy with Kiln. Like, we never used to have the old school, like, let's all sit around the table and - and pick on your code. But a lot of times, you know, a lot of communication between developers is communication about code and having a tool to do it, means that it - that the code review itself becomes asynchronous. It doesn't - you don't have to interrupt the programmer who might be, like, really writing something awesome right now in order to get a code review done. And it's integrated and you can search, and it's advanced in their hosts.
And okay. Here's some comparison charts. FogBugz, better than Excel; better than Mantis; puppy. You don't - you should adopt a puppy. You shouldn't be buying puppies. Okay, not that I don't know what that is. Heroin, pretty much the same. Kiln 2.0 is integrated and awesome, and it has asset management and works with FogBugz and nothing compares to it, and there's way too many slides here. Can we cut out some of these slides from the next - thanks? Yeah, like that. We don't - oh, I like this slide.
Okay, pricing. There's two products. FogBugz and Kiln, they're $25 each. But if you want the glorious package that includes both, it's only $30, so that would be the better deal. You can also install it on your server, if you are so inclined. Then you just pay a one-time fee and a - and a maintenance contract. Now, yeah, I'm done.
Okay. So, I want to take some questions. We have a mike set up here so if you have a question that you think might be interesting to the people that are watching this movie, go - go stand at the mike and say it. Otherwise if you don't - if you're sort of stuck in your chair, or don't want to stand up, just shout out your question, and I will repeat it for the benefit of the people watching at home.
Question, yes?
Question: Can I integrate Subversion into Kiln - an existing repository?
Answer: Can you integrate Subversion into an existing repository from Subversion into Kiln? You're going to want to import it, but it's not as bad as I make it seem. The Subversion to Mercurial import scripts are awesome in that they will allow you to re-import. And the new re-imports are - are efficient. They only get the changes that you haven't seen. So, you've got two choices. One is, you take your Subversion repository and just import it all. Or slightly more recommended, you import it into multiple repositories by sort of breaking it into common subdirectories because a lot of people have the Subversion repositories. They have everything in the world in one repository. And then, if you've still got people using Subversion, that's fine. They can keep using Subversion while you try Mercurial and you get their changes, and you sort of pull them in by re-importing their changes, and - or, you know, you just all switch to Mercurial. Does that approximately answer that?
Yeah, yes?
Question: Did you say something about the relation between Mercurial and Kiln?
Answer: Right.
Question: What have you added and how does it--
Answer: Right. What's the relationship between Mercurial and Kiln? I'll go a little broader than that. Distributed Version Control, there are two projects. They're two major open-source projects that are popular, Git and Mercurial. They started on them about the same day. They're very, very similar. We like them both. The reason we chose Mercurial is that it has better support for Windows developers, and the Git team is Linus Torvalds and he actually has a policy against adding features that will help Windows developers. So - so we - for that reason, we see we have a slight - not that it doesn't work, but we have a slight favoritism towards Mercurial, but we also like Git. So, that's Mercurial.
The tool itself, the interfaces, and stuff like that, we support totally native Mercurial, which means any Mercurial tool that comes out, works on the graphical user interface that I was showing you here, the Windows user interface is an open source thing called Tortoise HG, which is the Mercurial GUI package that's most commonly used on Windows.
The - what we've added is some extensions to Kiln, and so there's that large binary file extension. And there's extensions that allow you to use FogBugz authentications. You can have a common account and common security between FogBugz and Kiln, so that's like one of those commercial things. But then, we've got this whole graphical user interface thing where we're actually kind of managing your - your repositories for you. So, this is the repository management. And if you are using our hosting, these would be the repositories we're hosting for you. There would be security and permissions on them. There's the ability to see an activity feed where you can set up a filter saying, you know, show me all the stuff - actually it's the activity feed - show me all the stuff that people are checking in on my team. And you can narrow it down to certain people. So, there's just a whole bunch of kind of useful tools that we built here on the web interface to Kiln.
And then, there's that whole code review feature. And there's a code search feature so you can search your code. So basically what we're doing is, we're adding the kind of stuff on top of Mercurial that, you know, kind of commercial shops want to use that, you know, little open-source teams of - of four people don't really care about that much.
Yes?
Question: Kiln on Linux?
Answer: Kiln on Linux. Yeah, the client, no problem. The server itself, it requires Windows. The Kiln server requires Windows, or you can have us host it for you. But in terms of, like, what your developers are using, Linux Mac, Windows, it all works fine.
Question: Any plans for the server going to Linux?
Answer: Any plans for the server to go to Linux? It seems like so much work. The truth is, a surprising number of our customers are moving to our - our hosting of the software just so they're not constantly reinstalling things. And the cost for us with FogBugz, we have - the FogBugz server runs on Linux or - or Windows. And just because Windows servers are more consistent, and that they always tend to have the same things on them, whereas Linux servers are all over the map in terms of what they have on them, our experience has been that it costs us about ten times as much to support a Linux server - you know, an arbitrary Linux server than a Windows server. So, it's - it's not something we currently are all that excited about, I'm afraid.
Yeah? Yes?
Question: You talked about a single sign in and…
Answer: Yeah.
Question: Is there a way - if our company is hosting the services you use, like, normal Windows…
Answer: Yes, yes. We have LDAP support and works with Active Directory and I think it works with OpenLDAP because it's just LDAP support.
Yeah, yes?
Question: What is an Electric Dag, or something like that?
Answer: Oh, an electric dag. Yeah, that's really cool. I was going to leave that as homework, but I guess I can possibly show you that. I don't remember where--
Ben, where would I have a good one? Where should I look?
Ben: Probably What Time Is It, I think.
How come I don't have a project for What Time Is It? Is it website? No.
Ben: No, it's under the...
Oh, okay. Yeah, here I am. There's not that much to see here. Let's say that you're looking at a - you're looking at a particular case and you're wondering what - which - who - who - I am here and who - do I have that fixed? Usually, you have - this little thing on the left is the Dag. It's a word from Version Control. It says here's the code, and then it went here, and then it went here, and then it went here. At this point, we forked and then we merged. And usually, it gets way more complicated than that, and you have a lot more kind of weird forking and merging, and stuff like that because on a distributed version control system, people are working on their own copies and the stuff you get like these massive railroad tracks, and the massive railroad tracks makes it kind of hard to see who did what, and do I have that thing, and Fred is telling me that he fixed that thing, and I don't even know if the particular bizarre - if that fixes it. And so, the electrified dag basically let's you point to things on here and see - let's you point to things on here and see - where do I point - here - and see where they're included. It will just take that one particular path and light it up, and - and fade out everything else that's - that's not on that railroad track and it's a really convenient way to see what fixes and what change sets you have or a particular version has.
Yep?
Question: …anything about the cases and sub cases? And this is something that I…
Answer: Yeah, cases and sub cases, I didn't really mention that. Basically, the - the way that works, is you make a case. Let's say that this is a feature main case. And then, you click over here and say add sub case. And you can make it sub case 1, sub case 2, sub case 3, et cetera, and then, you can - you can go - you can go as far as you want, sub - sub case. And all the stuff that you might think would work, still works. Like for example, let's say that you have an estimate column here, and you're putting in estimates for these, like, individual tasks that you're going to work on. So that's a one-hour feature and this is a four-hour feature, and you can see we're actually rolling up the summary of the estimates into the major one. And this main case, maybe it has some work as well. It's got, like, another two days of work in addition to the summary work. So, we'll actually keep track of that. We'll say, hey this is 15 hours of work, plus five hours worth of sub cases in all, and we'll kind of track all that stuff for you for scheduling purposes.
Yeah?
Question: [Inaudible]
Answer: Yeah. Well, I know you're downloading the same amount, but on disk, it's going to take less space because you're just getting the source on the disk.
Question: [Inaudible]
Answer: That doesn't seem to - it doesn't seem to work that way. The compress - the thing is, the diffs are compressed, so that gets to you like with text, with source code, that gets you 60 percent, 70 percent compression right there. So, realistically, it doesn't unless you throw away a lot of code. It - it turns out that most people with Mercurial and Git make smaller repositories. So repositories are project or component not really like all the source code for your whole company like you might do with Subversion. So, it - it really turns out to - to be surprisingly not a problem and the performance benefits you get are kind of unbelievable. I mean, I know of, like, major search engine companies that I won't mention that have distributed development all over the world and they're using version control systems that start with "P" that I won't mention, and they - they literally cannot work togetther on code between large cities in Europe and Mountain View, California because the check-ins are - are so inefficient because there's no - because they have to diff - they're basically diffing it cross continents on a gigantic code base and that becomes, like, ridiculously inefficient. So they do, like, nightly merges if they're lucky. And this is just real painful to them. And with distributed version control, I mean this stuff is all designed for, like, laptop users in Nigeria or whatever, and the only thing you're ever sending is a little tiny diff.
Yeah.
Question: You haven't mentioned much about the Wiki.
Answer: Yeah,
Question: I'm intrigued by the possibility of putting requirements in there?
Answer: Yeah. Putting - do you put requirements in the Wiki or--
Question: [Inaudible]
Answer: Right. In practice, people have done requirements, specs, test cases and stuff like that. They've done that in the Wiki. One thing that's changed with FogBugz 8 is that the cases are now editable in FogBugz, the regular cases. So, a good way to do test-case management requirements are like a tiny feature spec where you don't need, you know, pictures, is - actually I just opened a case and just, and, you know, just write some text and then that actually - it does have a history like a Wiki and anybody can edit it and all that kind of stuff, and that gets you the advantage over the Wiki of having all the meta data like who is it assigned to, and what priority is it, and what priority is it, and when is it due, and which release is it for. So, if you have smaller requirements, or simpler things, simpler, like little one-off feature ideas, it's like, hey, let's add a spellchecker. Those little things can go in cases. And when you need kind of a larger more complicated document, you put it in a Wiki.
Question: Okay.
Answer: I'll talk a little bit - I did show you that the Wiki had a new editor, but it's also got a kind of exhaustive - a really good plug-in architecture. So, just to show you the kind of plug-ins, and we're shipping a lot of our functionality now in the form of plug-ins. One thing you can do - let's say, going into this particular case - this is just a typical example of a - of a plug-in. You can - I don't have Balsamiq on here. That would be cool. You can insert - a lot of times, people want to have conversations about a Wiki article, like - like, they'll send it around for - they'll send something around for review, and they'll say, "Hey, look at this Wiki I wrote. Put in your comments." So now you can say sort of an insert comments right there, and - and then in view mode, there will be a little comment block where you can say, I think this is just great, and kind of keep a little history of the comments in there which you can then kind of hide and show, and so there's - and this is just an example of the kind of - the kind of plug-ins you write. There's one called Balsamiq, which lets you make wire frame specs. Wire frame UI designs and stuff like that. There's just a lot of core stuff is done in plug-ins on the Wiki.
Yeah?
Question: So we're using Trac right now, and Trac has the ability to insert, like, a lot of the …
Answer: Right.
Question: … do you have the ability to do - to bring in links to cases into your Wiki?
Answer: Yeah, but it's just a - it's just a plain link. I don't think we have anything fancier than just like - okay, I'm linking to a case and the case is - and then the link will become bi-directional and stuff like that. I'm not sure what Trac has, but basically if I go in here and I say edit, and I want to make this thing link to that case that I just closed, then I'd say enter, link, and I put in the case number, and I can - you know, I could - I could search for it right here. That becomes a link to the case. You can hover over it and see the case. You can go look at the case. The case has a link back to the Wiki right here. So, it becomes a bi-directional link. That's a pretty good way to tie together, you know, small requirements with larger requirements, documents, or vice versa.
Yes, over there?
Question: [Inaudible]
Answer: Yeah. What would I recommend - the question was what would I recommend for a team moving to DVCS, that to be successful? Well, that's - I think the entire topic of the next talk. So, I'm not even going into too much of that, but Ben will be talking about a lot of that stuff. Most of it has to do with if you don't really grok DVCS when you set up the repository structure, it's going to be especially harder and you're not going to be really getting the benefits out of it. So, Ben is going to be introducing a lot of that stuff in the next talk so I'll just - I'll just defer to that.
Yes?
Question: [Inaudible]
Answer: How do you differentiate between a product backlog and a Sprint backlog? I don't know. What - what's the - is there a difference?
Question: [Inaudible]
Answer: Yeah. We have - there's only one kind of backlog that I know of, and that's just basically Sprint users have this concept of keeping - sorry - Scum users have this concept of keeping a backlog of stuff that you want to do if you ever get around to it, and prioritizing them by numbers, and that's a part of our Scrum plug-in. I'm pretty sure there's only one kind of backlog.
Yeah?
Question: you can also use tags [Inaudible]
Answer: Yeah. You can tag things into Sprint. We actually recommend if you're doing a Sprint that you use a milestone for your Sprints so you have a list of cases and pretend that these are features, though they actually are. You take a few of them and you go in here and you say, set milestone and make a new one. And we're just going to call it Sprint, you know, 25, and you could set a date if you wanted to, or you don't have to. And you've just created a new milestone that reflects that Sprint.
Yes, in the back?
Question: Do you have any plans to integrate with IDEs, such as Eclipse?
Answer: Do we have any plans to integrate with IDEs, such as Eclipse? We have really good integration with Eclipse through Mylyn, which is the way Eclipse talks to boat-tracking systems. We have - I don't - I don't quite know what Eclipse and Mercurial have. I'm sure there's something. We have - in Visual Studio, there's a new version of the FogBugz Visual Studio Plug-in, which is a lot better and it lets you get all your cases and see them in a little docking window in Visual Studio, and there are a couple of different options, open-source options, that are available for using Mercurial with Visual Studio. So, essentially on the Mercurial side, there's all the common stuff is all supported because it's one of these big major open-source projects. I'm sure there's an Emacs mode on the - on the FogBugz side through Eclipse. With Eclipse, you use Mylyn; and with Visual Studio, you use our plug-in.
Yeah?
Question: Any comments around, like, high availability DR tools?
Answer: High availability - which DR?
Questions: Like disaster recovery.
Answer: Disaster recovery of these tools, like when we're hosting them?
Question: No.
Answer: When you're hosting them?
Question: Yes.
Answer: Like, how would you create high availability? I'll - I'll start with FogBugz that - I mean, depending on how crucial your FogBugz data is to you, you can do what we do, which is you have a data center and you do a log shipping to some offsite location so that you have a second physical location that has the database, and that's what we do in our own data center. In the - in the case of Kiln, you actually wind up - Mercurial with distributed version control - you wind up with a much, much more secure environment because of the probability that lots of people will have a complete copy of the repository. So, you - if you have a team of people working on a project, you almost don't have to back up your central repository at all because what will happen is, the entire central repository goes south, you just take all developers and tell them to push all their changes to your new repository and now you have everything and your whole history is reconstructed because you haven't lost anything because everybody has the whole repository, or somebody - somebody must. And you can do similar things like push your repository to an offsite place and you can even set up automatic triggers with Mercurial to push all changes in a repository to a - a secondary location. But this actually took me a while to wrap my head around it that - that you're making so many more copies of things when you use distributed version control that the possibility of actually losing something becomes very slim. And Mercurial, unlike Subversion - actually Mercurial and Git basically use essentially cryptographically secure ways to make sure that change has - has not been tampered with. And so, there is a prove - now this is way out of my range - but it is actually cryptographically provable that if you're looking at a particular revision, that - that you can prove what the exact history of - of changes that went into that were. And I'm totally speaking out of school, but unlike traditional systems like Subversion where you could go into the repo and edit a file, you can't do that anymore because all kinds of hashes would break, people would be suspicious.
Yeah?
Question: We're a mixed hardware and software company project effort to 80 percent, 85 percent building software and 15 percent hardware, but -
Answer: Right.
Question: - we have, you know, real hardware communities, lead times, other things to worry about seems like a gravitational pull on the Gant Chart --
Answer: Gant Charts, oops.
Question: - and do you have other customers that have this - this scenario where you have real --
Answer: Yeah.
Question: - real-world time?
Answer: Yeah, yeah. So, there's - there's a bunch of features for that actually already built into our schedules. You can create external dependency. So, for example, if you're waiting for some event to happen in the real world before developers can do things, you can go put in external dependency into FogBugz. But generally what happens in the type of scenario you're describing, whether it's hardware, software, or even if it's just like another line of business, like it's an insurance company and they're building an oilrig or whatever. You know, one of those like not software lines of business, and you're just the software team, and they keep asking you for data to put into your Gant Charts. Well, what most people will do is that they will kind of up flow the data from evidence-based scheduling into Microsoft Project or - or Artemus or whatever, the crazy thing they're using there so that they can make their Gant Charts. And now you have really good data of what your best guess; you know, worst case; best case; and, you know, average scenario are. You're actually - at least, we can give you some visibility into what your line item should be on their big Gant Charts.
Question: Do you - do you have other concerns that [Inaudible]
Answer: Yes, absolutely. We definitely have hardware vendors who manage the hardware side. It's actually kind of surprising what people use FogBugz for. There's a Canadian Airline called Porter that uses us for all their customer service. And they're not even doing software development. Just because the email - the email features are awesome, the - we definitely have people that work - certainly in the VLSI and chip design, and that kind of stuff.
All the way in the back.
Question: Yeah. What do you recommend for doing source control of SQL server schema?
Answer: What do we recommend for doing source control of SQL server schema? I don't know off the top of my head. We tend to try not to - not to have them live in SQL server personally and that's how we do it ourselves. We have - in the case of FogBugz, we actually have, you know, text files that know how to look at any database and upgrade it to be a FogBugz database by adding tables, and columns, and stuff. But there are - there are real answers. It's just that I just don't know what they are right now. Usually, it's a matter of capturing the SQL server schema in some kind of text form and checking that in. There are tools for that. I don't - I wish I knew what they were, but I - but I don't.
Yeah, one last question.
Question: Yeah. Evidence-based scheduling. You've talked a lot about that.
Answer: Yeah.
Question: Is - it's great for managing expectations of management and your realistic
Answer: Yeah.
Question: Well, we found it's breaking down as we moved to more of a Scrum approach where the is estimating a case [Inaudible]
Answer: Right.
Question: And there's a pool of those that anybody can take and pull up and work on them.
Answer: Yeah.
Question: And so, do you guys have a recommended way to approach - because in the way it's set up now, eight persons --
Answer: Uh-huh.
Question: [Inaudible]
Answer: Right.
Question: - and then they track their time --
Answer: Yeah.
Question: - and then, you know, [Inaudible]
Answer: Theoretically, evidence-based scheduling knows about the concept that the estimator might not be the same person that actually implements the task. And so, the data that it's going to use in terms of your track record and the Monte - for the - that it's going to use for the Monte Carlo simulations is based on who estimated it, not who did it. On the other hand, if you're frequently finding yourselves in a situation where there is a different person estimating tasks than actually implementing them, those estimates, I can tell you right now, are not going to be as good -
Question: Right.
Answer: - as they would be if the - Now, if you're - how long are your sprints? You said you're doing Scrum. What kind of - what length sprints are you doing? Like, two weeks?
Question: Two weeks.
Answer: So, the truth is that evidence-based scheduling is not - although it's awesome, it is not - I wouldn't say it's required on the average Scrum team. You know, a Scrum team, it's really for a larger, longer length projects. And if you can really kind of iterate every two weeks and kind of knock something out, and people are happy with that, the idea is that you're kind of not worrying about the schedules because at least you're doing things in the right order and everybody is kind of happy. So, there's sort of a - it's not - let's say - let me put it this way. It's a - it's a good methodology for when, like, the really agile development methods just aren't going to work because you're building an oilrig for, you know, there are reporting requirements or somebody actually needs your software to be ready on a certain date and it's got to have a bunch of things, or you want to do kind of a lot of planning ahead, or you just want to try to figure out which of several major features to do based on how long they take. But a team that is really, like, super agile and fast iterations and stuff like that is not likely to - not likely to need it really.
Yeah?
Question: That's kind of where we [Inaudible]
Answer: Yeah. There is no shame in that. We don't use it as a stack overflow - shameful.
Okay. I'm going to - I'm sort of out of time here. But there are a bunch of us here from FogCreek and we'll be taking your questions afterwards if you have anything more about FogBugz specifically. But for now, I want to turn over the podium to Ben Pod, my colleague, who some of you will remember from the movie Aardvark. Anybody see that movie, Aardvark, Aardvark, Aardvark, Aardvark? He's the one that wrote the bug in the movie. That was the plot there. And he's going to - he's actually the inventor of - of Kiln and the lead developer on that project, and he's going to give you a little bit of talk about distributed version control and hopefully - and it's relative to Kiln, but everything he's talking about is going to be just as useful if you choose to use Git or use Mercurial without Kiln, or one of the other bizarre versions of Distributed Version Control.
So, please welcome Ben Pollack. (Watch Benjamin's talk on Distributed Version Control)