The Most Rewarding 3-Year Undertaking Ever

Last week I finished studying the Bible all the way through. It took about 3 years (one or two chapters per day) and was very rewarding. I am fascinated about all the things the prophets said would happen that happened. It has really strengthened my faith in Jesus Christ. It's all so clear now.

My Testimony

Those of you who knew me in school or at IBM or ISS may be amazed at how I've changed. I'm not referring to physical appearance. I'm talking about my attitude, happiness, outlook on life, health -- that kind of stuff. I've done many things I'm not proud of. I've even been the date from hell at least once. My biggest life-long problem has been controlling my temper and dealing with anger.

While at the University of Alabama, an ex-girlfriend told me "you sure are an angry young man." That was an eye opener. I struggled to change myself for years but had no success.

Somewhere in the early 2000's I started hanging out with a better crowd. Three men in particular had a great impact on me: Eric Smith, Garry Holton, and Carl Jameson. I met these guys at the Atlanta Street Baptist Church. They pointed me back to Christ. I turned my anger problem over to God by renewing my dedication to Jesus Christ. Since then life has been fantastic.

I thought life was good before, but I never knew how great it could be. I couldn't see what I was missing. I'm much more easy going now; full of peace. I'm much happier.

Hurricanes and Rockets: awesome power

I had the privilege of watching Discovery rocket off its pad and disappear into orbit. Wow, what a complex and powerful machine! I saw the cloud of smoke engulf the ship. I heard a bang and then saw Discovery lift out of that cloud. The flame from the engines was so bright it hurt my eyes. As its column of smoke grew taller its roar grew louder. 3000 miles per hour now. I could barely see the solid rocket boosters fall off. 5500 miles per hour. It was just a spec. I don't hear its roar any longer; it dissipated as slowly as it grew loud. Then, I could see Discovery no longer as it was off chasing the International Space Station. The astronauts were taking off their orange assent/descent suits before I found my car in the parking lot. 17,500 miles per hour and the astronauts are changing clothes.

A couple of weeks earlier I was at the beach when hurricane Dennis was out in the gulf. Dennis was making the waves pound the shore way over on the east coast of Florida. I was in awe of the power of the waves and even more in awe of the power of a storm having an impact so far away.

Rockets are pretty impressive, but nature can be much more powerful.

I have been reading my way through the Bible for about a year and a half. At the beach I had just gotten to Psalm 93 (New Living Translation);

The LORD is king! He is robed in majesty.
    Indeed, the LORD is robed in majesty and armed with strength.
The world is firmly established;
    it cannot be shaken.

Your throne, O LORD, has been established from time immemorial.
    You yourself are from the everlasting past.

The mighty oceans have roared, O LORD.
    The mighty oceans roar like thunder;
    the mighty oceans roar as they pound the shore.

But mightier than the violent raging of the seas,
    mightier than the breakers on the shore--
    the LORD above is mightier than these!

Your royal decrees cannot be changed.
    The nature of your reign, O LORD, is holiness forever.

Now, that's something to be in awe of.

Be specific

We were having a problem at work that we just couldn't solve. I had my best engineers on it. I looked at it myself multiple times. Toby tried lots of stuff, even reinstalling things. We thought about it and worked on it for three weeks. I even got BEA's support staff involved. The application worked just fine on the previous imaged hard drive. We couldn't find anything on the new hard drive image that could break our application. How frustrating. I was feeling like we had gained no ground and were spending all our time on this issue. We finally figured it out, but what's interesting is how:

I prayed specifically to gain some ground on one of two issues we were having. As soon as I was done praying, BAM! God gave me the solution. I walked over to the computer, tried it, and it worked!

Wow.

I believe God answered that one so swiftly because He knew there would be co-workers around that would ask how I came up with the solution -- that He would be praised and so that it could be shown that He is active in our lives and in our company.

I often pray about work. It's generally a generic prayer for help or wisdom or for the lost, but the results are far greater when I get specific.

Easter: The Resurrection of the Christ

The Greek historian and physician Luke wrote a fascinating account of the resurrection of the Christ, which picks up where Mel Gibson's wildly popular film The Passion of the Christ leaves off. After his resurrection from the dead, Jesus appeared 11 times over 40 days to more than 500 people! It's a fact. Jesus is alive. Read Luke's account.

Everyone has sinned and the punishment for that sin is separation from God. Jesus took our punishment for us -- he died in our place for our sin. He was separated from God for a short time, but he conquered death because he was sinless. Everyone who accepts this free gift gets to spend eternity in paradise. Everyone who rejects this gift will suffer eternal punishment for their sin. They will suffer eternal separation from God. And that is hell.

There is an afterlife and everyone, including you, will be in it. Where will you be then?

Attributes of a Great Manager

I've been thinking about what makes a good software development manager for quite some time. A friend recently asked for my thoughts on this so I thought I'd jot them down. Let me know what you think.

People management is top priority. Developing people is what it's all about. The job is to empower, delegate, motivate, reward and recognize. And it is to be done frequently. And that is all. It's okay to be a customer advocate given spare time, but that should not be the manager's job. It's good to use the application his group is building. And it's good to understand the competition. But QA and competitive analysis is not the manager's job. It's okay for the manager to be a proxy for the customer for a short while, but that's the product manager's job. And the manager can be the benevolent dictator (final arbiter) if there is no one else to do it.

A manager should encourage, reward, even expect continuous learning. This means more than just paying for appropriate books and sending a few employees to training each year. It also means encouraging employees to keep learning and sharing what they learn. Recognize employees that are involved in study groups and user groups. Encourage the use of software development best practices.

A good manager will take time to meet regularly with employees, in small groups and one-on-one. You'll likely find a good manager eating lunch with his employees or chatting in the break room. You may even find him playing video games, ping-pong or foosball. High levels of communication is very important. It's the best way to understand the organization's health, to discover employees' career goals and aspirations, and to find out what type of work they enjoy. Without this understanding, how can a manager mentor and develop his people?

Regularly is also the most appropriate time to give feedback. Managers should comment on good and bad work right away. Don't wait for the end of the project. And certainly don't wait for the annual review.

Perhaps the most difficult aspect of good management for most of us techies, oddly enough, is fostering creativity. We are all pretty creative. But we get in ruts and lose our edge. A good manager spends his time thinking of how to keep our brains creative.

I don't think a manager necessarily has to have been a programmer, though it can help. It helps to be technical enough to be credible, to have something in common with the engineers. And it helps to know when someone is trying to pull the wool over your eyes. It's useful to know how to do all the stuff I say don't do (below), but develop those skills in others rather than do the task yourself.

One of my favorite quotes is from Andy Hunt.

Management's responsibility: remove the obstacle. Team's responsibility: identify the obstacle.

It's the idea of management serving the employees, not the other way around. The employees get the real work done. Managers make sure the right folks are trained and aware of what needs to get done. Managers make sure employees have the information they need to do the job well.

A manager should not spend his time "doing" stuff. Richard Lowe expresses this best: "This means ALL tasks must be delegated, except for those tasks directly related to getting other people to do their jobs." The mistake is "doing actual work instead of managing." If a manager's job is to observe, motivate, reward, develop people, improve morale, and all that other stuff, then, if done well, there will be no time for anything else.

Managers should never make design or coding decisions. Nor should they make architectural decisions except to enforce whatever constraints come from product-family concerns or testability, reliability, redundancy, serviceability, availability, and other requirements of that ilk. But managers often can't resist. Where does this come from?

Often we take good developers and promote them into management. Not because they possess the attributes of a good manager, but for all the wrong reasons. "Fred did a great job on the hoohoppy project. He handled all the technical details. He's our most senior developer and we have this open management position..." This makes about as much since as, well, as me trying to make a weak analogy right now.

Of course it isn't always quite so black and white. Sometimes a manager may find himself in a situation in which no one knows how to do some necessary and urgent task and either the manager has a real junior team or has some seriously tight deadline. In this situation, I don't mind if a manager pitches in to learn how to get this thing done. But it shouldn't become the manager's job. Rather, he should teach someone else to do it and delegate that responsibility.

What about project management? Isn't that a manager's job? I don't think so, though that's where it usually ends up. Good managers should train up their team-leads to handle that responsibility. Managers should ensure that it gets done and should teach others how to do it if necessary. It's the same with risk management. Ensure it gets done, but don't do it yourself. "We need to have more trust in one another," Lou Gerstner Jr in his tenure as CEO of IBM wrote to employees -- "confidence that our colleagues can lead and do the right thing, so we don't have to attend every meeting and check every detail."

There is more, of course. But it all involves thinking about what needs doing, rather than the doing itself.

It should go without saying: Treat employees with a great deal of respect. Treat them like the intelligent human beings they are. Sadly, I've seen a few managers that just don't understand this. Or they never spend time in self-reflection, examining their behavior. So I'll leave you with this thought. Managers, reflect on your organization's effectiveness. And reflect on your effectiveness.

P.S. Before anyone points out that I don't walk the walk, let me add that all of this is very hard. To do it well requires either a tremendous amount of courage or like-minded upper management who will hold us accountable.

Iteration Planning – Story Breakdown

There is a certain set of things you should consider during Iteration Planning when breaking stories down into tasks. Here's a short check-list I created a while back to post in the XP-lab as a reminder. Other teams have done the same but with different items to consider. In our retrospectives we notice things we should have thought of and those items get added to this list.

Consider:

• Acceptance tests
• Risks
• DB impacts
• Design sessions needed
• ...
• Error conditions
• Boundary conditions
• Time-zero
• Front-end/back-end impact
• Permissions
• Audit logs
• ...

Programmer Velocity

Authors: Andrew Fuqua, Greg Houston, Matt Di Iorio

---

Each team may define programmer velocity differently. But within a team, every engineer must use the same definition – their velocities must take the same things into account. If not done consistently, tracking and load balancing will not work and the Iteration and Release plans will be unreliable. If you have not yet started using XP, this paper is not for you – Read this after reading the XP trilogy (see bibliography). If you are using XP, read on – this is how one team in ISS defined Programmer Velocity. This is what the XP literature calls a "local adaptation". More recently, teams in ISS use team velocity and do not track programmer velocity. But back when we did, this is how we did it.

Ideal Engineering Days

Before discussing velocity, we must define Ideal Engineering Days (IEDs). An IED is when everyone just leaves you alone to program. Unhook the phone; close Outlook, ICQ and AIM; cancel meetings; and put up a do not disturb sign.

Naturally, we never get 5 full IEDs in a week. You do have to go to meetings. You do have to check email and voicemail. And you do have to help others. In our case, the Events team, helping others takes the form of pair programming. If your team is not pairing, helping others takes the form of peer reviews, code walk-throughs, integration testing, and chalk talks.

Still, you might think that the more IEDs you can get in per week the better. Not necessarily – You may be neglecting something. You might not be spending enough time helping others, refactoring bad code and writing tests. If that's the case, your code will be difficult to maintain and your team will suffer.

We estimate tasks and stories in Ideal Engineering Days. We might also call these XPUs (eXtreme Programming Units). Ideally, stories should be broken down into tasks that will each take ½ to 2 IEDs.

Programmer Velocity

Ok, so we estimate in IEDs. What about all those meetings, phone calls, emails and interruptions? What about peer-reviews or pair programming? How do we account for those, and how do we deal with inaccurate estimates? Programmer Velocity must take all of that into account. Reread this paragraph to see what must be included in your velocity.

In the last iteration, I completed my tasks (those that I signed up for and am responsible for), originally estimated at four IEDs. It was a two-week iteration. So, I spent about three days each week pairing with others on their tasks, writing documents like this, attending meetings and fixing defects. In this example, my Velocity is 4.

Velocity is simply the original estimate in IEDs for the tasks you owned and completed last iteration. This is the definition used in the green "Planning eXtreme Programming" book. If I completed tasks originally estimated at five IEDs, then my velocity is five. It doesn't matter how long it actually took. What you count is the original estimate of the stories/tasks actually completed.

Each programmer does a different number of IEDs per week. Everyone's velocity is different.

•  Velocity takes into account how much "other stuff" you do. The more "other stuff" you do, the lower your velocity.
•  Velocity is also a measure of how over-conservative or ultra-aggressive your estimates are.
•  Velocity is measured. After each iteration, look to see how many IEDs you originally estimated for the tasks you completed. If you picked up more tasks during the iteration, add the original estimate for those into your velocity. If you did not complete all of the tasks you signed up for, do not add the estimates for those into your velocity. Remember; only count IEDs for tasks you led. Don't guess – measure it.

You may want to compute velocity over as few as one or as many as three iterations. If your velocity is stable or steadily moving in one direction, use "yesterday's weather" [Planning Extreme Programming, pages 33-34, 90, 117-118]: Your velocity for this iteration is your measured velocity from last iteration. If your velocity is unstable , first make it stable. If you can't, then measure velocity over more iterations.

There is no ideal value for velocity. However, if your velocity is more than 5 in a 10-day iteration, consider whether you are spending enough time refactoring, testing and helping others.

Managers, note: Velocity cannot be used to measure programmer productivity or value. "It's a result, not a control variable." [c2 wiki] It would be inappropriate to compare programmers by their velocity.

How to use an Individual's Velocity

After programmers sign up for and estimate tasks, the tracker balances everyone's workload. Programmers that have too much work give some tasks to other programmers. Balancing activity goes like this:

Total IEDs estimated for this iteration (my tasks)

VS.

My velocity.

Move tasks from programmers that have signed up for too many IEDs to those that have too few. Also, add or remove stories from the iteration to fit the actual size of the iteration.


A common mistake is to divide the number of days in the iteration by 2 "because we're pairing." Don't do this. If you have a 15-day iteration, do not divide 15 by 2 and say there are only 7 days. There are 15. Pairing is accounted for in everyone's velocity. If you multiply the task estimates by velocity AND divide days by two, you are overcompensating (double counting).

Example: Fred's velocity is 2.5. Fred should sign up as lead for 2.5 IEDs worth of tasks. These are tasks that he "owns" and is responsible for. Do not count tasks he intends to do as a partner this iteration – that's accounted for in his velocity.

Example: My velocity is 3.5. I sign up for tasks. My estimate comes out at 5 IEDs. I'm over loaded so I give a 1.5 IED task to someone else. Now I'm balanced. During the iteration, I get behind and need to defer ½ IED of a task to a later iteration. At the end of the iteration, I measure my new velocity. Since I didn't complete all of my tasks, my new velocity is 3 – the original estimate for those tasks I did finish.

Example: Chet's velocity is 6. So is Joe's. Chet and Joe correctly sign up for 6 IEDs for this iteration. Halfway through the iteration, the tracker notices that Chet is falling behind. Chet agrees to give one of his tasks, a 1.5 IED, to Joe, who is making great progress on his tasks. At the end of the iteration, Chet completed tasks originally estimated at 4.5 IEDs. Joe completed tasks originally estimated at 7.5 IEDs. Now, Chet's new velocity is 4.5 and Joe's is 7.5.

When tracking, I prefer to only track and balance a programmer's primary tasks – I don't try to balance work as pairs and I don't sign up as pairs. I like to keep it light. However, signing up and tracking as pairs actually gets people to pair up, so it may be worth the extra effort for your team to track and balance pairing activity as well.

Miscellany

Engineers should pair with others as much as others pair with them. For example, if Sue signs up to lead 5 IEDs, she should pair with someone else for around 5 IEDs. This will happen naturally if the tracker balances the workload.

If Sue doesn't help others with their tasks as much as others help her, some other team member has to pick up the pairing slack. This inflates Sue's velocity and deflates someone or everyone else's. This is not a problem as long as Sue is consistent – otherwise velocities will be unstable and less useful.

Special assignments, such as writing essays like this, can also impact your velocity. If a pair will be working on the assignment and if you can estimate it like a programming task, then treat it just like any other task in the iteration. If not, budget some of your time during the iteration for the special assignment – treat that time as unavailable. At the end of the iteration, measure your velocity as if you had fewer calendar days than everyone else.

Variations

Your team does not have to use the velocity definition presented above. You can come up with your own. I used to use a variation that used lots of messy division. It was based on the (now deprecated) Load Factor math from the white eXtreme Programming Explained book. I don't recommend it.

Here is another approach you could use. If your team thinks and estimates in terms of average size stories or tasks, then your team may want to define programmer velocity in terms of points per iteration, w here 1 point is an average size unit of work. For example, I might be able to do one point per iteration, whereas Charlie might crank out 1.5. My velocity would be 1 and Charlie's would be 1.5. Since tasks are not all the same size, Charlie can take on a task that is 50% larger than average, whereas I should work on an average one-pointer or two ½-pointers.

Pick an approach that works well for your team.

Story Velocity v. Task Velocity

Above, I defined velocity as the original estimate in IEDs for the tasks you owned and completed last iteration. You also have a story-velocity. Your team may decide to track Story Velocity or both Story and Task Velocities. If you decide to only track one, Story Velocity is more useful in that it can be used for both Release and Iteration planning. Task velocity is only useful for Iteration Planning. You can't do Release planning without your story velocity. (You cannot predict your end-date if you don't know your Story velocity.) I personally believe this and it's also what Ron Jeffries said at XP Universe 2001.

Why wouldn't the two velocities be the same? Because stories are estimated at a higher (and less accurate) level of granularity. Also, story estimation is done less often and may have been done with different assumptions or different team members. Story estimation should be done by the whole team as a group, but some teams only have a sub-set of the team estimate the stories (not a good idea). Conversely, task estimates are done more often and are usually done by only 1 or 2 programmers. They are done at the start of the iteration in which they'll be implemented, which is the most accurate time to estimate.

Use Task velocity in Iteration Planning to see if the team or any team member is over-committed for the iteration.

Use Story velocity in Release Planning to get a rough idea of what stories are in what iteration.

Update BOTH velocities at the end of EVERY iteration.

Team Velocity v. Individual Velocity

In the definition above I said "you or your team". Some teams track Individual velocity (more paperwork, more granularity, more accurate balancing). Other teams track Team velocity (less work, lightweight).
It's up to your team to decide whether to use Team or Individual Velocity.

Bibliography

Kent Beck, eXtreme Programming Explained, Addison-Wesley, 2000. (The white book.)

Ron Jeffries, Ann Anderson, Chet Hendrickson, eXtreme Programming Installed, Addison-Wesley, 2000. (The purple book.) Pages 64 and 65 are especially useful.

Kent Beck, Martin Fowler, Planning eXtreme Programming, Addison-Wesley, 2000. (The green book.)

Refactoring

Authors: Chris Simpkins, Andrew Fuqua

---

A few months into development our director paid us a visit. Being a wise veteran of the software industry, she was concerned how well our system tolerated changes in requirements and scope. Many projects handle such changes poorly. Without hesitation, we confidently explained that our system could not only tolerate a significant amount of change, but that we could keep the system working continuously while making the changes. How could we be so bold? Are we hypercompetent überprogrammers? Or foolhardy script-kiddies who don't fully understand the question we were answering? The key to our confidence lies in our consistent application of refactoring. If you'd like to learn more about refactoring, and perhaps become half as good a programmer as us in the process, read this brief overview, then visit the resources listed at the end of this paper. 

What Is It?

Refactoring is changing code you've already written without breaking anything. More formally, it is changing existing code to improve its internal structure while preserving its external behavior. You are already refactoring, though you may not know it: You find a method that needs another parameter to do its job. You find a class which should be split in two. Making these kinds of changes is what refactoring is all about.

Often we make such changes haphazardly, perhaps even ripping up the entire system with the intention of putting it back together in an improved form. Often we break something along the way. But we don't have to. We can introduce discipline into this process. Like with design patterns, we can use a set of common "refactorings" – commonly needed changes to code. Each "refactoring pattern" describes the applicability of the refactoring and a proven step-by-step process for it. We don't have to re-invent the process of changing code every time we do it; we can follow a recipe developed and refined by masters of the art.

What's So Special About It?

"You said we're already doing it – why formalize?" The "father" of refactoring once wondered the same thing. Martin Fowler developed the idea of formalized refactoring after watching Kent Beck make large changes in a system by applying a series of very small changes.

Beck would break a large change into smaller steps and follow a painstaking process of making one change, running his tests to make sure everything still worked, and then repeating this "change, test" process until all the steps were done.

At that time Fowler's own method of refactoring was the slash-and-burn method described earlier – tossing up the code and putting it back together again – with all the debugging headaches that entails.

Fowler noticed that Beck was much faster and more successful at making changes using his incremental approach. The key to understanding why Beck's approach is better is realizing that smaller changes are much easier to understand and control. If you make large changes and the system breaks, you have to do a lot of sleuthing to find exactly what broke the system. But if you make very small changes, testing your system after each one, you know exactly what caused the problem and it's easy to undo the change.

A major tenet of software engineering is breaking large problems into chunks manageable by the human brain. Refactoring applies this principle to the process of changing existing code.

Can I Use It on my Project?

Yes, if you've satisfied a couple of prerequisites. The biggest prerequisite is that you have automated unit tests in place for the code you're changing. This is vital. You must be able to assert that your changes didn't break anything.

Another prerequisite is having some sort of source code control in place. A major benefit of refactoring in small steps is that it allows you to abort the process if you find it's not working. Source control allows you to do this by returning your code to its pre-change state.

How Will It Help me on my Project?

How many software projects have you worked on where the finished product was exactly as envisioned in the initial requirements? All software systems change during the course of development, and a large part of a team's success lies in its ability to deal with change. Refactoring will make your code better, allow you to spend less time on up front design, and give you confidence to make changes. To be successful, we must embrace change – not fear it. It's a fact of life. You have two choices: adapt, or die. Refactoring helps you adapt.

I'm Convinced! (or Just Curious) Where Can I Learn More?

Fowler, Martin. Refactoring: Improving the Design of Existing Code. Reading , Mass. : Addison-Wesley, 1999. This is the book on refactoring.

Martin Fowler's Refactoring web site contains a wealth of information, including the PhD thesis that started the formal Refactoring movement, a constantly expanding catalog of refactorings, and links to several Refactoring resources.

A refactoring plug-in for (X)Emacs. Yet another reason we should all be using the One True Editor (tm) :-)

Continuous Integration

Authors: Matt Di Iorio, Andrew Fuqua, Charlie Hubbard

---

Have you ever been on a project that practiced big-bang integration – wait until all the features are done, then throw it all together and see what happens? It usually turns out just like the name implies – a totally chaotic, primordial soup of software. Just getting the code to compile is an amazing feat. Then come the bugs. How painful it is to harness the power of nature to integrate your code!

Or say you have a daily build. But for some reason your daily build is broken daily. So you spend half the day tracking down the slacker who checked in the bad code.

"Ah, but my daily build never breaks", you say. Great! But have you ever spent more than a day tracking down one bug? I thought so.

Our team has overcome these problems using simple techniques: We check-in frequently. And we build and test continuously.

"We measure success one build at a time"

Our automated build and test process runs continuously – not daily. Well, it actually runs every half-hour, but that's close enough to continuous for me.

By building and testing continuously, we find it trivial to figure out what broke the build. Given the short time span between builds, the system is usually built with only one developer's changes at a time. If it broke, we readily know why.

By checking in frequently, we avoid file contention. This means never having to merge two sets of changes.

By doing both of these, we never need more than a few minutes to integrate a piece of code into the system. The code in MKS is always buildable – it always works – every minute of every day. I can't stress this enough: it is never in a broken state. That certainly makes integration easy. And since we check-in frequently, we never have much to integrate.

XP's practices build on each other. That's what makes the process so strong. Like XP, continuous integration's practices are multiplicative. Continuous integration draws its strength from good unit tests and frequent check-ins: The tests are more valuable because they are run more often. The frequent builds are more valuable because by running the tests they do more work each time.

Our House

You will get some benefit just by increasing check-in and build/test frequency. We discovered that some additional things we do make continuous integration even better. You may have different needs and different practices. But here is what we do.

Farewell Make

We broke our love affair with Make. She was never a very accommodating partner – always wanting things to be perfect, never compromising. I wanted spaces; She wanted tabs. We just couldn't go on.

We started using a tool called Ant. It's an extensible, cross-platform build tool written in Java. When we discovered that it uses XML for the build file, we just had to use it. Ant pulls from MKS, compiles the code, builds and signs the jar files, runs the tests, and lets us know if anything failed. A Windows Scheduled Task kicks it off every 30 minutes.

One cool Ant feature is the ability to write "build listeners". We wrote a listener to yell at us when the build fails. To notify the developers we use a "net send" command. We wanted to make it intrusive as possible so the developers couldn't ignore failures.

Developers need to build and test on their workstations before checking in their changes. Our IDE allows us to build and test. But we also wanted each developer be able to build with the same process as on the "official" build machine. This required us to modify our architecture but it helps keep the build running smoothly.

"Just the facts, ma'am"

Once you start running an automated build, you've got to see the results. Our results are on a web page. Prominently shown at the top is the Success / Failure message. Then come the details. If the build was not successful, you can see which files didn't compile or which tests failed.

When the build does fail, it only takes us a couple minutes to find and fix the problem. (Remember all the benefits I mentioned above?) In keeping with the spirit of "continuous", we don't wait 20 minutes for the next regularly scheduled build. We can kick off an unscheduled build at any time from our build page.

Our build page even has a cool plot of the number of unit tests run each day. That helps us focus on writing more tests.

Keep it up

It's always good to be on the lookout for things that you can do to make your life easier (or safer). For example, Mike Slifcak suggested we build the database from scratch before running all the tests. This ensures our production code and our install code don't get out of sync. Now we have continuous integration of the install and production code. See how much you can automate.

Our main goal with the build process was to get the most value by doing the least amount of work. There were lots things we could have added to our build process, like automatically logging all the changes between builds. But that wasn't necessary and would only give us more to maintain. Stay light!

Links

http://www.martinfowler.com/articles/continuousIntegration.html
Martin Fowler's article on continuous integration that got us started.

http://jakarta.apache.org/ant/
Here's the great great Ant.