Continuous integration with extreme builds

A known best practice in software development is Continuous Integration. It is better to integrate often than it is to integrate rarely because the effort to resolve integration issues grows exponentially with the volume of change that is integrated.

Now let’s do some math. Suppose we have a project that runs a complete build (including a smoke test) in 10 minutes. Given an exponential growth of integration issues with a growing volume of changes, suppose the following average times to resolve integration issues:

  • 1 hour interval: 10 minutes to resolve integration issues
  • 2 hour interval: 40 minutes
  • 3 hour interval: 90 minutes = 1 hours 30 minutes
  • 4 hour interval: 160 minutes = 2 hours 40 minutes
  • 8 hour interval: 640 minutes = 10 hours 40 minutes

So when running a nightly build, it is not possible to resolve the integration issues the next day anymore.

Now suppose we have a large system where the build takes 1 hour to run. So the minimum build interval is 2 hours. But a large system also has a large development team, so the volume of change in 2 hours results in more than 40 minutes to resolve integration issues. Let’s say it takes 50 minutes. So the build runs for 1 hour and the repairs take 50 minutes. So you have 4 runs per day, some of them will not even result in a successful build before the next run.
You can probably imagine what happens if the build+repair time starts exceeding the build interval: the project will become instable because nobody can trust the build anymore. Increasing the build interval will make things exponentially worse!

This may seem like a theoretical problem, but I have worked for several organizations where the build run takes more than 1 hour to run, sometimes even 4 hours. There you have the challenge to make the organization productive in software development that I am involved with…

About Frank Schophuizen (fschop)

Hi, my name is Frank Schophuizen and I am working as a consultant in CM, Agile and ALM for TOPIC Embedded Systems. I have over 30 years experience in software development in the technology industry, with the last 15 years mainly in process improvement, deployment and integration of methods and tools in the area of Agile (SAFe, Scrum), CM and ALM. I am strongly interested in the complexities of collaboration and integration in multi-project and multi-site organizations. I have worked with various technology companies such as Philips, ASML, NXP and Vanderlande, and with various tool vendors such as Atlassian (e.g. Jira, Confluence),IBM Rational (e.g. ClearCase, Synergy, Jazz products) as well as open source tools (e.g. SVN, Git, Jenkins, Trac, Eclipse). I am living in Eindhoven, the Netherlands, with my wife. We have 3 adult children. My main hobbies are classical music and photography.
This entry was posted in integration. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.