Wednesday, September 30, 2009

Build and Continuous Integration on Small Projects

Continuous Integration (CI) is a concept whose value is usually described in the context of projects of considerable size in terms of modules, environments, number of developers etc. The CI is seen on one hand as a tool to keep efforts of many developers and teams coordinated, and on another, to keep an eye on the complex environment in which code has to live. Yet even on a trivially small project like GNUBridge, consisting of one developer, one module,  and less than 10,000 lines of code, CI has provided considerable value.

First, it forced me to spell out the assumptions and weaknesses of the build process. My manual process involved running junit tests in the IDE, then building a jar and ftping it to the website. Even in a simple process like this is prone to problems. First, a lot of bad things can happen between the time junit tests are run in the IDE and a jar is built. Key resources could be excluded from the jar. Classes could be compiled by the wrong compiler. Also, the manual FTP step could introduce a problem, like having ascii transfer mode. When I was automating the build in Ant, I made sure the jar was produced first, and then the unit tests were being run against the jar. If the tests pass, the jar gets published by the CI server.

Second, CI forced a separation of the application from my development environment. There are various features in any single environment that are not apparent until the build is attempted on another box in alien environment. In my case, the integration tests were instantiating invisible GUI components, and build was failing on the CI server which did not have X. This forced me to look at the architecture, and put to a test the MVC pattern that GNUBridge was supposed to follow. GNUBridge passed the test - in a couple of hours I had views replaced with mocks, with no significant detriment to code coverage statistics. The CI server also had a limit on the CPU time and memory that was considerably smaller than my box. Out of the window went expensive performance benchmarks. If GNUBridge ever needs them, they can be replaced with lightweight benchmarks. This saved me a couple of minutes in the test suite running time, making it practical once again to run the full suite every 20 minutes or so. Both of these restrictions imposed by the CI server forced discipline that made the architecture and development environment better.

In addition to architectural improvements, the separation between build and the development box had two more advantages. For one, it is now be assured that the tests do not rely on any artifacts that may be laying around. Second, it is easy for anyone else to get started by having everything that's needed available in the source code repository, and only needing the basic tools - subversion client, JDK and Ant to get going.

Along the same lines, any developer wishing to get a quick look at the state of the project, can see the commit frequency and what's being worked on. While I haven't implemented code metrics yet, when they come, it'll be one more axis along which the code can improve.

Another unimplemented feature which will reduce risk of bugs getting by are automatically run stochastic tests. These are exploratory tests with random inputs that discover interesting scenarios and verify that program's behavior is correct in situations that I may not have thought about or found relevant at the time I was writing my developer tests. I will be making a separate post on the importance of stochastic tests in GNUBridge. For stochastic tests to provide the most value, there needs to be a sufficiently large sample, but it's impractical for developer to run them frequently. So CI server is where they belong. 

Finally, while target audience are the developers, the CI server also provides important functionality to end users. They can get the bleeding edge product, tested in a fresh environment, along with an automated "what's new" feature listing pulled from commit comments. When reporting a problem, they can go back to an older version one commit at a time, making it much easier to pinpoint the source of a problem. There's an RSS feed that automatically alerts them of new features being available.

As with most of my software experiences, the full glory of these benefits was not apparent to me when I set out to do CI for GNUBridge. My motivation at the time was the automation of the stochastic tests, which ironically, still remains to be implemented. Based on the insights gained on this project, I think there are valid reasons to consider CI even on small projects.

  • January 10, 2010: CI server just exposed a bug in static initialization that was not a problem when working within the IDE.

  • January 24, 2010: CI server exposed a dependency of a production class on a test class

    1. I have a question for you about the use of stochastic tests on the CI server. You mention early on that their are limits to CPU use on the CI server that are much lower than your personal machine. Won't these limits reduce the effectiveness of using stochastic tests? That is you will only be able to run 1000 of them instead of a million and are therefore much less likely to find a problem than you would be on a less limited machine?

      Also, can you give us some details about the service you're using to host your CI server?


    2. Good point. I see CI server solution as more scalable when additional developers contribute to the project. In the meantime, search space is so large, that better test design is where the time is best invested.

      There are very limited options for free CI hosting I could find. Gnubridge is hosted with Parabuild. Another one I'd consider looking into is runcoderun.