Some time ago we ran a survey asking a few questions about the build process, specifically the tools that are used to do incremental builds and how much time those builds take. We had over 600 responses, so now it’s time to count the results.

This is the first time that we’ve published results on the incremental build process, so the information is more likely to serve as a guide than an authoritative information source. That being said, the information is still quite interesting, and if it serves to start a conversation that improves the process of even one team, then we’re proud to have helped out. If you haven’t answered the 3-question survey yet, take two minutes and go for it – and do let your community know about it – as more answers trickle in we’ll update this post with the new data. If you’d like to play with the results on your own we‘ve provided all the data and our calculations in a handy Excel sheet that you can download here.

The first question in the survey was “What build tool do you use for incremental builds on your largest current project?” The breakdown follows:

Chart 1: “Which build tool is used most often for incremental builds?”

This does not include tools that scored less than 10 answers in the survey. Those are:

• Buildr
• Shell scripts
• javac
• Make
• NAnt
• Savant
• Hudson
• PHP

Maven and Ant are responsible for over half of the incremental builds in the wild. It also seems that Maven has overtaken Ant by popularity, although you should take it with a grain of salt as the questions were not phrased for this purpose. Unfortunately I couldn’t find any external data to confirm or contradict these results. Please do let us know if you know of other surveys on this topic.

The other half of incremental builds are done inside IDEs. Although some of those are just driving Ant or Maven inside the IDE, there is also a large number of developers that use the IDE as the primary build tool during development (we’ll analyze this in detail in the end of the report).

It might be valuable to reiterate here that this is a poll where respondents are self-selecting, and therefore it may not accurately reflect the actual marketplace when it comes to determining market share.  Since we’re more interested in the incremental build process itself, that’s something we can live with.  That being said, there is a chance that people who have fast (or slow) builds may be more likely to complete a poll like this.  Since we’re not sure if this is serious enough to sway any results, we’ll just display the data, and let you decide.  So, while 53% of developers are using Ant or Maven for their incremental builds, everyone else uses their IDEs: Eclipse is dominating the IDE landscape with 32% , followed by IntelliJ IDEA at 10% and NetBeans at 5%.

With that in mind, we asked, “How long does an incremental build take?”

Chart 2: “How long does an incremental build take?”

This is relatively good news. Nearly half of our respondents (44%) indicated that their incremental build process takes less than 30 seconds. 40% of respondents have incremental builds lasting from 1 to 3 minutes. Only 16% of incremental builds last over 4 minutes.

The average length of a build is 1.9 minutes with the standard deviation of 2.8.

To finish up, we asked, “In an hour of coding, how many times do you run an incremental build?”

Chart 3: “How often do Java developers run incremental builds?”

A healthy 31% of respondents don’t have to run the build at all (e.g. it’s run automatically on save). The rest of the numbers are all over the place, so we’ll wait with the analysis until we can put them in context.

The average number of incremental builds an hour is 3.9 times, with a standard deviation of 4.1.

It’s time to crunch some data. We assigned numeric values to each of the intervals (e.g. “2.5″ for the “2-3″ interval) and multiplied the number of incremental builds an hour by the amount of time one incremental build takes (basically, Chart 2 times Chart 3), thus finding the approximate amount of time respondents spend building in each hour of development. This was done per respondent, so if someone said that their build takes 4-5 minutes and they build twice an hour, we’d see a result of 9 minutes per hour. We broke down the data into the following intervals:

Chart 4: “Time spent on incremental builds during an hour of coding”

The average total time taken by incremental builds in an hour is exactly 6 minutes, but the standard deviation is 10.1, rendering this number unreliable. We can, however, divide the respondents in three quite well defined groups:

• Less than 1 minute an hour. 34% of respondents basically don’t spend any time on the incremental build process.
• 1 to 5 minutes an hour. 34% of respondents spend a “reasonable” amount of time on incremental builds – under 5 minutes an hour. This group spends an average of 3 minutes an hour on incremental builds, which corresponds to about 5% of total development time.
• Over 5 minutes an hour. 32% of respondents spend over 5 minutes an hour on incremental builds. The weighted average in this group is over 13 minutes an hour. This group of developers is spending about 22% of their development time on incremental builds.

Chart 5: “Time spent on incremental builds, per hour, by build tool”

It is clear from this chart that Ant and Maven take significantly more time than IDE builds. Both take about 8 minutes an hour, which corresponds to 13% of total development time. There seems to be little difference between the two, perhaps because the projects where you have to use Ant or Maven for incremental builds are large and complex.

Eclipse is definitely the fastest with 2.9 minutes an hour, which corresponds to about 5% of total development time. Eclipse is the only IDE supporting true incremental build on save using a fast embedded compiler, so these results are expected. It is likely that the true number is even lower, as some of the respondents use Eclipse to launch the Ant or Maven builds (we’ll return to this with the next chart).

IntelliJ IDEA falls in between with 5.7 minutes an hour and about 10% of total development time. It does not have true background compilation and often the IDE builds just launch Ant or Maven behind the scenes.

Chart 6: “Incremental build length breakdown per build tool”

This final chart shows the proportion of respondents in one of the three groups we defined with Chart 4 broken down per build tool.

Finally some things become clear. About 61% of Eclipse builds happen instantly (taking less than 1 minute per hour). We can assume that those are the respondents using compile-on-save, whereas the rest use Eclipse to launch Ant or Maven builds. Considering that 32% of respondents are using Eclipse (see Chart 1), this means that about 20% of all of the respondents use Eclipse with compile-on-save and thus benefit from the instant incremental builds.

The breakdowns of Ant and Maven times are quite similar, with Maven being slightly slower. Moreover the breakdown for IntelliJ IDEA is also similar, supporting the hypotheses that IntelliJ IDEA builds launch Ant or Maven in the background. The proportion itself likely corresponds to the size and complexity of projects, with smaller ones building quickly and larger ones taking build time of over 5 minutes and hour.

It is hard to draw any simple conclusions from this survey, as different groups have different problems, so instead let’s put together the most interesting and reliable numbers we got:

• 56% of respondents have builds that last over half a minute, each.
• 20% of respondents use Eclipse with compile-on-save and thus benefit from the instant incremental builds.
• 34% of respondents spend an average of 3 minutes an hour on incremental builds, which corresponds to about 5% of total development time, or 1.5 weeks per year (40-hour weeks).
• 32% of respondents spend more than 5 minutes per hour on incremental builds.  Of the developers spending more than 5 mins per hour, 13 minutes per hour is the average amount of time spent.  13 mins per hour equals 22% of total development time, or 6.5 weeks per year (40 hour workweeks).
• Weeks spent on incremental builds are calculated by assuming 48 work weeks per year (minus vacation) and an average of 5 hours of development time per day.
• Java developers spend 1.5 to 6.5 work weeks a year (with an average of 3.8 work weeks,  or 152 hours, annually) waiting for builds, unless they are using Eclipse with compile-on-save.

Tags: Ant, Build Tool, eclipse, IntelliJ, JRebel, Maven, NetBeans

This entry was posted on Wednesday, October 21st, 2009 at 11:30 am and is filed under blog, news. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.