This is a story of an interesting support request handled by our guru Lauri Tulmin. The inquiry was about a rare occurring ArrayIndexOutOfBoundsException that JRebel seemed to cause in Wicket. After some investigation he discovered that the root exception was thrown from the following Wicket code:

private final Map<IModifiable, Entry> modifiableToEntry = 
   new ConcurrentHashMap();
 
  public void start(Duration pollFrequency) {
    this.task = new Task("ModificationWatcher");
 
    this.task.run(pollFrequency, new ICode() {
      public void run(Logger log) {
        Iterator iter = new ArrayList(
          ModificationWatcher.this.modifiableToEntry.values()).iterator();
        while (iter.hasNext())

Specifically the exception was thrown from the constructor of ArrayList. How was that possible?

Let’s take a step back and review the reasons for this code. One issue with collections is that when iterating a ConcurrentModificationException is thrown if a collection is changed (usually by a different thread). This is done to protect the Iterator from unpredictable behaviour. A common idiom to avoid it is to copy the collection before iteration like that:

for(Iterator i = new ArrayList(collection).iterator(); i.hasNext();) {...}

Note that the collection must either to be synchronized or otherwise suitable for a multi-threaded environment.

This idiom is a very common one, as easily proven by a simple Google Code search. In fact we used it several times in the JRebel code and is present in several places in Wicket as well. So how could this cause an ArrayIndexOutOfBoundsException?

When Lauri investigated in depth he found out that this idiom is inherently unsafe in a multi-threaded environment (even when the underlying collection is synchronized!). The reason for that is the way ArrayList was constructed before Java 1.6. In my 1.5 Java SDK source code it looks like this:

public ArrayList(Collection<? extends E> collection) {
  int size = collection.size();
  firstIndex = 0;
  array = newElementArray(size + (size / 10));
  collection.toArray(array);
  lastIndex = size;
  modCount = 1;
}

The issue is that there is a race condition between the time the size is recorded and the collection.toArray(array) is called. During that time it is conceivable (and indeed possible) that the size of the collection is changed by a different thread. If the size is now bigger, the array copying in toArray() fails with the dreaded ArrayIndexOutOfBoundsException. When researching the issue further we found that the constructor in Java 1.6 ArrayList has been changed to avoid this issue, so at least on Oracle Java 1.6 or later you are safe. Unfortunately I couldn’t find a specific bug referring to that issue, so likely as not it’s an accidental fix.

So what should you do to avoid this issue? One way is to put the synchronized block around the whole loop, but this requires that you only access that collection from other threads in the same synchronized block. An easier solution is to use the toArray()method like this:

for (Iterator i = Arrays.asList(collection.toArray()).iterator(); i.hasNext();)

Over the Reloading Java Classes (RJC) article series, we’ve examined how class reloading happens, from objects and classloaders to application servers and solutions that reduce Turnaround. It’s been pretty in-depth, so we’d like to take a step back, move to more shallow ground, and take a look at why we would want class reloading to work in the first place. What are the real costs of missing class reloading on Java teams?

As we discover, this is a challenging question to answer — but we’ll give it a shot here, using the attempts of others to guide us.

Other Articles in the Reloading Java Classes Series

• RJC101: Objects, Classes and ClassLoaders
• RJC201: How do Classloader leaks happen?
• RJC301: Classloaders in Web Development — Tomcat, GlassFish, OSGi, Tapestry 5 and so on
• RJC401: HotSwap and JRebel — Behind the Scenes
• RJC501: How Much Does Turnaround Cost?

The Easily-Measurable Cost of Turnaround: Builds and Redeploys

Last year we ran two surveys asking developers how much time they spend building and redeploying their application. Although we’ve been talking with developers about this for a long time, and everyone understands that some amount of time is spent on builds and redeploys, it was still surprising to see the results – including actual numbers of minutes, days, and weeks spent on the process. For example, during an average hour of coding developers spend:

• An average of 6 minutes building the application [Source: The Build Tool Report].
• An average of 10.5 minutes redeploying the application [Source: The Java EE Container Redeploy & Restart Report].

To be fair, there are large deviations on both sides of these averages — the survey data we used only came from 600 and 1100 respondents respectively, and they used a variety of technologies which you can read about in the reports themselves. Some folks don’t spend as much as the averages, while others spend much more, and your project could be anywhere in that range.

Given that, the averages should apply pretty nicely to Java EE development as an industry (with the usual disclaimer of “lies, damn lies and statistics”). Let’s take a look at an awful scenario — where everyone in the industry is right on the average:

• Build (6 mins) + Redeploy (10.5 mins) = 16.5 minutes per hour of development or 27.5% of coding time
• Assuming that developers only code 5 hours per day x 5 days per week x 48 weeks a year = 1200 coding hours a year
• 1200 hours * 27.5% = 330 hours a year or 330/40 = 8.25 full-time work-weeks a year spent on turnaround, per developer
• Assuming $20/h average salary, this adds up to 330 * 20 = $6600 per developer per year [Source: conservative estimation backed by http://www.worldsalaries.org/computerprogrammer.shtml]
• Larry Ellison recently quoted the number of Java developers around the world at 9 million.

If you’d like to work out the time or cash spent annually by the industry on Turnaround, these are some numbers you can start with.

Context Switching

Programming is complicated. In fact it’s so complicated that most people are not capable of doing it.  Those who are capable must mentally juggle a lot of context while writing code.. context including specifications, GUI considerations, relevant library calls, relevant application classes, methods and variables and so on.

This information is held in the the working (aka short-term) memory. Since the working memory is not meant for long term storage, switching away from a current task means that you will start losing the context surrounding the code, and then spend time trying to get back into it. The rest of the article will discuss how quickly context is lost, how long it takes to restore it, and what effect it has on the quality of your work.

How quickly does working memory degrade?

In 1959, an article titled, “Short-Term Retention of Individual Verbal Items” by L. Peterson and M. Peterson appeared in the Journal of Experimental Psychology [Source: http://www-test.unifr.ch/psycho/site/assets/files/Allg/ExU/Peterson_1959.pdf]. It showed that on average, after only 3 seconds, about half of our working memory is lost. The following graph plots the amount of working memory preserved after every subsequent 3 seconds:

After 15 seconds, less than 10% of the original memory is preserved. While these results may not be directly applicable to the programming context, it is clear that memory degradation occurs after seconds, not minutes. Translated, it means that short distractions cause dramatic losses in working memory.

How long it takes to restore context after an interruption?

Although there’s not a direct answer to that question, going through the relevant literature gives us some clues:

[…] the recovery time after a phone call is at least 15 minutes. [...] If more than 10 interrupts occur during a day, the time between the interrupts becomes too short to accomplish product development work.
Rini van Solingen et al, Interrupts: Just a Minute Never Is

To take a call you need to completely switch away from your working environment and focus on the conversation. Other interrupts are less disruptive than that:

Recovering from an email interrupt, and returning to work at the same work rate as before the interrupt — 64 seconds.
Thomas Jackson et al, Case Study: Evaluating the Effect of Email Interruptions within the Workplace

Instant message — 11 to 25 seconds.
Thomas Jackson, Instant Messaging Implications in the Transition from a Private Consumer Activity to a Communication Tool for Business

FYI – These quotes are not based on software developers, but rather more generic office workers. Take that as you’d like. Since context recovery is a process, it takes some time to get back to maximum speed after an interrupt:

The trouble is, getting into “the zone” is not easy. When you try to measure it, it looks like it takes an average of 15 minutes to start working at maximum productivity. [...] The other trouble is that it’s so easy to get knocked out of the zone. Noise, phone calls, going out for lunch, having to drive 5 minutes to Starbucks for coffee, and interruptions by coworkers [...] all knock you out of the zone.
Joel Spolsky, Where do These People Get Their (Unoriginal) Ideas?

So what about development Turnaround — should that be considered an interrupt and how much does it add to the cost?

From experience, any pause will cause the working memory to start fading, and longer pauses will cause developers to multi-task (even if the other task is reading Slashdot). This will mean that the context surrounding a particular task is at least partially lost and needs to be restored. A longer pause means more memory degradation, increased likelihood of a task switch, and the assumption that recovery time is also longer. Just to illustrate, let’s assume that it takes 50% of the length of the pause to recover the context after getting back to work. In that case, the total cost of turnaround for a java developer dealing with average build and redeploy delays looks something like:

• 16.5 (mins per hour spent on Build & Redeploy phases) + 50% = 24.75 minutes per hour of development or 41.25% of coding time
• 1200 (coding hours per year) * 41.25% = 495 hours per year spent on Turnaround (divide that by 40 hours per week to get 12.375 full-time work-weeks a year spent on turnaround)
• Assuming a $20/h average salary this adds up to 495 * 20 = $9900 per developer per year

These numbers make a lot of assumptions. I’m not convinced that the average hourly wage of someone reading this blog is $20. I’m also not convinced that these numbers make an air-tight argument — but the point here is: the social cost of Turnaround is more than just the time spent on building, redeploying, & restarting.

Finally, I’m not convinced that developers keep all the code they wrote on a day full of distractions – I get the feeling that long turnaround times, distractions, and interruptions have a negative effect on code quality, and on the mental abilities of other developers. Though we couldn’t find data measuring this specifically, there is some indication that this impact is also not trivial, and I’d like to finish with it:

In a series of tests carried out by Dr Glenn Wilson, Reader in Personality at the Institute of Psychiatry, University of London, an average worker’s functioning IQ falls ten points when distracted by ringing telephones and incoming emails. This drop in IQ is more than double the four point drop seen following studies on the impact of smoking marijuana. Similarly, research on sleep deprivation suggests that an IQ drop of ten points is equal to missing an entire night of sleep. This IQ drop was even more significant in the men who took part in the tests.
Dr Glenn Wilson

Many programmers appear to be continually frustrated in attempts to work. They are plagued by noise and interruption, and pessimistic that the situation will ever be improved. The data recorded about actual interruptions supports the view that the so-called “work-day” is made up largely of frustration time.
T. DeMarco and T. Lister, Programmer performance and the effects of the workplace,

Conclusions

It’s been a long trip from the beginning of the series to this point. We can only hope that we answered some of your questions and maybe posed some new questions to answer. Probably the most important point that we were trying to bring to your attention is that reducing Java turnaround is a complex and expensive problem, which deserves a lot more attention than it’s getting at the moment

What are your thoughts on all this?

Resources

• Concerning Interruptions – Stephen B. Jenkins, Institute for Aerospace Research, National Research Council of Canada
• Interruptions.net Literature — comprehensive index of published papers concerning interruptions

Other Articles in the Reloading Java Classes Series

• RJC101: Objects, Classes and ClassLoaders
• RJC201: How do Classloader leaks happen?
• RJC301: Classloaders in Web Development — Tomcat, GlassFish, OSGi, Tapestry 5 and so on
• RJC401: HotSwap and JRebel — Behind the Scenes
• RJC501: How Much Does Turnaround Cost?