Classloaders are one of those subjects that affect every Java developers’ daily life, but aren’t necessarily something that most people understand to the core. The fact that you’re interested in this, my friend, makes you an exception to the rule. Luckily, our CTO Jevgeni is as well, and he regularly eats classloaders for breakfast. The great folks from Jazoon 2010 in Zurich have recently made his “Do you REALLY get class loaders?” presentation available, which we’re happy to share with you below. Jevgeni has also previously written a series of articles on class loading that you can read at your leisure. If you want to see the video stream and the slides at the same time, click anywhere other than the “Play” button and you will be taken to the Parleys website (which also happens to have lots of other great presentations).

And hey, since you’re already here, why not see how JRebel handles classloaders first hand?

Here’s the presentation abstract:

Class loaders are at the core of the Java language. Java EE containers, OSGi, NetBeans modules, Tapestry 5, Grails and many others use class loaders heavily. Yet when something goes wrong, would you know how to solve it? In this session we’ll take a tour of the Java class loading mechanism, both from JVM and developer point of view. We’ll see how different delegation systems are built, how synchronization works, what is the difference between finding classes and resources, what wrong assumptions has been made and are now supported. Next we will look at typical problems that you get with class loading and how to solve them. ClassNoDefError, IncompatibleClassChangeError, LinkageError and many others are symptoms of specific things going wrong that you can usually find and fix. For each problem we’ll go through a hands on demo with a corresponding solution. We’ll also take a look at how and why classloaders leak and how can you remedy that.

Hey folks!

Next week we’re happy to be a part of a webinar with the great team over at MuleSoft, and we wanted to extend the invite out to all of you! Using JRebel together with TCat makes life a lot easier for Tomcat users, so take this chance to learn more and register for the event!

~~
Do you ever ask yourself these questions: “Can I eliminate the redeploy phase from Tomcat’s development cycle? How can I more efficiently diagnose performance bottlenecks in my Tomcat application? How can I simplify configuring Tomcat for my application?”

Tomcat users spend 160 hours per year waiting for redeploys, not to mention the time getting back into the flow of coding. They also spend quite a bit of time configuring Tomcat for deployment to production, and investigating and resolving performance and availability issues — without deep diagnostic data.

In a webinar on July 28th at 10am PT, MuleSoft and ZeroTurnaround have teamed up to provide web application developers using Tomcat with the ultimate “one-two punch” for developer productivity.

Register now!

Join us for a 45-minute demo-driven webinar, where you will see how JRebel and Tcat Server can help you:

* Instantly see the effects of your code changes without having to restart Tomcat
* Discover and quickly resolve performance issues with deep diagnostic capabilities
* Visually browse and edit Tomcat configuration files

Speakers:
Sateesh Narahari, Director of Product Management at MuleSoft
Toomas Romer, Co-Founder, Lead Developer at ZeroTurnaround

Logistics:
Date: July 28, 2010
Time: 10am PT / 1pm ET / 7pm GMT (find your time zone)

About MuleSoft:
MuleSoft is the Web Middleware Company, providing enterprise-class software based on the world’s most popular open source application infrastructure products, Mule ESB and Apache Tomcat. MuleSoft brings an ideal combination of simplicity and power to today’s web applications with Mule ESB, Tcat Server and iBeans. MuleSoft’s products boast more than 1.5 million downloads and over 2,500 production deployments by leading organizations such as Walmart.com, Nestlé, Honeywell and DHL, as well as 5 of the world’s top 10 banks. MuleSoft is headquartered in San Francisco with offices worldwide.

About ZeroTurnaround:
ZeroTurnaround builds technologies that make the Java platform more productive: both for development and production. Our Rebel technology integrates directly into the JVMs, application servers, and development tools that most teams use – making them more competitive. JRebel saves Java EE developers between 3-7 weeks of wasted development time (yup; 40-hour workweeks), has been downloaded hundreds of thousands of times, and pays for itself in under a week. Our customers are the who’s who of the finance, web application, and technology industries, including most global banks plus the Bank of America, American Airlines, Lufthansa, LinkedIn, HP, Siemans, Logica, Kayak, Oracle, IBM, and more. For rolling out or rolling back upgrades to live applications instantly, check out LiveRebel, in private beta now.

This is a story of an interesting support request handled by our guru Lauri Tulmin. The inquiry was about a rare occurring ArrayIndexOutOfBoundsException that JRebel seemed to cause in Wicket. After some investigation he discovered that the root exception was thrown from the following Wicket code:

private final Map<IModifiable, Entry> modifiableToEntry = 
   new ConcurrentHashMap();
 
  public void start(Duration pollFrequency) {
    this.task = new Task("ModificationWatcher");
 
    this.task.run(pollFrequency, new ICode() {
      public void run(Logger log) {
        Iterator iter = new ArrayList(
          ModificationWatcher.this.modifiableToEntry.values()).iterator();
        while (iter.hasNext())

Specifically the exception was thrown from the constructor of ArrayList. How was that possible?

Let’s take a step back and review the reasons for this code. One issue with collections is that when iterating a ConcurrentModificationException is thrown if a collection is changed (usually by a different thread). This is done to protect the Iterator from unpredictable behaviour. A common idiom to avoid it is to copy the collection before iteration like that:

for(Iterator i = new ArrayList(collection).iterator(); i.hasNext();) {...}

Note that the collection must either to be synchronized or otherwise suitable for a multi-threaded environment.

This idiom is a very common one, as easily proven by a simple Google Code search. In fact we used it several times in the JRebel code and is present in several places in Wicket as well. So how could this cause an ArrayIndexOutOfBoundsException?

When Lauri investigated in depth he found out that this idiom is inherently unsafe in a multi-threaded environment (even when the underlying collection is synchronized!). The reason for that is the way ArrayList was constructed before Java 1.6. In my 1.5 Java SDK source code it looks like this:

public ArrayList(Collection<? extends E> collection) {
  int size = collection.size();
  firstIndex = 0;
  array = newElementArray(size + (size / 10));
  collection.toArray(array);
  lastIndex = size;
  modCount = 1;
}

The issue is that there is a race condition between the time the size is recorded and the collection.toArray(array) is called. During that time it is conceivable (and indeed possible) that the size of the collection is changed by a different thread. If the size is now bigger, the array copying in toArray() fails with the dreaded ArrayIndexOutOfBoundsException. When researching the issue further we found that the constructor in Java 1.6 ArrayList has been changed to avoid this issue, so at least on Oracle Java 1.6 or later you are safe. Unfortunately I couldn’t find a specific bug referring to that issue, so likely as not it’s an accidental fix.

So what should you do to avoid this issue? One way is to put the synchronized block around the whole loop, but this requires that you only access that collection from other threads in the same synchronized block. An easier solution is to use the toArray()method like this:

for (Iterator i = Arrays.asList(collection.toArray()).iterator(); i.hasNext();)

Over the Reloading Java Classes (RJC) article series, we’ve examined how class reloading happens, from objects and classloaders to application servers and solutions that reduce Turnaround. It’s been pretty in-depth, so we’d like to take a step back, move to more shallow ground, and take a look at why we would want class reloading to work in the first place. What are the real costs of missing class reloading on Java teams?

As we discover, this is a challenging question to answer — but we’ll give it a shot here, using the attempts of others to guide us.

Other Articles in the Reloading Java Classes Series

• RJC101: Objects, Classes and ClassLoaders
• RJC201: How do Classloader leaks happen?
• RJC301: Classloaders in Web Development — Tomcat, GlassFish, OSGi, Tapestry 5 and so on
• RJC401: HotSwap and JRebel — Behind the Scenes
• RJC501: How Much Does Turnaround Cost?

The Easily-Measurable Cost of Turnaround: Builds and Redeploys

Last year we ran two surveys asking developers how much time they spend building and redeploying their application. Although we’ve been talking with developers about this for a long time, and everyone understands that some amount of time is spent on builds and redeploys, it was still surprising to see the results – including actual numbers of minutes, days, and weeks spent on the process. For example, during an average hour of coding developers spend:

• An average of 6 minutes building the application [Source: The Build Tool Report].
• An average of 10.5 minutes redeploying the application [Source: The Java EE Container Redeploy & Restart Report].

To be fair, there are large deviations on both sides of these averages — the survey data we used only came from 600 and 1100 respondents respectively, and they used a variety of technologies which you can read about in the reports themselves. Some folks don’t spend as much as the averages, while others spend much more, and your project could be anywhere in that range.

Given that, the averages should apply pretty nicely to Java EE development as an industry (with the usual disclaimer of “lies, damn lies and statistics”). Let’s take a look at an awful scenario — where everyone in the industry is right on the average:

• Build (6 mins) + Redeploy (10.5 mins) = 16.5 minutes per hour of development or 27.5% of coding time
• Assuming that developers only code 5 hours per day x 5 days per week x 48 weeks a year = 1200 coding hours a year
• 1200 hours * 27.5% = 330 hours a year or 330/40 = 8.25 full-time work-weeks a year spent on turnaround, per developer
• Assuming $20/h average salary, this adds up to 330 * 20 = $6600 per developer per year [Source: conservative estimation backed by http://www.worldsalaries.org/computerprogrammer.shtml]
• Larry Ellison recently quoted the number of Java developers around the world at 9 million.

If you’d like to work out the time or cash spent annually by the industry on Turnaround, these are some numbers you can start with.

Context Switching

Programming is complicated. In fact it’s so complicated that most people are not capable of doing it.  Those who are capable must mentally juggle a lot of context while writing code.. context including specifications, GUI considerations, relevant library calls, relevant application classes, methods and variables and so on.

This information is held in the the working (aka short-term) memory. Since the working memory is not meant for long term storage, switching away from a current task means that you will start losing the context surrounding the code, and then spend time trying to get back into it. The rest of the article will discuss how quickly context is lost, how long it takes to restore it, and what effect it has on the quality of your work.

How quickly does working memory degrade?

In 1959, an article titled, “Short-Term Retention of Individual Verbal Items” by L. Peterson and M. Peterson appeared in the Journal of Experimental Psychology [Source: http://www-test.unifr.ch/psycho/site/assets/files/Allg/ExU/Peterson_1959.pdf]. It showed that on average, after only 3 seconds, about half of our working memory is lost. The following graph plots the amount of working memory preserved after every subsequent 3 seconds:

After 15 seconds, less than 10% of the original memory is preserved. While these results may not be directly applicable to the programming context, it is clear that memory degradation occurs after seconds, not minutes. Translated, it means that short distractions cause dramatic losses in working memory.

How long it takes to restore context after an interruption?

Although there’s not a direct answer to that question, going through the relevant literature gives us some clues:

[…] the recovery time after a phone call is at least 15 minutes. [...] If more than 10 interrupts occur during a day, the time between the interrupts becomes too short to accomplish product development work.
Rini van Solingen et al, Interrupts: Just a Minute Never Is

To take a call you need to completely switch away from your working environment and focus on the conversation. Other interrupts are less disruptive than that:

Recovering from an email interrupt, and returning to work at the same work rate as before the interrupt — 64 seconds.
Thomas Jackson et al, Case Study: Evaluating the Effect of Email Interruptions within the Workplace

Instant message — 11 to 25 seconds.
Thomas Jackson, Instant Messaging Implications in the Transition from a Private Consumer Activity to a Communication Tool for Business

FYI – These quotes are not based on software developers, but rather more generic office workers. Take that as you’d like. Since context recovery is a process, it takes some time to get back to maximum speed after an interrupt:

The trouble is, getting into “the zone” is not easy. When you try to measure it, it looks like it takes an average of 15 minutes to start working at maximum productivity. [...] The other trouble is that it’s so easy to get knocked out of the zone. Noise, phone calls, going out for lunch, having to drive 5 minutes to Starbucks for coffee, and interruptions by coworkers [...] all knock you out of the zone.
Joel Spolsky, Where do These People Get Their (Unoriginal) Ideas?

So what about development Turnaround — should that be considered an interrupt and how much does it add to the cost?

From experience, any pause will cause the working memory to start fading, and longer pauses will cause developers to multi-task (even if the other task is reading Slashdot). This will mean that the context surrounding a particular task is at least partially lost and needs to be restored. A longer pause means more memory degradation, increased likelihood of a task switch, and the assumption that recovery time is also longer. Just to illustrate, let’s assume that it takes 50% of the length of the pause to recover the context after getting back to work. In that case, the total cost of turnaround for a java developer dealing with average build and redeploy delays looks something like:

• 16.5 (mins per hour spent on Build & Redeploy phases) + 50% = 24.75 minutes per hour of development or 41.25% of coding time
• 1200 (coding hours per year) * 41.25% = 495 hours per year spent on Turnaround (divide that by 40 hours per week to get 12.375 full-time work-weeks a year spent on turnaround)
• Assuming a $20/h average salary this adds up to 495 * 20 = $9900 per developer per year

These numbers make a lot of assumptions. I’m not convinced that the average hourly wage of someone reading this blog is $20. I’m also not convinced that these numbers make an air-tight argument — but the point here is: the social cost of Turnaround is more than just the time spent on building, redeploying, & restarting.

Finally, I’m not convinced that developers keep all the code they wrote on a day full of distractions – I get the feeling that long turnaround times, distractions, and interruptions have a negative effect on code quality, and on the mental abilities of other developers. Though we couldn’t find data measuring this specifically, there is some indication that this impact is also not trivial, and I’d like to finish with it:

In a series of tests carried out by Dr Glenn Wilson, Reader in Personality at the Institute of Psychiatry, University of London, an average worker’s functioning IQ falls ten points when distracted by ringing telephones and incoming emails. This drop in IQ is more than double the four point drop seen following studies on the impact of smoking marijuana. Similarly, research on sleep deprivation suggests that an IQ drop of ten points is equal to missing an entire night of sleep. This IQ drop was even more significant in the men who took part in the tests.
Dr Glenn Wilson

Many programmers appear to be continually frustrated in attempts to work. They are plagued by noise and interruption, and pessimistic that the situation will ever be improved. The data recorded about actual interruptions supports the view that the so-called “work-day” is made up largely of frustration time.
T. DeMarco and T. Lister, Programmer performance and the effects of the workplace,

Conclusions

It’s been a long trip from the beginning of the series to this point. We can only hope that we answered some of your questions and maybe posed some new questions to answer. Probably the most important point that we were trying to bring to your attention is that reducing Java turnaround is a complex and expensive problem, which deserves a lot more attention than it’s getting at the moment

What are your thoughts on all this?

Resources

• Concerning Interruptions – Stephen B. Jenkins, Institute for Aerospace Research, National Research Council of Canada
• Interruptions.net Literature — comprehensive index of published papers concerning interruptions

Other Articles in the Reloading Java Classes Series

• RJC101: Objects, Classes and ClassLoaders
• RJC201: How do Classloader leaks happen?
• RJC301: Classloaders in Web Development — Tomcat, GlassFish, OSGi, Tapestry 5 and so on
• RJC401: HotSwap and JRebel — Behind the Scenes
• RJC501: How Much Does Turnaround Cost?

One common comment we hear when talking about JRebel is that class updates should be implemented in the standard JVM (see feature comparison and behind-the-scenes notes to find out more about JRebel and HotSwap). However even if Oracle or IBM would announce tomorrow that they implemented the support for full schema change HotSwap in the next version of the JVM, JRebel would still be a worthwhile investment. Why?

1. Adding new classes

HotSwap protocol works by specifying the class name and the updated bytecode. It works for existing classes because there’s a relatively unique correspondence between the class name and its content. HotSwap ignores the fact that each class is loaded in a specific class loader. However to add a new class you need to choose that class loader, which HotSwap doesn’t support.

In exploded deployment (where WARs, EARs and JARs are deployed unpacked as directories) new classes can be loaded directly from the file system. But in packaged deployment (WARs, EARs and JARs deployed as archives) adding new classes is a challenge. Even adding the new class to the archive will not do the trick, as the archives are commonly unpacked when deployed and updating the archive either causes redeployment or no change whatsoever.

To solve this problem in a standard & compatible way you’d need:

• A new standard API for Java EE that would allow you to find the corresponding class loader for a specified application. This would need to be implemented by the application server.
• Extensions to the HotSwap protocol to allow sending new classes for a specified class loader, OR a different API/protocol to allow remote interaction with the IDE. This would need to be implemented in the JVM.
• each IDE to be aware of what application the new classes belong to. This would need to be supported by the IDE.

JRebel solves this by

• Extending the application server class loaders to search for classes from JRebel sources.
• Using the rebel.xml configuration file to match each specific class loader to the classes in the IDE.

2. Resources

The HotSwap protocol doesn’t make any mention of the resources packaged along with classes. Java applications include quite a few of these (XMLs, property files, graphic files, etc). If you use packaged deployment these resources will not be updated until you make a full build and redeploy.

To solve this problem in a standard & compatible way you’d need:

• A new standard API for Java EE that would allow you to find the corresponding class loader for a specified application. This would need to be implemented by the application server.
• Extensions to the ClassLoader API to allow adding external sources for finding resources. This would need to be implemented by the application server.
• Extensions to the HotSwap protocol to allow sending resources for a specified class loader or a different API/protocol to allow remote interaction with the IDE. This would need to be implemented in the JVM. A big problem with this approach is that resource names are often not unique and several distinct resources with the same name in the same application are quite common.
• The IDE to be aware of which application the resources belong to. This would need to be supported by the IDE. Possibly you’d need to know even more about resources to solve the problem described in the previous point.

JRebel solves this by

• Extending the application server class loaders to search for resources from JRebel sources.
• Using the rebel.xml configuration file to match specific class loader to the resources in the IDE. JRebel solves the problem with multiple resources by giving the user control via rebel.xml to affect the order in which those resources are returned to the application.

3. Web resources

Besides the class loader resources, that are looked up via the ClassLoader API, every web application has a wealth of web resources like HTML, JSP, CSS, JS, images and so on. Unlike the class loader resources these are available in two ways:

1. Served to the end user via the HTTP protocol.
2. Read by the application using the ServletContext API.

If you use packaged deployment, then to update these resources you would need to do a full build and redeploy. HotSwap isn’t aware of these resources (and probably shouldn’t be), since ClassLoaders do not participate in any way in finding these resources.

To solve this problem in a standard & compatible way you’d need:

• A new standard API for Java EE that would allow you to find the corresponding ServletContext or similar API for a specified application. This would need to be implemented by the application server.
• A remote API for updating/injecting web resources into a specified ServletContext. These resources must also be served via HTTP when requested. This would need to be implemented by the application server. Luckily, unlike the class loader resources, web resources are uniquely identified by their name.
• The IDE to be aware of which application the web resources belong to. This would need to be supported by the IDE.

JRebel solves this by

• Extending the application server ServletContext implementation and HTTP server implementations to search for web resources from JRebel sources.
• Using the rebel.xml configuration file to match a specific web application to the web resources in the IDE.

4. Caches

So let’s say the powers that be in the Java world have teamed up and implemented everything we described so far. You start your application, add a new method to the class, open your browser and stare at a juicy MethodNotFoundError. As it turns out, Java servers and frameworks tend to aggressively cache:

• Class structure information. This means that the servers/frameworks won’t be aware of the new methods and will fail. A good example is JSP EL and Scriptlets, which will fail in most implementations when trying to access a new method.
• Resources. Here we have both positive caches (the updated content is ignored) and negative caches (the new resource isn’t visible). Almost every server and some frameworks will include both of these caches.
• Configuration. Many resources (and class annotations) are only scanned once, and all changes are ignored afterwards. E.g. Spring XML configuration or Annotations.

This means that even after both JVM and the application server APIs are updated to support application updates, you still need to re-engineer a lot of Java code to accept the fact that classes and resources can change.

JRebel solves this by

• Purging the class structure information using listeners called when a class is updated and a lot of bytecode instrumentation to change the current framework implementations.
• Integrating with servers and frameworks to intelligently disable or alter the positive and negative caches. JRebel introduces its own update-aware cache to speed up resource lookup and delivery.
• Deeply integrating with frameworks to allow updates to the configuration. E.g. for Spring, JRebel causes the XMLs to be reread and the classpath to be rescanned.

5. Managed Components (EJBs, Spring beans,…)

Finally, when all is said and done, we still have the framework- or server-managed components like EJBs. These components often have:

• Some kind of state associated with an instance or an instance pool.
• Layers of proxies implementing aspects of functionality.
• Dependency injection.

These features mean that updating the class schema is nowhere near enough. When a class is updated:

• State may need to be updated to correspond to changes in the configuration. E.g. another method should now be executed inside a transaction.
• Proxies may need to be reconstructed or updated to apply to the added methods. Otherwise you might be looking at the MethodNotFoundError again.
• New dependencies must be injected. Otherwise NullPointerExceptions will abound.

Implementing this in current servers and frameworks will mean a substantial re-engineering across the codebase. JRebel solves this by re-implementing some server/framework features to allow for such updates to take place.

Conclusions

Although it could be possible to implement all of JRebel’s features in the Java ecosystem, it would take much more than just an update to the JVM. You’d need to re-engineer standards, introduce new APIs, implement them across app servers and then re-engineer much of the existing server/framework code base to take the possibility of updates into account. JRebel not only allows updates to class schema to take place, but also solves all those problems so that you can just update your code and see the change immediately. On average, that saves 2.5 minutes per redeploy — 5 times per hour of coding, and as Craig Pardey noticed, a whole bunch of context switching as well.

In this article we’ll review how dynamic classloaders are used in real servers, containers and frameworks to reload Java classes and applications.  We’ll also touch on how to get faster reloads and redeploys by using them in optimal ways.

Other Articles in the Reloading Java Classes Series

• RJC101: Objects, Classes and ClassLoaders
• RJC201: How do Classloader leaks happen?
• RJC301: Classloaders in Web Development — Tomcat, GlassFish, OSGi, Tapestry 5 and so on
• RJC401: HotSwap and JRebel — Behind the Scenes
• RJC501: How Much Does Turnaround Cost?

(more…)

Other Articles in the Reloading Java Classes Series

• RJC101: Objects, Classes and ClassLoaders
• RJC201: How do Classloader leaks happen?
• RJC301: Classloaders in Web Development — Tomcat, GlassFish, OSGi, Tapestry 5 and so on
• RJC401: HotSwap and JRebel — Behind the Scenes
• RJC501: How Much Does Turnaround Cost?

A video presentation “Do you really get class loaders” by Jevgeni Kabanov

From ClassLoaders to Classes

If you have programmed in Java for some time you know that memory leaks do happen. Usually it’s the case of a collection somewhere with references to objects (e.g. listeners) that should have been cleared, but never were. Classloaders are a very special case of this, and unfortunately, with the current state of the Java platform, these leaks are both inevitable and costly: routinely causing OutOfMemoryError’s in production applications after just a few redeploys.

Let’s get started. Recalling RJC101: to reload a class we threw away the old classloader and created a new one, copying the object graph as best we could:

Every object had a reference to its class, which in turn had a reference to its classloader. However we didn’t mention that every classloader in turn has a reference to each of the classes it has loaded, each of which holds static fields defined in the class:

This means that

1. If a classloader is leaked it will hold on to all its classes and all their static fields. Static fields commonly hold caches, singleton objects, and various configuration and application states. Even if your application doesn’t have any large static caches, it doesn’t mean that the framework you use doesn’t hold them for you (e.g. Log4J is a common culprit as it’s often put in the server classpath). This explains why leaking a classloader can be so expensive.
2. To leak a classloader it’s enough to leave a reference to any object, created from a class, loaded by that classloader. Even if that object seems completely harmless (e.g. doesn’t have a single field), it will still hold on to its classloader and all the application state. A single place in the application that survives the redeploy and doesn’t do a proper cleanup is enough to sprout the leak. In a typical application there will be several such places, some of them almost impossible to fix due to the way third-party libraries are built. Leaking a classloader is therefore, quite common.

To examine this from a different perspective let’s return to the code example from our previous article. Breeze through it to quickly catch up.

Introducing the Leak

We will use the exact same Main class as before to show what a simple leak could look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

public class Main {
  private static IExample example1;
  private static IExample example2;
 
  public static void main(String[] args)  {
    example1 = ExampleFactory.newInstance().copy();
 
    while (true) {
      example2 = ExampleFactory.newInstance().copy();
 
      System.out.println("1) " +
        example1.message() + " = " + example1.plusPlus());
      System.out.println("2) " +
        example2.message() + " = " + example2.plusPlus());
      System.out.println();
 
      Thread.currentThread().sleep(3000);
    }
  }
}

The ExampleFactory class is also exactly the same, but here’s where things get leaky. Let’s introduce a new class called Leak and a corresponding interface ILeak:

1
2
3
4
5
6
7
8
9
10

interface ILeak {
}
 
public class Leak implements ILeak {
  private ILeak leak;
 
  public Leak(ILeak leak) {
    this.leak = leak;
  }
}

As you can see it’s not a terribly complicated class: it just forms a chain of objects, with each doing nothing more than holding a reference to the previous one. We will modify the Example class to include a reference to the Leak object and throw in a large array to take up memory (it represents a large cache). Let’s omit some methods shown in the previous article for brevity:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

public class Example implements IExample {
  private int counter;
  private ILeak leak;
 
  private static final long[] cache = new long[1000000];
 
  /* message(), counter(), plusPlus() impls */
 
  public ILeak leak() {
    return new Leak(leak);
  }
 
  public IExample copy(IExample example) {
    if (example != null) {
      counter = example.counter();
      leak = example.leak();
    }
    return this;
  }
}

The important things to note about Example class are:

1. Example holds a reference to Leak, but Leak has no references to Example.
2. When Example is copied (method copy() is called) a new Leak object is created holding a reference to the previous one.

If you try to run this code an OutOfMemoryError will be thrown after just a few iterations:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at example.Example.<clinit>(Example.java:8)

With the right tools, we can look deeper and see how this happens.

Post Mortem

Since Java 5.0, we’ve been able to use the jmap command line tool included in the JDK distribution to dump the heap of a running application (or for that matter even extract the Java heap from a core dump). However, since our application is crashing we will need a feature that was introduced in Java 6.0: dumping the heap on OutOfMemoryError. To do that we only need to add -XX:+HeapDumpOnOutOfMemoryError to the JVM command line:

java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid37266.hprof ...
Heap dump file created [57715044 bytes in 1.707 secs]
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at example.Example.<clinit>(Example.java:8)

After we have the heap dump we can analyze it. There are a number of tools (including jhat, a small web-based analyzer included with the JDK), but here we will use the more sophisticated Eclipse Memory Analyzer (EMA).

After loading the heap dump into the EMA we can look at the Dominator Tree analysis. It is a very useful analysis that will usually reliably identify the biggest memory consumers in the heap and what objects hold a reference to them. In our case it seems quite obvious that the Leak class is the one that consumes most of the heap:

Now let’s run a search for all of the Leak objects and see what are they holding to. To do that we run a search List objects -> with outgoing references for “example.Leak”:

The results include several Leak objects. Expanding the outgoing references we can see that each of them holds on to a separate instance of Example through a bunch of intermediate objects:

You may notice that one of the intermediate objects is ExampleFactory$1, which refers to the anonymous subclass of URLClassLoader we created in the ExampleFactory class. In fact what is happening is exactly the situation we described in the beginning of the article:

• Each Leak object is leaking. They are holding on to their classloaders
• The classloaders are holding onto the Example class they have loaded:

Conclusions

Though this example is slightly contrived, the main idea to take away is that it’s easy to leak a single object in Java. Each leak has the potential to leak the whole classloader if the application is redeployed or otherwise a new classloader is created. Since preventing such leaks is very challenging, it’s a better idea to use Eclipse Memory Analyzer and your understanding of classloaders to hunt them down after you get an OutOfMemoryError on redeploy.

This article addressed the following questions:

• How does reloading a class cause the classloader to leak?
• What are some consequences of leaking classloaders?
• What tools can be used to troubleshoot these memory leaks?

Resources

• How to fix the dreaded “java.lang.OutOfMemoryError: PermGen space” exception (classloader leaks)
• Download full source code

Other Articles in the Reloading Java Classes Series

• RJC101: Objects, Classes and ClassLoaders
• RJC201: How do Classloader leaks happen?
• RJC301: Classloaders in Web Development — Tomcat, GlassFish, OSGi, Tapestry 5 and so on
• RJC401: HotSwap and JRebel — Behind the Scenes
• RJC501: How Much Does Turnaround Cost?