« Real-time data integration using Change Data Capture | Main | Manifesto for secure data sharing »

August 27, 2009

When optimizing - don't forget the Java Virtual Machine (JVM)

Recently, I was working on a project that was coming to a close. It was related to optimizing a database using a Java based in-memory cache to reduce the load. The application had to process up to a million objects per day and was characterized by its heavy use of memory and the high number of read, write and update operations. These operations were found to be the most costly, which meant that optimization efforts were concentrated here.

The project had already achieved impressive performance increases, but one question remained unanswered - would changing the JVM increase performance?

As the code was written in Java, the first idea which came to mind was to optimize the Java Virtual Machine (JVM). Many optimization configuration settings are available for each vendors JVM offering. e.g. Sun Microsystems. Some examples of optimization parameters include: memory allocation, garbage collector configuration, thread configuration and a lot more. Andrig T. Miller presents his experiences here and stackoverflow has useful tips.

In this post I don't want to concentrate on the individual tweaking of JVM configuration settings as I wanted to test how the application performs on other vendors JVMs, rather than the default one from Sun Microsystems. It is often better to replace the engine than to tweak an existing one (In the software world it is usually not true).

TESTED JVMs

First I had to define the list of JVMs which I wanted to test. They had to be available for free as I didn't have time to ask any company to send me their version of JVM for evaluation. I tested the following JVMs:

Sun Java was the obvious choice, it is commonly used in industry, so there is no need for explanation. IBM's JVM was chosen as IBM has a proud heritage and has been developing its own JVMs for years. Oracle JVM is pushing hard into the market, so I had to check it. Why OpenJDK? I like open source, I'm using Fedora Linux and it was in the repository.

HOW I TESTED

Now it was time to test these toys. But how to test them? I had two choices: write my own testing suite or use existing tools which are proven to be reliable in testing of Java performance. I decided to do both.

Simple application

As for my own testing framework, I wrote a very simple Java application which was performing many add/delete/modify operations on a HashMap object. Additionally this application was executing the garbage collector every X iterations, why? To replicate the operations that were limiting performance in the in-memory cache software, especially the frequent execution of the garbage collector.

Testing tools

Regarding some existing tools, I quickly researched and found two suitable solutions. The first one was Phoronix Test Suite (PTS), which is a very nice framework for performance testing using:

Application to speedup

As for the final testing tool, I decided to use the in-memory cache application which required speeding up. The results from this test would determine if the application should be migrated to another JVM.

RESULTS

So what about the results?

Using the Phoronix Test Suite SunJDK6, OpenJDK were almost equally fast. Oracle RockIt was much slower than these two, but IBM JDK6 was the slowest performing JVM. So don't use these two with PTS, you will not see any speedup.

My own simple testing application surprised me in terms of performance. IBM JDK6 was the fastest JVM in this test and by fastest I mean really fast. Average time of an iteration in my app was 13 seconds (Sun JDK6, OpenJDK, JrockIt). IBM JVM averaged 4 seconds.

And now for the most important results for the in-memory cache application to be sped up. The results can be seen in the graph below:

TotalTimeOnly

What we can see there? A big surprise, our application was performing almost the same on each JVM. The presented results are average, I ran the application 10 times on each JVM. No real improvement, so it did not make sense to migrate to another JVM and invest time in proving its stability.

CONCLUSION

The main finding was that no speedup was observed for the in-memory cache application that I was working on. This resulted in no JVM migration and subsequent testing. The project was closed faster and there were no lingering doubts about whether a change in JVM would increase performance.

Interestingly, the simple application that I developed executed 3X faster on IBM JDK. This indicates that application characteristics can result in significant differences in their performance on JVMs.

The main point is to test the application that you want to tune on different JVMs. It is simple and quick to do and you may be surprised by the performance increases that you see.

This is my experience – please share yours?

Bookmark and Share

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a01156f69dc6b970c0120a57b65be970c

Listed below are links to weblogs that reference When optimizing - don't forget the Java Virtual Machine (JVM):

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

That's an interesting post. However my skeptical nature doesn't allow me to digest the results as served. They are actually meaningless without more details how the measurement was done.

It always takes some time after a startup to warm up a JVM. During this phase necessary class loading is performed, highly utilized code is compiled to native code and heap grows. When testing a server application you usually want to exclude those warm up phase from results since lifetime of such JVM process is usually long. As far as I know many people benchmarking java applications assume this phase can least as much as 10 minutes.

Moreover even after JVM is warmed up there is still plenty issues to be aware of. You listed them at the beginning so I know you're aware of them and probably tried to reduce their impact.

I'm not a performance expert but I realize how much you can tailor a JVM to your application by various switches. Isn't it a better approach than run an application on a JVM it wasn't tested on?

As an example I can say that recently I micro benchmarked a piece of code and only turning off JIT compilation slowed down the execution by factor 10. Including the JIT compilation in the results without including benefits from that compilation corrupts the results. This way one can say a JVM compiling more byte code to native code is actually slower.

Well, movement to a new JVM can mean a lot of things. Some JVM implementation print internal log messages differently. For example JRockit verbose messages are very difficult to read. If you hit a class loader problem you might be better off debugging the app on a Sun VM.

Most developers also do not have the luxury of choosing the VM they will target their app to. Your post however raises a nice question. I will try to profile some home grown apps across VMs if I have the time. Perhaps that will help compare stats.

Probably JVM implementation does matter, but without information about JVM parameters and about GC runs under it's difficult to judge. as i know ibm use specific heap management, so it could be faster just because it doesn't waste time on a minor gc runs.

Hi Pawel,

Thanks for the comment. You are right, my blog post lack some details regarding my testing procedure. Sorry for that, however I will explain them now:

- Hardware configuration was: Intel Core2Duo CPU 2.16Ghz, 4GB of RAM. Software was running on Fedora 11.
- As you noticed I'm aware of different switches and I know that they may cause a big difference. For example. I ran these tests on Sun 1.6 JVM with new G1 Garbage collector and I was shocked as it was slower :) I didn't post the results of this experiment with G1 because it crashed our production ready application, G1 is currently experimental so I don't want to blame anyone for that.
- Secondly, I wanted to test the default configuration of JVMs. Such configuration should be optimal in most cases, I wanted to check it.
- What is more, my custom tests of "Simple application" and "Application to speedup" ran for about 15min each, this diagram may be misleading as it shows average time of single, time consuming procedure which was the measure here.

Regarding the question, if fine tuning a JVM is better than choosing a different one, I want to say that you are right. However at the beginning of the post I said that we already tried that. Our app is fast now, but as always it could be faster. With this in mind I decided to try other JVMs as I didn't do it before. It is clear that tests could be performed in more detail, but as always it requires time which may be better spent on other areas of the project.

Right. All I was trying to say is that modern JVMs are so complicated that it's very hard to reliably test application's performance and get a justified conclusion. You have to know the JVM internals and be very scrupulous.

Like I said, since you mentioned some of the obstacles at the beginning of the post it looks like you handled at least some of them. However I don't like the idea of hiding this information from the reader. It gives a fake belief that performance test a java application is a trivial thing to do. Which is not.

CertPal, running on a different JVM can mean much more than inconviniences like the one with logging facility you mentioned.

Take as an example bug 287720 from Eclipse bugzilla (https://bugs.eclipse.org/bugs/show_bug.cgi?id=287720). The whole application crashes just because it points to a class that some version of JRE is missing. Imagine such a class load happens long after the application is started on a production server.

A couple of idiosyncrasies in the IBM JDK mean by default it is not as fast. Nearly all the while gencon is considerably faster than the default opthruput policy but only when you set a large initial nursery size -Xmn. You can get a large increase of speed... But basically no JVM change is likely to have anything like the effect that tuning your application will have.

How to install Microsoft Java Virtual Machine? I need it to install Matlab but can't seem to get it to instal?
I haven't installed MATLAB in years but now I'm trying to and every time I run the installer it says it needs a new version of Java Virtual Machine (it used Microsoft's version). It reboots my computer and gives me the same message over and over. http://ki2.appealtextparliament.com/map-1.html

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been posted. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment