Matt Brasier, C2B2 Head of Consultants with niche Java EE consultancy C2B2 Consulting, compares the performance of JBoss Messaging, JBoss MQ and Sun Java System Message Queue, in their out of the box configurations. This first article looks at the performance of these three implementations when sending messages to a JMS queue. The results indicate that the overhead of establishing a connection to a server, and looking up a queue is lowest in JBoss MQ, but that JBoss Messaging is the fastest at actually sending a message. However the time taken to send messages tends towards 1-2ms in all implementations. The testing also highlights the importance of configuring your JMS server to protect your application server from running out of memory.
Introduction
JMS, the Java Message Service, is a standard API for asynchronous messaging within Java. As part of the Java EE specification, each Java EE certified application server must have a JMS implementation. The open source Java EE application servers JBoss and Glassfish, both provide for ‘pluggable’ JMS implementations, but developers usually give little thought to their choice of implementation and accept the default. This paper looks at the performance of a number of open source JMS implementations in their out-of-the-box configurations:
• JBoss MQ 4.2.2GA
• JBoss Messaging 1.4.0 SP3
• Sun Java System Message Queue 4.1 (Glassfish)
The choice was made to test the out of the box configurations, as this is the configuration most frequently used when deploying an application. In the future we plan to test these same JMS implementations in a number of configurations, including ones where the configurations are as similar as possible, this will include configurations that have been tuned by our consultants to deliver high performance. The results of these future tests will be released in further white papers.
Testing
Testing was performed using two 'large' Amazon EC2 servers, one to run the application server and the other to run the test client. The test client was a stand-alone java application that spawns a number of threads which enqueue messages to a specified queue. The test code connects to the queue, enqueues a number of messages and then disconnects. This is repeated five times and the averages are calculated. Between each test run, the JMS queues are cleared of messages and the application servers are restarted.
The application servers were installed onto the first EC2 machine and left in their default configurations, JBoss Messaging was installed into JBoss 4.2.2 using the instructions that come with the download. The 'default' configuration was used for testing JBoss MQ, and the 'messaging' configuration was used for testing JBoss Messaging. Both JBoss and Glassfish were running on Sun Java 1.6.0_0. No changes were made to the heap size or other JVM parameters.
The client was configured to spawn 5 threads, and posted messages containing strings of around 30 characters.
The code for the JMS client is available here
Results
The results show that the best performing JMS implementation for applications that open a JMS connection, and keep it open for a lengthy amount of time, is JBoss Messaging. For applications that open a JMS connection, send one, or a small number of JMS messages, and then disconnect, reconnecting to send more later, is JBoss MQ. The Sun Java System Message Queue (OpenMQ) performed worse than both JBoss Messaging and JBoss MQ for small batches of messages, but similar to JBoss MQ for larger batches, this is likely due to it using IIOP as the communications protocol when looking up and connecting to the queue.
The table below shows the time taken in ms for each imlpementation to post different numbers of messages to the queue.
| Implementation\ Batch Size | 1 | 10 | 100 | 1000 | 10000 |
| JBoss Messaging | 99 | 106 | 485 | 1367 | 15595 |
| JBoss MQ | 55 | 48 | 423 | 2224 | 21183 |
| GlassFish OpenMQ | 267 | 304 | 972 | 2760 | 21183 |
It is noticable that the time taken to post one message and ten messages are of the same order, which indicates that the overhead in looking up the queue and establishing a connection to the JMS server is the dominating factor when sending small numbers of messages.
Because of the exponential nature of the data we have collected, it makes more sense to divide the time taken to post all the messages by the number of messages posted, giving us a time per message value for a particular number of messages.
The graph below compares the time taken per message, when sending different numbers of messages per connection.
On the graph, MS (blue) is the time taken for JBoss messaging, MQ (pink) is the time taken for JBoss MQ, and GF (yellow) is the time taken for Sun Java System Message Queue (Glassfish). This graph does not clearly show the performance when large numbers of messages are being sent over each connection, which is better seen in the raw data.
| Messages | JBoss Messaging | JBoss MQ | GlassFish OpenMQ |
| 1 | 99 | 56 | 268 |
| 10 | 10.6 | 4.8 | 30.4 |
| 100 | 4.8 | 4.2 | 9.7 |
| 1000 | 1.36 | 2.2 | 2.7 |
| 10000 | 1.36 | 2 | 2.1 |
We can see that the time taken to post a single message, in the default configuration, tends towards 1.36ms for JBoss Messaging, and towards 2ms for JBoss MQ and Glassfish OpenMQ.
As can be seen in the results, JBoss Messaging out-performs JBoss MQ and Sun Java Service Message Queue for large batches of messages. The number of messages used in each batch is limited to 10000, as beyond this number, the servers begin to exhibit some instability. The number of messages was increased by a factor of 10 each time, and the point at which each server failed, and the nature of that failure were noted.
| Implementation | Failure point | Nature of failure |
| JBoss MQ | 1000000 | Server ran out of memory and stopped responding. |
| JBoss Messaging | 100000 | Server threw OutOfMemory errors, client stopped responding. |
| Glassfish OpenMQ | 100000 | Client receives errors indicating that the server is no longer accepting messages. The server continues to respond to other requests. |
MS 100000 – server runs out of heap space.
GF 100000 – client receives errors indicating queue is full, server continues to function correctly.
The configurations could easily be tuned to prevent these errors, but this would not be compatible with our goal of using an out-of-the box configuration.
Conclusion
The three JMS implementations all behave differently, the performance characteristics can essentially be split into a number of areas.
• Establishing a connection
• Sending messages
• Failure characteristics
JBoss MQ was the fastest at establishing a connection, but JBoss Messaging out-performed it at sending messages. The recommended apporach when sending or receiving messages is to open a JMS connection when the client starts, and close it when it shuts down. If this approach is taken, then JBoss Messaging will likely perform the best if the default configuration is used, however there is little difference. If a new connection to the JMS server is established every time a message is sent, then JBoss MQ will perform faster.
When overloaded with messages, JBoss MQ and JBoss messaging suffered failures that brought the whole application server down. OpenMQ, in contrast, prevented the client from sending any more messages once the message queue reached a size limit. This behaviour protected the application server from failure.
This testing has demonstrated that all of the JMS implementations exhibit similar performance when connections are held over long periods of time, but that if you make heavy use of JMS, your choice of JMS implementation should be based upon your expected usage profile. We also showed it is important to tune your configuration to ensure that failures in the JMS subsystem do not affect the rest of the server. The second part of this performace profiling will look at performance of receiving messages using the default configuration, and the third part will look at the performance gains that can be achieved by tuning.
