C2B2 logo icon

Guerilla JMX Monitoring

What if things go wrong and you have no monitoring in place? That's exactly what happened to one of our customers.

Very often in the course of work at C2B2, we find that customers are running their middleware with no monitoring in place. Often, there is some monitoring, but it is not at the JVM level, just at the OS level which is incredibly limiting. You will know the amount of RAM that the JVM is taking up, but you have no idea if the heap is 20%, 50% or 90% full.

This lack of insight always takes me by surprise, since monitoring is one of the first things we recommend to set up for any business. Why wouldn't you want to know how your business-critical infrastructure is doing in advance of that 3am phone call? Why wouldn't you want to have a week of data to pore over telling you exactly what went wrong at 2:48am leading to the outage, and how to put it right and prevent it happening again before your morning coffee?

The advantages of having a good monitoring solution in place are huge, and the benefits just get compounded. Black Friday is now, officially, a "thing" in the UK. Thanks, Amazon. This year saw a lot of businesses underestimating Black Friday and getting it very wrong on the most lucrative day of the year!
 

Is your business carrying unnecessary risk that could be mitigated by a monitoring solution? Discover our monitoring services and find a bespoke solution to meet your operational needs.


Having good monitoring in place enables such wonderful cloud-based goodies as auto-scaling, meaning you can react to these kinds of situations and increase your capacity to cope with all the extra transactions.

But what if things go wrong and you have no monitoring in place? That's exactly what happened to one customer of ours recently.

The scenario was fairly simple. Hundreds of ActiveMQ brokers were in place in remote offices where applications would queue messages to local queues. Those messages were then bridged to a central ActiveMQ, like a hub and spoke topology.

The problem was that remote queues would get backed up and messages wouldn't be bridged across to the central broker.

The solution to find out what was going on with the ActiveMQ broker was to use the Java Management eXtensions API (JMX). Below, I've included the Java class I put together for that purpose, to read data from MBeans exposed over JMX (written for Java 6):

The script is very basic and was written to serve a specific purpose. There's very little to actually tie it in to ActiveMQ, so if you want to modify it to make a generic version, it shouldn't be too difficult.

The output will be a CSV file with attributes for the broker (add or remove MBean attribute names from the brokerAttributes array as needed) and a separate CSV file for each queue on the broker.

The important lines in the above are (modified below):

 String host = "localhost";
    String port = "1099";
    String url = "service:jmx:rmi:///jndi/rmi://" + host + ":" + port + "/jmxrmi";

    JMXServiceURL serviceUrl = new JMXServiceURL(url);
    JMXConnector jmxConnector = JMXConnectorFactory.connect(serviceUrl, null);

    try {
    
       MBeanServerConnection mbeanConn =
          jmxConnector.getMBeanServerConnection();
       ObjectName mBean = new
          ObjectName("mbean name");
       String attr = "mbean attribute name";
       mbeanConn.getAttribute(mBean, attr);
    
    } catch (Exception e) {
       e.printStackTrace();  
    } finally {  
       jmxConnector.close();  
    }

The above code is connecting to a JMX mbean server using the remote URL built at the top. Then an ObjectName is created with the name of the mbean I'm interested in, and then below that, a String of the attribute I'm interested in is created. I then use the getAttribute method with the MBean and attribute I want.

In a similar vein, I wrote a script using Groovy to monitor Tomcat for a different customer, when we were just interested in the number of HTTP connections.

I include both here because both are useful, but the difference between the Groovy and Java versions is stark. Both are doing essentially the same thing but, for a use case like this, Groovy lets you get straight to code which actually solves the problem, though perhaps Java 8 will go some way to improving the situation.

The important lines in Groovy to get an MBean attribute are:

 def port = 8090
    def host = '192.168.150.154'
    def fileName = 'HTTPMbean.txt'
    def connection = new JmxBuilder().client(port: port, host: host)
    
    connection.connect()
    
    // Get the MBeanServer.
    def mbeanConn = connection.MBeanServerConnection
    
    //Create GroovyMBean. 
    def mbean = new GroovyMBean(mbeanConn, "mbean name")
    
    println "$mbean.attributeName"

You can find names of new MBeans to monitor, and their attributes, with JConsole as shown in the screenshot below:

This is certainly not the sort of monitoring we would recommend long-term. These solutions were one-offs, designed to help in the solving of specific problems. If you ever find yourself in the position of needing to use scripts like these to get some insight into your system, ask yourself why you don't already have that insight and the pile of historical data to go with it!