C2B2 logo icon

Monitoring Tomcat with JavaMelody

Andy Overton describes an on-premise monitoring solution he deployed for a customer using a small number of Tomcat instances for a business transformation project.

In this post, troubleshooting specialist, Andy Overton describes an on-premises monitoring solution he deployed for a customer using a small number of Tomcat instances for a business transformation project involving a mobile application deployment. Using a step-by-step approach, he walks through his JavaMelody configuration and how he implements alerts in tandem with Jenkins.

Introduction

Whilst working with a customer recently I was looking for a simple, lightweight monitoring solution for monitoring a couple of Tomcat instances when I came across JavaMelody.

https://github.com/javamelody/javamelody/wiki

After initial setup - which is as simple as adding a couple of JAR files to your application - you immediately get a whole host of information readily available with no configuration whatsoever.

Playing about with it for a while and being impressed, I decided to write this blog because I thought I might be able to use my experiences to throw some light on a few of the more complex configurations (e-mail notifications, alerts etc.).

Technology Landscape

I’m going to start from scratch so you can follow along. To begin with, all of this was done on a VM with the following software versions:

  • OS – Ubuntu 16.04
  • Tomcat – 8.0.35
  • JDK - 1.8.0_77
  • JavaMelody – 1.59.0
  • Jenkins - 1.651.2

Tomcat Setup

  1. Download from http://tomcat.apache.org/download-80.cgi
  2. Add an admin user:
  3. Add the following line to tomcat-users.xml:
<role rolename="manager-gui"/>
<user username="admin" password="admin" roles="manager-gui"/>

Start Tomcat by running <TOMCAT_DIR>/bin/startup.sh

The management interface should now be available at: http://localhost:8080/manager

JavaMelody Setup

Download from https://github.com/javamelody/javamelody/releases and unzip.

Add the files javamelody.jar and jrobin-x.jar to to the WEB-INF/lib directory of the WAR file you want to monitor.

I used a simple testapp used for testing clustering. Obviously we’re not testing clustering here but it doesn’t actually matter what the application does for our purposes.

Download the clusterjsp.war from here (or use your own application):

Drop the WAR file in the <TOMCAT_DIR>/webapps directory and it should auto-deploy.

Point a browser to http://localhost:8080/clusterjsp/monitoring and you should see a screen similar to this screen grab from github:

First Look

For new users, I'll just offer a quick run-down of my out-of-the-box experience. First thing you see are the graphs you have immediately available:

  • Used memory
  • CPU
  • HTTP Sessions
  • Active Threads
  • Active JDBC connections
  • Used JDBC connections
  • HTTP hits per minute
  • HTTP mean times (ms)
  • % of HTTP errors
  • SQL hits per minute
  • SQL mean times (ms)
  • % of SQL errors

You can access additional graphs for such things as garbage collection, threads, memory transfers and disk space via the 'Other Charts' link, and helpfully these can be easily expanded with a mouse click. Less helpfully, there's no auto-refresh so you do need to update the charts manually.

If you scroll down, you'll find that 'System Data' will make additional data available and here you can perform the following tasks:

  • Execute the garbage collector
  • Generate a heap dump
  • View a memory histogram
  • Invalidate http sessions
  • View http sessions
  • View the application deployment descriptor
  • View MBean data
  • View OS processes
  • View the JNDI tree

You can also view the debugging logs from this page - offering useful information on how JavaMelody is operating.

Reporting Configuration Guide

JavaMelody features a reporting mechanism that will produce a PDF report of the monitored application which can be generated on an ad-hoc basis or be scheduled for daily, weekly or monthly delivery.

To add this capability simply copy the file itext-2.1.7.jar, located in the directory src/test/test-webapp/WEB-INF/lib/ of the supplied javamelody.zip file to <TOMCAT_DIR>/lib and restart Tomcat.

This will add 'PDF' as a new option at the top of the monitoring screen.

Setting up an SMTP Server

In order to set up a schedule for those reports to be generated and sent via email, you first need to set up a Send Only SMTP server.

Install the software: sudo apt-get install mailutils

This will bring up a basic installation GUI and here you can select 'Internet Site' as the mail server configuration type. Then simply set the system mail name to the hostname of the server.

You'll then need to edit the configuration file /etc/postfix/main.cf and alter the following line from inet_interfaces = all to inet_interfaces = localhost

Restart postfix with: sudo service postfix restart

You can test it with the following command (replacing the e-mail address):
echo "This is a test email" | mail -s "TEST" your_email_address

Scheduling the Report

With the email done, the next step is to schedule JavaMelody to send out daily e-mails of the PDF report. Firstly we need to download a couple of additional libraries.

JavaMail library: http://repo1.maven.org/maven2/javax/mail/mail/1.4.1/mail-1.4.1.jar

Activation library: http://repo1.maven.org/maven2/javax/activation/activation/1.1/activation-1.1.jar

When you have these, copy both files to <TOMCAT_DIR>/lib and add the following code to <TOMCAT_DIR>/conf/context.xml (replacing the e-mail address):

<Resource name="mail/MySession" auth="Container" type="javax.mail.Session"
                                 mail.smtp.host="localhost"
                                 mail.from="no-reply@test.com"
/>
<Parameter name="javamelody.admin-emails" value="your_email_address" override="false"/>
<Parameter name="javamelody.mail-session" value="mail/MySession" override="false"/>
<Parameter name="javamelody.mail-periods" value="day" override="false"/>

Once the server is started, you can send a test mail by calling this action:

http://<host>/<context>/monitoring?action=mail_test

Alerts (Using Jenkins)

Alerting takes a little more setting up and isn’t provided by JavaMelody itself. Instead, it's provided by Jenkins with a Monitoring add-on, so first of all, you'll need to download Jenkins from:

https://jenkins.io/index.html

Use the following command to run Jenkins (we need to run on a different port as we have Tomcat running on the default 8080):  java -jar jenkins.war --httpPort=9090

Jenkins is now available at: http://localhost:9090

The nest step is to install the following plug-ins for Jenkins:

  • Monitoring – Needed for linking in with JavaMelody
  • Groovy – Needed to run Groovy code. This is required for setting up the alerts.
  • Email Extension – Needed to customise the e-mails Jenkins sends out

To install the monitoring plugin:

  1. Click 'Manage Jenkins'
  2. Select 'Manage Plugins'
  3. Select 'Available'
  4. Find and select the 'Monitoring Plugin'
  5. Click 'Install without restart'

Then follow the same procedure for Groovy and Email Extension. 

Groovy Configuration

Now, let's make sure the Groovy runtime is installed and configured by using sudo apt-get install groovy to install it to /usr/share/groovy

In order to run our Groovy scripts and call JavaMelody methods we'll need log4j and JavaMelody on the Groovy classpath. JavaMelody uses an old version for log4j (1.2.9) which can be downloaded from:

http://archive.apache.org/dist/logging/log4j

To configure Groovy:

  1. Go to Manage Jenkins, select 'Configure System'
  2. Under the Groovy section, select 'Groovy Installations'
  3. Add a name for your installation.
  4. Set GROOVY_HOME to /usr/share/groovy

Email Extension Plugin Configuration

  1. Go to Manage Jenkins, select 'Configure System'
  2. Under Jenkins location, set the URL to: http://hostname:9090 (replacing hostname with your hostname)
  3. Set the System Admin e-mail address to: donotreply@jenkins.com (or something similar – this is the address that alert e-mails will be sent from.
  4. Under the Extended E-mail Notification section, set SMTP server to localhost

Creating Alerts

Next up we'll set up a test alert, which triggers when there are more than 0 HTTP sessions - obviously not realistic, but good for demo and testing purposes.

From the main Jenkins menu:

  1. Select 'New Item'
  2. Select 'Freestyle' project
  3. Add the following details:
    • Name - High Session Count Alert
    • Description - Test alert triggered when there are more than 0 HTTP sessions
  4. Under 'Build Triggers', select 'Build' and 'Periodically'

    Now you can schedule how often to run your alert check. The syntax is exactly like a cronjob. Here we will set it to run our check every 10 minutes using the following: */10 * * * *
     
  5. Under 'Build', click 'Add build step'
  6. Select 'Execute Groovy' script
  7. Set the 'Groovy Version' to whatever you called it previously
  8. Add the following Groovy code:
import net.bull.javamelody.*;
url = "http://localhost:8080/clusterTest/monitoring";
sessions = new RemoteCall(url).collectSessionInformations(null);
if (sessions.size() > 0) throw new Exception("Oh No - More than zero sessions!!!");

This simple piece of code calls the URL of JavaMelody, retrieves the sessions information and then if that size is greater than zero, throws an Exception. Add javamelody.jar and log4j jar to the classpath (under Advanced) e.g.:

/home/andy/javamelody/javamelody.jar:/home/andy/logging-log4j-1.2.9/dist/lib/log4j-1.2.9.jar

Under 'Post-Build Actions', select 'Add post build action', then select 'Email Notification', add the email address to send the alert to and finally, Save.

Testing

In order to test the alert triggers as required simply call your application e.g.

http://localhost:8080/clusterTest/jsp/sessionTest.jsp

You should receive an e-mail with the subject 'Build failed in Jenkins', which looks something like this:

------------------------------------------
Started by user anonymous
Building in workspace <http://192.168.43.129:9090/job/High%20Session%20Count%20Alert%202/ws/>
[workspace] $ /usr/share/groovy/bin/groovy -cp /home/andy/javamelody/javamelody.jar:/home/andy/logging-log4j-1.2.9/dist/lib/log4j-1.2.9.jar "<http://192.168.43.129:9090/job/High%20Session%20Count%20Alert%202/ws/hudson4959397560302939243.groovy">
Caught: java.lang.Exception: Alert-Start
Oh No - More than zero sessions!!! Number of sessions: [SessionInformations[id=9BBFCF23C5126EDDBD44B371F1B11FD0, remoteAddr=127.0.0.1, serializedSize=229]]
 Alert-End
java.lang.Exception: Alert-Start
Oh No - More than zero sessions!!! Number of sessions: [SessionInformations[id=9BBFCF23C5126EDDBD44B371F1B11FD0, remoteAddr=127.0.0.1, serializedSize=229]]
 Alert-End
        at hudson4959397560302939243.run(hudson4959397560302939243.groovy:7)
Build step 'Execute Groovy script' marked build as failure

As Jenkins is generally used as a build tool, the outgoing e-mail isn’t the most user friendly when we’re looking to use it for alerting purposes. So, the final thing we will look at is altering the outgoing e-mail into something more legible.

Editing the Outgoing Email

First of all we will alter the Groovy script so that we can strip out the stack trace and additional information that we don’t need as we’re alerting on a specific condition of our app, not the underlying JavaMelody code.

In order to do so we will use Alert-Start and Alert-End to indicate the start and end of the alert message we want to put in the e-mail we will send out. Later we will use a regular expression to extract this from the whole Exception.

Go to the High Session Count Alert project and alter the last line of the Groovy script, changing it from:

if (sessions.size() > 0) throw new Exception("Oh No - More than zero sessions!!!");

to

if (sessions.size() > 0) throw new Exception("Alert-Start\nOh No - More than zero sessions!!! Number of sessions: " + sessions.size() + “\nAlert-End”);
  1. Click Configure
  2. Delete the e-mail notification post-build action
  3. Add a new one - Editable Email Notification
  4. Set Project Recipient List, add your e-mail address
  5. Set the Default Subject to - JavaMelody - High Session Count ALERT
  6. Set the Default Content to the following:
Build URL : ${BUILD_URL}

Alert : ${PROJECT_NAME}

Description: ${JOB_DESCRIPTION}

${BUILD_LOG_EXCERPT, start="^.*Alert-Start.*$", end="^.*Alert-End.*$"}

The key thing here is the BUILD_EXCERPT. This takes in 2 regular expressions to indicate the start and end lines within the build log. This is where we strip out all of the extraneous stack trace info and just get the message between the Alert-Start and Alert-End tags.

To see a list of all available email tokens and what they display, you can click the "?" (question mark) next to the Default Content section.

Conclusion

Hopefully, this post has given you a good starting point for using JavaMelody and Jenkins to monitor your Tomcat instances. There is a lot more that I haven’t covered but I’ll leave that as an exercise for the reader to dig a little deeper.

I’ve been impressed by it as a simple to set up, free monitoring tool. Configuring the alerts is a bit more of an effort but it’s nothing too difficult and it’s a tool I’d certainly recommend.