As you may realise by now, there are a lot of areas in JBoss EAP 6.4.0 you can configure and customise. - so what do you need to be thinking about when deciding what you want to implement in a production environment?
Well, the areas I want to look at in this, the third and final part of the series are:
- What we need in our Production environments?
- What we need to ask of our developers?
For a JBoss deployment to be production-ready from a logging perspective, we need to think about several key areas:
- What areas are the priority for us to monitor
- What housekeeping should be in place
- What can we do to troubleshoot issues when they arise
For most organisations, monitoring solutions are in place that can be configured to connect to the server (usually through an agent,) read the log, and alert for keywords such as ERROR and FATAL. You could also set up the monitoring solution to be more specific, and alert on certain phrases only.
It therefore makes sense for any JBoss server that the log being monitored by a monitoring solution is a single log that contains all messages for these log levels - and that can be easily parsed. From an administrative point of view this is also what I would want to see. One log that contains all I need to know about the current running of the system.
One of the most common operational challenges our clients face is how to monitor middleware environments effectively. Not only is monitoring essential for root-cause analysis of system failures, but is a key activity for capacity planning, failure mitigation, and performance benchmarking. Find out more about our monitoring services.
By default, when installing JBoss as a service we get two logs - a console log and a server log. The console log shows everything that has happened since the last restart, and the server log shows everything that has happened. For me only one of these logs is required and it’s the server log.
This is the mainstay of your information about the system, and should be the only one you need to worry about. So for me – I ignore and limit the information sent to the console log when running JBoss as a service, and concentrate on the server log.
Another thought here is to copy daily server logs to a central server. This can be useful if any trend analysis is required or if you are troubleshooting across a domain.
This may sound obvious, but as the monitoring will alert – the log needs to be clear of errors when you first start monitoring it in production. It is never sensible to start with errors already occurring.
If you do not have any log rotation or housekeeping and endlessly keep logs then eventually disk space will be an issue.
There is generally little point in keeping logs in production for more than 14 days - and more often for more than 7 days. If you are monitoring the system effectively, then the alerts will be seen immediately and dealt with. If any logs need to be kept for Problem Management or Root Cause Analysis scenarios then these can be moved away manually.
One thing to realise here is that if JBoss is running, then if you remove the active log file (moving it to an archive directory perhaps) it won’t automatically regenerate. The best practice is to copy it and empty it whilst in situ.
Luckily, JBoss provides a number of different Log Handlers for us to use to make the housekeeping easy. There are several handlers that can be used to rotate the log on size or time. Now in 6.4 there is also a handler (Periodic Size) that can do either – and acts on whichever triggers the rotation first.
If there is an issue on the server that we need to look into more closely, then we have the ability to add specific log categories and raise the logging as we need them. This will take affect dynamically so we can turn up logging when the system is exhibiting a problem to troubleshoot - and then turn it down when finished.
This is particularly beneficial so that we do not swamp the logs with messages we don’t care about as it can also cause performance headaches - and could cause logs to increase in size substantially, leading to disk space issues.
We also have the potential here to log specific log categories to a different handler and hence a different log file so we can see our troubleshooting messages in a different file outside of the standard logging mechanism which then won’t interfere with normal monitoring.
Personally I like to troubleshoot against a separate debug log, and have a Log Handler previously set up that I can utilise if and when required. This way you can place that log elsewhere, perhaps on a different file system or disk so it interferes less with the normal running of the system.
For this you would create a new Handler and use that handler for specific log categories when required (see the examples in the previous post in this series for how to create a handler and associate a log category with that handler).
For boot errors EAP 6.4.0 has a introduced a CLI command read-boot-errors. It is a Management command and can be used to monitor the boot errors.
This allows a script to be used to see if any boot errors have occurred. Particularly useful if you are starting up a number of servers at the same time.
As we have seen in the previous posts in this series, the Management Interface Logging is turned off by default.
I like this to be turned on. If you are running a large environment, it provides another avenue for troubleshooting and auditing. It could be that a problem occurred due to the wrong CLI command being issued. This could have been ad hoc at the time or may be a script being run automatically. Hence, to see all activity on the server at the time an issue may have occurred is invaluable.
When we talk about Log Levels (as defined previously) and the types of messages that should fall into each level, it’s the developers that often don’t adhere quite as strictly as they should.
I am not a developer - but as an administrator who is the first point of call if issues are flagged in the log, I want to look at the logs on JBoss when an application is running, and ask these questions :
1. How noisy is the log ?
Try the log at different log levels and see whether each level has what you would consider the right information for that level. For example, do INFO messages look like they should be INFO or perhaps really be DEBUG ?
I have seen many applications that are ‘noisy’ and that make the log virtually unreadable and very difficult to diagnose when issues are occurring.
2. If Stack Traces are logged for an error – are they useful for the context of the error?
3. And finally on Stack Traces, are they necessary anyway?
Can the ERROR description define the issue enough that you don’t need to see the entire Stack Trace.
Don’t be afraid to push these issues back to the developers to change. If it affects your ability to properly monitor and troubleshoot a production application then it isn’t production ready in my eyes.
Hopefully some aspects of this series of posts have given you pause for thought and helped you along your way for implementing a production logging configuration that provides an environment that is well monitored and has easier troubleshooting. JBoss has a lot of flexibility where monitoring is concerned and you can get lost in the plethora of options available. My advice? Keep it simple, straightforward and uncluttered. Let it work for you not against you!