C2B2 logo icon

JBoss Logging and Best Practice for EAP 6.4 (Part 1 of 3)

If you’ve ever treated logging as an afterthought, or only given it serious consideration when the actual troubleshooting begins, this three-part guide will give you everything you need to implement a JBoss EAP 6.4 production logging configuration from the word go.

PART 1 | PART 2 | PART 3

Written from an administrator point of view, this post will take you step-by-step through the best practices for your production environment and make troubleshooting a far easier proposition!

Logging is crucial to your environment; it can assist you greatly in understanding your system, helps detail items that are either a cause for concern now, or that might be in the future, and is a perfect tool for root cause analysis if fatal errors occur.

Because of this you need log files that are readable, clean, and show what is useful to see. I have seen lots of JBoss log files where the log is spewing out so many errors with full stack traces and thousands upon thousands of INFO messages, that your ability to find the actual nub of the problem takes a lot of time - if you can decipher the log at all!

So, here are some questions to ask about logging:

  • What do we get out of the box in EAP 6.4?
  • How can we configure it?
  • What do we need in our Production environments?
  • What do we need to ask of our developers?

A lot of the decisions you make here will be specific to your environment. For example, how critical the applications are, your monitoring configuration, and the ease of troubleshooting are key to how you want your logging to be configured. These are decisions only you can make about the environment you administer and support.

For this article, I will be looking at JBoss EAP 6.4.0 running on CentOS 7.1.1503 and as such, this post will take a Linux slant. The reason I am using 6.4.x is that there are a few enhancements to the logging introduced in this version that I wanted to include.

Note: When I use $JBOSS_HOME I mean the directory in which JBoss is installed, and when I use $JBOSS_LOG_DIR I mean the directory in which the logs are being stored.

  • For Standalone this is usually in $JBOSS_HOME/standalone/logs.
  • For Domain mode this will be in $JBOSS_HOME/domain/logs for the process controller and host logs, and $JBOSS_HOME/domain/server//logs for the server log.

 

Out of the Box

As of JBoss EAP 6.4.0 the following logging is set by default:

GC Log

For a standalone server, GC logging is enabled and is defined in:

$JBOSS_HOME/bin/standalone.sh

The following options are given to the JVM:

-verbose:gc –Xloggc:”$JBOSS_LOG_DIR/gc.log” –XX:+PrintGCDetails –XX:+PrintGCDateStamps –XX:+UseGCLogFileRotation –XX:NumberOfGCLogFiles=5 –XX:GCLogFileSize=3M –XX:-TraceClassUnloading

For a domain server, GC logging is not enabled for the Process Controller, Host Controller or the Servers. You will need to enable these yourself through JVM properties on the Domain Controller for the level you want (i.e. Server level, Server Group level, Host level, Domain level)

Boot Log

For a standalone server the boot log file is defined as:

-Dorg.jboss.boot.log.file=$JBOSS_HOME/standalone/server/server.log

This is specified in the standalone.sh file but is also defined in the logging.properties file.

For a domain server the boot log file is defined as going to a different log depending on the process running.

For the Process Controller it is:

-Dorg.jboss.boot.log.file=$JBOSS_LOG_DIR/process-controller.log

For the Host Controller it is:

-Dorg.jboss.boot.log.file=$JBOSS_LOG_DIR/host-controller.log

This is specified in the domain.sh file but also in the logging.properties file.


Logging.properties

The logging.properties file provides the configuration definitions.

For a standalone server the logging.properties file is defined as:

-Dlogging.configuration=file:/$JBOSS_HOME/standalone/configuration/logging.properties

This is specified in the standalone.sh file

The logging.properties file for the standalone server contains the default information as to what is being logged, telling us the log categories configured and their log levels, the handler configurations and any formatters.

It is the logging.properties file that defines the FILE handler as going to:

$JBOSS_HOME/standalone/log/server.log

...and the CONSOLE handler as going to:

SYSTEM_OUT

The logging properties file for a domain server is in:

$JBOSS_HOME/domain/configuration/logging.properties

This is specified in the domain.sh

This file contains boot logging configuration only for the Process Controller and the Host Controller.

There is a default server configuration file (which is the same as the standalone logging.properties file) in:

$JBOSS_HOME/domain/configuration/default-server-logging.properties

NOTE: The logging.properties file is only active until the logging subsystem is loaded. You will notice by looking at the logging subsystem in the standalone.xml or the domain.xml that it is the same configuration as you see in the logging.properties file.


Console Log

The Console Log is used when running the scripts in $JBOSS_HOME/bin/init.d which are used when installing JBoss as a service, otherwise it logs to the screen if you are running JBoss from the standalone.sh or domain.sh scripts.

Both the jboss-as-domain.sh and jboss-as-standalone.sh files define the console log to be stored in:

/var/log/jboss-as/console.log

Log Levels

Whilst JBoss supports all log levels there are 6 main ones that get used (This information is taken from the Admin and config guide).

Log Level
Description

TRACE

Used for messages providing detailed information about the running state of an application

DEBUG

Used for messages that indicate the progress of individual requests or activities of an application.

INFO

Used for messages that indicate the overall progress of an application.

WARN

Used to indicate a situation that is not in error but is not considered ideal.  May indicate circumstances that may lead to errors in the future.

ERROR

Used to indicate an error that has occurred that could prevent the current activity or request from completing but will not prevent the application running

FATAL

Used to indicate events that could cause critical service failure.


NOTE: VERBOSE is not a log level that JBoss supports.
 
JBoss CLI Logging
By default the JBoss CLI logging is turned off. The configuration for this is in:

$JBOSS_HOME/bin/jboss-cli-logging.properties
 
Summary
So, to sum up the first of my blogs about JBoss logging, I have looked at the default configuration we have when first installing JBoss. The second part will look at how we can configure the logging from these default settings, and then I’ll examine some best practice in the final article.
 
References
 
PART 1 | PART 2 | PART 3