Written from an administrator point of view, this post will take you step-by-step through the best practices for your production environment and make troubleshooting a far easier proposition!
Logging is crucial to your environment; it can assist you greatly in understanding your system, helps detail items that are either a cause for concern now, or that might be in the future, and is a perfect tool for root cause analysis if fatal errors occur.
Because of this you need log files that are readable, clean, and show what is useful to see. I have seen lots of JBoss log files where the log is spewing out so many errors with full stack traces and thousands upon thousands of INFO messages, that your ability to find the actual nub of the problem takes a lot of time - if you can decipher the log at all!
So, here are some questions to ask about logging:
- What do we get out of the box in EAP 6.4?
- How can we configure it?
- What do we need in our Production environments?
- What do we need to ask of our developers?
A lot of the decisions you make here will be specific to your environment. For example, how critical the applications are, your monitoring configuration, and the ease of troubleshooting are key to how you want your logging to be configured. These are decisions only you can make about the environment you administer and support.
For this article, I will be looking at JBoss EAP 6.4.0 running on CentOS 7.1.1503 and as such, this post will take a Linux slant. The reason I am using 6.4.x is that there are a few enhancements to the logging introduced in this version that I wanted to include.
Note: When I use $JBOSS_HOME I mean the directory in which JBoss is installed, and when I use $JBOSS_LOG_DIR I mean the directory in which the logs are being stored.
- For Standalone this is usually in $JBOSS_HOME/standalone/logs.
- For Domain mode this will be in $JBOSS_HOME/domain/logs for the process controller and host logs, and $JBOSS_HOME/domain/server//logs for the server log.
Out of the Box
As of JBoss EAP 6.4.0 the following logging is set by default:
For a standalone server, GC logging is enabled and is defined in:
The following options are given to the JVM:
-verbose:gc –Xloggc:”$JBOSS_LOG_DIR/gc.log” –XX:+PrintGCDetails –XX:+PrintGCDateStamps –XX:+UseGCLogFileRotation –XX:NumberOfGCLogFiles=5 –XX:GCLogFileSize=3M –XX:-TraceClassUnloading
For a domain server, GC logging is not enabled for the Process Controller, Host Controller or the Servers. You will need to enable these yourself through JVM properties on the Domain Controller for the level you want (i.e. Server level, Server Group level, Host level, Domain level)
For a standalone server the boot log file is defined as:
This is specified in the standalone.sh file but is also defined in the logging.properties file.
For a domain server the boot log file is defined as going to a different log depending on the process running.
For the Process Controller it is:
For the Host Controller it is:
This is specified in the domain.sh file but also in the logging.properties file.
The logging.properties file provides the configuration definitions.
For a standalone server the logging.properties file is defined as:
This is specified in the standalone.sh file
The logging.properties file for the standalone server contains the default information as to what is being logged, telling us the log categories configured and their log levels, the handler configurations and any formatters.
It is the logging.properties file that defines the FILE handler as going to:
...and the CONSOLE handler as going to:
The logging properties file for a domain server is in:
This is specified in the domain.sh
This file contains boot logging configuration only for the Process Controller and the Host Controller.
There is a default server configuration file (which is the same as the standalone logging.properties file) in:
NOTE: The logging.properties file is only active until the logging subsystem is loaded. You will notice by looking at the logging subsystem in the standalone.xml or the domain.xml that it is the same configuration as you see in the logging.properties file.
The Console Log is used when running the scripts in $JBOSS_HOME/bin/init.d which are used when installing JBoss as a service, otherwise it logs to the screen if you are running JBoss from the standalone.sh or domain.sh scripts.
Both the jboss-as-domain.sh and jboss-as-standalone.sh files define the console log to be stored in:
Whilst JBoss supports all log levels there are 6 main ones that get used (This information is taken from the Admin and config guide).
Used for messages providing detailed information about the running state of an application
Used for messages that indicate the progress of individual requests or activities of an application.
Used for messages that indicate the overall progress of an application.
Used to indicate a situation that is not in error but is not considered ideal. May indicate circumstances that may lead to errors in the future.
Used to indicate an error that has occurred that could prevent the current activity or request from completing but will not prevent the application running
Used to indicate events that could cause critical service failure.
NOTE: VERBOSE is not a log level that JBoss supports.
JBoss CLI Logging