C2B2 logo icon

JBoss Logging and Best Practice for EAP 6.4 (Part 2 of 3)

Following on from Brian's previous post in the series, which showed you the default logging configuration for JBoss EAP 6.4.0, this post takes a look at how you can configure some of the core components.

PART 1 | PART 2 | PART 3

I'll be taking a standard common approach for the configuration purposes of this post, and will leave more advanced configuration for future posts. So for now, we will primarily look at the configuration for a standalone deployment.


GC Log
The GC Log can be configured in the standalone.conf for standalone servers and in JVM properties for the domain servers.
For the standalone server these can be overridden as a whole by updating the JAVA_OPTS in the standalone.conf file.  (Note – you will need *all* the options you require)
The standalone.sh script checks for the presence of a ‘-verbose:gc’ entry in JAVA_OPTS.  So if this exists in the standalone.conf file then it will bypass the GC configuration in the standalone.sh.

An example additional line in the standalone.conf is :
## Specify options to pass to the Java VM.#if [ "x$JAVA_OPTS" = "x" ]; then   JAVA_OPTS="-Xms1303m -Xmx1303m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true"   JAVA_OPTS="$JAVA_OPTS -Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS -Djava.awt.headless=true"   JAVA_OPTS="$JAVA_OPTS -Djboss.modules.policy-permissions=true"   JAVA_OPTS="$JAVA_OPTS -verbose:gc -Xloggc:/opt/jboss/jboss-eap-6.4/standalone/log/gctest.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=3M -XX:-TraceClassUnloading"else   echo "JAVA_OPTS already set in environment; overriding default settings with values: $JAVA_OPTS"fi

Note: I have changed the name of the log to gctest.log. 

We can then see these options shown in the process :

$ ps -ef | grep jajboss     4438  4355 16 10:42 pts/0    00:00:07 java -D[Standalone] -server -XX:+UseCompressedOops -Xms1303m -Xmx1303m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.modules.policy-permissions=true -verbose:gc -Xloggc:/opt/jboss/jboss-eap-6.4/standalone/log/gctest.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=3M -XX:-TraceClassUnloading -Dorg.jboss.boot.log.file=/opt/jboss/jboss-eap-6.4/standalone/log/server.log -Dlogging.configuration=file:/opt/jboss/jboss-eap-6.4/standalone/configuration/logging.properties -jar /opt/jboss/jboss-eap-6.4/jboss-modules.jar -mp /opt/jboss/jboss-eap-6.4/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone -Djboss.home.dir=/opt/jboss/jboss-eap-6.4 -Djboss.server.base.dir=/opt/jboss/jboss-eap-6.4/standalone
Boot Log

For this post, rather than show how to modify the boot logging, it is worth mentioning the new CLI command introduced in 6.4 – ‘read-boot-errors’.

This is part of the management core service and looks at the log, and reports back errors relating to the start of the server. This is very useful as it can be scripted using CLI to look at numerous servers and check them, pulling the information centrally.

To test this using the standalone server, I renamed the h2 directory so the server could not find the h2 module :

$ pwd/opt/jboss/jboss-eap-6.4/modules/system/layers/base/com/h2database$ mv h2 h2old

I then started the JBoss server and ran the cli command :

$ ./jboss-cli.sh --connect[standalone@localhost:9999 /] /core-service=management:read-boot-errors{    "outcome" => "success",    "result" => [        {            "failed-operation" => {                "operation" => "add",                "address" => [                    ("subsystem" => "datasources"),                    ("jdbc-driver" => "h2")                ]            },            "failure-timestamp" => 1460370253333L,            "failure-description" => "JBAS010441: Failed to load module for driver [com.h2database.h2]"        },        {            "failed-operation" => {                "operation" => "add",                "address" => [                    ("subsystem" => "datasources"),                    ("data-source" => "ExampleDS")                ]            },            "failure-timestamp" => 1460370254540L,            "failure-description" => "{"JBAS014771: Services with missing/unavailable dependencies" => ["jboss.data-source.java:jboss/datasources/ExampleDS is missing [jboss.jdbc-driver.h2]","jboss.driver-demander.java:jboss/datasources/ExampleDS is missing [jboss.jdbc-driver.h2]"]}",            "services-missing-dependencies" => [                "jboss.data-source.java:jboss/datasources/ExampleDS is missing [jboss.jdbc-driver.h2]",                "jboss.driver-demander.java:jboss/datasources/ExampleDS is missing [jboss.jdbc-driver.h2]"            ]        },        {            "failed-operation" => {                "operation" => "enable",                "address" => [                    ("subsystem" => "datasources"),                    ("data-source" => "ExampleDS")                ]            },            "failure-timestamp" => 1460370254542L,            "failure-description" => "{"JBAS014879: One or more services were unable to start due to one or more indirect dependencies not being available." => {"Services that were unable to start:" => ["jboss.data-source.reference-factory.ExampleDS","jboss.naming.context.java.jboss.datasources.ExampleDS"],"Services that may be the cause:" => ["jboss.jdbc-driver.h2"]}}",            "missing-transitive-dependency-problems" => {                "Services that were unable to start:" => [                    "jboss.data-source.reference-factory.ExampleDS",                    "jboss.naming.context.java.jboss.datasources.ExampleDS"                ],                "Services that may be the cause:" => ["jboss.jdbc-driver.h2"]            }        }    ]}
You can see the boot errors are shown and pinpoint the area you need to investigate.
Console Log
As mentioned in the previous post, the console log gets used by default when using the jboss-as-standalone.sh or jboss-as-domain.sh scripts. The file is placed in the /var/log/jboss-as/ directory.
When setting up JBoss to run as a service you will use the jboss-as.conf script. The easiest way to modify where the console log goes is to modify this script which feeds the configuration into the jboss-as-standalone.sh and jboss-as-domain.sh scripts.
Edit the jboss-as.conf file and uncomment the JBOSS_CONSOLE_LOG configuration, and modify as appropriate.

In my example below I have uncommented the line and changed the filename to test.log.

# General configuration for the init.d scripts,# not necessarily for JBoss AS itself.# The username who should own the process.#JBOSS_USER=jboss# The amount of time to wait for startup## STARTUP_WAIT=30# The amount of time to wait for shutdown## SHUTDOWN_WAIT=30# Location to keep the console log## JBOSS_CONSOLE_LOG=/var/log/jboss-as/console.logJBOSS_CONSOLE_LOG=/var/log/jboss-as/test.log

When I now stop and start the service you can then see in my directory the new filename alongside the old.

# pwd/var/log/jboss-as# lltotal 16-rw-r--r--. 1 root root 5679 Apr 11 12:28 console.log-rw-r--r--. 1 root root 4776 Apr 11 12:35 test.log


There are 7 types of Handlers you can create and you can create multiple handlers of each type. For this example we will create a new ‘Size’ Handler Type. We will do this through the CLI and see the results in the Console.

To start – our server is running and we have connected using the CLI. To add a new Handler we use the add command for the new handler name.  For the most part we will keep the default values :

[standalone@localhost:9999 /] /subsystem=logging/size-rotating-file-handler=NEWSIZE:add(file={"path"=>"newsize.log", "relative-to"=>"jboss.server.log.dir"},level="DEBUG",enabled=true, append=false, rotate-size=5m,max-backup-index=10,rotate-on-boot=true,suffix=".yyyy-MM-dd-HH"){"outcome" => "success"}

We have created a handler called ‘NEWSIZE’ that will write to the file ‘newsize.log’ at DEBUG level and rotate if the file gets to 5Mb and keep a backup of 10 files.

We can check the values for the handler we have created :

[standalone@localhost:9999 /] /subsystem=logging/size-rotating-file-handler=NEWSIZE:read-resource{    "outcome" => "success",    "result" => {        "append" => false,        "autoflush" => true,        "enabled" => true,        "encoding" => undefined,        "file" => {            "path" => "newsize.log",            "relative-to" => "jboss.server.log.dir"        },        "filter" => undefined,        "filter-spec" => undefined,        "formatter" => "%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%E%n",        "level" => "DEBUG",        "max-backup-index" => 10,        "name" => "NEWSIZE",        "named-formatter" => undefined,        "rotate-on-boot" => true,        "rotate-size" => "5m",        "suffix" => ".yyyy-MM-dd-HH"    }}

In the Console we can see the Handler added :

We can see our new log created on the file system :

[root@localhost init.d]# ll /opt/jboss/jboss-eap-6.4/standalone/log/total 156-rw-rw-r--. 1 jboss jboss   1669 Apr 11 09:50 backupgc.log.current-rw-rw-r--. 1 jboss jboss   1500 Apr 11 10:05 gc.log.0.current-rw-rw-r--. 1 jboss jboss   1494 Apr 11 12:35 gctest.log.0.current-rw-r--r--. 1 jboss jboss      0 Apr 11 12:59 newsize.log-rw-rw-r--. 1 jboss jboss 133362 Apr 11 12:35 server.log-rw-rw-r--. 1 jboss jboss  10419 Feb  4 19:50 server.log.2016-02-04

If we want to modify an entry we can use the write-attribute command. So if we want to change the size of the files to 10Mb we can use the following:

[standalone@localhost:9999 /] /subsystem=logging/size-rotating-file-handler=NEWSIZE:write-attribute(name=rotate-size,value=10m)
If we want to remove the handler entirely, we can use the remove command:
[standalone@localhost:9999 /] /subsystem=logging/size-rotating-file-handler=NEWSIZE:remove
Log Categories

You can define a log category against a particular handler and level of message you want to see. This is useful when troubleshooting if you know the area you want to analyse, and want to see a higher level of logging just for that area.

For this example we will add a log category for org.apache.coyote and attach it to our NEWSIZE handler we have just created.

To add a new log category we need to use the add command with the new category:
[standalone@localhost:9999 /] /subsystem=logging/logger=org.apache.coyote:add(category=org.apache.coyote,level=DEBUG,handlers=[NEWSIZE]){"outcome" => "success"}We can check the new category :[standalone@localhost:9999 /] /subsystem=logging/logger=org.apache.coyote:read-resource {    "outcome" => "success",    "result" => {        "category" => "org.apache.coyote",        "filter" => undefined,        "filter-spec" => undefined,        "handlers" => ["NEWSIZE"],        "level" => "DEBUG",        "use-parent-handlers" => true    }}

We can see this new category in the console:


If we want to modify an entry we can use the write-attribute command. So if we want to change the log level we can use the following:
[standalone@localhost:9999 /] /subsystem=logging/logger=org.apache.coyote:write-attribute(name=level, value=TRACE)

If we want to remove the handler entirely we can use the remove command:

CLI Logging

To log activity through the CLI and through the Console, you can easily enable the Management Interface logging using a CLI command.

[standalone@localhost:9999 /] /core-service=management/access=audit/logger=audit-log:write-attribute(name=enabled,value=true)

This produces a management audit log file created at $JBOSS_HOME/standalone/data/audit-log.log

You can also modify the $JBOSS_HOME/bin/jboss-cli-logging.properties file for just the CLI logging.  Change the log level to INFO and uncomment the handler.

# Additional logger names to configure (root logger is always configured)loggers=org,org.jboss.as.clilogger.org.level=OFF# assign a lower level to enable CLI logginglogger.org.jboss.as.cli.level=INFO# Root logger levellogger.level=${jboss.cli.log.level:INFO}# Root logger handlers# uncomment to enable logging to the filelogger.handlers=FILE

Once done and CLI restarted then the file jboss-cli.log will be created with CLI information stored.

Advanced Configuration

As mentioned earlier, there are a number of more advanced logging configurations that could be achieved. As these are less standard and commonplace, they have been left in favour of being covered in future blog posts.

  • Logging Profiles and their Configuration
  • SysLog Handlers
  • Log Category Filtering
  • Asynchronous logging

To summarise this blog series so far: We have seen what the default logging configuration is in JBoss EAP 6.4.0 and now know how to reconfigure the most common aspects for different types of logging. Part three will look at the recommendations for which configuration changes you should make.