This is part two in a series of blogs loking at JBoss EAP 5 and 6.
In Part One of this series on JBoss (available here : http://blog.c2b2.co.uk/2013/01/setting-up-jboss-eap-5-cluster-with.html) I looked at setting up a basic two-node cluster with JBoss EAP 5 and load balancing using mod_jk.
In this blog I will look at things to consider when moving into a production environment along with a few practical examples of some of the lesser known techniques.
Note - I will be configuring the system created in my first blog so if you are using a different version of JBoss or a different configuration then files may have different names or be in different places.
Things to consider when moving into a production environment
Security - When building a production system security is paramount. JBoss out of the box is built for development purposes. This means there are many services which although they may be useful in development are simply not needed in production and these are often left in leaving security holes. You need to ensure you remove unwanted services and also make sure that only the people you want to access the system can access it.
Monitoring & Fault Diagnosis - Whilst we may strive for a completely faultless system it's very rare things go accordingly! Therefore you need monitoring in place to ensure that you can (hopefully) be alerted before something goes wrong or in the case when something does go wrong you can diagnose the cause.
Performance - Application performance is generally measured in terms of response time and throughput although you must also take into account machine performance and JVM performance.
Availability - You need to ensure that your system is available at all times. In the first blog in the series we looked at using clustering and mod_jk to help provide failover and load balancing. This helps ensure that if one server goes down further calls are handled by another server and that server calls are balanced amongst all servers to prevent overloading one.
Scalability refers to the capability of a system to increase total throughput under an increased load when resources are added. There are two ways to scale your system:
Vertically - Scaling vertically means adding more resources to the machines you already have (e.g adding more memory or a better processor) or replacing them with better kit.
Horizontal - Scaling horizontally means adding more machines. This will require more configuration and a more complicated programming model.
However scaling vertically will cost more money and there's only so far you can go before you need to scale out. You also need to ensure you don't introduce single points of failure.
Configuring JBoss
When configuring any application server you have to take into account what you are trying to achieve and what is best suited for the applications you intend to run. One size does not fit all!
It can be a long and complex process and i'm afraid I can't give you a one-stop solution. However, I can give you some pointers to some of the things you can do to prepare your system for production.
Firstly we want to set up a baseline server configuration by copying one of the server configurations in $JBOSS_HOME/server. Deciding which server configuration to use is entirely dependent on your requirements. For this blog I will be using the all configuration and calling it myserver1 as in my first blog. The idea is that once I have one server stripped down and working as I require I can then make copies for other nodes in my cluster.
I would advise that when making changes try starting and stopping JBoss in between to make sure it's still starting up ok. If you make mass changes all at once and then find it won't start or you're getting some error this can be difficult to track down and you may find you need to start over.
Removing unused services
There are a large number of services and other parts of JBoss that are generally not needed or recommended in a production environment. Obviously every system is different so ensure the service isn't needed before removing it.
Removing unnecessary services can helps increase performance as well as close a number of security holes.
The following is a list of the services that can generally be removed from JBoss in a production environment:
Mail service
Quartz scheduler service
Scheduler service
Web console
UUID generator
HTTP Invoker
Hot Deployment Scanner
ROOT.war home page
JBoss Messaging
Web Services
I won't go into detail about how to remove each and every service as details on this can easily be found elsewhere but there are a few lesser known items that can also be removed from JBoss and these I will detail.
Removing the Dynamic ClassLoading Service
There is a simple http service that supports requests for classes for RMI dynamic class loading (org.jboss.web.WebService).
This can be disabled to help secure the system.
To remove the service, edit the file :
$JBOSS_HOME/server/myserver1/conf/jboss-service.xml
Look for the Class Loading section and remove the mbean :
<mbean code="org.jboss.web.WebService" name="jboss:service=WebService">
You will also need to remove the dependency from the EJB deployer. Simply edit the file
$JBOSS/HOME/server/myserver1/deployers/ejb-deployer-jboss-beans.xml
remove the line
<property name="webServiceName">jboss:service=WebService</property>
Switch from lazy JSP compilation to background compilation
If you make a change to a JSP page which has already been compiled JBoss can recompile that page in the background. The previously compiled JSP page will still be available to serve requests. Once the new page has been compiled successfully it will replace the old page. This helps improve availablity of your JSP pages on a production server.
Open the file
$JBOSS_HOME/server/myserver1/deployers/jboss-web.deployer/web.xml
Look for the jsp servlet (org.apache.jasper.servlet.JspServlet).
Ensure the development parameter is set to false.
If development is false and checkInterval is greater than 0, background compilations are enabled.
checkInterval is the time in seconds between checks to see if a JSP needs to be recompiled, default is 0
Add the following init-param :
<init-param>
<param-name>checkInterval</param-name>
<param-value>300</param-value>
</init-param>
Securing the consoles
It is recommended to either remove the consoles entirely, protect them using JAAS or ensure that only users on the local machine can access them.
Replacing components
Hypersonic database - Hypersonic is only intended for development use. It should be replaced with a database of your choosing. Details of how to do this depend on the database but details are readily available on the internet. Note - There are a number of services that use the DefaultDS (default data source), so these will have to be updated to point to your new database.
Monitoring service - I would recommend replacing the default monitoring service with a more substantial tool such as Hyperic or RHQ.
JVM Tuning and Fault Diagnosis
There are a multitude of Java runtime parameters that can be set to make your server perform better. There are also many settings that will help with fault diagnosis.
These can all be set in your servers run.conf file found in
$JBOSS_HOME/server/myserver1
Note : these parameters should be set according to your particular needs. The values given are just examples.
Set your minimum and maximum heap size to be the same :
-Xms1303m -Xmx1303m
Set minimum and maximum perm gen to the same to avoid allocation failures & subsequent full GC's
-XX:PermSize=256m -XX:MaxPermSize=256m
Set the minimum and maximum eden space - tune as appropriate for throughput
-XX:NewSize=300m -XX:MaxNewSize=300m
Log Garbage Collection information
-verbose:gc
-Xloggc:/opt/jboss/jboss-eap-5.1/jboss-as/server/$JBOSSCONF/log/gc.log
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
Ensure a heap dump is generated on an out of memory error
-XX:-HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/opt/dumps/jboss.hprof
Reduce logging overhead
Whilst logging to the console is useful for development purposes and for debugging it's not necessary in a production environment. Therefore we can delete the console appender.
Open the file
$JBOSS_HOME/server/myserver1/conf/jboss-log4j.xml
Delete the console appender :
<appender name="CONSOLE">
...
...
</appender>
Delete the corresponding appender-ref from the <root> element at the bottom of the file:
<appender-ref ref="CONSOLE"/>
Conclusion
Well, that's it for now. There are a myriad of other changes that can be made but this is meant as an introduction to setting up a server for production. There are whole books written on the subject but hopefully I've covered some of the main points and given you a flavour of the kind of things that can be done.
Remember, JBoss out of the box (or out of the zip file!) is not designed to be run 'as is'. You should also note that there is no ideal configuration. It will take time and effort to get the best configuration for what you require.
You should take time to work out what you want in terms of services, then strip back your config to match that. You should test, test and test again to ensure that your configuration matches the requirements with regards to the applications you will be running on your server. You should tune your JVM to ensure you have the best settings for your particular configuration. You should ensure you have monitoring in place to keep a check on everything and ensure that if your server does crash you have as much information as possible at hand to diagnose what caused it. And finally you should ensure that you consider security throughout all aspects of your system from design, development and deployment through to how your server is set up and what you can do to secure it.
In the next blog in this series I will be looking at JBoss EAP 6 and creating a basic 2 node cluster with load balancing. I will be comparing this to what I did with EAP 5 in the first in this blog series.
Andy Overton
Senior Managed Services Consultant
C2B2
Nice post Andy - thank you.
ReplyDelete