17 February 2015

JBoss EAP 6 Domain Mode

When Red Hat redesigned JBoss 5, one of the key things they did was to combine all the configuration from separate files within their own modules to a single XML file. When you start JBoss by calling standalone.sh, that single XML file holds all the configuration it uses, which makes it much easier to track down any misconfiguration.

Clustering in standalone mode is very straightforward; simply edit the JGroups subsystem in the appropriate configuration file (standalone-ha or standalone-full-ha) as Andy Overton has outlined in his previous blog. Providing they both have the same configuration, the servers will discover each other and you're done. The downside comes when you have large clusters to manage and need to make the same configuration change in many places!

In domain mode, however, things are a little different.


9 February 2015

Automating SOA Suite Installations

Automation is a useful ability in many fields, with installation of software being no different. Once the time has been put in to create an automating installation script, automation can save you a great deal of time in avoiding what can be seen as a repetitive and time consuming task. In this blog I’ll describe how much of the installation process can be automated, and ways to make the automation reusable.

3 February 2015

Clustering WebSockets on WildFly - Part 2


Hello all

I know that I said in my previous blog that I was doing a two-part series on how to put together a clustered web application running on WildFly on EC2. Having spent some more time hacking around, I've realised that if I was to dump everything else into one more blog it would be quite long-winded. As a result, this is now part two of three. This blog will be focusing on setting up and using Infinispan.

Recap


In the previous post, we talked about the basics of setting up a WebSocket application running on WildFly and ran through some of the front-end and back-end code for that. We also looked at how to tweak the JGroups configuration for WildFly's clustering on EC2.


Infinispan


Infinispan is a highly scalable and highly available in-memory key/value data store written in Java - and is the open source project behind JBoss Data Grid. Infinispan can be used as a distributed cache in front of an SQL database or as its own NoSQL data grid. When using it as a data grid, it is possible to configure Infinispan to persist data to disk-based data stores in order to ensure that data is not lost.

Infinispan is typically used in two different modes:

1. Library mode is where you use Infinispan within your own application's source code.
2. Client/Server mode is where you have Infinispan running separately from your application as its own server. This is the mode that we will use for our setup.

The Infinispan Server has four endpoints - or protocols - that clients can make use of. The protocols are:

1. HotRod
2. REST
3. Memcached
4. WebSocket

In this case, we will make use of the WebSocket protocol. Given that we have previously looked at setting up a WebSocket server on WildFly, it seems like a natural fit to now look at making use of a WebSocket client in Java to store our data.

Setup changes


In the first blog, I put together a diagram of the architecture set-up that I was intending to use. I did not include anything in there regarding how and where we were going to deal with our storage for the application.

This is an updated revision of the architecture:

In practice, we would expect multiple Infinispan instances running within our architecture to ensure the availability of our data. For this demo however, we will just be using a single server.


WebSocket requests


There isn't a large amount of documentation on making use of the WebSocket server for Infinispan - at least for using a Java client - so it was a nice exercise to dig around the Infinispan server codebase to dig out what our request would have to look like.

As we can see from the above gist, we need to send a request (as a string) that has a JSON structure with some specific parameters:
  • The "opCode" parameter tells the server what operation you are attempting to perform - put/get/remove/notify are the options. This is so that the server knows which of its internal handlers to invoke in order to deal with your operation. 
  • The "mime" parameter tells the server the mime type that you are using. The server cannot deal with "application/json". So I used "text/plain". 
  • The "cacheName" parameter tells the server the name of the cache you want to perform your operation on. 
  • The "key" parameter states the key you wish to perform your operation on. 
  • The "value" parameter is only specified for a put operation and will be the value that you wish to use.
Any time when we do send a message to the server, we have to ensure that we create a structure which follows this one.


InfinispanEndpoint class


In the application that runs on WildFly, we have put together an InfinispanEndpoint class. This class will be the one that deals with communicating with the Infinispan server.

Similar to the @ServerEndpoint annotation used on the server side, we would use a @ClientEndpoint annotation for the Java client. This is annotated on the class level. We also have the same method level annotations available, such as @OnOpen, @OnClose, @OnMessage etc.

For starters, let's look at how we start the client. In this case, we are instantiating the client from our application code, and we will provide the WebSocket URI to connect to - so we are passing that as a parameter to our constructor.

For the other annotated methods, all that we are doing is logging messages in our application - apart from the onMessage() method, which is where we do a little bit more work. What we have done in addition is set up a separate MessageHandler interface; we can instantiate different instances in our Getter and Storer classes that will then deal with these messages appropriately. For now though, let's look at what we have to do within this endpoint class.

The important method to look at is sendMessage() (in testing, the WebSocket wouldn't be opened in time before we tried to send a message so we just pause for a short time) as this is the one that sends the message to the server. The API for doing so is identical to the server-side one, because we are dealing with the same type of Session object!

I also put together a separate client (running in a main() method) to test this functionality. What this client does is put a key/value pair, wait 10s and then try and get the same key. Once it gets the server response it will just dump that message to screen - this was also so I could understand what the server responses looked like. Here is how I am handling the message:

From the gist below, we can see how the messages that we send to Infinispan are constructed and then sent out, as well as the successful response from the server.
And there we go, we have now built a Java client to connect to the Infinispan WebSocket Server. If you are unclear on how some of the other internal client wiring works, I followed this thread on StackOverflow quite closely for some ideas. It quite neatly explains how to set things up.

In the next part, we will look at putting all of these parts together in a full application running on Amazon EC2.

Thanks for reading!

Navin Surtani 
C2B2 Consultant 

27 January 2015

MIDDLEWARE INSIGHT - C2B2 Newsletter Issue 20



FEATURED NEWS


Clustering WebSockets on Wildfly - read more

Can Chef be used to help provide Continuous Delivery of Oracle SOA Suite Applications? - read more



JAVA EE & OPEN SOURCE


Java EE 8
What’s up with Java EE 8? A Whistle-Stop Tour - read Part 1 and Part 2 on Voxxed 
A Look at the Proposed Java EE 8 Security API - read more here 
The most popular upcoming Java EE 8 technologies according to ZEEF users - read more on Arjan Tijms' blog
GlassFish
Using JASPIC to secure a web application in GlassFish, read more on the C2B2 Blog
Vasilis Souvatzis's Java EE 7 Thesis Using GlassFish - see more on the Aquarium Blog  
Getting started with Payara Server - a drop in replacement for GlassFish Server - see the video here
What's in store for the Vampire fish? Read more on Payara Blog
A GlassFish Alternative with Teeth - read more on Voxxed
London JavaEE and GlassFish User Group with Peter Pilgrim ' Digital Java EE 7 New and Noteworthy' - find out more and register here
Tomcat
Self-Signed Certificate for Apache TomEE (and Tomcat) - read more on Alex Soto's blog  
Alternative Logging Frameworks for Application Servers: Tomcat - read the article by Andy Pielage 
Other
Guerilla JMX Monitoring - read more by Mike Croft
Java EE VS Spring smackdown - find out more on the Payara Blog
Lightweight Integration with Java EE and Apache Camel - read more on Voxxed
Thinking About Picking Up a New JVM Language? A Masterpost to Guide Java Devs - read more on Voxxed
Pivotal cuts funding for open-source JVM language Groovy - read more here ; see the related article 'Open Source doesn’t need more funding – it needs better business models' on Jaxenter.com 
Spring Framework 4.1.4 released - read more on the Spring Blog
Why I Don't Like Open Source - read the article by Remy Sharp 
DDD (Domain-Driven Design) + Java EE "Hanginar" on Thursday - read more by Reza Rahman
File Uploads Using JSF 2.2 and Java EE 7 - find out more on The Aquarium Blog 

ORACLE


Can Chef be used to help provide Continuous Delivery of Oracle SOA Suite Applications? - read more on the C2B2 Blog
Purging data from Oracle SOA Suite 11g - Part 1 read the article by Irfan Suleman
Set-up a 12c SOA/BPM Infrastructure - read the article by Rene van Wijk  
WebLogic Server and the Oracle Maven Repository - read more on the WebLogic Server Blog
Additional new material WebLogic Community - read more on the WebLogic Community
Oracle presents: Avatar 2.0 – where to next? Read more on Jaxenter.com

JBOSS & RED HAT

Configuring RBAC in JBoss EAP and Wildfly - Part Two - read more on the C2B2 Blog
Clustering WebSockets on Wildfly - read the article by Navin Surtani
Hibernate Search 5: Adding Full-Text Query Super-Powers to Your JPA! - see presentation video and slides by Sanne Grinovero
Java EE Webcast: Hibernate OGM - see more on Voxxed
Java EE, Docker, WildFly and Microservices on Docker - read more on Markus Eisele's Blog
Openshift - New Platform, New Docs: Be a Part of It! - read more on the Openshift Blog
Containers, microservices, and orchestrating the whole symphony - read the article by Uri Cohen
Vagrant with Docker provider, using WildFly and Java EE 7 image - read more on Arun Gupta's Blog
Simple Java EE (JSF) Login Page with JBoss PicketLink Security - read more by Lincoln Baxter

  DATA GRIDS & BIG DATA


Payara 4.1.151 sneak preview - Hazelcast session persistence - find out more on the Payara Blog
Tapping Big Data to Your Own Advantage - read more on Dzone
How To Setup Big Data Tooling For JBoss Developer Studio 8 - read more on Eric Schabell's Blog
DBA skills must evolve, morph to cope with big data technologies - read more on TechTarget

23 January 2015

Using JASPIC to secure a web application in GlassFish

Introduction

In this blog post I will provide a brief introduction to JASPIC and then take a walk through setting up a basic demo using JASPIC to secure a simple web application in GlassFish.

What is JASPIC?

JASPIC stands for Java Authentication Service Provider Interface for Containers. The original JSR for JASPIC was created back in 2002 but it wasn't completed until 2007 and wasn't included in Java EE until Java EE 6 back in 2009.

JSR 196 defines a standard service-provider interface (SPI) and standardises how an authentication module is integrated into a Java EE container.

It is supported by all the popular web containers and is mandatory for the full Java EE 6 profile.

It provides a message processing model and details a number of interaction points on the client and server.

A compatible web container will use the SPI at these points to delegate the corresponding message security processing to a server authentication module (SAM).

Walk-through

I will be using the following software whilst doing this walk-through. If you are using different versions then you may see different results.

Ubuntu 14.04 LTS
Eclipse Luna (with Glassfish Tools - available here - http://marketplace.eclipse.org/content/glassfish-tools-luna
JDK 1.7.0_25
GlassFish 4.1

Creating the Runtime Environment

First of all, we will create a runtime environment so we can run GlassFish from within Eclipse. In order to do so:

In Eclipse go to:
File->New->Other->Server
Select Glassfish 4
Set the JDK
Set the Server directory to <GF_INSTALL_DIR>/glassfish
All other defaults should be OK.

Creating the web application

OK. So, next we will create a very basic web-app consisting of a single servlet.

In Eclipse go to:
File -> New -> Dynamic Web Project

Name - JASPICTest

Make sure the Target runtime is GlassFish 4.
Press Next twice to accept the defaults and then select Generate web.xml deployment descriptor on the last page.
Click Finish.

Right-click on your newly created project and select New → Servlet

package - uk.co.c2b2
Class Name - TestServlet
Click Next and Finish.

This will create a basic servlet. Add the following code to the doGet method:

PrintWriter out = response.getWriter();

try
{
out.println("<html>");
out.println("<head>");
out.println("<title>Test Servlet</title>");
out.println("</head>");
out.println("<body>");
out.println("You have accessed TestServlet at " + request.getContextPath () + "<br>");

Principal userPrincipal = request.getUserPrincipal();
boolean adminUser = request.isUserInRole("admin");
String userName;

if (userPrincipal != null)
{
userName = userPrincipal.getName();
}
else
{
userName = "Unknown User";
}

out.println("You are currently authenticated as: " + userName + "<br>");

if (adminUser)
{
out.println("<br>As you're admin you can view this.<br>");
}
else
{
out.println("<br>Sorry, you're not admin. Nothing to see here.<br>");
}

out.println("</body>");
out.println("</html>");
}
finally
{
out.close();
}

This is very basic but will allow us to see the relevant authentication data being returned by the server.

Testing your application

Start your GlassFish server - Right click on your server and click Start. (Note - If you can't see the Servers tab ensure you have the Java EE perspective open).

To run your servlet right click on TestServlet, select Run As -> Run on Server

Make sure your GlassFish server is selected and hit Finish. Click on Always use this server when running this project to make things simpler in future.

Go to:

http://localhost:8080/JASPICTest/TestServlet

and you should get the following response:

You have accessed TestServlet at /JASPICTest2
You are currently authenticated as: Unknown User

Sorry, you're not admin. Nothing to see here.

Creating the Server Authentication Module

The Server Authentication Module (SAM) must implement the javax.security.auth.message.module.ServerAuthModule interface as defined by JSR 196. The interface can be found here:

https://docs.oracle.com/javaee/6/api/javax/security/auth/message/module/ServerAuthModule.html

The SAM is invoked indirectly by the message processing runtime at the validateRequest and secureResponse interaction points.

Create a new Java project - TestAuthModule

Right click project, select Build Path-> Configure Build Path->Libraries.

Click Add External JARs

Add the following dependencies:

javax.servlet-api.jar
javax.security.auth.message-api.jar

Both can be found in:

<GF_INSTALL_DIR>/glassfish/modules

These will only be used to compile the code, you don't need to package them up as the web container will already contain copies.

Right click on your new project and create a new Java class TestAuthModule which implements the interface - javax.security.auth.message.module.ServerAuthModule

For now we won't add in anything aside from replacing the auto-generated validateRequest method with the following:

public AuthStatus validateRequest(MessageInfo messageInfo, Subject clientSubject,
Subject serviceSubject) throws AuthException
{
    System.out.println("validateRequest called");
 
    return AuthStatus.SUCCESS;
}

Export your new project as a jar file. Right click on the TestAuthModel project, select Export->Java->JarFile.

Remove the classpath and project resources and call the output file TestAuthModule.jar

Export the jar file to <GF_INSTALL_DIR>/glassfish/lib

Restart Glassfish

Configuring the SAM

Next up we need to create a message security provider in GlassFish and then link this to our web app.

Go to the GlassFish admin console:

http://localhost:4848

Go to Configurations->server-config->Security->Message Security

Select HttpServlet

Select the Providers tab. Click New.

Provider ID - TestSAM
Provider Type - server
Class Name - uk.co.c2b2.TestAuthModule

Updating the web app

In the JASPICTest app, edit glassfish-web.xml

Add the following to indicate that the TestSAM you have just set up should be used for this app.

<glassfish-web-app httpservlet-security-provider="TestSAM">

Testing the changes

In Eclipse - Right click on TestServlet and select Run As -> Run On Server

You should now see in the Glassfish console output:

INFO: validateRequest called

This shows that Glassfish our SAM is now being used.

Locking down resources

OK, so at the moment we have a web app that is linked to our SAM but we haven't actually said to secure anything and even if we did our SAM simply authenticates anyone!

So, lets implement some (albeit very basic) security.

NOTE - This is only for demo purposes and to show how JASPIC works, it is most definitely not intended to be a way of doing security!.

First of all, let's lock down our servlet. We want to lock it down to only users with the role admin or standard. To do so, add the following to the application web.xml:

<security-constraint>
<web-resource-collection>
<web-resource-name>JASPICTest</web-resource-name>
<url-pattern>/*</url-pattern>
</web-resource-collection>
<auth-constraint>
<role-name>standard</role-name>
<role-name>admin</role-name>
</auth-constraint>
</security-constraint>

<security-role>
<role-name>standard</role-name>
</security-role>
<security-role>
<role-name>admin</role-name>
</security-role>

There is one additional (rather ugly) step we need to do to make our app work. In order for GlassFish to accept the roles that our authentication module puts into the JAAS Subject we have to map them to groups.

In order to do so, add the following to glassfish-web.xml

<security-role-mapping>
        <role-name>standard</role-name>
<group-name>standard</group-name>
</security-role-mapping>
<security-role-mapping>
<role-name>admin</role-name>
<group-name>admin</group-name>
</security-role-mapping>

Next up we will alter our SAM and implement the methods.

Each of the methods is implemented as follows:

initialize - Simply takes the CallBackHandler and instantiates our local handler.
getSupportedMessageTypes - Returns the HTTP servlet request and response types.
secureResponse - Simply returns Success.
cleanSubject - Clears all principals from the Subject.

validateRequest - This is the main method of interest. In order to pass in the user and role I have just added them as servlet request parameters for testing purposes. This method extracts those values and then calls authenticateUser.

authenticateUser - NOTE - This method doesn't actually do any authentication! It simply takes the user and group, creates callback classes from them and passes them to the callback handler.

Once we have added the above our TestAuthModule looks like this:

public class TestAuthModule implements ServerAuthModule
{
@SuppressWarnings("rawtypes")
protected static final Class[] supportedMessageTypes = new Class[] {
HttpServletRequest.class, HttpServletResponse.class };

private CallbackHandler handler;

public void initialize(MessagePolicy requestPolicy, MessagePolicy responsePolicy,
CallbackHandler handler, @SuppressWarnings("rawtypes") Map options) throws AuthException
{
System.out.println("initialize called.");
this.handler = handler;
}

@SuppressWarnings("rawtypes")
public Class[] getSupportedMessageTypes()
{
return supportedMessageTypes;
}

public AuthStatus validateRequest(MessageInfo messageInfo, Subject clientSubject,
Subject serverSubject) throws AuthException
{
HttpServletRequest request = (HttpServletRequest)messageInfo.getRequestMessage();

String user = request.getParameter("user");
String group = request.getParameter("group");

System.out.println("validateRequest called.");
System.out.println("User = " + user);
System.out.println("Group = " + group);

authenticateUser(user, group, clientSubject, serverSubject);

return AuthStatus.SUCCESS;
}

public AuthStatus secureResponse(MessageInfo msgInfo, Subject service) throws AuthException
{
System.out.println("secureResponse called.");
return AuthStatus.SEND_SUCCESS;
}

public void cleanSubject(MessageInfo msgInfo, Subject subject) throws AuthException
{
if (subject != null)
{
subject.getPrincipals().clear();
}
}

private void authenticateUser(String user, String group, Subject clientSubject,
Subject serverSubject)
{
System.out.println("Authenticating user " + user + " in group " + group);

CallerPrincipalCallback callerPrincipalCallback = new CallerPrincipalCallback(clientSubject,
user);

GroupPrincipalCallback groupPrincipalCallback = new GroupPrincipalCallback(clientSubject,
new String[] { group });

try
{
handler.handle(new Callback[] { callerPrincipalCallback, groupPrincipalCallback });
}
catch (Exception e)
{
e.printStackTrace();
}
}
}

Testing

Now all we need to do is test our new module.

First of all build it, jar it up and copy it over to <GF_INSTALL_DIR>/glassfish/lib as before.

Restart Glassfish.

Now we can test by passing in different dummy credentials.

You should see the following:

If you go to:

http://localhost:8080/JASPICTest/TestServlet?user=Andy&group=standard

You should see the "Sorry, you're not admin. Nothing to see here." message.

If you go to:

http://localhost:8080/JASPICTest/TestServlet?user=Andy&group=admin

You should see the "As you're admin you can view this." message.

And if you go to:

http://localhost:8080/JASPICTest/TestServlet?user=Andy&group=xxx

You should see a HTTP Status 403 - Forbidden message.

Wrapping Up

Hopefully this has given you a taster of how to use JASPIC to secure your applications and you can see how relatively straightforward it is to put the basic building blocks in place.

If you're looking for an example of a SAM that does authentication then there is one available from Oracle here:

http://docs.oracle.com/cd/E19798-01/821-1752/gizeb/index.html

Although JASPIC is yet to take off it's a good first step towards standardising security in web containers and avoids the need for each to have their own proprietary solution, although there is still the issue of different containers using different deployment descriptors hindering the portability of apps.














21 January 2015

Can Chef be used to help provide Continuous Delivery of Oracle SOA Suite Applications?


It was one of those rare occasions that I was in the office when my boss started a conversation with me that went along the lines of:

Steve: I’ve been thinking…..
Alan: Yes? (I wonder where this conversations going)
Steve: Alan, you have been working with Chef for customer ‘X’ for about a year now’
Alan: Yes? (Still not sure where this conversation is going)
Steve: I know you have been working on deployments to large scale JBoss EAP 5.x/6.x (Standalone) clusters, but would Chef be a suitable tool to help provide ‘Continuous Delivery of Oracle SOA Suite Applications?
Alan: Give me a couple of days to review our WLST scripts used to deploy Oracle SOA applications (currently executed manually) to assess if Chef can be used and I’ll write a blog about it.

Background

I have been working with one of our customers for just over a year helping them to achieve their goal towards providing ‘Continuous Delivery’ for a number of their core business applications. Continuous focuses on automating all the build and deployment steps up to and including the UAT environments. The Dev Ops teams have been using git, Maven, Jira, Stash and Jenkins for a number of years so have a well-established ‘Continuous Integration’ process, but lacked the tools to automate the deployment.

In order to implement Continuous Delivery, a tool was required to deploy the applications to JBoss EAP 5.x and 6.x (Standalone) application server clusters with up to 100 nodes per environment, enter Chef See previous blog 'Who’s pulling the strings when it comes to provisioning IT systems, Puppet or Chef? '

Chef is a software provisioning system developed by Opscode which allows the infrastructure is modelled as code on the Chef server, with clients installed on the servers to be managed (Nodes). The Chef clients communicate with the Chef server to determine what changes need to be made to the nodes configuration. The infrastructure is modelled using the following Chef objects:

  • Cookbooks contain Chef DSL/ruby code in recipes, libraries and definitions to define what resources/operations need to be carried out on a node. The ruby code references from attributes defined in either the cookbook, roles, data bags or environments.
  • Data Bags are used to define global attributes that can be used by all code defined in the cookbooks.
  • Environments are associated to nodes managed by Chef and define the cookbooks and any attributes specific for that environment. The cookbook definitions can be pinned to a version, allowing for different versions to be applied to the DEV, TEST and PROD environments.
  • Roles describe the purpose of a server and defines the run-list and order of recipes to be applied to the node. 
Nodes are registered with the Chef server and assigned a role(s) and environment

Chef provides a command line utility called knife, which is used to manage the Chef objects and environment. The knife ssh subcommand is used to invoke SSH commands (in parallel) on a subset of nodes, based on the results of a search query made to the Chef server. This allows you to invoke a Chef client run on nodes to provision/deploy an environment.

As an example, say we have an application that processes ‘Insurance Claims’ and comprises of a web tier deployed to a 2 node cluster and business logic tier to a 5 node cluster.

In the above example the following Chef Roles would be created, ‘ClaimsUI’ and ‘ClaimsBusinessLogic’, defining the run-list of recipes that are executed to deploy each component.

A Chef Environment ‘ClaimsDev’ would be created where the environment specific properties for the application would be defined, such as database connection details, web server urls etc. The nodes would be assigned the appropriate roles and environment, so for example, to deploy the Claims UI to the development environment, the following knife command would be issued:

knife ssh -x afryer “chef_environment:ClaimsDev AND role: ClaimsUI” “sudo -u root chef-client -l info”

The knife ssh command queries the Chef Server returning a list of matching nodes in the Claims Development environment that need the Claims Web interface to be deployed. An ssh session is started on each node returned as the user ‘afryer’ and runs the Chef client with sudo access. The Chef client connects to the Chef server and updates the attributes in the node’s hash map and executes the recipes defined in the ClaimsDev Role’s run-list. The recipes read the attributes from the nodes hash map (defined in cookbooks/environments/roles) and performs the operations in the recipes, deploying the Claims Web component on the required nodes.
We now have a mechanism to deploy/provision applications to an environment on multiple nodes from a single command line call. Configuring the knife tool on the server running Jenkins and executing a shell from a Jenkins task to run the appropriate knife ssh command enables ‘Continuous Delivery’ to be achieved in the Dev, Test and UAT environments.

So can Chef be used to provision Oracle SOA Suite? Oracle SOA Suite is based on the Oracle Weblogic server which consists of an Administration Server and Managed Servers in a Domain. The Administration Server is used to configure Managed Servers in a Domain and deploy applications to them. The WebLogic Scripting Tool (WLST) is a command-line interface used to automate domain configuration, application deployment and configuration, see Oracle WebLogic Scripting Tool for more information. This still requires all the Oracle Weblogic binaries to be installed on the hosts in the Domain, but the majority of the configuration will be done via the Administration Server typically using WLST.

Chef has an execute resource that can run any OS script/command which means that WLST scripts can be executed from a Chef client run. The properties required to drive the deployment would need to be modelled as attributes in Chef Environments/Cookbooks, which would be read from the node hash at runtime by the Chef recipes and passed to the required WLST scripts. Hence Chef would be able to install the Weblogic binaries on all the nodes in a Domain, perform any configuration on the individual Managed servers, executing any WLST scripts via the Administration Server.
In summary, Chef can deliver a solution that automates the provisioning/deployment of Oracle SOA Suite applications in a repeatable manner, significantly reducing deployment times. This lends itself to provisioning consistent configuration across multiple environments leading to a reduction in time spent trying to debug configuration errors which can occur with manual deployments. With careful design of the recipes in a granular way, it can also help to promote the building of standard environments quickly for new projects using a certified middleware stack. This enforces good practices across all projects, improving the quality of the systems delivered, reducing development and support costs.

In the next blog I’ll create a simple Chef cookbook to deploy and configure Oracle Weblogic SOA Suite in a development environment.


16 January 2015

Purging data from Oracle SOA Suite 11g - Part 1

The following questions will be answered:
    • How does Oracle SOA Suite 11g (PS6 11.1.1.7) store data?
    • What data does Oracle SOA Suite 11g (PS6 11.1.1.7) store?
    • Why do you need to purge Oracle SOA Suite 11g (PS6 11.1.1.7) data?
    • What are the purging options available for Oracle SOA Suite 11g (PS6 11.1.1.7)?
    • Which data will be purged by the Oracle SOA Suite 11g (PS6 11.1.1.7) purge script?
    • List of composite instance states that will be considered for purging by the purge script
    • How to install the Oracle SOA Suite 11g (PS6 11.1.1.7) purge script?
    • How to execute the Oracle SOA Suite 11g (PS6 11.1.1.7) purge script?
    • What is Looped purging (Oracle SOA Suite 11g (PS6 11.1.1.7) purge script)?
    • What is Parallel purging (Oracle SOA Suite 11g (PS6 11.1.1.7) purge script)?
    • Description of parameters used by the Oracle SOA Suite 11g (PS6 11.1.1.7) purge script
    • Example 1: Executing the Oracle SOA Suite 11g (PS6 11.1.1.7) purge script for all composites
    • Example 2: Executing the Oracle SOA Suite 11g (PS6 11.1.1.7) purge script for a specific composite


How does Oracle SOA Suite 11g (PS6 11.1.1.7) store data?

SOA Suite uses a database schema called SOAINFRA (collection of database objects such as tables, views, procedures, functions etc.) to store data required for the running of SOA Suite applications. The SOAINFRA (SOA Infrastructure) schema is also referred to as the ‘dehydration store’ acting as the persistence layer for capturing SOA Suite data.

What data does Oracle SOA Suite 11g (PS6 11.1.1.7) store?

Composite instances utilising the SOA Suite Service Engines (BPEL, mediator, human task, rules, BPM, OSB, EDN etc.) will write data to tables residing within the SOAINFRA schema. Each of the engines will either write data to specific engine tables (e.g. the CUBE_INSTANCE table is used solely by the BPEL engine) or common tables that are shared by the SOA Suite engines such as the AUDIT_TRAIL table.

Few examples (below) of the type of data that is stored within the SOAINFRA schema:


  • Message payload (e.g. input, output)
  • Scope (e.g. variables)
  • Auditing (e.g. data flow timestamps)
     
  • Faults        
  • Deferred (messages that can be recovered)
  • Metrics

Why do you need to purge Oracle SOA Suite 11g (PS6 11.1.1.7) data?

Data within the Oracle SOA Suite database can grow to substantial levels in a short space of time. Payload sizes and volume of data will have an impact on available disk space which in turn will affect the performance of SOA Suite. For example, EM console can often become slow to navigate, increasing number of messages becoming stuck or requiring recovery, JTA transaction problems etc.


Purging itself, can become challenging if the data has not been maintained due to the large number of composite instances. Therefore, establishing a purge strategy and implementing it on a regular basis will help maintain the health of SOA Suite keeping the environment running efficiently.  

What are the purging options available for Oracle SOA Suite 11g (PS6 11.1.1.7)?

Oracle provides three options for purging Oracle SOA Suite 11g data:


  • EM Console: Within the Enterprise Manager console the ‘Delete with Options’ can be used to manually delete many instances at once however, this may lead to transaction timeouts and is not recommended for large volumes.
  • Purge Script: This is the process of deleting instances that are no longer required using stored procedures that are provided with Oracle SOA Suite 11g out of the box.
  • Partitioning: Instances are segregated based on user defined criteria within the database, when a partition is not required it will be dropped freeing the disk space.

Which data will be purged by the Oracle SOA Suite 11g (PS6 11.1.1.7) purge script?


The purge script will delete composite instances that are in the following states:
  • Completed
  • Faulted
  • Terminated by user
  • Stale
  • Unknown


The purge script will NOT delete composite instances that are in the following states:
  • Running (in-flight)
  • Suspended
  • Pending Recovery


List of composite instance states that will be considered for purging by the purge script:

Instance
State
Instance
State Description
Will Oracle purge script delete instances in this state?
0
Running
no
1
Completed
yes
2
Running with faults
no
3
Completed with faults
yes
4
Running with recovery required
no
5
Completed with recovery required
no
6
Running with faults and recovery required
no
7
Completed with faults and recovery required
no
8
Running with suspended
no
9
Completed with suspended
no
10
Running with faults and suspended
no
11
Completed with faults and suspended
no
12
Running with recovery required and suspended
no
13
Completed with recovery required and suspended
no
14
Running with faults, recovery required, and suspended
no
15
Completed with faults, recovery required, and suspended
no
16
Running with terminated
yes
17
Completed with terminated
no
18
Running with faults and terminated
no
19
Completed with faults and terminated
yes
20
Running with recovery required and terminated
no
21
Completed with recovery required and terminated
no
22
Running with faults, recovery required, and terminated
no
23
Completed with faults, recovery required, and terminated
no
24
Running with suspended and terminated
no
25
Completed with suspended and terminated
no
26
Running with faulted, suspended, and terminated
no
27
Completed with faulted, suspended, and terminated
no
28
Running with recovery required, suspended, and terminated
no
29
Completed with recovery required, suspended, and terminated
no
30
Running with faulted, recovery required, suspended, and terminated
no
31
Completed with faulted, recovery required, suspended, and terminated
no
32
Unknown
yes
64
-
yes

How to install the Oracle SOA Suite 11g (PS6 11.1.1.7) purge script?

The following details will be required:


  • Database host details:
    • hostname (IP address)
    • username
    • password


  • SOA Database details:
    • SOAINFRA schema prefix 
    • SOAINFRA schema password


  • Full path of the SOA Suite home folder
  • Full path of the directory where the Oracle purge script will write log information to (a folder on the database host)


‘DEV’ was the soainfra schema prefix used for the examples below.


    1. Log into the Database host server.

    1. Connect to the database as administrator using SQL*Plus:
sqlplus / as sysdba

    1. Grant privileges to the soainfra (database) user that will be executing the scripts:
GRANT EXECUTE ON DBMS_LOCK TO <PREFIX>_SOAINFRA;
GRANT CREATE JOB TO <PREFIX>_SOAINFRA;
GRANT CREATE EXTERNAL JOB TO <PREFIX>_SOAINFRA;

    1. Exit SQL*Plus and go to the location of the Oracle purge script:
exit


$cd <SOA_HOME>/rcu/integration/soainfra/sql/soa_purge/

    1. Connect to the database as the soainfra user using SQL*Plus:
sqlplus <PREFIX>_SOAINFRA/<password>

@soa_purge_scripts.sql

Procedure created.
Function created.
Type created.
Type body created.
PL/SQL procedure successfully completed.
Package created.
Package body created.

    1. Exit SQL*Plus and create a directory where the log files (generated by the Oracle purge script) should be written to:
exit
$mkdir -p <LOG_LOCATION>/PurgeLogs

    1. Connect to the database with SQL*Plus as SYSDBA and declare the directory:
sqlplus / as sysdba

CREATE OR REPLACE DIRECTORY SOA_PURGE_DIR AS '<LOG_LOCATION>/PurgeLogs';

GRANT READ, WRITE ON DIRECTORY SOA_PURGE_DIR TO <PREFIX>_SOAINFRA;

All the database objects required for purging data using the Oracle purge script are now loaded into the SOAINFRA schema ready for use.

How to execute the Oracle SOA Suite 11g (PS6 11.1.1.7) purge script?


There are two options for running the purge script:
  • Looped
  • Parallel 

What is Looped purging (Oracle SOA Suite 11g (PS6 11.1.1.7) purge script)?

Looped purge is a single threaded PL/SQL script that will iterate through the SOAINFRA tables and delete instances matching the parameters specified.

What is Parallel purging (Oracle SOA Suite 11g (PS6 11.1.1.7) purge script)?
Parallel purge is essentially the same as the looped purge. It is meant to be more efficient as it uses the dbms_scheduler package to spawn multiple purge jobs all working on a distinct subset of data. There are 2 more parameters that can be specified in addition to the ones used by the looped purge. This is designed to purge large data volumes hosted on high-end database nodes with multiple CPUs and a good IO sub-system. A maintenance window should be used as it requires a lot of resources.


purge script parameters
type
mandatory optional
default
usage?
parallel or
both
description
min_creation_date
timestamp 
 M

both
Beginning creation date for the composite instances.
max_creation_date
timestamp
 M

both
Ending creation date for the composite instances.
batch_size
integer
 O
20000 
both
Batch size used to loop the purge, how many instances will be deleted in 1 loop.
The way the script is run is the following: it reads / and writes to a temporary table.
This is how many records will be used one cycle before going to the next cycle.
This is NOT how many records to delete in 1 purge operation.
max_runtime
integer
 O
 60
both
Expiration time at which the purge script will stop running. Specified in minutes
retention_period
timestamp
 O
null
both
Retention period is only used for BPEL instances.
The value for this parameter must be greater than or equal to max_creation_date.
Used as a further level of filtering. Specify a retention period if you want to retain the composite instances based on the modification date.
purge_partitioned_component
boolean
O
false 
both
Users can invoke the same purge to delete partitioned data.
composite_name
string
null 
both
The name of the SOA composite application.
You can purge the instances of a specific SOA composite application and leave the instances of other composites unpurged. This action enables you to purge certain flows more frequently than others due to high volume or retention period characteristics.
composite_revision
string 
null 
both
The revision number of the SOA composite application.
soa_partition_name
string 
null 
both
The partition in which the SOA composite application is included.
ignore_state
boolean 
false 
both
If this is set to true then all instances will be purged regardless of state.
DOP
integer 
 4
parallel only
Defines the number of purge jobs to run at the same time.
As a rule of thumb, the number of jobs should not exceed the number of CPUs on the node by more than one. For example, on a quad core / 4 thread CPU RDBMS box we will set it to 3.
max_count
integer 
O
1000000
parallel only
Defines the maximum number of rows to process (not the number of rows deleted). A temp table is created and then jobs are scheduled to purge based on the data.

Example 1: Executing the Oracle SOA Suite 11g (PS6 11.1.1.7) purge script to purge data for all composites
We are required to delete all composite instances which were created between 1st June 2010 and 30th June 2010. In addition, there is a requirement not to delete instances that have been modified after 30th June 2010. The script must finish running after an hour due to business hours resuming shortly afterwards.


min_creation_date      = 1st June 2010
max_creation_date     = 30 June 2010
retention_period         = 1st July 2010


The above will in effect delete all "composite instances" where the created time of the instance is between 1st June 2010 and 30 June 2010 and the modified date of the BPEL instances is less than 1st July 2010.


  1. looped execution of purge script

DECLARE


    max_creation_date     timestamp;
    min_creation_date     timestamp;
    batch_size            integer;
    max_runtime           integer;
    retention_period      timestamp;


 BEGIN
    min_creation_date  := to_timestamp('2010-06-01','YYYY-MM-DD');
    max_creation_date  := to_timestamp('2010-06-30','YYYY-MM-DD');
    max_runtime        := 60;
    retention_period   := to_timestamp('2010-07-01','YYYY-MM-DD');
    batch_size         := 10000;
  
    soa.delete_instances(
            min_creation_date  => min_creation_date,
            max_creation_date  => max_creation_date,
            batch_size         => batch_size,
            max_runtime        => max_runtime,
            retention_period   => retention_period);
 
END;
/


  1. Parallel execution of purge script

DECLARE


    max_creation_date           timestamp;
    min_creation_date           timestamp;
    batch_size                  integer;
    max_runtime                 integer;
    retention_period            timestamp;
    DOP                         integer;
    max_count                   integer;
    purge_partitioned_component boolean;


 BEGIN
    min_creation_date           := to_timestamp('2010-06-01','YYYY-MM-DD');
    max_creation_date           := to_timestamp('2010-06-30','YYYY-MM-DD');
    max_runtime                 := 60;
    retention_period            := to_timestamp('2010-07-01','YYYY-MM-DD');
    batch_size                  := 10000;
    DOP                         := 3;
    max_count                   := 1000000;
    purge_partitioned_component := false);


  
    soa.delete_instances_in_parallel (
        min_creation_date           => min_creation_date,
        max_creation_date           => max_creation_date,
        batch_size                  => batch_size,
        max_runtime                 => max_runtime,
        retention_period            => retention_period,
        DOP                         => DOP,
        max_count                   => max_count,
        purge_partitioned_component => purge_partitioned_component);
              
END;
/

Example 2: Executing the Oracle SOA Suite 11g (PS6 11.1.1.7) purge script to purge data from a specific composite

Same as Example 1 Scenario but with an additional requirement of only purging data from the composite named OrderBookingComposite . No other composite data should be purged.


Composite details can be gathered by querying the COMPOSITE_INSTANCE table within the SOAINFRA schema.
The column named COMPOSITE_DN (distinguished name) holds the details required by the purge script:

Format: <soa_partition name>/<composite name>!<composite_revision>
Example: default/OrderBookingComposite!1.0


  1. looped execution of purge script for a specific composite
DECLARE
min_creation_date           timestamp;
max_creation_date           timestamp;
batch_size                  number;
max_runtime                 number;
retention_period            timestamp;
purge_partitioned_component boolean;
composite_name              varchar2(200);
composite_revision          varchar2(200);
soa_partition_name          varchar2(200);


 BEGIN
    min_creation_date           := to_timestamp('2010-06-01','YYYY-MM-DD');
    max_creation_date           := to_timestamp('2010-06-30','YYYY-MM-DD');
    max_runtime                 := 60;
    retention_period            := to_timestamp('2010-07-01','YYYY-MM-DD');
    batch_size                  := 10000;
    purge_partitioned_component := true;
    composite_name              := 'OrderBookingComposite';
    composite_revision          := '1.0';
    soa_partition_name          := 'default';


  
    soa.delete_instances(
        min_creation_date           => min_creation_date,
        max_creation_date           => max_creation_date,
        batch_size                  => batch_size,
        max_runtime                 => max_runtime,
        retention_period            => retention_period,
        purge_partitioned_component => purge_partitioned_component,
        composite_name              => composite_name,
        composite_revision          => composite_revision,
        soa_partition_name          => soa_partition_name);
 
END;
/

  1. looped execution of purge script for a specific composite
DECLARE
min_creation_date           timestamp;
max_creation_date           timestamp;
batch_size                  number;
max_runtime                 number;
retention_period            timestamp;
DOP                         integer;
max_count                   integer;
purge_partitioned_component boolean;
composite_name              varchar2(200);
composite_revision          varchar2(200);
soa_partition_name          varchar2(200);


 BEGIN
    min_creation_date           := to_timestamp('2010-06-01','YYYY-MM-DD');
    max_creation_date           := to_timestamp('2010-06-30','YYYY-MM-DD');
    max_runtime                 := 60;
    retention_period            := to_timestamp('2010-07-01','YYYY-MM-DD');
    batch_size                  := 10000;
    DOP                         := 3
    max_count                   := 1000000;
    purge_partitioned_component := true;
    composite_name              := 'OrderBookingComposite';
    composite_revision          := '1.0';
    soa_partition_name          := 'default';


  
    soa.delete_instances(
        min_creation_date           => min_creation_date,
        max_creation_date           => max_creation_date,
        batch_size                  => batch_size,
        max_runtime                 => max_runtime,
        retention_period            => retention_period,
        DOP                         => DOP,
        max_count                   => max_count,
        purge_partitioned_component => purge_partitioned_component,
        composite_name              => composite_name,
        soa_partition_name          => soa_partition_name);
 
END;
/

Conclusion

This blog has provided a basic understanding of the purge script contained within Oracle SOA Suite 11g (PS6 11.1.1.7).

A long term purging strategy needs to be implemented and in order to do so, a good understanding of the workings of the purge script is required along with an awareness of the issues related to the script.

Therefore, leading on from part 1 there will be few more blogs that will cover other the following:


  • Part 2: How does the Oracle SOA Suite 11g (PS6 11.1.1.7) purge script work?


  • Part 3: How to establish a long term purge strategy for Oracle SOA Suite 11g (PS6 11.1.1.7)

    By 
    Irfan Suleman