BPM – Page 2 – Efficient technology for the enterprise

Web Service Security on BiPRO Day

At the upcoming BiPRO day on 11.June 2013
I am going to give a presentation introducing the most important standards in the area of web service security. The aim is to show the purpose of the standards and how they work together to create secure and interoperable message based web service solutions.

About BiPRO day:
“Einmal im Jahr treffen sich die Mitglieder des BiPRO e.V. sowie Interessierte aus der Versicherungs- und Finanzdienstleistungsbranche zum BiPRO-Tag. Dabei stehen aktuelle und zukünftige Themenfelder der Prozessoptimierung allgemein und des BiPRO e.V. im Speziellen im Vordergrund. Dazu zählen Vorträge und Präsentationen aus laufenden und bevorstehenden Projekten, die Vorstellung neuer Normen sowie Berichte über Norm-Implementierungen bei den Mitgliedsunternehmen des Vereins.”

Integrated Process Management with Open Source

If you ever tried to create an execution environment to automate business- or integration processes based on Open Source products, you know that this is not an easy task. Although Open Source products like Activiti or Apache Camel are of high quality, they do not run with production grade quality out-of-the-box. For serious usage scenarios typically a lot of work is required to integrate those products into a sound platform. This fact hinders companies to use those great products and turn to closed source alternatives from Oracle, Appian or Inubit, just to name a few.

Now there is an interesting alternative called oparo. oparo is an integrated process automation platform based on rock solid Open Source products. oparo is not limited to BPMN processes only. It rather focuses on the entire process spanning business, workflow, mediation and integration.

The platform does all the plumbing required to turn single products such as Activiti, Apache Camel, Apache ActiveMQ, Lucene/Solr, etc. into a platform that can be used out-of-the box. Even better, oparo is entirely ASF2.0 licensed (today and tomorrow) which offers broad usage options and does not involve any hidden costs for enterprise features.
oparo shields the process engineer (the guy who analyses and automates processes) as much as possible from low level technical tasks such as connecting and transforming Camel and Activiti message payloads. It offers a unified development approach for the process engineer to focus on business functionality instead of technical plumbing. Moreover it comprises additional valuable services such as process flow tracking, humantask integration or a registry. Due to oparos service binding approach, those services can be easily integrated in existing IT landscapes using almost any technology (e.g. .NET, JEE, HTML5/JS/CSS). The runtime is scalable (in terms of technology and licenses), the set up is automated and the whole platform is based on proven standards.

If that sounds promising, you can give it a try. You can find more information and a downloadable jumpstart distribution at oparo – the efficient process platform (German only)

Business Process Evolution and Versioning

(Automated) business processes evolve over time! And they usually evolve faster than IT systems do.
So how can business process changes be delivered to the users quickly?

Let’s look at an example:
Assume we have a process for vacation planning for the staff of a large company. Initially the process was automated based on the knowledge of the human resource department. After 2 months new insights require a process change. The process should be optimized to speed up the decison whether vacation is granted or not. The process has evolved and the changes have to be put in place as soon as possible. This is a common situation and actually one of the promises of business process management is: Deliver business value fast.

Sounds simple, but how can we deliver the changed process?

There are serveral options to put the changed process in place:

Option 1: Parallel
The changed process coexists with the initial one for a period of time. Existing process instances must continue with the inital process definition.

Example: Users of the process are gradually trained to use the changed process. Some departments can still use the initial process, some use the new one. The process is triggered by IT systems as well. Those systems should have a smooth upgrade path.

Action: Create a new version of the process and deploy it in parallel to the one already in place.

|--- Startable V1 -------->
|--- Instances V1 -------->
                 |--- Startable V2 --------->
                 |--- Instances V2  -------->

Option 2: Merge
The changed process replaces the initial one. Existing process instances must continue using the changed process definition.

Example: Law changes render invalid the initial process. As of now all processes, including already running instances, must run with the latest process definition.

Action: Create a new version of the process and migrate existing instances to the new process definition.

|--- Startable V1 ------|--- Startable V2 --------->
|--- Instances V1 ------|--- Instances V1 + V2 ---->

Option 3: Phase Out
The changed process replaces the initial one. Existing process instances must continue with the inital process definition.

Example: Process analysis caused the process to be optimized, so that it can be executed in less time. All users should immediately use the changed process.
To keep effort low, already running process instances should continue running with the inital process definition.

Action: Create a new version of the process and deploy it in addition to the one already in place. Prevent the initial process version to be started by disabling the start events.

|--- Startable V1 --------|
|--- Instances V1 --------------------|
                          |--- Startable V2 --------->
                          |--- Instances V2  -------->

Be aware of endpoints:
If process versions are provided in parallel like in scenario 1 and 3 and connected to technical endpoints, for instance filedrops or web services, those endpoints might collide. Changing the structure of an endpoint, for instance the message payload, might cause incompatibility as well. In those cases (which are likely to happen) the endpoints must be versioned. Alternatively a dispatching mechanism can be used to route messages to the appropriate process version.

As you can see versioning is am important concept for process evolution. Which strategy to use depends on the process and the particular business requirements. The options introduced in this blog post might help to take the right decision. Make sure your process platform supports the options you need.

Agile Process Management with Open Source

Are you interested to know how to combine process management, agility and Open Source software? Then the roadshow Agile Process Management with Open Source is for you. It is going to take place in several German cities during autumn 2012. I am going to present ways to achive efficiency in the area of process automation using proven Open Source technologies paired with agile approaches. In times where CIOs have to think twice before they spend IT budget, undoubtely an interesting topic to talk about. It have some interesting ideas to share and hope for inspiring discussions.

Combining Groovy and XSLT for Data Transformation

In the blog post Beautiful Transformations with Groovy I described how easy it is to create data transformations with Groovy. But sometimes organisations invested massively in XSLT transformation and want to reuse their existing XSLT templates. Read on for an an example that shows how to do that.

Assume we want to transform the following XML file to HTML:


  
    Germany
    Fast and nice
  
  
    Spain
    Large and handy
  
  
    Italy
    Small and cheap

Lets further assume the result should look like this:

Does it make sense? I don’t know, but that’s not really important. 😉

We have the following XSLT template to perform the transformation:

All you need is a Groovy script like the one below to transform the xml file to html using the given xslt.

// Load xslt
def xslt= new File("template.xsl").getText()

// Create transformer
def transformer = TransformerFactory.newInstance().newTransformer(new StreamSource(new StringReader(xslt)))

// Load xml
def xml= new File("cars.xml").getText()

// Set output file
def html = new FileOutputStream("output.html")

// Perform transformation
transformer.transform(new StreamSource(new StringReader(xml)), new StreamResult(html))

This is self-explanatory, isn’t it?
As XSLT is somewhat limited when it comes to more complex transformations, it can be extended by custom processors which can we implemented in Java or Groovy. A custom processor in Groovy can be implemented like this:

public class AgeProcessor{
    public def process(ExpressionContext context,int n){
        return "Age: " + (2012 - n) + " years";
    }
}

The processor is hooked up to the XSLT using the expressions in line 3 and 28 of the above XSLT file.
The examples above show how to reuse existing XSLT in Groovy. Are you interested to see the same same transformation in pure Groovy? (sorry, I could not resist ;-))
Here is the code:

// Load xml
def cars = new XmlSlurper().parse(new File("cars.xml"))

// Set output file
def writer = new FileWriter("output.html")

// Perform transformation
def builder = new MarkupBuilder (writer);
builder.html(xmlns:"http://www.w3.org/1999/xhtml") {
    head {
        title "Cars collection"
    }
    body {
        h1("Cars")
        ul(){
            cars.car.each{car ->
                li(car.@name.toString() + "," + car.country + "," + car.description + ", Age: " + (2012 - car.@year.toInteger()) + " years")
            }
        }
    }
 }

It is shorter and self-contained. It is also more intuitive and therefore easier to maintain. But if you have the requirement to support XSLT in Groovy you now know how to do that.

Agility through Business Process Automation?

Sometimes business process automation (BPA) is described as the silver bullet to improve agility and time to market. Especially large vendors spend huge amounts of marketing budget to promote their BPM tool suites, “360 Degree”- and “Zero Code”-approaches.

But why does BPM increasy agility? Is it really easier to adapt processes to business changes if a process has been automated using a BPM suite?

Sometimes yes, in the narrow range of allowed options. Often no, cause IT-coded processes are not as flexible as people in an organisation. But that does not mean that business process automation is a bad idea at all. There are areas in which process automation makes perfect sense.

Especially processes for which the following factors apply:

clearly structured and predictable
repetetive
frequently executed

Interestingly, often agility does not come from automated processes itself, but rather from the people who have their hands free for other more sophisticated ad hoc processes. We have experienced that in a large project for an international organisation from the public sector. Provided people have the right skills, BPA can help turning people from “routine workers” to “knowledge workers” (see It is All Taylor’s Fault). BPA allowed them to automate their repetetive tasks. It was a great improvement and productivity gain for the people and the organisation. The key was to give them a tooling that they were able to control, even without much help from IT guys.

Knowledge workers do not need their processes automated. They need other tools mostly to get the right structured information at the right time. IT can help in this regard, but not via BPA. I would call this Business Process Facilitation (BPF) rather than automation.

BPF means giving the people tools to do their job in a efficient manner without imposing predefined processes on them. In other words, it leaves the process and decision power with the people not the machines. User centered dashboards, search engines and adaptive case management tools are examples for BPF. We have experienced this in another project in which we evaluated the value of process automation using a BPM-Suite. In this highly dynamic environment the decision was to not implement BPM as it would have hindered agilty. Instead we implemented BPF to support the knowledge workers. The system mainly focused on efficient data management and decision making.

All in all it is not black and white, not Taylorism against knowledge work. Success comes from a combination of both. The key is focusing on things that are beneficial for the people and organisation. Sometimes it is automation, sometimes not. Process automation is cleary no silver bullet, but if applied wisely and with the right focus it can help organisations to improve efficiency.

BPMN 2.0 Model Interchange with Enterprise Architect, Signavio Modeler and Activiti Designer

I have been using the Enterprise Architect modeling tool for many projects. I like the tool especially due to its quick modeling capabilities and productivity. Therefore is ideal for workshops with participants from IT and business. Performing live modeling together with the domain experts is a very efficient approach to gather accurate information related to processes and IT systems.

So I am very pleased to see that Enterprise Architect 9 now supports the creation of BPMN 2.0 models as well. And even better, models can be easily exported in form of BPMN 2.0 interchange format via Project – Model Publisher in the Enterprise Architect menubar (by the way, import is not possible. :-().
Anyway, I gave it a try and exported a simple model from EA:

To prove that the interchange works, I imported the model in my favorite BPMN tool the Signavio Modeler. This is the result:

That’s not too bad, is it? But it seems that some model details are missing. The dataInputAssociation and dataOutputAssociation, the scriptFormat and the groovy script attached to the ScriptTask were not imported.
It seems that (at least for those two tools) BPMN2.0 model interchange is not yet sufficiently possible.

Update:
I imported the bpmn file into the Activiti Designer Eclipse Plugin version 5.7.1 as you can see below:

The Groovy script was successfully imported, but the dataInputAssociation and dataOutputAssociation was not recognized and therefore removed from the process definition. The original file is solely used for the import. After importing an *.activiti file is created and used as THE model. On saving the model, a fresh BPMN XML file is generated. That is, you loose control over the file at the XML level. I personally don’t like this, as I think executable BPMN files should be treated as source code and need to be fully controlled by developers. A designer tool can help, as long as it does not destroy the orginal source files.

Summary so far:
Sparx Enterprise Architect – Comprehensive modeling, good export, no import
Signavio Modeler – Comprehensive modeling, good export, incomplete import
Activiti Designer – Limited modeling, good export, incomplete import

None of the tools has sound reverse engineering or roundtrip capabilities yet.
For descriptive (level 1) and analytical (level 2) modeling Sparx Enterprise Architect and Signavio Modeler are great.
For executable modeling (level 3) a plain XML editor seems to be the only reasonable option for the time being.

Beautiful Transformations with Groovy

Data transformations are the daily business in ETL and ESB scenarios. If you have a service- or business process boundary it is very likely that data has to be transformed between different representation.
Typical integration scenarios have to deal with a huge amount of different formats (flat file, xml, csv, json, even binary). To make things even worse they tend to change from time to time. Therefore it is important to have tools that are flexible and easy to understand. I found Groovy to be very handy for the transformation job. Let’s have a look at a typical example. Assume we have a flat file in the following format:

Id Product     Amount
 1 Cheese      15
 2 Sausage     2
 3 Bread       3
 4 Butter      4

What we want is an xml representation like this:


  
    
    Cheese
    15
  
  
    
    Sausage
    2
  
  
    
    Bread
    3
  
  
    
    Butter
    4

The groovy script below does the job. The great thing is not only that is works but also that its easy to develop and maintain. A developer can see the source and target structure at the same time.

def reader = new StringReader(flatfile);
def writer = new StringWriter();
def builder = new groovy.xml.MarkupBuilder(writer)

builder.order(){
    reader.eachLine() {
        def line = it
        item(){  
            id(id:line.getAt(0..1).trim())
            name(line.getAt(3..14).trim())
            amount(line.substring(15).trim())
        }
    }
}

println writer.toString();

Thanks to closures an the MarkupBuilder class the transformation can be written in a fluent and intuitive way. Typical other transformations are as simple as that, making Groovy a first choice for data transformations.

The Future of ServiceMix

In the ServiceMix forum there is an interesting debate going on about the Future of ServiceMix. I’ve collected some quotes from the thread. I know it is sometimes dangerous to cite people without giving the full context, but I think it helps to get a rough idea about the direction in which ServiceMix is heading. If you are interested in more details please read to original post.

Quotes:

“The idea is really to make ServiceMix the best possible container for deploying Camel based integration.”

“This also means that there’s really not much to reuse from smx4, so smx5 would have its own new and dedicated svn area.”

“Dropping JBI is already a great step in that direction.”

“Additional things that could be pushed inside ServiceMix 5 would be a Tomcat based container deployment option (for those that don’t need OSGi), a new manual similar to what we have in Karaf (maybe reusing parts of it).”

“For advanced use cases, we should definitely guide users towards Karaf, but for a set of basic things, just using Tomcat and simple WARs may be the simplest thing that works for them!”

“To build the added value I’ve been talking about, we don’t really need the NMR imho. I’m convinced we can do it directly on top of camel without the need for another layer, mostly using Camel interceptors. That way, OSGi is not really needed anymore, though it’s still the best way to deploy Camel routes using Karaf and would remain the main and standlone ServiceMix distribution.”

“Providing a way to easily deploy Activiti would definitely be a good thing. I don’t think it should part of the default distribution, as I’d like to keep it as light as possible though.”

Here is my personal summary:

ServiceMix 5 will primarily focus on Camel.
JBI is finally dead.
Karaf and OSGi will represent the primary runtime.
Tomcat will act as an alternative container.
Activiti will not be integrated.

Consequences:

If you develop for ServiceMix 4 today, focus on OSGi and Camel instead of JBI and ServixMix features since those are the stable parts of ServiceMix.

If you have ServiceMix 3 and JBI solutions in place think about migration towards Karaf and Camel. If you plan to integrate Activiti, make sure the required functionality is not already covered by Camel to avoid unnecessary complexity (see blog post Combining Activiti and Camel).

Combining Activiti and Camel?

It seems that that ServiceMix/Camel and Activiti are growing together as you can read in the blog post Supersize Activiti with Mule ESB and Apache Camel and Deploy Activiti as OSGi Bundles

Activiti has its roots in business process management, whereas Camel comes from the area of enterprise integration. Both allow to describe and automate processes. Activiti uses BPMN, a graphical modeling language with an XML representation for persistence and execution. Camel uses a textual DSL which can also be expressed in XML. Grahpical modeling is also possible with the Fuse IDE.

BPMN example:

The BPMN XML representation is not intended to be written directly and can be hard to read.

Camel example:

from("file://start")
.choice()
  .when(header("goldCustomer").isEqualTo("true"))
    .to("bean:processor?method=grantDiscount")
    .to("direct:out")
  .otherwise()
    .to("direct:out");

from("direct:out")
.to("file://end");

Basically both examples are identical. The Camel example is more explicit and shows more technical details which are hidden in the BPMN example.

Especially non technical users tend to think that a graphical process notation is easier to understand because it looks better. It might be true for simple processes like the one above. But for more complex processes textual DSLs are often easier to write and understand.

Activiti and Camel overlap in many areas. Does it make sense to use them side by side anyway? I think yes, because neither Activiti nor Camel is complete in terms of features required by a fully featured SOA/BPM runtime. Actually they complement each other very well in some areas.

Unique features added by Camel:

Connectivity / Adaptors to connect to external systems using a variety of different protocols
Predefined EIP patterns for message routing

Unique features added by Activiti:

Support for long running stateful processes
Human workflow integration

In order to avoid unnecessary complexity it helps to recall the difference between mediation- and business processes which I descibed in my blog post from 2006.

For mediation processes Camel is appropriate in most cases, even long running processes can be executed as long as they are stateless. But if you need long running stateful processes and human interaction Activiti is an interesting option.

Both Camel and Activiti processes can be modeled using prescriptive and analytical BPMN. The software designer has the responsibility to decide which technology is appropriate for implementing the process. If this is not done right you will end up with unnecessary complexity, limited agility and maintainability.

It would be ideal to have an Open Source SOA/BPM Platform that enables end to end process automation without technology boundaries. As long as this is not available, combining Activiti and Camel is an interesting option. Proper solution design and differentiation of process types is key to create sustainable solutions.