Integrated Process Management with Open Source

If you ever tried to create an execution environment to automate business- or integration processes based on Open Source products, you know that this is not an easy task. Although Open Source products like Activiti or Apache Camel are of high quality, they do not run with production grade quality out-of-the-box. For serious usage scenarios typically a lot of work is required to integrate those products into a sound platform. This fact hinders companies to use those great products and turn to closed source alternatives from Oracle, Appian or Inubit, just to name a few.

Now there is an interesting alternative called oparo. oparo is an integrated process automation platform based on rock solid Open Source products. oparo is not limited to BPMN processes only. It rather focuses on the entire process spanning business, workflow, mediation and integration.

The platform does all the plumbing required to turn single products such as Activiti, Apache Camel, Apache ActiveMQ, Lucene/Solr, etc. into a platform that can be used out-of-the box. Even better, oparo is entirely ASF2.0 licensed (today and tomorrow) which offers broad usage options and does not involve any hidden costs for enterprise features.
oparo shields the process engineer (the guy who analyses and automates processes) as much as possible from low level technical tasks such as connecting and transforming Camel and Activiti message payloads. It offers a unified development approach for the process engineer to focus on business functionality instead of technical plumbing. Moreover it comprises additional valuable services such as process flow tracking, humantask integration or a registry. Due to oparos service binding approach, those services can be easily integrated in existing IT landscapes using almost any technology (e.g. .NET, JEE, HTML5/JS/CSS). The runtime is scalable (in terms of technology and licenses), the set up is automated and the whole platform is based on proven standards.

If that sounds promising, you can give it a try. You can find more information and a downloadable jumpstart distribution at oparo – the efficient process platform (German only)

Business Process Evolution and Versioning

(Automated) business processes evolve over time! And they usually evolve faster than IT systems do.
So how can business process changes be delivered to the users quickly?

Let’s look at an example:
Assume we have a process for vacation planning for the staff of a large company. Initially the process was automated based on the knowledge of the human resource department. After 2 months new insights require a process change. The process should be optimized to speed up the decison whether vacation is granted or not. The process has evolved and the changes have to be put in place as soon as possible. This is a common situation and actually one of the promises of business process management is: Deliver business value fast.

Sounds simple, but how can we deliver the changed process?

There are serveral options to put the changed process in place:

Option 1: Parallel
The changed process coexists with the initial one for a period of time. Existing process instances must continue with the inital process definition.

Example: Users of the process are gradually trained to use the changed process. Some departments can still use the initial process, some use the new one. The process is triggered by IT systems as well. Those systems should have a smooth upgrade path.

Action: Create a new version of the process and deploy it in parallel to the one already in place.

|--- Startable V1 -------->
|--- Instances V1 -------->
                 |--- Startable V2 --------->
                 |--- Instances V2  -------->

Option 2: Merge
The changed process replaces the initial one. Existing process instances must continue using the changed process definition.

Example: Law changes render invalid the initial process. As of now all processes, including already running instances, must run with the latest process definition.

Action: Create a new version of the process and migrate existing instances to the new process definition.

|--- Startable V1 ------|--- Startable V2 --------->
|--- Instances V1 ------|--- Instances V1 + V2 ---->

Option 3: Phase Out
The changed process replaces the initial one. Existing process instances must continue with the inital process definition.

Example: Process analysis caused the process to be optimized, so that it can be executed in less time. All users should immediately use the changed process.
To keep effort low, already running process instances should continue running with the inital process definition.

Action: Create a new version of the process and deploy it in addition to the one already in place. Prevent the initial process version to be started by disabling the start events.

|--- Startable V1 --------|
|--- Instances V1 --------------------|
                          |--- Startable V2 --------->
                          |--- Instances V2  -------->

Be aware of endpoints:
If process versions are provided in parallel like in scenario 1 and 3 and connected to technical endpoints, for instance filedrops or web services, those endpoints might collide. Changing the structure of an endpoint, for instance the message payload, might cause incompatibility as well. In those cases (which are likely to happen) the endpoints must be versioned. Alternatively a dispatching mechanism can be used to route messages to the appropriate process version.

As you can see versioning is am important concept for process evolution. Which strategy to use depends on the process and the particular business requirements. The options introduced in this blog post might help to take the right decision. Make sure your process platform supports the options you need.

Scrum and Silverlight in Reinsurance

Today I would like to share a success story of a project I accompanied as Scrum Coach and Solution Architect from analysis to production.

Main success factors were:

  • Scrum (Agile Development)
  • Cross functional Team
  • Service-oriented Design (SOA)
  • Silverlight RIA

You can read more in the case study Modernization of business partner management.

Core Values of Great Software

Over the last decade I have seen many software solutions. Some excellent and some really bad ones. But most solutions were somewhere in the middle. They just did what they were supposed to do.

But is this really good enough? I don’t think so. Especially today where time to market is critical and the budgets are limited, functioning software is the bare minimum. On top of that other core values are getting more and more important.

  • Functionality – This is obvious.  The software must do what people expect it to do in a reliable manner.
  • Evolvability – This is also known as sustainability. A value that is important in many areas of our daily life. And also in software development. It means that we don’t want to reinvent the wheel over and over again. We don’t want to produce waste. Software must be flexible so that it can be reassembled even if technologies change. Investements have to be saved. Evolvability can be primarily achieved through proper software design. Especially service-oriented approaches are promising here.
  • Production efficiency – This is the most underestimated value. It means that just work is not enough, rather it must be quick and easy to produce. For a long time IT is has a reputation of being complicated and expensive. As a reaction to that companies are outsourcing to cheaper countries or applying more flexibile development processes auch as Scrum to improve efficiency. Especially the latter has proven to be successful. Choosing the right software is also a good way to come closer to that goal. For example frameworks should not only assessed by its functionality (often done by developers) but also on the ease of use and developing performance.

Although those values are important for every software project, they are especially important in todays SOA/BPM undertakings as they promise to increase agility and time to market. This can only be achieved with software solutions that are flexible and easy to use. Ideally even for business people without a developer background.

Schema first contract design

One of the most important artifacts in Service-oriented Architecture (SOA) is the service contract. The contract defines how service producers and consumers interact with each other. Most people agree that a contract first approach is the preferred way of creating service contracts. WSDL based service contracts are very common. In many projects they are generated by the respective frameworks, such as Apache CXF or Microsoft WCF. This is dangerous, as the developer looses control over the contract. This can result in missing interoperability and limited reusabilty expecially in cross platform scenarios.
From what I see, most developers know how to work with their favourite IDE, but only a few know how to create a proper service contract from scratch. I think that is caused by the tool vendors, which in most cases have better support for code based approaches.
In this post I would like to show how to create a portable contract from scratch and share some best practices.

The domain model

The first thing to think about is the domain or entity model that the service should expose. The model should be modeled in XML Schema. XML Schema is standard, portable and supported on all platforms. In fact it is a good idea to use XML Schema as the single source of model information.

The following listing defines the entity Product:
 



  
    
      
        
          Name of the product
        
      
      
        
          Quantity on stock
        
      
    
  

The entity Product has two attributes name and quantity. Please note that the xml elements (line 10 and 15) are abbreviated. This helps to reduce the message size on the wire, expecially important in WANs. The downside is obviously reduced readability. That is not really a problem as the contract should be documented anyway using the standard XML Schema annotation elements as shown in line 11 to 13.
This is the result:

Wouldn’t it be nice to have a common base class in order to create a more generic service contract?
Luckily XML Schema allows that. Let’s create another schema BaseTypes.xsd in which we define the base types.



  
    
      
      
    
  

  
    
      
      
      
      
    
  

  
    
      
        
          Unique key
        
      
      
        
          Status of the entity
        
      
    
  

This base type Entity (line 24) defines common attributes for all entities. In XML Schema one can define enumerations as well (line 8 and 15). The enumeration EntityStatus defines the state of the entity. This is important in scenarios in which the entities are detached from the database. By marking the entities, it is easy to implement change tracking on the consumer side and just return the changed or deleted parts to the server. This saves network and processing time. EntityType is used later in our service interface to request different types of entities.
We can now import the base types into Types.xsd and change the Product type:





  
    
      
        
          
            
              Name of the product
            
          
          
            
              Quantity on stock
            
          
        
      
    
  

You can see the import in line 9. In line 12 and 13 the base type is added to the Product type.

This is the result:
Entity with base class

Now we have our model completely defined in XML Schema. By using imports and type inheritance one can create very modular entity models. Those models can be reused without any change by interface definitions such as WSDL and WADL for SOAP or REST services. Moreover the code models (Java, C#, etc.) can be fully generated from the schema. Every good framework has a generator tool on board. For instance Apache Axis/wsdl2java, Apache CXF/wsd2java, Silverlight/slsvcutil, .NET/svcutil, etc. The schema can be stored in a service repository in order to increase reusability amongst the organisation.

The messages
Every service operation receives a message and returns a message to the sender. Those messages should also be defined using XML Schema. The following snippet shows the message definitions. The fault messages are ommitted for the sake of simplification.


    
      
    
  

  
    
      
    
  

    
        
            This represents a request to read business entities
        
        
            
                
                    
                        Type of the entities to read
                    
                
                
                    
                        Unique keys of the entities to read
                    
                
            
        
    

    
        
            This represents the read response
        
        
            
                
                    
                        List of entities
                    
                
            
        
    

Those are our messages:

ReadEntitiesRequest and ReadEntitiesResponse are bulk operations that allow to read multiple entities with one call. As you can see ReadEntitiesResponse does not return Products but Entities. Do you remember? This is our base class type for all entities. By using the base class, the message can return arbitrary entity types in the same message. This is an example for a strongly typed but yet extensible interface.

The schemas can be reused in WSDL files:



  
  
  
  
      
  
  
    
  
  
  
    
      
      
    
  

  
        
    
      
      
        
      
      
        
      
    
  
  
    
      
    
  

The type information and message definitions are completely externalized to individual schemas. As a result the WSDL is small. The imported types (line 4) are used to define the WSDL messages (line 7 and 10). A WADL file would look like this:



  
    
    
  

  
    
      
        

            
              
                This is the product id
              
            
          
        
        
          
        
      
    
  

The types are imported (line 8 and 9) and used to define the return message (25).
By using the respective tool, such as wsdl2java or svcutil, we can completely generate the model and messages for the client and server on JEE, .NET and other platforms.

Summary
As you can see it is not too difficult to create schema based service contracts in a platform agnostic way. If you do, you will be rewarded with a great level of control, portability and reusability. Powerful features of XML Schema, such as inheritance, allow to create robust and extensible service contracts, which are the basis for interoperability. And if you want, you can even use an XML editor. 😉

Forms Over Data vs. Domain Driven Design

A common challenge in Service-oriented architecture (SOA) is the design of service contracts. Beside many variations we see two basic types of contracts.

Forms over data (FOD) is an approach in which the forms in the user interface directly reflects the data structures in a database. The service-layer, if present, primarily offers create, read, update and delete (CRUD) operations for the individual database entities.

FOD is fostered by technologies such as Microsoft Access, Oracle Forms, .NET RIA Services. In JEE it is recognized as an anti pattern addressed by design (anti-)patterns such as Face Off or Session Facade.
FOD is tempting because large parts of the service-layer can be generated from an existing data model. By using the right tools, developing an FOD-application can be quick and easy.

Domain driven design (DDD) is an approach that utilizes a service-layer whose data objects are inspired by the business domain. The database structures are hidden behind a service-layer and not known by the service consumer, for instance the user interface. The domain model is usually modeled using XML Schema. DDD is typically used in SOA environments.

DDD-Pros

  1. Portable domain model – If modeled in XML-Schema the domain object can be generated on JEE, .NET and other platforms.
  2. Business alignment – A business oriented domain model can be better understood by business users, thus increasing agility.
  3. Higher degree of reusability – Services representing business concepts are more likely to be reused, for instance in BPM environments.
  4. Decoupling of user interface and database – Data structures can be changed (up to a certain degree, of course) without affecting the clients. Changes can be handled in the service layer.
  5. Maintenance – Proper domain driven services are designed to last. Paired with a versioning approach service maintenance can be simplified.

FOD-Pros

  1. Development effort – FOD is often supported out-of-the-box in many products, so that no using it would require more knowledge and development effort.

Although there are always areas in which one or the other approach makes more sense, you can clearly see my preference. I think forms over data can be particularly useful for small applications and prototypes. Especially with existing database models. For large enterprise applications one should prefer a domain driven service design in combination with a sound SOA approach.

Increase Efficiency using Groovy, DSL and SOA

Recently SpringSource published a case study of a project which I supported as system architect and lead developer from concept to production.
The case study shows how organisations can greatly increase efficiency by the use of modern technology approaches such as Groovy, SOA and Domain Specific Languages (DSL).
You can read the case study here.

Rules, Processes or Services?

Together Services, Business Processes (BPM) and Business Rules (BRM) can be a powerful combination to keep the SOA promise of creating agile business applications.
I often hear questions about how those three can be combined in a reasonable way.
In their article Implementation of business rules and business processes in SOA Boris Lublinsky and Didier Le Tien give some practical advice.