Scrum and Silverlight in Reinsurance

Today I would like to share a success story of a project I accompanied as Scrum Coach and Solution Architect from analysis to production.

Main success factors were:

  • Scrum (Agile Development)
  • Cross functional Team
  • Service-oriented Design (SOA)
  • Silverlight RIA

You can read more in the case study Modernization of business partner management.

Lightweight Collaboration Tools

When you work with distributed teams it is important to have lightweight tools for efficient communication. Here are some free tools that I would recommend:

Surveymonkey – Online polls
DFN Scheduler – Scheduling
Skype – Video and voice conferencing
Chatzy – Private web based chatrooms (port 80 only, mobile access)
Trello – Scrum and Kanban boards (highly interactive)
Kunagi – Scrum boards (includes tools like planning poker)
Teamviewer – Online meetings and remote control (free for non commercial use only)

Impediment Number One to Agile Adoption

Scrum is in many respects very different from traditional project management approaches, especially waterfall models. It requires a different mindset in which it is ok to say:

“I don’t know yet exactly how long the project is going take, but give me some time to get to know the requirements and the team. After some Sprints we will give you a solid estimation based on empirical knowledge. Trust us, we do the best we can to deliver quality on time”.

For many especially “classic-minded” project managers such a statement is unimaginable. They simply can’t understand the culture of Scrum as it is very different from what they are used to.
After applying Scrum in several projects over the last years and after giving many Scrum workshops I think that the only way of learning Scrum is by doing it, ideally accompanied by a skilled coach. Books and certifications help, but they do not create deep understanding.
And here begins the dilemma. Especially managers in larger organisations never work on projects, they manage. Therefore it is hard for them to leave their classic mindset. This leads to non-supportive behavior which often blocks the way to agile adoption in large enterprises. In a recent interview Scrum in larger organisations Jeff Sutherland describes the challenges to change the management mindset.

He says:
“… major challenges you will have to deal with when you implement Scrum in a large organization is to change the mindset in the organization in general and on management-level in particular …”

For the reasons given above this nut is hard to crack. To me it seems that this is impediment number one on the way to agile adoption in larger enterprises.

RIA goes to Hollywood

Most RIA technologies today use an asynchronous model to communicate with the server, primarily to keep the UI responsive no matter what the server is doing. This principle is also known as the Hollywood principle. Don’t call us, we call you means that instead of continuously polling, the server calls back when the operation finished. In order to be notified when the server finished its job, callbacks are used.

For instance in Silverlight this leads to code like this

void ClickLoad()
{
    LoadCustomer(1,
    (result) =>
    {
        // Process result
    },
    (error) =>
    {
        // Process error
    });
}

void LoadCustomer(int id, Action<Customer> success, Action<Exception> error)
{
    LoadFromServer("select * from customers where id = " + id,
    (result) =>
    {
        success(result[0] as Customer);
    },
    (exp) =>
    {
         error(exp);
    });
}

void LoadFromServer(string query, Action<List<object>> success, Action<Exception> error)
{
    server.LoadCompleted += (s, e) =>
    {
        if (e.Error != null)
            error(e.Error);
        else
            success(e.Data);
    };

    server.LoadAsync(query);
}

Not very nice, is it? In jquery we find a similar pattern.

$.ajax({
  url: 'ajax/load.html',
  data: "query=1"
  success: function(result) {
    // Process result
  }
  error: function(XMLHttpRequest, textStatus, errorThrown) {
    // Process error
  }
});

Usually an application consists of several layers. In that case the callbacks have to be routed back to the original caller, leading to code that is neither easy to read nor easy to maintain.

From what is already visible, it is very likely that Silverlight and WinRT are going to have great overlap. The combination of XAML, C# and WinRT plus the tooling will be very familiar to all Silverlight users.
That means, it is likely that the asynchronous programming model will be the predominant model for client/server communication for Windows Metro style apps. In order to simplify this, C#5.0 is going to include the await/async keywords, that will make callback chaining obsolete.

Example:


void ClickLoad()
{
    try
    {
        Customer result = await LoadCustomer(1);
        // Process result
    }
    catch(Exception e)
    {
        // Process error
    }
}

async Task<Customer> LoadCustomer(int id)
{
    List<object> result = await LoadFromServer("select * from customers where id = " + id);
    return result[0] as Customer;
}

async Task<List<object>> LoadFromServer(string query)
{
    return await server.LoadAsync(query);
}

This is much cleaner than the examples above. Actually it is a synchronous programming model supporting an asynchronous runtime model. As asynchronous calls are very common in RIA applications the async/await keywords are going to change the way we program async calls in the near future, leading to code that is easier to develop and maintain.

I can’t await async in C# 5.0 😉

Combining Groovy and XSLT for Data Transformation

In the blog post Beautiful Transformations with Groovy I described how easy it is to create data transformations with Groovy. But sometimes organisations invested massively in XSLT transformation and want to reuse their existing XSLT templates. Read on for an an example that shows how to do that.

Assume we want to transform the following XML file to HTML:


  
    Germany
    Fast and nice
  
  
    Spain
    Large and handy
  
  
    Italy
    Small and cheap

Lets further assume the result should look like this:

Does it make sense? I don’t know, but that’s not really important. 😉

We have the following XSLT template to perform the transformation:

All you need is a Groovy script like the one below to transform the xml file to html using the given xslt.

// Load xslt
def xslt= new File("template.xsl").getText()

// Create transformer
def transformer = TransformerFactory.newInstance().newTransformer(new StreamSource(new StringReader(xslt)))

// Load xml
def xml= new File("cars.xml").getText()

// Set output file
def html = new FileOutputStream("output.html")

// Perform transformation
transformer.transform(new StreamSource(new StringReader(xml)), new StreamResult(html))

This is self-explanatory, isn’t it?
As XSLT is somewhat limited when it comes to more complex transformations, it can be extended by custom processors which can we implemented in Java or Groovy. A custom processor in Groovy can be implemented like this:

public class AgeProcessor{
    public def process(ExpressionContext context,int n){
        return "Age: " + (2012 - n) + " years";
    }
}

The processor is hooked up to the XSLT using the expressions in line 3 and 28 of the above XSLT file.
The examples above show how to reuse existing XSLT in Groovy. Are you interested to see the same same transformation in pure Groovy? (sorry, I could not resist ;-))
Here is the code:

// Load xml
def cars = new XmlSlurper().parse(new File("cars.xml"))

// Set output file
def writer = new FileWriter("output.html")

// Perform transformation
def builder = new MarkupBuilder (writer);
builder.html(xmlns:"http://www.w3.org/1999/xhtml") {
    head {
        title "Cars collection"
    }
    body {
        h1("Cars")
        ul(){
            cars.car.each{car ->
                li(car.@name.toString() + "," + car.country + "," + car.description + ", Age: " + (2012 - car.@year.toInteger()) + " years")
            }
        }
    }
 }

It is shorter and self-contained. It is also more intuitive and therefore easier to maintain. But if you have the requirement to support XSLT in Groovy you now know how to do that.

Trapped in the Comfort Zone

Many agile techniques such as Kaizen, Sashimi or Kanban correspond to terms and principles found in asian culture. A less known principle is:

“Do not develop an attachment to any one weapon or any one school of fighting”
– Miyamoto Musashi

In the context of agile it means that one should change the process if it helps to achieve the goals. This is something most developers would agree to as processes are often seen as impediments.
The same applies to technology. Translated to the technical world it reads: Do not stick to your favourite technology if there is something better suited to meet the project or customer needs. This is something many developers would not immediately agree to. Developers usually love sticking to their JEE, Spring, .NET, SOAP, REST, [any other technology] with which they grew up. They often argue that learning a new technology is time consuming and therefore hardly possible to change.
I think that is wrong. Provided a developer has a sound background, he or she can become productive in a new technology within a short time. I’ve seen developers switching from JEE to .NET and vice versa without problems. This is possible because technology always evolves. Most new frameworks and programming languages do not reinvent the wheel. The are always based on similar common principles which remain valid and stable over time. It is more a matter of mindset that keeps people trapped in their technology comfort zone.

Is that a problem?

Sometimes yes, especially when paired with Groupthink, it hinders innovation and production efficiency.

How can this prevented?

1. Make sure you have people with long standing experience in different technology domains in your team. People who worked with multiple technologies are usually more willing to reflect technology decisions and align them to the requirements of the business.

2. Don’t start a project with a strong technology committment. Let the team decide which technology is best suited to solve the business problem. Of course in conformance with the corporate standards.

3. Ensure that the team has the freedom to decide which tools they want to use.

Having the option to change weapons (processes, tools, frameworks, etc.) if needed, improves the likeliness of successful project delivery.

Bug or Change – Cause of Conflict in Agile Projects

According to the second rule of the agile manifesto working software is more important than comprehensive documentation. This is definitely true!
To be clear, this does not mean that software developed by agile teams is not documented. If comprehensive documentation brings value to the organisation, agile teams produce this as well. Specifications are written as well in agile projects. Why? Because it is not (and never was) a good idea to start development without knowing what to implement.
But contrary to waterfall projects in which much of the specification is written upfront, in Scrum the specification is written as part of the sprints. And due to the close collaboration in cross functional teams, the specification can be much more lightweight without loss of quality for the final product. This is all great. But there is a challenge to keep in mind.

When it comes to testing (acceptance, performance, etc.), either as part of the sprint (which is definitely the preferred way), or later when the product moves towards production, the testers have to find and classify bugs. Usually they do this based on the specification. In case a functionality is missing or not working as described in the specification it is a bug.
With a lightweight specification that don’t decribe every little detail it is sometimes hard to tell whether someting is a missing feature or rather a change for a later version of the product. This situation can cause conflicts.

But not necessarily. The important factor is that the testers are part of the agile team context from the beginning, so that they share the knowledge and experience with the rest of the team. In a culture of trust, the team can easily negotiate whether a finding is a bug or a change. If the team is commited to deliver quality (the Scrum Master has the responsibility to educate the team to do so), this model works properly.

This strategy correlates with the conflict resolution scenario Use collaboration to resolve the conflict described in the interesting blog post Know These Five Causes of Conflict written by Karen Ruby.

Quote:
“However, if trust is there, this conflict resolution scenario can be the best way to resolve conflicts once and for all. When both parties come together, communicate, and trust each other a definitive resolution to their conflict can occur.”

How to Staff an Agile Team

Although there is a greater likeliness of success in Scrum projects than in non-Scum projects, Scrum projects sometimes fail as well. If you ask the people involved in failed Scrum projects, they quickly accuse Scrum of being the cause for the failure. They claim to have done everything that Scrum requires, but failed, so the method is blamed.
In agile software development the most important factor is the team, the team and … … yes, the team. But a common misconception is that you just have to put together a few people to get a team that performs well. This is completely wrong, as a group of people is something very much different than a team. A group is just a bunch of individuals who neither strive for the same aim nor have a deep common understanding of the project. And they often do not trust each other enough to perform well. A team is different. People in a team trust each other, they strive for a common aim and share a deep understanding. And they have fun doing what they do. But how can a group be turned into a team?

There are well known social processes that every group has to undergo to become a team. One of those processes is the Tuckman principle of forming, storming, norming, performing (and adjourning). In their readable article Teamwork: Why teams are more successful than groups. Dr. Eberhard Huber and Sven Lindenhahn describe key factors of successful agile teams. This very much matches my expericence both as team member and coach of Scrum teams. It is essential that the group undergoes a productive storming phase in which an internal hierarchy and decision making structure is cultivated. This can be hard and tiring, but is essential for success. The key is to bring together the right mixture of individuals who have the interpersonal skills to find their place in a larger group of people in a constructive way. The ability to make compromises is important.

And even agile teams need leadership! Not from the outside in form of a project manager, but rather from inside the team. There must be people with the interpersonal and technical skills to take leadership. The authority can’t be given by management, but needs to be earned every day. Personality is key.

If you staff your next agile team, make sure you have the right mixture of skills on board. I am not talking about just technical skills, that’s self-evident. Technical knowledge can be easily shared in an agile context. What I mean are personal skills such as:

  • Ability to make compromises
  • Ability to accept constructive criticism
  • Ability to take responsibility
  • Ability to take leadership when needed

Don’t expect that the team works in harmony from day one. It is absolutely normal that the team members argue a lot, especially at the beginning. This is not a sign of bad team constellation. It is rather a natural step towards a productive agile team.

Agility through Business Process Automation?

Sometimes business process automation (BPA) is described as the silver bullet to improve agility and time to market. Especially large vendors spend huge amounts of marketing budget to promote their BPM tool suites, “360 Degree”- and “Zero Code”-approaches.

But why does BPM increasy agility? Is it really easier to adapt processes to business changes if a process has been automated using a BPM suite?

Sometimes yes, in the narrow range of allowed options. Often no, cause IT-coded processes are not as flexible as people in an organisation. But that does not mean that business process automation is a bad idea at all. There are areas in which process automation makes perfect sense.

Especially processes for which the following factors apply:

  • clearly structured and predictable
  • repetetive
  • frequently executed

Interestingly, often agility does not come from automated processes itself, but rather from the people who have their hands free for other more sophisticated ad hoc processes. We have experienced that in a large project for an international organisation from the public sector. Provided people have the right skills, BPA can help turning people from “routine workers” to “knowledge workers” (see It is All Taylor’s Fault). BPA allowed them to automate their repetetive tasks. It was a great improvement and productivity gain for the people and the organisation. The key was to give them a tooling that they were able to control, even without much help from IT guys.

Knowledge workers do not need their processes automated. They need other tools mostly to get the right structured information at the right time. IT can help in this regard, but not via BPA. I would call this Business Process Facilitation (BPF) rather than automation.

BPF means giving the people tools to do their job in a efficient manner without imposing predefined processes on them. In other words, it leaves the process and decision power with the people not the machines. User centered dashboards, search engines and adaptive case management tools are examples for BPF. We have experienced this in another project in which we evaluated the value of process automation using a BPM-Suite. In this highly dynamic environment the decision was to not implement BPM as it would have hindered agilty. Instead we implemented BPF to support the knowledge workers. The system mainly focused on efficient data management and decision making.

All in all it is not black and white, not Taylorism against knowledge work. Success comes from a combination of both. The key is focusing on things that are beneficial for the people and organisation. Sometimes it is automation, sometimes not. Process automation is cleary no silver bullet, but if applied wisely and with the right focus it can help organisations to improve efficiency.