The secrets of evolvable software

TL;DR: Modularization is the key to evolvable software. This blog post sheds some light on different aspects of modularization of software systems.

Motivation

The IT business is very dynamic. Changes happen everytime and everywhere. Programming languages, frameworks, tools, infrastructure – it feels like almost everything is changing all the time. For enterprises this is a challenge as it constantly requires modernization and maintenance. Ok, this is part of the nature of software systems today you might say, so why should we care?

Because from a business perspective it would be nice to have systems which do not require expensive rewrites now and then for cost reasons. Instead it would be better to have software that is able to evolve over time. Of cource maintenance is always required for example to get security patches. But if the software is build with evolvability in mind it can be adapted to future needs more easily (without major reimplementation). Evolvable software is structured in a way that enables change. It is one of the core values of Clean Code. Simply put evolvability is an important trait of modern software systems.

How can we get there? Let’s have a look at several aspects and good practices to create evolvable software.

Design

The key of evolvable software is in its design. And because the business domain is much more stable than technologies, the business is a good basis for design considerations. Domain Driven Design (DDD) means exactly that. Instead of using technical concepts such as message brokers, databases and so on we use business concepts such as products, sales or invoicing to organize our software system.

Especially important is the strategic design and in particular the concept of Bounded Contexts which is the central design pattern in DDD. Bounded Contexts foster modularity already at the design level. Domain design is carried out together with the experts from the particular domains and can be documented ideally with a graphical notation such as UML.

Codebase

Based on the domain design the codebase can be created and organized. I often see systems in which design model and code are decoupled which makes it difficult to understand the codebase. Moreover code is often structured based on technical concepts such as controllers, entities and so on. Although technical terms can be used at a lower level, the main structuring concept should be the domain. It is important to keep the design structure from the business domain at the code level to foster maintainability and evolvability.

Modularity is of utmost importance when it comes to evolvable software systems. This has to do with the fact that even the best programmers in the world are not able to understand software that is too large and complex. Modularity helps to chunk systems into smaller parts that are more likely to be understood.

Assume a Java based system. What are the options for modularity. At the language level we have packages. Actually packages are intended to structure a codebase into logical units. By keeping classes package private and exposing dedicated interfaces you can enforce encapsulation of your modules. Unfortunately it does not properly work with subpackages, but if a module is not too large this is not neccessarily a problem. By enforcing architectural constraints on your codebase with tools such as ArchUnit you can even relax encapsulation while keeping the codebase clean. If you want versioned artifacts, you can create libaries for instance with maven or gradle. If you need more independence put the libraries into separate repositories.

Services and APIs

Is that already a service? If the module has a dedicated interface contract I would say yes. But that is more of a philosophical question 😉

If you want to expose the service in an interoperable way, just add an HTTPS-Endpoint with REST- or RPC-style. No matter whether you choose a code- or contract-first approach, always go for an API-first aproach, as this gives you a better an more though-out structure which is usually closer to the business domain. My personal favourite is contract-first, for instance based on OpenAPI, because it is technology agnostic and opens the way for alternative implementation languages. Do you remember? We are talking about evolvable software. Even programming languages are changing over time. And even asynchronous interfaces and message broker based services deserve an API-contract, which can be created using AsyncAPI.

Frontend Components

Discussions about services most of the time happen on the server side in the form of service oriented architecture or microservices. But user interfaces can be modularized as well. Most modern UI frameworks such as Angular or React have a component model on board. UI components have interfaces as well. Those interfaces comprise everything that communicates with the outside world of the component such as attributes, events, cookie/local/session storage and window messages that should be properly documented. Modular distribution can be achieved by creating npm packages. If you want to increase interoperability and freedom of technology implement ui components as web components.

Containers

Containerization is a strong trend for good reasons. But over the last years the industry has learned that it is not always the best option to deploy each service or module as a separate unit. Deploying and running a lot of small units requires more infrastructure which can be costly to implement and maintain. In the past the pendulum swang from monoliths to microservices architectures which are both extreme in its implementations. Today so called moduliths are combining modularity with monolithic deployment models. Starting with a modulith can be a good option, evolving it to a microservice-based deployment model if and only if required. You can deploy a modulith standalone or using Docker and Kubernetes the same way as you would do with a single service if this is the runtime of your choice. Anyway it is important to find a proper granularity of modules. Self contained systems aligned to the business domains are often a good option in this regard. This also helps to keep cognitive load and responsibility manageable at the team level. This is again a matter of design.  

Limits

What happens when the programming Plattform changes completely, let’s say from .NET to JEE or from Java to Python? Even then you can save parts of your investments. Design outcomes and standards-based artifacts such as OpenAPI contracts are platform agnostic and can be reused. And as the business domain is usually quiet stable and not connected to changes in technology , you have a good chance to reuse your design outcomes as is.

Summary

Modularization is essential when it comes to creating evolvable software. Technologies are powerful enough to do it. Modular software is primarily a matter of design and the will to invest some effort into it. From the many projects that I have seen in my career, I am convinced that it is one of the most important things to do when creating professional software systems.

Aliens and Avatars – Explorative Product Deveploment in Action

On this years W-JAX I had the pleasure to host a session about explorative product development. The session was about the appropriate mixture of methods and software architecture especially, but not only, for projects in the pioneering phase. A good example of the Creative Software Workbench in action. You can see the session recording on entwickler.de (subscription required).

Lean Coffee at Jax Innovation Forum and DevOpsCon

This autumn I am going to moderate the Jax Innovation Forum and the DevOps Transformation Day again. On both days we will also have Lean Coffee Sessions, a format that incorporates input from the attendees. It is very energetic, interesting and fun. On top of that we conduct it even in hybrid mode, in which you can participate no matter whether you are on site or in the office. If you are interested in those kinds of events we can also carry it out it in your organisation. The event will be adjusted according to your corporate needs. You can find more information here.

The Data Lakehouse – Best of Data Lake and Data Warehouse

Almost every company today utilizes a kind of data warehouse or business intelligence solution for data analysis and reporting. Those solutions are primarily based on relational data, ETL jobs and reporting. Although powerful they are limited when it comes to very large data sets or realtime processing.

Some years ago the paradigm of Data Lakes was born to process very large data sets. Data Lakes are based on the idea of raw data processing, streaming data, ELT and machine learning.

What about combining the strengths of both into something even more powerful? This is what is called the Data Lakehouse, a term conceived by Databricks.

Evolution of data storage, from data warehouses to data lakes to data lakehouses
Data Lakehouse. Source: https://databricks.com/de/glossary/data-lakehouse

As the name suggests, it combines the strengths of Data Warehouses with the power of Data Lakes. Although the term Data Lakehouse was not really used in 2020, we built a Data Lakehouse for a logistics company already then.

One of the main datasets in this project comprised 16 years or freight offers plus live data. The historical data was transferred from Oracle Databases to a new Data Lake. In addition stream sources were set up to ingest live data directly from the source applications into the Data Lake. The result was a huge active archive including historical and live data based on Hadoop, Spark, Kafka and HBase. The raw data was stored and continuously transformed into a normalized form ready to be processed by reporting and machine learning jobs. A logical structure, metadata and governance were added using Apache Atlas and Avro schemas. Reporting and end user security was implemented using Microsoft Power BI.

The result was something we would probably call a Data Lakehouse today. The combination of BI and Data Lake was very successful, so we created a success story to describe it.
To me is seems that Data Lakehouse is a very useful concept. It is an evolutionary step towards an integrated solution for processing and analysis of massive amounts of data by applying good practices in terms of governance, security and reporting. Surely something BI-Teams should have an eye on.

JAX Agile Day 2022

We are doing agile for over 20 years now. It shows that Agile is not just a flash in the pan like many other trends in IT. Agile has brought innovation to many organizations and changed the way we develop software today. This is already a great success . According to the principle of inspect and adapt the agile movement is still evolving and entering more and more domains. That’s why the main theme of this year’s Agile Day is Agile beyond Development. We are widening the scope to aspects and departments outside of development to take a more systemic view on modern software development.

So, this year the JAX Agile Day is going to be great again. You can see all sessions here. I am very glad to be your host on 2. Mai in Mainz or wherever you like, because the day is going to be streamed directly to you office as well. We are also making the day more interactive for you by integrating a Lean Coffee Session.

And that’s not all. For the upcoming conferences we are going to rename the day and open the agenda to create a space for strong innovation in professional software development. But this is something that has to be revealed later. For the time being I hope you take part in the conference and to meet you there.

Agile Day at W-JAX

From 8.November 21 to 12.November 21 W-JAX is taking place in Munich. W-JAX is the winter version of JAX, which is the leading conference for professional software development in Germany. This year W-JAX will be hybrid, that is people can attend personally or online from whereever they are.

I have the pleasure to moderate this years agile day again. The main topic of the day is agile beyond development. Many organisations are applying agile frameworks and techniques with good results. But business agility is often not achieved. That’s why we have choosen inspiring sessions about agility at the business level.

My session is about Kanban Flight Levels and OKRs. I am going to show how to establish coordination and alignment levels on top of agile teams.

The last session will be an interactive lean coffee session in which you can bring in and discuss your own topics.

I am looking forward to this great conference day and hope to meet you there.

Agile Culture Articles

In 2021 many organisations are trying to become more agile. They do that by applying techniques such as Scrum or Kanban. In the articles I wanted to shed some light on more hidden aspects of cultural development which I think is the real success factor of agile organizational development.

Recently I published a series of articles about agile culture development in the German IT magazines Java Magazin and Windows Developer.

Those are the Java Magazin issues that contain the article:

I am glad to see that this time the article was published in two magazines. Those are the Windows Developer magazines.

If you are interested in the content, please visit entwickler.kiosk, or even better, get in touch with me so that we can talk personally about agile culture development in your organisation. 🙂