On this years W-JAX I had the pleasure to host a session about explorative product development. The session was about the appropriate mixture of methods and software architecture especially, but not only, for projects in the pioneering phase. A good example of the Creative Software Workbench in action. You can see the session recording on entwickler.de (subscription required).
This autumn I am going to moderate the Jax Innovation Forum and the DevOps Transformation Day again. On both days we will also have Lean Coffee Sessions, a format that incorporates input from the attendees. It is very energetic, interesting and fun. On top of that we conduct it even in hybrid mode, in which you can participate no matter whether you are on site or in the office. If you are interested in those kinds of events we can also carry it out it in your organisation. The event will be adjusted according to your corporate needs. You can find more information here.
In the issue 08/22 of the German Javamagazin you can read my latest article about the future of Agile.
You can read it also online on entwickler.de.
Enjoy the read.
Save the date: Monday 2.May 13:30 Rheingoldhalle Mainz or Online
Almost every company today utilizes a kind of data warehouse or business intelligence solution for data analysis and reporting. Those solutions are primarily based on relational data, ETL jobs and reporting. Although powerful they are limited when it comes to very large data sets or realtime processing.
Some years ago the paradigm of Data Lakes was born to process very large data sets. Data Lakes are based on the idea of raw data processing, streaming data, ELT and machine learning.
What about combining the strengths of both into something even more powerful? This is what is called the Data Lakehouse, a term conceived by Databricks.
As the name suggests, it combines the strengths of Data Warehouses with the power of Data Lakes. Although the term Data Lakehouse was not really used in 2020, we built a Data Lakehouse for a logistics company already then.
One of the main datasets in this project comprised 16 years or freight offers plus live data. The historical data was transferred from Oracle Databases to a new Data Lake. In addition stream sources were set up to ingest live data directly from the source applications into the Data Lake. The result was a huge active archive including historical and live data based on Hadoop, Spark, Kafka and HBase. The raw data was stored and continuously transformed into a normalized form ready to be processed by reporting and machine learning jobs. A logical structure, metadata and governance were added using Apache Atlas and Avro schemas. Reporting and end user security was implemented using Microsoft Power BI.
The result was something we would probably call a Data Lakehouse today. The combination of BI and Data Lake was very successful, so we created a success story to describe it.
To me is seems that Data Lakehouse is a very useful concept. It is an evolutionary step towards an integrated solution for processing and analysis of massive amounts of data by applying good practices in terms of governance, security and reporting. Surely something BI-Teams should have an eye on.
We are doing agile for over 20 years now. It shows that Agile is not just a flash in the pan like many other trends in IT. Agile has brought innovation to many organizations and changed the way we develop software today. This is already a great success . According to the principle of inspect and adapt the agile movement is still evolving and entering more and more domains. That’s why the main theme of this year’s Agile Day is Agile beyond Development. We are widening the scope to aspects and departments outside of development to take a more systemic view on modern software development.
So, this year the JAX Agile Day is going to be great again. You can see all sessions here. I am very glad to be your host on 2. Mai in Mainz or wherever you like, because the day is going to be streamed directly to you office as well. We are also making the day more interactive for you by integrating a Lean Coffee Session.
And that’s not all. For the upcoming conferences we are going to rename the day and open the agenda to create a space for strong innovation in professional software development. But this is something that has to be revealed later. For the time being I hope you take part in the conference and to meet you there.