The Data Lakehouse – Best of Data Lake and Data Warehouse

Almost every company today utilizes a kind of data warehouse or business intelligence solution for data analysis and reporting. Those solutions are primarily based on relational data, ETL jobs and reporting. Although powerful they are limited when it comes to very large data sets or realtime processing.

Some years ago the paradigm of Data Lakes was born to process very large data sets. Data Lakes are based on the idea of raw data processing, streaming data, ELT and machine learning.

What about combining the strengths of both into something even more powerful? This is what is called the Data Lakehouse, a term conceived by Databricks.

Evolution of data storage, from data warehouses to data lakes to data lakehouses
Data Lakehouse. Source: https://databricks.com/de/glossary/data-lakehouse

As the name suggests, it combines the strengths of Data Warehouses with the power of Data Lakes. Although the term Data Lakehouse was not really used in 2020, we built a Data Lakehouse for a logistics company already then.

One of the main datasets in this project comprised 16 years or freight offers plus live data. The historical data was transferred from Oracle Databases to a new Data Lake. In addition stream sources were set up to ingest live data directly from the source applications into the Data Lake. The result was a huge active archive including historical and live data based on Hadoop, Spark, Kafka and HBase. The raw data was stored and continuously transformed into a normalized form ready to be processed by reporting and machine learning jobs. A logical structure, metadata and governance were added using Apache Atlas and Avro schemas. Reporting and end user security was implemented using Microsoft Power BI.

The result was something we would probably call a Data Lakehouse today. The combination of BI and Data Lake was very successful, so we created a success story to describe it.
To me is seems that Data Lakehouse is a very useful concept. It is an evolutionary step towards an integrated solution for processing and analysis of massive amounts of data by applying good practices in terms of governance, security and reporting. Surely something BI-Teams should have an eye on.

JAX Agile Day 2022

We are doing agile for over 20 years now. It shows that Agile is not just a flash in the pan like many other trends in IT. Agile has brought innovation to many organizations and changed the way we develop software today. This is already a great success . According to the principle of inspect and adapt the agile movement is still evolving and entering more and more domains. That’s why the main theme of this year’s Agile Day is Agile beyond Development. We are widening the scope to aspects and departments outside of development to take a more systemic view on modern software development.

So, this year the JAX Agile Day is going to be great again. You can see all sessions here. I am very glad to be your host on 2. Mai in Mainz or wherever you like, because the day is going to be streamed directly to you office as well. We are also making the day more interactive for you by integrating a Lean Coffee Session.

And that’s not all. For the upcoming conferences we are going to rename the day and open the agenda to create a space for strong innovation in professional software development. But this is something that has to be revealed later. For the time being I hope you take part in the conference and to meet you there.

Agile Day at W-JAX

From 8.November 21 to 12.November 21 W-JAX is taking place in Munich. W-JAX is the winter version of JAX, which is the leading conference for professional software development in Germany. This year W-JAX will be hybrid, that is people can attend personally or online from whereever they are.

I have the pleasure to moderate this years agile day again. The main topic of the day is agile beyond development. Many organisations are applying agile frameworks and techniques with good results. But business agility is often not achieved. That’s why we have choosen inspiring sessions about agility at the business level.

My session is about Kanban Flight Levels and OKRs. I am going to show how to establish coordination and alignment levels on top of agile teams.

The last session will be an interactive lean coffee session in which you can bring in and discuss your own topics.

I am looking forward to this great conference day and hope to meet you there.

Agile Culture Articles

In 2021 many organisations are trying to become more agile. They do that by applying techniques such as Scrum or Kanban. In the articles I wanted to shed some light on more hidden aspects of cultural development which I think is the real success factor of agile organizational development.

Recently I published a series of articles about agile culture development in the German IT magazines Java Magazin and Windows Developer.

Those are the Java Magazin issues that contain the article:

I am glad to see that this time the article was published in two magazines. Those are the Windows Developer magazines.

If you are interested in the content, please visit entwickler.kiosk, or even better, get in touch with me so that we can talk personally about agile culture development in your organisation. 🙂

JAX 2020: Big Data and Agile Culture

This year JAX is taking place from 7. September to 11. September in Mainz. W-JAX is taking place from 2. November to 6. November in Munich. Due to the Corona situaton it will be a special experience as the conferences are going to be held in an hybrid manner (on-site and online). In my sessions I am going to talk about Big Data and Agile Culture.

In the Big Data session I am going to show you how to set up an Open Source Big Data platform from scratch. You will see how popular technologies such as Hadoop, Spark, Hive, Kafka and others work together. We are going to implement a typical end-to-end use case live together. You’ll get a solid understanding of what these technologies do and how they work together to form a platform.

The Agile session covers aspects of culture as a building block of agile organisation development. I am going to talk about what culture actually is, why it is an essential part of “being agile” and how to develop it. Moreover I am going to share experiences and common pitfalls on the journey of agile culture development.

I am glad to be there and hope to meet you on-site or online.

Success Story: Big Data in Logistics

In the years 2019 and 2020 I had the pleasure to support TIMOCOM in the implementation of their brand new Big Data Platform.
TIMOCOM is an international logistics platform provider and a true champion in its area.

When we started the initiative the company had an existing BI-System to perform reporting and statistical analysis. The aim was to extend the capapabilities of the company to collect, store and analyse huge amount of data. A Big Data solution comprising best of breed open source products was chosen. The new technology stack is able to scale not only technically but also business wise as it ist completely license cost free. It is based on technologies such as Java, Python, Hadoop, Hive, Kafka, Spark and HBase.
A major challenge in the beginning was that the staff had almost no knowledge of the applied technologies. To cope with this situation and to establish the solution quickly and in high quality, we’ve set up a Creative Software Workbench (CSW). A CSW combines the areas of modern technology, agile methodology and team dynamics to create an enviroment in which digital products can be created in the best possible way. It is based on more than 25 years of practical experience from many successful and of course some not so successful projects. In this enviroment agile engineering and active learning are important parts which helped us to master the Big Data ecosystem in a reasonable amount of time.

The new platform enables the company to gain new insights from their data today an tomorrow. It is an important step in the future to support their data driven business model.
You can read about the project in the success story “Wissen aus Daten”. I am glad that I can add this story to our list of success stories. If you want to know more about it, don’t hesitate to contact me.


Workshop: Big Data you can Touch

Today I released the brand new Workshop:Big Data you can Touch.

If you start researching about Big Data Platforms, you will find an overwhelming amount of possible technologies. But if you dig deeper you’ll find that many platforms are based on the same proven Open Source products.

This workshop teaches how to set up your own Big Data platform using professional Open Source products. Together we’ll build a end-to-end use case using a Lambda-Architecture and Machine Learning.

It is intended for all people who are generally intested in Big Data platforms, e.g. developers, architects, analysts or decision makers, who want to know how those technologies work together.

The workshop takes 4 hours and can be booked as On-Site-Training and Online-Webinar. Hope to see you there…

Upcoming events:.

7. Mai 13:00 – 17:00 Webinar: Big Data zum Anfassen

13. Mai 13:00 – 17:00 Webinar: Big Data zum Anfassen

21. Mai 13:00 – 17:00 Webinar: Big Data zum Anfassen