JAX 2020: Big Data and Agile Culture

This year JAX is taking place from 7. September to 11. September in Mainz. W-JAX is taking place from 2. November to 6. November in Munich. Due to the Corona situaton it will be a special experience as the conferences are going to be held in an hybrid manner (on-site and online). In my sessions I am going to talk about Big Data and Agile Culture.

In the Big Data session I am going to show you how to set up an Open Source Big Data platform from scratch. You will see how popular technologies such as Hadoop, Spark, Hive, Kafka and others work together. We are going to implement a typical end-to-end use case live together. You’ll get a solid understanding of what these technologies do and how they work together to form a platform.

The Agile session covers aspects of culture as a building block of agile organisation development. I am going to talk about what culture actually is, why it is an essential part of “being agile” and how to develop it. Moreover I am going to share experiences and common pitfalls on the journey of agile culture development.

I am glad to be there and hope to meet you on-site or online.

Success Story: Big Data in Logistics

In the years 2019 and 2020 I had the pleasure to support TIMOCOM in the implementation of their brand new Big Data Platform.
TIMOCOM is an international logistics platform provider and a true champion in its area.

When we started the initiative the company had an existing BI-System to perform reporting and statistical analysis. The aim was to extend the capapabilities of the company to collect, store and analyse huge amount of data. A Big Data solution comprising best of breed open source products was chosen. The new technology stack is able to scale not only technically but also business wise as it ist completely license cost free. It is based on technologies such as Java, Python, Hadoop, Hive, Kafka, Spark and HBase.
A major challenge in the beginning was that the staff had almost no knowledge of the applied technologies. To cope with this situation and to establish the solution quickly and in high quality, we’ve set up a Creative Software Workbench (CSW). A CSW combines the areas of modern technology, agile methodology and team dynamics to create an enviroment in which digital products can be created in the best possible way. It is based on more than 25 years of practical experience from many successful and of course some not so successful projects. In this enviroment agile engineering and active learning are important parts which helped us to master the Big Data ecosystem in a reasonable amount of time.

The new platform enables the company to gain new insights from their data today an tomorrow. It is an important step in the future to support their data driven business model.
You can read about the project in the success story “Wissen aus Daten”. I am glad that I can add this story to our list of success stories. If you want to know more about it, don’t hesitate to contact me.


Workshop: Big Data you can Touch

Today I released the brand new Workshop:Big Data you can Touch.

If you start researching about Big Data Platforms, you will find an overwhelming amount of possible technologies. But if you dig deeper you’ll find that many platforms are based on the same proven Open Source products.

This workshop teaches how to set up your own Big Data platform using professional Open Source products. Together we’ll build a end-to-end use case using a Lambda-Architecture and Machine Learning.

It is intended for all people who are generally intested in Big Data platforms, e.g. developers, architects, analysts or decision makers, who want to know how those technologies work together.

The workshop takes 4 hours and can be booked as On-Site-Training and Online-Webinar. Hope to see you there…

Upcoming events:.

7. Mai 13:00 – 17:00 Webinar: Big Data zum Anfassen

13. Mai 13:00 – 17:00 Webinar: Big Data zum Anfassen

21. Mai 13:00 – 17:00 Webinar: Big Data zum Anfassen

JAX 2019: Agile Team Architecture and Big Data

JAX is one of the most known conferences for Java, architecture and software innovation in Germany. Im am glad to be invited this year to give some sessions. Between the 6th and 10th May 2019 JAX will be taking place at Rheingold Halle in Mainz.

Agile product teams are becoming more and more mission critical. On the 6th I am going to give a presentation about the way agile product teams can be built by applying software architecture principles such a resilience and performance to teams.

When people start learning Big Data technologies for many it seems to be complex due to the sheer amount of products in the Big Data ecosystem. On the 8th I am going to show a simple Big Data Stack to get started with. I am going to set up a working stack from scratch and implement a working lambda architecture.

You can see the timeslots on the JAX website. I look forward to seeing you there.

Free Avro Schema Viewer

Avro is a data serialization system. The core is Avro schema which can be used to describe the structure of datasets very much like XML Schema or JSON Schema. Avro is primarily used in Big Data scenarios for which it offers special features like schema evolution. This is a typical Avro file:

{"namespace": "net.pleus.domain",
    "name": "customer",
    "version":"1.0",
    "doc" : "Customer Dataset",
    "type": "record", 
    "fields": [
        {"name": "id", "type": "int","default":"-1", "doc":"Unique id of the customer"},
        {"name": "name", "type": ["string", "null"],"default":null,"aliases":["fullname"],"doc":"Customer's name (optional)"},
        {"name":"address", "default":null, "doc":"Address information",
            "type":{
                "type":"record",
                "name":"address",
                "fields":[
                    {"name": "street", "type" : "string","default":"unknown", "doc":"Street"},        
                    {"name": "city", "type": "string","default":"unknown", "doc":"City"}
                ]
            }
        },
        {"name": "contact", "default":null, "doc":"List of contact options", "type": {
            "type": "array",
            "items": {
                "type":"record",
                "name":"contact",
                "fields":[
                    {"name": "type", "type" : { "name":"values" , "type": "enum", "namespace" : "net.pleus.contacts", "symbols" : ["EMAIL", "PHONE", "MESSENGER"]}, "doc" : "Type of contact"},        
                    {"name": "url", "type": "string","default":"unknown", "doc":"The contact url"}
                ]
            }            
        }}
    ]
   }

As you can see it is a JSON file that is structured according to the Avro specification. Although this verbose form might be suitable for technical people, often such structure definitions have to be discussed with non-technical people from the business domain. This is especially the case if you follow a domain driven approach.

To make it easier to discuss and negotiate Avro structure definitions without creating redundant model representations, I’ve created an easy to use Avro Viewer. If you drop the file shown above you will see a visual representation that just shows the essence of the schema.

You can see records, enums, arrays, defaults, aliases, documentation and so on without the JSON markup noise.
Avro Viewer is free to use. It is just plain HTML+JavaScript+CSS. If required you can download the source and modify it to suit your needs.

Fintech Success Story

In 2017 PLEUS Consulting supported Yareto GmbH in the development of their new independent comparision site for the automotive finance industry.
Yareto is a fintech company that specializes in automotive financing. In 2016 the corporate startup was founded to build a brand new comparision site for the automotive finance industry. The site enables car dealers to compare credit offers in the areas of sales financing and purchase financing. Lenders get access to sales channels they couldn’t serve before.

PLEUS Consulting supported Yareto in setting up a Creative Software Workbench. The Creative Software Workbench aligns technology, processes and people in a way that creates an environment in which high quality digital products can be developed in short time.

The front-ends were developed using modern web technologies such as Java-/Typescript, HTML5, CSS and Angular. For the backend Java Enterprise (JEE) and a Sustainable Service Design approach was utilized to design and build a backend with a high degree of reuse and scalability. The service landscape was established using Domain Driven Design principles. Operations was performed using cloud platforms.

On the technical side, PLEUS Consulting supported the teams as Lead Developer. In the area of agile techniques, PLEUS Consulting supported the development teams as Agile Coach. The combination of those roles worked quite well especially in the phases of seed and growth. With these roles the company received thorough support in the areas of technology and methodology.

The project has shown that with a combination of modern technologies, agile approaches and the right people a very short concept to market cycle can be achieved, creating competitive advantages. This is what the Creative Software Workbench is all about.

You can read more details about the project in the official success story. More info about the Creative Software Workbench can be found on the official website.

JAX 2018: Management of Requirements and Quality the Agile Way

Between the 23th and 27th April 2018 JAX will be taking place at Rheingold Halle in Mainz.

On the 23th I am going to give a presentation about the way agile companies turn ideas into valuable software in a short time and how this differs from classic requirements engineering practices.

On the 24th I am going to show how agile teams build high quality software. I’ll cover techniques and principles that help to ship software increments to the customer after every Sprint.

You can see the timeslots on the JAX website. I look forward to seeing you there.