Revolutionise the Data Lake through Kafka

DIN 10:45 - 11:15

Zaal 6

When an organization establishes a data lake that serves as its source of truth, relying on batch jobs to collect data may prove inadequate resulting in suboptimal decision-making driven by outdated information. This session will dive into the basics of integrating event streaming into the Delta Lake, with Kafka at the center of it all.

As we shift from on-premises to cloud-based infrastructure, obtaining near real-time data in a scalable and dependable manner becomes imperative. This presents a unique opportunity to explore creative solutions that can endure the test of time. Join us as we explore the intricacies of incorporating data from various sources, such as SAP, into the Delta Lake, with Apache Kafka serving as the cornerstone of this process.

Through this solution, you will discover how to combine various components of AWS, and Confluent using several use cases. The process starts with the architecture of a real-time replication, followed by an overview of the connectors setup on Kafka Connect clusters in AWS Elastic Containers service (ECS). Then go together through the process of extracting data from various sources e.g. SAP tables with delta-enabled capabilities and/or Website data into Kafka Topics. Next, the Kafka Connector performs pivotal transformations, reaching the final step of piecing together the data in the Delta Lake. This way, the data can be used for real-time analysis and recommendations in client-facing applications.

During this session, we'll explore the advantages of emphasizing automation to streamline the implementation process using Infrastructure as Code - Terraform, and the overview of integrating event streaming into the Delta Lake.

  • Thema

    Het is niet nieuw als we zeggen dat ieder bedrijf beschikt over een enorme bulk aan informatie. Om waarde te kunnen creëren aan data dient het proces van ontdekken, interpreteren en communiceren van significante patronen doorlopen te worden. Het zijn de organisaties die erin slagen om gebruikers op diverse niveaus deze data te laten analyseren die hun kansen grijpen. Begrijpelijke analysetools zorgen voor begrip tussen de ene en de andere medewerker, ongeacht het niveau. Maar vermoedelijk nog belangrijker: de juiste analyse leidt tot slimme campagnes en bevlogen medewerkers, zaken die gericht zijn in te zetten voor het behouden van klanten.