Real-Time Data Processing with Kafka Training Course

Share this course

Duration

3 Days

Course Overview

Developers who code in Java, Python or Scala spend three days building event pipelines on Apache Kafka that react the moment data arrives. Day one compares streaming with batch work, explains event-driven design and the broker, topic, partition and offset model, and writes a producer and consumer app. Day two processes streams with the Kafka Streams API, links Kafka to databases and analytics tools through Kafka Connect, and secures and monitors a cluster. Day three scales for high volume with partitions and replication, studies log aggregation and event sourcing cases, and ends with teams presenting a Kafka design for a business use case.

Format of Training

  • Whiteboard sessions on brokers, topics and partitions before any code
  • Coding labs that write producers, consumers and Kafka Streams jobs
  • Demo of an end-to-end pipeline wired up with Kafka Connect
  • Cluster security and scaling exercises, then a team design pitch

Course Objectives

  1. Explain how real-time processing and event-driven architectures work.
  2. Describe Apache Kafka’s design and its core parts.
  3. Set up and manage Kafka clusters.
  4. Produce, consume and process live data streams.
  5. Connect Kafka to outside systems such as databases and stream processing tools.
  6. Apply Kafka to practical cases such as log aggregation, data pipelines and monitoring.
  7. Deploy and scale Kafka in production.

Prerequisites

Course Outline

Day 1: Introduction to Real-Time Data Processing and Kafka

Session 1: Fundamentals of Real-Time Data Processing

  • Real-time processing compared with batch processing
  • What event-driven architectures are

Session 2: Apache Kafka Overview

  • How Kafka is built: brokers, topics, partitions and offsets
  • Preparing a Kafka environment

Session 3: Producing and Consuming Data Streams

  • Coding producers and consumers in Kafka
  • Writing a Kafka producer and consumer application

Day 2: Advanced Kafka Features and Integration

Session 1: Kafka Streams and Processing

  • Processing streams with the Kafka Streams API
  • Running live data transformations in Kafka Streams

Session 2: Integrating Kafka with Other Systems

  • Linking Kafka to databases, monitoring tools and analytics platforms
  • Demo: an end-to-end pipeline built with Kafka Connect

Session 3: Securing and Monitoring Kafka Clusters

  • Role-based access control, encryption and monitoring
  • Locking down and watching a Kafka cluster

Day 3: Scaling and Real-World Applications

Session 1: Scaling Kafka for High-Volume Data

  • Setting partitions, replication and fault tolerance
  • Tuning Kafka so it scales

Session 2: Real-World Kafka Applications

  • Cases on log aggregation, event sourcing and real-time analytics
  • Group task: designing a Kafka solution for a business case and presenting it

Session 3: Best Practices and Future Trends

  • Tuning Kafka for production use
  • Talk: new ideas and directions in real-time data processing

Book This Course

Customise This Course for Your Team

Teams can bring sample events from their own apps, sensors or logs so the producer and stream labs work on familiar data. If your Kafka cluster is already running, Day 3 can focus on scaling and securing that setup instead of a fresh one. Let us know your event sources and we will plan the labs around them.
Need help with the right course to choose?

Explore more opportunities

Introduction to Data Engineering Training Course

ETL Processes for Data Pipelines Training Course

Data Modeling for Analytics Training Course

Building Scalable Data Pipelines with Apache Airflow Training Course

Data Warehousing with Snowflake and Redshift Training Course

Real-Time Data Processing with Kafka Training Course

Real-Time Data Processing with Kafka Training Course