15 Topics
10 Days
Programme Overview
Training Description
Who Should Attend
This course is ideal for;
- Data Engineers and Architects
- Software Developers and DevOps Engineers
- Data Scientists working with streaming data
- Systems Administrators responsible for data infrastructure
- Big Data and Analytics Professionals
- Technical Leaders and Managers
- Anyone involved in designing or implementing real-time data solutions.
Session Objectives
- Understand the core concepts and architecture of real-time data processing.
- Learn the fundamental principles and components of Apache Kafka.
- Acquire skills in developing stream processing applications using Kafka Streams.
- Comprehend the architecture and capabilities of Apache Flink.
- Explore strategies for building robust data pipelines with Flink's DataStream API.
- Understand the critical role of state management and fault tolerance in streaming.
- Gain insights into advanced windowing, watermarking, and time concepts in Flink.
- Develop a practical understanding of joining and aggregating data streams.
- Master the use of Flink's Table API and SQL for streaming analytics.
- Acquire skills in deploying, monitoring, and managing Flink and Kafka Streams applications.
- Learn to apply best practices for building production-ready, scalable streaming solutions.
- Comprehend techniques for integrating Flink with other data sources and sinks.
- Explore strategies for ensuring data quality and consistency in real-time pipelines.
- Understand the importance of performance tuning for both Kafka and Flink.
- Develop the ability to lead and implement a successful Real-Time Data Processing with Flink and Kafka Streams project.
About the Course
In the fast-paced world of data, mastering Real-Time Data Processing with Flink and Kafka Streams is a transformative skill for organizations seeking to derive immediate insights from continuous data streams, enabling them to react to events as they happen and gain a significant competitive advantage. The convergence of Apache Kafka for high-throughput messaging and Apache Flink for powerful stream processing provides a robust, scalable, and fault-tolerant architecture for building mission-critical, real-time applications across various industries, from finance to IoT. This comprehensive training course is meticulously designed to equip data engineers, software developers, and data scientists with the advanced knowledge and practical strategies required to design, develop, and deploy production-grade stream processing pipelines using these leading technologies. Without robust expertise in Real-Time Data Processing with Flink and Kafka Streams, organizations risk latency in their data analysis, outdated business intelligence, and a failure to capitalize on the valuable, time-sensitive insights hidden within their data, underscoring the vital need for specialized expertise in this critical domain.
Curriculum & Topics
15 Topics | 10 Days
-
Subtopic 1.1:
The evolution of data processing: from batch to real-time.
-
Subtopic 1.2:
Key concepts: event-driven architecture, streaming data, and event time.
-
Subtopic 1.3:
Use cases for real-time analytics and stream processing.
-
Subtopic 1.4:
Overview of the modern streaming ecosystem.
-
Subtopic 1.5:
Choosing between different streaming technologies.
-
Subtopic 2.1:
Kafka's architecture: topics, partitions, producers, and consumers.
-
Subtopic 2.2:
Setting up a Kafka cluster and Zookeeper.
-
Subtopic 2.3:
The role of Kafka as a central nervous system for data.
-
Subtopic 2.4:
Using Kafka command-line tools for topic management.
-
Subtopic 2.5:
Kafka's performance characteristics and durability.
-
Subtopic 3.1:
What is Kafka Streams and its place in the Kafka ecosystem?
-
Subtopic 3.2:
The Streams DSL (Domain-Specific Language).
-
Subtopic 3.3:
KStream and KTable abstractions and their differences.
-
Subtopic 3.4:
Building a simple stream processing application.
-
Subtopic 3.5:
Deploying and running a Kafka Streams application.
-
Subtopic 4.1:
State stores and local state management.
-
Subtopic 4.2:
Windowing: hopping, tumbling, and session windows.
-
Subtopic 4.3:
Stream-to-stream and stream-to-table joins.
-
Subtopic 4.4:
Handling late-arriving data.
-
Subtopic 4.5:
Processor API for fine-grained control.
-
Subtopic 5.1:
Flink's architecture: JobManager, TaskManagers, and slots.
-
Subtopic 5.2:
The DataStream API vs. the DataSet API.
-
Subtopic 5.3:
Flink's programming model: sources, transformations, and sinks.
-
Subtopic 5.4:
Key differentiators of Flink: state, time, and fault tolerance.
-
Subtopic 5.5:
Setting up a Flink development environment.
-
Subtopic 6.1:
Flink sources: reading from Kafka, files, and sockets.
-
Subtopic 6.2:
Common transformations: map, filter, flatMap, and keyBy.
-
Subtopic 6.3:
Implementing aggregations and reductions.
-
Subtopic 6.4:
Flink sinks: writing to Kafka, databases, and files.
-
Subtopic 6.5:
Writing and submitting a Flink job.
-
Subtopic 7.1:
Understanding processing time, event time, and ingestion time.
-
Subtopic 7.2:
Watermarks and their role in handling out-of-order events.
-
Subtopic 7.3:
Types of windows: tumbling, sliding, and session windows.
-
Subtopic 7.4:
Triggers and evictors for advanced windowing control.
-
Subtopic 7.5:
Implementing a real-world windowing scenario.
-
Subtopic 8.1:
Managed state vs. raw state.
-
Subtopic 8.2:
Working with keyed state and operator state.
-
Subtopic 8.3:
Checkpointing and state backend configuration.
-
Subtopic 8.4:
Savepoints for versioning and upgrades.
-
Subtopic 8.5:
Ensuring exactly-once state consistency.
-
Subtopic 9.1:
Stream-to-stream joins with windows.
-
Subtopic 9.2:
Stream-to-table joins with an external data source.
-
Subtopic 9.3:
Connecting two different data streams.
-
Subtopic 9.4:
Patterns for enriching a data stream with a static dataset.
-
Subtopic 9.5:
Best practices for designing join logic.
-
Subtopic 10.1:
Introduction to the Table API and Flink SQL.
-
Subtopic 10.2:
Integrating with the DataStream API.
-
Subtopic 10.3:
Using Flink SQL for declarative stream processing.
-
Subtopic 10.4:
Connecting to various data catalogs and sources.
-
Subtopic 10.5:
Building a real-time dashboard using Flink SQL.
-
Subtopic 11.1:
Packaging and deploying Kafka Streams jobs.
-
Subtopic 11.2:
Monitoring Kafka Streams applications.
-
Subtopic 11.3:
Managing and scaling Kafka Streams instances.
-
Subtopic 11.4:
Configuration best practices for production.
-
Subtopic 11.5:
Strategies for rolling upgrades and application health checks.
-
Subtopic 12.1:
Flink deployment modes: standalone, YARN, Kubernetes.
-
Subtopic 12.2:
Monitoring Flink jobs using the Flink UI.
-
Subtopic 12.3:
High availability configurations for Flink clusters.
-
Subtopic 12.4:
Setting up logging and metrics.
-
Subtopic 12.5:
CI/CD pipelines for Flink projects.
-
Subtopic 13.1:
The Flink Kafka connector: architecture and configuration.
-
Subtopic 13.2:
Best practices for building end-to-end pipelines.
-
Subtopic 13.3:
Understanding data format compatibility.
-
Subtopic 13.4:
Using both Flink and Kafka Streams in a single ecosystem.
-
Subtopic 13.5:
Performance tuning the Flink-Kafka connection.
-
Subtopic 14.1:
Handling complex event processing (CEP) with Flink.
-
Subtopic 14.2:
Introduction to the Flink ML and Graph APIs.
-
Subtopic 14.3:
Stream processing with Python (PyFlink).
-
Subtopic 14.4:
Integrating with other cloud services and data platforms.
-
Subtopic 14.5:
The future of real-time data processing.
-
Subtopic 15.1:
Participants work in teams to design a complete streaming pipeline.
-
Subtopic 15.2:
Exercise: ingest data from a simulated source, process with Flink, and write to a dashboard.
-
Subtopic 15.3:
Implement windowing, state management, and fault tolerance.
-
Subtopic 15.4:
Deploy and monitor the application on a cluster.
-
Subtopic 15.5:
Present the final project and discuss design choices.
