15 Topics
10 Days
75 Sessions
Programme Overview
Training Description
Who Should Attend
- Cloud architects
- Solutions architects
- Data scientists
- Machine learning engineers
- Business intelligence professionals
- Database professionals
- DevOps and platform engineers
- Data platform engineers
- Enterprise architects
- Data governance professionals
- IT and technology managers
- Professionals responsible for building or modernizing enterprise data platforms
Session Objectives
- Understand the core principles of Lakehouse architecture
- Learn how to implement scalable ELT pipelines on data lakes
- Integrate Delta Lake, Apache Iceberg, or Apache Hudi for data reliability
- Design efficient data lake schemas and partitioning strategies
- Use Apache Spark and SQL for large-scale data transformation
- Manage metadata, transactions, and schema evolution
- Enable real-time and batch data access on a unified platform
- Orchestrate workflows with tools like Airflow and dbt
- Ensure data quality, governance, and lineage
- Optimize performance, cost, and data retrieval speeds
- Build end-to-end pipelines for BI, analytics, and ML workloads
About the Course
This advanced training provides participants with the knowledge and practical skills required to design, build, operate, and optimize end-to-end data engineering solutions using the modern lakehouse architecture. The course combines the scalability and flexibility of data lakes with the reliability, governance, and analytical capabilities traditionally associated with data warehouses.Participants will explore the complete data engineering lifecycle, from data ingestion and transformation to storage, data quality, governance, analytics, and consumption. The training covers batch and streaming pipelines, medallion-style architectures, data modelling, metadata management, orchestration, performance optimization, security, and production operations across modern lakehouse environments.
Curriculum & Topics
15 Topics | 75 Sessions
-
Workshop 1.1:
Lakehouse vs. traditional data warehouse and data lake
-
Workshop 1.2:
Benefits of unifying storage and analytics
-
Workshop 1.3:
Open formats: Delta Lake, Iceberg, Hudi
-
Workshop 1.4:
Key components and ecosystem overview
-
Workshop 1.5:
Use cases in analytics, BI, and machine learning
-
Workshop 2.1:
ELT vs. ETL in Lakehouse environments
-
Workshop 2.2:
Streaming vs. batch ingestion
-
Workshop 2.3:
Tools for ingestion: Apache NiFi, Airbyte, Kafka, Spark
-
Workshop 2.4:
Handling schema drift and data validation
-
Workshop 2.5:
Data partitioning and file formats
-
Workshop 3.1:
Introduction to Spark DataFrames and SQL
-
Workshop 3.2:
Optimizing transformations and joins
-
Workshop 3.3:
Structured Streaming for real-time pipelines
-
Workshop 3.4:
Performance tuning and job optimization
-
Workshop 3.5:
Working with large-scale JSON, Parquet, and ORC files
-
Workshop 4.1:
ACID transactions and time travel
-
Workshop 4.2:
Schema enforcement and evolution
-
Workshop 4.3:
Managing Delta logs and versions
-
Workshop 4.4:
Vacuuming and data retention policies
-
Workshop 4.5:
Delta Lake vs. traditional lake formats
-
Workshop 5.1:
Table structure and metadata layers
-
Workshop 5.2:
Data compaction and clustering
-
Workshop 5.3:
Querying with Spark, Trino, Presto
-
Workshop 5.4:
Use cases for versioning and rollback
-
Workshop 5.5:
Performance comparisons and trade-offs
-
Workshop 6.1:
Bronze, Silver, and Gold layer modeling
-
Workshop 6.2:
Medallion architecture design principles
-
Workshop 6.3:
Data normalization and denormalization
-
Workshop 6.4:
Partitioning, bucketing, and clustering strategies
-
Workshop 6.5:
Handling slowly changing dimensions (SCDs)
-
Workshop 7.1:
Introduction to dbt core and dbt Cloud
-
Workshop 7.2:
Building modular SQL models
-
Workshop 7.3:
Testing and documenting transformations
-
Workshop 7.4:
Orchestrating dbt with Airflow
-
Workshop 7.5:
Lineage visualization and deployment best practices
-
Workshop 8.1:
Cataloging data with Unity Catalog, Hive Metastore, AWS Glue
-
Workshop 8.2:
Implementing data classifications and tags
-
Workshop 8.3:
Role-based access control and fine-grained permissions
-
Workshop 8.4:
Managing table versions and audit logs
-
Workshop 8.5:
Lineage tracking with OpenMetadata or DataHub
-
Workshop 9.1:
Creating DAGs in Airflow for Lakehouse pipelines
-
Workshop 9.2:
Managing dependencies and retries
-
Workshop 9.3:
Scheduling workflows and integrating alerts
-
Workshop 9.4:
Parameterization and configuration
-
Workshop 9.5:
Using Prefect or Dagster as alternatives
-
Workshop 10.1:
Writing validation rules with Great Expectations
-
Workshop 10.2:
Detecting anomalies and outliers in data
-
Workshop 10.3:
Monitoring freshness, volume, and distribution
-
Workshop 10.4:
Building dashboards with Superset or Grafana
-
Workshop 10.5:
Incident management and resolution
-
Workshop 11.1:
Streaming ingestion with Kafka and Spark Structured Streaming
-
Workshop 11.2:
Building real-time dashboards and microservices
-
Workshop 11.3:
Window functions and watermarking
-
Workshop 11.4:
Use cases in IoT, finance, and e-commerce
-
Workshop 11.5:
Trade-offs of stream vs. micro-batch
-
Workshop 12.1:
Integrating ML models into pipelines
-
Workshop 12.2:
Feature engineering and feature stores
-
Workshop 12.3:
Using MLflow for tracking and deployment
-
Workshop 12.4:
Batch inference vs. real-time scoring
-
Workshop 12.5:
Lakehouse as a foundation for MLOps
-
Workshop 13.1:
Query optimization with Z-Ordering and caching
-
Workshop 13.2:
Choosing the right file size and format
-
Workshop 13.3:
Auto-scaling compute resources
-
Workshop 13.4:
Storage tiering and archival
-
Workshop 13.5:
Tracking cost per query and job
-
Workshop 14.1:
Implementing encryption at rest and in transit
-
Workshop 14.2:
Fine-grained access with row- and column-level security
-
Workshop 14.3:
Data masking and tokenization techniques
-
Workshop 14.4:
GDPR and HIPAA compliance measures
-
Workshop 14.5:
Audit trails and anomaly detection
-
Workshop 15.1:
CI/CD and version control integration
-
Workshop 15.2:
Deploying and monitoring end-to-end workflows
-
Workshop 15.3:
Handling failures, retries, and rollbacks
-
Workshop 15.4:
Documentation and handover practices
-
Workshop 15.5:
Case study: From ingestion to BI using the Lakehouse stack
