12 Topics
5 Days
60 Sessions
Programme Overview
Training Description
Who Should Attend
- Cloud Data Engineers
- Data Warehouse Developers
- Data Scientists working with large-scale datasets
- Machine Learning Engineers
- BI and Analytics Professionals
- Data Integration Specialists
- Database Administrators
- Data Governance and Data Management Professionals
- MLOps Engineers
- DevOps Engineers supporting data platforms
- Technology and Data Platform Managers
Session Objectives
- Understand the architecture and core concepts of Delta Lake
- Explore ACID transactions and their role in reliable data lakes
- Implement data versioning and time travel for auditing and rollback
- Learn schema enforcement and evolution techniques
- Manage large-scale data pipelines using Delta Lake and Apache Spark
- Apply Delta Lake to batch and streaming workloads seamlessly
- Enable data quality, integrity, and reliability across data pipelines
- Monitor and optimize performance of Delta Lake tables
- Integrate Delta Lake with cloud-native data services and tools
- Leverage Delta Lake for real-time analytics and machine learning workflows
- Implement compliance-ready versioned data pipelines
About the Course
This advanced training course provides participants with the practical knowledge required to design, implement, and manage Delta Lake and modern lakehouse architectures for reliable, scalable, and high-performance data engineering. The course explores how Delta Lake enhances data lakes with capabilities such as ACID transactions, schema enforcement, schema evolution, time travel, data versioning, and reliable batch and streaming processing.
Participants will learn how Delta Lake integrates with Apache Spark to support large-scale data ingestion, transformation, analytics, and machine learning workloads. The course covers lakehouse architecture, Delta tables, transaction logs, data versioning, partitioning, optimization, schema management, streaming pipelines, and data lifecycle management.
Through practical exercises and enterprise scenarios, participants will learn how to build reliable lakehouse data pipelines, manage changing datasets, recover previous versions of data, optimize storage and query performance, and integrate Delta Lake into modern data engineering environments. The course also addresses governance, security, data quality, and best practices for production-scale lakehouse deployments.
Curriculum & Topics
12 Topics | 60 Sessions
-
Workshop 1.1:
Understanding the limitations of traditional data lakes
-
Workshop 1.2:
Key features of Delta Lake and its advantages
-
Workshop 1.3:
Comparison of Parquet, Delta, and other formats
-
Workshop 1.4:
Overview of open-source Delta Lake architecture
-
Workshop 1.5:
Use cases for versioned, reliable data lakes
-
Workshop 2.1:
Delta Lake components: transaction log, metadata, data files
-
Workshop 2.2:
Internals of the Delta log and checkpointing
-
Workshop 2.3:
How Delta Lake maintains consistency in distributed systems
-
Workshop 2.4:
File formats and directory structures
-
Workshop 2.5:
Integration with cloud storage (S3, ADLS, GCS)
-
Workshop 3.1:
Role of ACID properties in big data workloads
-
Workshop 3.2:
Write operations and multi-writer support
-
Workshop 3.3:
Commit protocol and transaction handling
-
Workshop 3.4:
Conflict detection and resolution
-
Workshop 3.5:
Read consistency and isolation levels
-
Workshop 4.1:
Enforcing schemas at write time
-
Workshop 4.2:
Preventing data corruption through column-level checks
-
Workshop 4.3:
Managing schema changes over time
-
Workshop 4.4:
AutoMerge and column mapping
-
Workshop 4.5:
Best practices for schema compatibility
-
Workshop 5.1:
Enabling and accessing historical versions of data
-
Workshop 5.2:
Using VERSION AS OF and TIMESTAMP AS OF queries
-
Workshop 5.3:
Auditing and rollback of data changes
-
Workshop 5.4:
Comparing snapshots and diffing datasets
-
Workshop 5.5:
Managing storage cost for time travel
-
Workshop 6.1:
Delta as a sink and source for streaming jobs
-
Workshop 6.2:
Writing streaming data to Delta tables
-
Workshop 6.3:
Reading Delta tables in structured streaming
-
Workshop 6.4:
Micro-batch vs. continuous processing
-
Workshop 6.5:
Handling late-arriving data and watermarking
-
Workshop 7.1:
File size optimization and compaction
-
Workshop 7.2:
Z-ordering and data skipping
-
Workshop 7.3:
Auto optimize and auto compaction
-
Workshop 7.4:
Vacuuming stale data files
-
Workshop 7.5:
Indexing strategies for faster queries
-
Workshop 8.1:
Tracking changes across Delta operations
-
Workshop 8.2:
Capturing metadata changes
-
Workshop 8.3:
Integration with data catalogs and governance tools
-
Workshop 8.4:
Using Delta logs for auditing
-
Workshop 8.5:
Visualization of lineage with Spark UI
-
Workshop 9.1:
Creating and managing Delta tables
-
Workshop 9.2:
MERGE, UPDATE, DELETE, and UPSERT operations
-
Workshop 9.3:
Partitioning strategies for Delta tables
-
Workshop 9.4:
DML command optimizations
-
Workshop 9.5:
Altering and renaming Delta tables
-
Workshop 10.1:
Using Delta Lake APIs with PySpark, Scala, and SQL
-
Workshop 10.2:
SparkSession and Delta-specific options
-
Workshop 10.3:
Caching and checkpointing with Delta
-
Workshop 10.4:
Delta operations in Spark notebooks
-
Workshop 10.5:
Debugging and logging Delta jobs
-
Workshop 11.1:
Running Delta Lake on Databricks
-
Workshop 11.2:
Delta Lake OSS with Apache Spark
-
Workshop 11.3:
Differences between open-source and managed Delta
-
Workshop 11.4:
Compatibility with MLflow and Koalas
-
Workshop 11.5:
Deployment scenarios and limitations
-
Workshop 12.1:
Integrating Delta Lake with Azure Synapse, AWS Glue, and GCP BigLake
-
Workshop 12.2:
Delta Sharing protocol for secure data exchange
-
Workshop 12.3:
Connecting Delta Lake to Power BI, Tableau, and Looker
-
Workshop 12.4:
Storage cost and performance optimization in the cloud
-
Workshop 12.5:
Managing access control and permissions
