15 Topics
5 Days
75 Sessions
Programme Overview
Training Description
Who Should Attend
- Business Intelligence Professionals
- Database Administrators
- Software Engineers
- DevOps Engineers
- Cloud Architects
- Solutions Architects
- Machine Learning Engineers
- IT Managers
- Data Platform Engineers
- Big Data Professionals
- Technology Consultants
- Professionals transitioning into cloud-based data engineering
Session Objectives
- Understand the fundamentals of serverless computing and its benefits for data engineering
- Learn how to design serverless data pipelines for batch and streaming data
- Explore integration with cloud-native services for data ingestion, transformation, and output
- Build event-driven architectures that respond to data triggers efficiently
- Master monitoring, logging, and alerting in serverless environments
- Apply cost optimization strategies for serverless workflows
- Ensure scalability, fault-tolerance, and reliability in serverless pipelines
- Implement real-time analytics and data lake ingestion
- Enforce security and governance across serverless components
- Use infrastructure as code to manage and automate deployments
- Gain hands-on experience through practical labs and a capstone project
About the Course
The Serverless Data Engineering Training Course equips data and technology professionals with the knowledge and practical skills required to design, build, deploy, and manage modern data pipelines using serverless cloud technologies. The course explores how organizations can process, transform, store, and analyze data without the operational overhead of managing traditional servers and infrastructure.
Participants will learn how to design scalable and cost-efficient data architectures, build event-driven pipelines, automate data processing, implement data quality controls, and integrate cloud-based analytics services. The training also covers security, governance, monitoring, performance optimization, and cost management for serverless data platforms.
Through hands-on exercises, architecture case studies, and practical projects, participants will develop the ability to build reliable serverless data solutions suitable for modern analytics, business intelligence, machine learning, and real-time data applications.
Curriculum & Topics
15 Topics | 75 Sessions
-
Workshop 1.1:
Overview of serverless computing
-
Workshop 1.2:
Benefits and challenges for data processing
-
Workshop 1.3:
Key cloud provider offerings: AWS, GCP, Azure
-
Workshop 1.4:
Serverless vs. container-based architectures
-
Workshop 1.5:
Use cases in modern data engineering
-
Workshop 2.1:
AWS Lambda, Azure Functions, Google Cloud Functions
-
Workshop 2.2:
Function lifecycle and execution model
-
Workshop 2.3:
Writing and deploying serverless functions
-
Workshop 2.4:
Managing concurrency and limits
-
Workshop 2.5:
Using frameworks like Serverless Framework and SAM
-
Workshop 3.1:
Sources of events: file drops, API calls, queues
-
Workshop 3.2:
Triggering pipelines on data events
-
Workshop 3.3:
Designing event producers and consumers
-
Workshop 3.4:
Ensuring idempotency and retry strategies
-
Workshop 3.5:
Chaining and fan-out patterns
-
Workshop 4.1:
Using Kinesis, Pub/Sub, and Event Hubs
-
Workshop 4.2:
Handling structured and unstructured data
-
Workshop 4.3:
Real-time vs. batch ingestion
-
Workshop 4.4:
Validating and transforming incoming data
-
Workshop 4.5:
Integrating with APIs and third-party data sources
-
Workshop 5.1:
Using Amazon S3, Azure Blob, and GCS
-
Workshop 5.2:
Object lifecycle management and versioning
-
Workshop 5.3:
Event notifications on file uploads
-
Workshop 5.4:
Data partitioning and organization
-
Workshop 5.5:
Storage security and access policies
-
Workshop 6.1:
Stream processing with Lambda and Kinesis
-
Workshop 6.2:
Aggregation and windowing techniques
-
Workshop 6.3:
Handling late-arriving and out-of-order data
-
Workshop 6.4:
Delivering processed results to sinks
-
Workshop 6.5:
Combining stream and batch processing
-
Workshop 7.1:
AWS Step Functions, Azure Durable Functions
-
Workshop 7.2:
Defining state machines and workflows
-
Workshop 7.3:
Error handling and retries in orchestration
-
Workshop 7.4:
Chaining multi-step pipelines
-
Workshop 7.5:
Visualizing and monitoring execution
-
Workshop 8.1:
Querying with Athena, BigQuery, Synapse Serverless
-
Workshop 8.2:
ETL and ELT design in serverless context
-
Workshop 8.3:
Leveraging Glue and Dataflow for transformation
-
Workshop 8.4:
Schema inference and metadata cataloging
-
Workshop 8.5:
Using PySpark and SQL for data prep
-
Workshop 9.1:
Logging with CloudWatch, Stackdriver, Azure Monitor
-
Workshop 9.2:
Tracing and profiling functions
-
Workshop 9.3:
Creating metrics and dashboards
-
Workshop 9.4:
Handling cold starts and latency issues
-
Workshop 9.5:
Alerting and anomaly detection
-
Workshop 10.1:
Understanding billing for serverless workloads
-
Workshop 10.2:
Reducing invocations and execution time
-
Workshop 10.3:
Managing data transfer costs
-
Workshop 10.4:
Setting budgets and usage alerts
-
Workshop 10.5:
Comparing serverless vs. managed alternatives
-
Workshop 11.1:
IAM policies for least privilege
-
Workshop 11.2:
Securing secrets and API keys
-
Workshop 11.3:
Encrypting data at rest and in transit
-
Workshop 11.4:
Managing authentication and authorization
-
Workshop 11.5:
Reviewing serverless security best practices
-
Workshop 12.1:
Using Terraform and CloudFormation
-
Workshop 12.2:
Creating reproducible deployments
-
Workshop 12.3:
Versioning infrastructure and code
-
Workshop 12.4:
Managing environments and secrets
-
Workshop 12.5:
Automated testing and validation
-
Workshop 13.1:
Creating APIs using API Gateway and Lambda
-
Workshop 13.2:
Designing REST and GraphQL endpoints
-
Workshop 13.3:
Rate limiting and throttling
-
Workshop 13.4:
Integrating with data stores
-
Workshop 13.5:
Securing APIs with OAuth and tokens
-
Workshop 14.1:
Deploying lightweight models with serverless
-
Workshop 14.2:
Triggering predictions on data events
-
Workshop 14.3:
Integrating with SageMaker, Vertex AI, ML.NET
-
Workshop 14.4:
Streaming inference vs. batch inference
-
Workshop 14.5:
Scaling ML workloads with autoscaling functions
-
Workshop 15.1:
Building a full serverless data pipeline
-
Workshop 15.2:
Real-world case studies from e-commerce and finance
-
Workshop 15.3:
End-to-end implementation and demo
-
Workshop 15.4:
Troubleshooting and final optimization
-
Workshop 15.5:
Presentation and feedback session
