15 Topics
5 Days
75 Sessions
Programme Overview
Training Description
Who Should Attend
Session Objectives
- Understand the fundamentals and architecture of Prefect
- Build and orchestrate robust, modular data workflows
- Integrate Prefect with cloud platforms and data tools
- Monitor, schedule, and retry tasks with precision
- Handle errors, state transitions, and conditional logic
- Optimize pipeline performance and resource usage
- Deploy workflows using Prefect Cloud or self-hosted options
- Apply Prefect to real-world use cases across industries
- Manage parameterized flows and dynamic tasks
- Build dashboards and observability for operational confidence
- Implement secure and scalable orchestration infrastructure
About the Course
This advanced training provides participants with practical skills to design, build, orchestrate, monitor, and optimize reliable data and automation workflows using Prefect. The course explores modern workflow orchestration concepts, including flows, tasks, deployments, scheduling, event-driven execution, retries, observability, and workflow management.Participants will learn how to transform complex data and business processes into maintainable, automated workflows. The training emphasizes production-ready orchestration, failure handling, dependency management, parameterization, concurrency, infrastructure configuration, monitoring, and integration with modern data engineering and cloud environments.
Curriculum & Topics
15 Topics | 75 Sessions
-
Workshop 1.1:
Understanding workflow orchestration in modern data environments
-
Workshop 1.2:
Overview of Prefect and its use cases
-
Workshop 1.3:
Prefect vs other orchestration tools (Airflow, Dagster, Luigi)
-
Workshop 1.4:
Setting up a local Prefect environment
-
Workshop 1.5:
Prefect 1.0 vs Prefect 2.0 features
-
Workshop 2.1:
Flows, tasks, states, and parameters
-
Workshop 2.2:
DAG-free flow design principles
-
Workshop 2.3:
Agents, deployments, and schedules
-
Workshop 2.4:
Execution layers and Prefect Cloud
-
Workshop 2.5:
Prefect API and orchestration backends
-
Workshop 3.1:
Writing tasks and defining flows
-
Workshop 3.2:
Parameterizing tasks and using inputs/outputs
-
Workshop 3.3:
Orchestrating sequential and parallel tasks
-
Workshop 3.4:
Using the @flow and @task decorators
-
Workshop 3.5:
Executing and monitoring flows
-
Workshop 4.1:
Setting retries and handling timeouts
-
Workshop 4.2:
Using triggers and conditional execution
-
Workshop 4.3:
Managing task state transitions
-
Workshop 4.4:
Implementing backoff strategies
-
Workshop 4.5:
Building fault-tolerant task chains
-
Workshop 5.1:
Time-based and event-based scheduling
-
Workshop 5.2:
Deploying flows using Prefect CLI and YAML
-
Workshop 5.3:
Version control and reproducible deployments
-
Workshop 5.4:
Configuring schedules with intervals, cron, and RRules
-
Workshop 5.5:
Triggering flows via API or Prefect Cloud UI
-
Workshop 6.1:
Enabling structured logging in flows
-
Workshop 6.2:
Capturing logs from failed tasks
-
Workshop 6.3:
Setting up Prefect notifications and alerts
-
Workshop 6.4:
Integrating with Slack, PagerDuty, and email
-
Workshop 6.5:
Using dashboards and log viewers
-
Workshop 7.1:
Task mapping for parallel execution
-
Workshop 7.2:
Building dynamic flows with loops and branching
-
Workshop 7.3:
Handling variable input lengths
-
Workshop 7.4:
Using result caching across mapped tasks
-
Workshop 7.5:
Reducing repetitive code with helper utilities
-
Workshop 8.1:
Integrating Prefect with Snowflake, DBT, BigQuery
-
Workshop 8.2:
Triggering external workflows using HTTP and CLI
-
Workshop 8.3:
Orchestrating file movement with S3, GCS, and FTP
-
Workshop 8.4:
Using Prefect with Airbyte, Great Expectations
-
Workshop 8.5:
Authenticating with secrets and credentials
-
Workshop 9.1:
Setting up Prefect Cloud workspace
-
Workshop 9.2:
Managing users, roles, and permissions
-
Workshop 9.3:
Deploying Prefect Server with Docker and Kubernetes
-
Workshop 9.4:
Comparing latency, cost, and control
-
Workshop 9.5:
Hybrid setups and scaling agents
-
Workshop 10.1:
Managing environments using .env files and profiles
-
Workshop 10.2:
Parameterizing flows for CI/CD pipelines
-
Workshop 10.3:
Registering flows with environment variables
-
Workshop 10.4:
Using Prefect blocks and secrets
-
Workshop 10.5:
Best practices for configuration management
-
Workshop 11.1:
Capturing and analyzing task errors
-
Workshop 11.2:
Building custom failure hooks
-
Workshop 11.3:
Retry logic for network and API failures
-
Workshop 11.4:
Creating fallback flows and safe exits
-
Workshop 11.5:
Building resilient pipelines for unreliable sources
-
Workshop 12.1:
Using Prefect Collections for reusable components
-
Workshop 12.2:
Creating reusable tasks across projects
-
Workshop 12.3:
Sharing flows and patterns with Prefect Hub
-
Workshop 12.4:
Designing flows with modular architecture
-
Workshop 12.5:
Testing flows using mocks and fixtures
-
Workshop 13.1:
Identifying bottlenecks in orchestration
-
Workshop 13.2:
Leveraging concurrency and async features
-
Workshop 13.3:
Managing flow runtime and execution priority
-
Workshop 13.4:
Flow and task memory management
-
Workshop 13.5:
Efficient logging and task completion metrics
-
Workshop 14.1:
Integrating Prefect into GitHub Actions, GitLab CI
-
Workshop 14.2:
Automating deployment via CI pipelines
-
Workshop 14.3:
Versioning flow deployments with Git
-
Workshop 14.4:
Building continuous test and deploy workflows
-
Workshop 14.5:
Using Docker for portable orchestration pipelines
-
Workshop 15.1:
Data ingestion from APIs and batch processing
-
Workshop 15.2:
Triggering ML model training pipelines
-
Workshop 15.3:
Orchestrating ETL and reporting dashboards
-
Workshop 15.4:
Final hands-on project with full pipeline deployment
-
Workshop 15.5:
Review, Q&A, and certification assessment
