- Introduction
-
Welcome! 0 hr 1 min
-
Why Dynamic Task Mapping? 0 hr 2 min
- Expand()
-
Map your tasks 0 hr 2 min
-
The true power of Expand() 0 hr 2 min
-
Define limits 0 hr 1 min
- Partial()
-
Define constant parameters 0 hr 2 min
-
Partial with Taskflow and non-Taskflow Operator
- Classic Operators
-
Mapping with a non-taskflow operator 0 hr 2 min
-
Practice: From mapped tasks to a classic operator
- Finishing up...
-
Quiz!
-
Summary
-
How was it?
Airflow: Dynamic Task Mapping
Learn to create tasks based on current data using Dynamic Task Mapping.
Welcome! We're so glad you're here 😍
One of the most anticipated features of Airflow introduced in version 2.3 was the Dynamic Task Mapping which is a way for a workflow to create a number of tasks at runtime based upon current data, rather than the DAG author having to know in advance how many tasks would be needed.
Here you will be able to differentiate between partial and the expand method which right before executing a mapped task creates n copies of the task, one for each input.
Also, let's see the difference between dynamic tasks and dynamic task mapping.
🎯Objectives
At the end of this course, you'll be able to:
- Differentiate partial and expand
- Differentiate task mapping and dynamic tasks
- Create tasks at runtime based upon current data
👥 Audience
Who should take this course:
- Data Engineers
- Data Analysts
- Data Scientists
Set aside 18 minutes to complete the course.
💻 Setup Requirements
You need to have the following:
- Docker and Docker compose on your computer (cf: get Docker)
- The Astro CLI
- Access to a web browser