- Introduction
-
Welcome 0 hr 1 min
-
Objectives
- Getting Started
-
Why SubDAGs? 0 hr 4 min
-
Creating our SubDAG 0 hr 4 min
-
Best Practices
- SubDAG in action
-
Combining the Tasks in SubDAG 0 hr 3 min
-
Communication between DAG/SubDAG and SubDAG 0 hr 5 min
-
Putting it all together 0 hr 7 min
-
Practice: Transferring Data from a Parent DAG/SubDAG to SubDAG
-
Additional Info
- Adios SubDAGs
-
Quick Overview of TaskGroups 0 hr 5 min
- Finishing Up
-
Quiz!
-
Resources
-
Summary
-
How was it
Airflow: SubDAGs
Learn how to use SubDAGs to write cleaner and more efficient DAGs.
Welcome! We're so glad you're here 😍
Subdags allow us to create reusable task patterns in DAGs.
Although it sounds exciting, it is a legacy feature and has been deprecated as it is quite complex to use and introduces performance issues.
For task grouping try using the Task Groups. But it's never a bad idea to know why subdags should be avoided. So let's find that out.
🎯Objectives
At the end of this course, you'll be able to:
- Combine tasks into SubDAG
- Perform repetitive tasks using SubDAGs
- Transfer data from Parent DAG/SubDAG to SubDAG
👥 Audience
Who should take this course:
- Data Engineers
- Data Analysts
- Software Engineers
Set aside 40 minutes to complete the course.
💻 Setup Requirements
You need to have the following:
- Docker and Docker compose on your computer (cf: get Docker)
- The Astro CLI
- Access to a web browser