- Introduction
-
Welcome!
- Data Orchestration Fundamentals
-
Why Orchestration?
- Airflow for Data Orchestration
-
Introduction to Airflow
-
Who uses Airflow and for what?
-
Considerations of using Airflow
- Demystifying Airflow
-
Running Airflow
-
How does Airflow Work?
- Wrap Up
-
Review
-
Final Quiz
-
How was it?
Introduction to Orchestration and Airflow
An overview of the world of data orchestration and Apache Airflow!
Welcome to the Introduction to Orchestration and Airflow Module!
In this Module, we'll take a comprehensive look at data orchestration, exploring its definition and historical progression to understand how modern solutions like Apache Airflow came about. We'll then delve into Airflow, a powerful tool for orchestrating complex workflows, by examining its key characteristics, the use cases it can be used for, and the personas that use it. We'll also explore the practical considerations for running Airflow in different scenarios and stages of team growth. Finally, we'll go through how data pipelines are built in Airflow using Directed Acyclic Graphs (DAGs), Tasks, and Operators.
🎯 Learning Objectives
At the end of this module, you'll be able to:
- Define data orchestration
- Visualize the historical journey of modern data orchestration
- Define what Apache Airflow is
- Identify the key characteristics of Apache Airflow
- Identify the key personas that use Airflow and for what purpose
- Identify critical use cases for Airflow
- Identify considerations for using Airflow for specific use-cases
- Identify how Airflow is run at each stage of a growing team
- Define three key Airflow terms: DAG, Task, and Operator
- Visualize how a DAG is structured using Tasks and Operators