Airflow Project: Dynamic Task Mapping with S3

About this course

Welcome! We're so glad you're here 😍

This Project shows an example implementation of sorting files in an S3 bucket into two different buckets based on logic involving the content of the files using dynamic task mapping method introduced in Airflow 2.4.

Learn how to create tasks dynamically without knowing the number of files in an S3 bucket before hand.

Also learn how taskflow api makes it easy to create dependencies and transfer data between tasks.

You can also read more about the project at https://registry.astronomer.io/dags/2-4-example-dag-expand-kwargs

At the end of this course, you'll be able to:

Create Connection for AWS
Use the AWS Provider Package to Interact with AWS-S3
Use Dynamic Task Mapping to Generate Dynamic Tasks
Transfer data from a source S3 bucket to the destination bucket.

👥 Audience

Who should take this course:

Data Engineers
Data Analysts
Software Engineers

Set aside 35 minutes to complete the course.

💻 Setup Requirements

You need to have the following:

Docker and Docker compose on your computer (cf: get Docker)
The Astro CLI
Access to a web browser
AWS Account

Curriculum0 hr 25 min

Introduction
Welcome! 0 hr 1 min
Objectives
Setting the Stage
Practice: Prerequisites
Setting up Astro and AWS Connection 0 hr 3 min
Time to move some files!
Exploring the AWS Provider 0 hr 5 min
Using AWS Provider- Operators 0 hr 6 min
Creating Tasks Dynamically 0 hr 6 min
DAG in action! 0 hr 5 min
The DAG file for you
Finishing Up!
Quiz!
Resources
Summary
How was it?

About this course