- Introduction
-
Welcome! 0 hr 2 min
- Why an ExternalPythonOperator?
-
The need for External Python Operator 0 hr 3 min
-
The new way of resolving Dependency Conflicts 🎥 0 hr 2 min
- Lights EPO Action!
-
Prepackaging your VirtualEnvironment 0 hr 5 min
-
Practice: New Dependency New Environment
-
Passing Xcoms between ExternalPythonOperator 0 hr 6 min
- More on EPO
-
Difference between EPO and KPO and VEO 0 hr 3 min
- Finishing Up
-
Quiz!
-
Resources
-
Summary
-
How was it?
Airflow: The ExternalPythonOperator
Learn how to run python tasks in a safe virtual environment.
Welcome! We're so glad you're here 😍
Want to run some of your tasks with a different set of Python libraries than the other tasks (and maybe other than the main Airflow environment) without having any conflicts?
Or maybe you are tired of waiting for the virtual environment to be created while using the VirtualEnvOperator.
Also maybe you don't have a Kubernetes Cluster to use a KubernetesPodOperator.
Well the good news is you can use the ExternalPythonOperator to execute Python callable inside a pre-defined virtual environment that is ideally preinstalled in the airflow environment without conflicting with other packages. It also allows you to use a different version of airflow.
🎯Objectives
At the end of this course, you'll be able to:
- Prepackage a virtual environment in airflow.
- Run tasks with a different set of Python packages using the ExternalPythonOperator .
- Use the ExternalPythonOperator to execute Python callable inside a pre-defined virtual environment.
- Resolve dependency conflicts for tasks using different version of tools.
👥 Audience
Who should take this course:
- Data Engineers
- Data Analysts
- Data Scientists
- Software Engineers
Set aside 35 minutes to complete the course.
💻 Setup Requirements
You need to have the following:
- Docker and Docker compose on your computer (cf: get Docker)
- The Astro CLI
- An internet connection with permission to access GitHub and download Python dependencies
- Access to a web browser