Apache Airflow Certification Course

NOTE: Participants need to bring laptops capable of running Docker, and must be comfortable with python and the command line in their operating system of choice. 2nd batch of material assumes knowledge of Apache Airflow, which people will get from the first course.

This is a 9-week, in-person course: The first 4 weeks will cover material for the Airflow Fundamentals certificate, while the last 5 weeks will cover material for the DAG Authoring certificate. Classes will be every Thursday from 6pm to 8pm,beginning on July 11th.

Course Description:

This 9-week comprehensive course is designed to prepare participants for Apache Airflow certifications, focusing on the Astronomer Airflow 101 and DAG Authoring certifications. Ideal for data engineers, data scientists, and IT professionals, this course offers a robust curriculum combining theoretical knowledge with practical application, ensuring a thorough understanding of Apache Airflow.

What is Apache Airflow?

Apache Airflow is an open-source platform used to programmatically author, schedule, and monitor workflows. It allows users to define workflows as Directed Acyclic Graphs (DAGs) of tasks, providing powerful tools to manage and automate data pipelines. Airflow is widely used for ETL processes, business operations, infrastructure management, and MLOps, offering a scalable and flexible solution for complex workflows.

Requirements:

• Proficiency in Python

• Comfort with Command Line Interface (CLI)

• Laptop

• Docker

Course Outline:

Weeks 1-4: Airflow 101 Certification

• Week 1: Introduction to Apache Airflow

• What is Airflow?

• What is a DAG?

• Example Use Cases

• ETL

• Business Operations

• Infrastructure Management

• MLOps

• Architecture/How It Works

• Week 2: Setting Up CLI and Exploring the User Interface

• Setting Up CLI

• User Interface

• Looking at DAGs

• Connections

• Variables

• DAG Overview

• Parameters

• Resources

• Academy

• Docs

• Registry

• Ask Astro

• Week 3: DAGs 101

• First DAG Walkthrough

• Scheduling

• Operators

• Connections

• Xcoms

• Variables

• Sensors

• Week 4: Scheduling Deep Dive

• All the Dates and Times

• Cron vs. Timedelta vs. Custom Timetables

• Idempotence and Determinism

• Backfilling and Catchup

Weeks 5-9: DAG Authoring Certification

• Week 5: Applying Business Logic

• Branching

• Grouping Tasks

• Trigger Rules

• Templating

• Task Priority

• Week 6: TaskFlow API

• Xcoms

• TaskFlow API

• Week 7: Advanced Dependency Management

• Branching

• Trigger Rules

• cross_downstream() and chain()

• ExternalTaskSensor

• TriggerDagRunOperator

• Week 8: Optimization

• Dynamic Tasks

• Parallelism, DAG_Concurrency, and Max_Active_Runs_Per_DAG

• Pools and Queues

• Async Operators

• Week 9: Reliability and Maintenance

• Retries

• SLAs

• Timeouts

Learning Outcomes:

By the end of this course, participants will:

• Have a solid understanding of Apache Airflow and its components.

• Be able to develop, deploy, and manage workflows using Airflow.

• Understand best practices and advanced techniques for DAG authoring.

• Be prepared to take the Astronomer Airflow 101 and DAG Authoring certification exams.

Embark on this journey to master Apache Airflow and achieve certification, enhancing your skills and career prospects in the data engineering field.

Host or Publisher The GEOINT Hub