Practical Apache Airflow

Develop Data Pipelining & Workflow. Build, schedule and monitor Data Pipelines using Apache Airflow in Python

Ratings 4.01 / 5.00

What You Will Learn!

Airflow, Developing Data Pipeline, Python, Pandas
Hands on data pipeline development and comparison with other technology
Installation, Configuration of Airflow
DAG's, Creating workflow, Operators, Tasks, dependency management, Hooks, Connections
Different Executors like Local, Celery and Sequential and differences
Airflow Architecture in detail
Advanced concepts like Xcoms, Branching, Variables and DAG Chaining
Authentication and Log storage to S3
Airflow on Docker

Description

Data engineering is a field that can be thought as a superset of business intelligence and data warehousing which brings more elements from software engineering. The reason data engineering exists today is because companies have massive treasure troves of data, but to provide value the data must be extracted. Data engineering provides the toolbox and is how we make sense of that data quickly and effectively.

When it comes to managing data collection, munging and consumption, data pipeline frameworks play a significant role and with the help of Apache Airflow, task of creating data pipeline is not only easy but its actually fun. Originated from AirBnb, Airflow soon became part of the very core of their tech stack.

The data infrastructure ecosystem has yet to show any sign of converging into something more manageable. It seems like we’re still in a huge phase of expansion where every new day bring new distributed database, new frameworks, new libraries and new teammates. As these systems get more complicated and evolve rapidly, it becomes even more important to have something like Apache Airflow that brings everything together in a sane place where every little piece of the puzzle can be orchestrated properly with sane APIs.

So in this course we will be learning as how to reach feature completeness with this amazing orchestration tool called Apache Airflow. You will not only learn to setup the environment but also learn how to create workflow pipeline with real world example so don't wait and sign-up today and get started.

Looking forward to seeing you in this course!