5 Course Combo Package To Kick-Start Career in Data Engineering with 7 Projects on AWS/Azure/GCP

✅ Project: Spotify Data Pipeline using AWS

👉🏻 Phase 1: In the Python for DE course we used simple Python to build this entire pipeline using AWS Lambda, S3, Glue Crawler, Athena

👉🏻 Phase 2: In the Data Warehouse for the DE course we replaced the load part with Snowpipe and Snowflake for analytics

👉🏻 Phase 3: In Apache Spark for DE course we are going to replace Lambda Part for transformation using AWS Glue ETL and write our pipeline using Apache Spark.

👉🏻 Phase 4: In Apache Airflow for the DE course we will replace the Lambda part and transformation using Apache Airflow to build a complete data pipeline

👉🏻 Phase 5: In the DE course we will orchestrate the entire pipeline using Apache Airflow, we will use Spark, Lambda, and Glue external but use Airflow or run it.

✅ Project: Azure Data Engineering using Databricks

If you are someone who wants to learn about Azure Cloud and Spark then this project is a goldmine, you will get to learn so many things from data ingestion, automating pipelines using Data factory, writing Spark code in data bricks

- Data ingestion in Azure
- Building pipeline using Azure Data factory
- How to write production-ready Spark Code
- How to build Delta Lake using Medallion Architecture in Databricks
- Writing SQL for Analytics

✅ Project: RealTime Data Pipeline using Apache Nifi and Snowflake Stream

If you want to learn the basics of docker and how to build a real-time data pipeline then this project will teach you everything

- Basics of docker
- Deploying Nifi and Creating a pipeline
- Snowpipe on Snowflake
- How to build SCD 1,2 & 3 using Snowflake

✅ Project: Crypto Data Pipeline using Google Cloud, Composer Airflow, BigQuery

You will learn how to integrate real-time crypto data using API and build incremental data pipelines using Google Cloud and Airflow

- Fetch real-time data from Crypto API
- Build a complete pipeline using Apache Airflow
- Deploy DAGs on Cloud Composer
- Build Data Warehouse on Google BigQuery
- Automate Pipeline to run every 10min
- Build visualization for final insights