GCP Data Engineering Project: MySQL to BigQuery via Dataflow

Build a realistic GCP data engineering project that ingests retail data from Cloud SQL for MySQL into BigQuery using Google Cloud Dataflow.

You will work through the project as a data engineer—from understanding the business context and source systems to defining the pipeline contract, designing incremental loads, setting up Google Cloud resources, implementing the solution, validating the output, and orchestrating repeatable runs.

In this project, you will:

• Prepare the source: Set up retail source tables in Cloud SQL for MySQL

• Configure networking: Connect Cloud SQL and Dataflow using private networking

• Build the ingestion pipeline: Move data from MySQL into BigQuery using Dataflow

• Load data incrementally: Use watermarks to process new and updated records

• Create BigQuery tables: Build staging and raw tables with partitioning and clustering

• Merge incremental data: Use BigQuery MERGE procedures to update target tables

• Orchestrate the pipeline: Deploy and schedule workflows using Cloud Composer and Apache Airflow

• Validate the results: Query the loaded data and verify initial and incremental runs in BigQuery

More than isolated service demonstrations

You will build the project in your own Google Cloud account using the provided source code, sample retail data, configuration files, SQL scripts, and infrastructure setup scripts.

The focus is not only on getting the pipeline to run. You will understand why each Google Cloud service is used, how Cloud SQL, Dataflow, BigQuery, and Cloud Composer work together, and how the pipeline handles new and updated source records during subsequent runs.

By the end of the course, you will have a complete GCP data engineering project that you can practice, adapt for your portfolio, and explain clearly in interviews.

This course contains a promotion.