AWS Data Engineering Project: End-to-End RDS to S3 with Glue

Build a realistic AWS data engineering project that ingests healthcare data from Amazon RDS PostgreSQL into an Amazon S3 data lake using AWS Glue.

You will work through the project as a data engineer—from understanding the business context and source systems to defining the pipeline contract, making design decisions, setting up AWS, implementing the solution, validating the output, and scheduling repeatable runs.

In this project, you will:

• Prepare the source: Set up a PostgreSQL database in Amazon RDS

• Build the Glue job: Ingest multiple source tables using AWS Glue and PySpark

• Load incrementally: Use watermarks to process new and updated records

• Store the data: Write the output to Amazon S3 as Parquet files

• Track each run: Add ingestion metadata for auditing and validation

• Catalog the data: Use AWS Glue Crawlers and the Glue Data Catalog

• Validate the output: Query the ingested data using Amazon Athena

• Schedule the pipeline: Use Glue Workflows and Triggers for repeatable runs

More than isolated service demonstrations

You will build the project inside your own AWS account using the provided source code, healthcare sample data, configuration files, and infrastructure setup scripts.

The focus is not only on getting the pipeline to run. You will understand why each AWS service is used, how the components work together, and how the pipeline handles new and updated source records during subsequent runs.

By the end of the course, you will have a complete AWS data engineering project that you can practice, adapt for your portfolio, and explain clearly during interviews.

This course contains a promotion.