Cloud Data Engineering with AWS and Azure Training Course

Share this course

Duration

5 Days

Course Overview

Engineers moving pipelines off local servers spend five days building, automating and securing them on the two largest public clouds. Day one covers elasticity and serverless design, prepares AWS and Azure accounts and manages S3 and Blob Storage. Day two works in AWS Glue with crawlers, the Data Catalog, ETL jobs and PySpark scripts. Day three builds pipelines and data flows in Azure Data Factory. Day four orchestrates multi-step pipelines with triggers, tunes cost and speed, and adds access controls and encryption. Day five reviews cases from finance, retail and healthcare, deploys a full pipeline and looks at new tools.

Format of Training

  • Trainer-led builds that run the same task in AWS and then in Azure
  • Storage lab on S3 buckets and Blob Storage, plus PySpark scripting in Glue
  • Demos of an automated Glue pipeline and an end-to-end Data Factory flow
  • Cost and security exercises, then a group business problem on Day 5

Course Objectives

  1. Explain how cloud data engineering is structured and why it works.
  2. Run ETL workflows in AWS Glue and Azure Data Factory.
  3. Store and manage data in AWS S3 and Azure Blob Storage.
  4. Automate and tune data pipelines.
  5. Connect cloud pipelines to analytics platforms.
  6. Apply good practice for security, scale and cost control in the cloud.
  7. Design and deploy data solutions in the cloud.

Prerequisites

Course Outline

Day 1: Introduction to Cloud Data Engineering

Session 1: Fundamentals of Cloud-Based Data Engineering

  • What cloud platforms are and why teams use them
  • Main ideas: elasticity, scalability and serverless design

Session 2: AWS and Azure Ecosystems

  • A first look at AWS Glue, S3 and Azure Data Factory
  • Preparing AWS and Azure environments

Session 3: Data Storage in the Cloud

  • How AWS S3 and Azure Blob Storage work
  • Lab: managing cloud storage buckets and keeping them secure

Day 2: Building Data Pipelines with AWS Glue

Session 1: Introduction to AWS Glue

  • The parts of Glue: Crawlers, Data Catalog and ETL Jobs
  • Setting up AWS Glue for a simple ETL job

Session 2: Transforming Data with Glue

  • Reshaping data with Python and PySpark
  • Writing a transformation script and running it

Session 3: Automating Glue Workflows

  • Scheduling Glue jobs and keeping an eye on them
  • Demo: an automated data pipeline from start to end

Day 3: Designing Workflows with Azure Data Factory

Session 1: Introduction to Azure Data Factory

  • The parts and design of Azure Data Factory
  • Setting up a data pipeline in Azure Data Factory

Session 2: Data Transformation in Azure

  • Building data flows and transformations
  • Reshaping data in Azure Data Factory

Session 3: Integrating Azure Data Factory with Other Tools

  • Linking to Azure Blob Storage, databases and APIs
  • Demo: an end-to-end pipeline

Day 4: Advanced Cloud Data Engineering

Session 1: Orchestrating Complex Pipelines

  • Triggers, event-based workflows and conditional logic
  • Running a multi-step pipeline in AWS and Azure

Session 2: Optimizing Performance and Costs

  • Ways to tune pipelines and keep costs down
  • Demo: reading pipeline performance figures

Session 3: Securing Cloud Data Workflows

  • Putting access controls, encryption and compliance in place
  • Locking down a cloud data pipeline

Day 5: Real-World Applications and Deployment

Session 1: Real-World Use Cases

  • Cases from finance, retail and healthcare
  • Group task: solving a business problem with cloud data engineering

Session 2: Deploying Cloud Data Solutions

  • Good practice for deploying and monitoring
  • Putting a complete data pipeline live in AWS or Azure

Session 3: Future Trends in Cloud Data Engineering

  • New ideas and tools
  • Talk: keeping up as cloud platforms change

Book This Course

Customise This Course for Your Team

Teams committed to a single cloud can spend all five days in AWS or in Azure, with the freed time going to deeper labs. The Day 4 and Day 5 exercises can also use a pipeline your engineers are planning right now. Tell us which cloud you run on and we will set the schedule.
Need help with the right course to choose?

Explore more opportunities

Introduction to Data Engineering Training Course

ETL Processes for Data Pipelines Training Course

Data Modeling for Analytics Training Course

Building Scalable Data Pipelines with Apache Airflow Training Course

Data Warehousing with Snowflake and Redshift Training Course

Real-Time Data Processing with Kafka Training Course

Cloud Data Engineering with AWS and Azure Training Course