Cloud Data Engineering with AWS and Azure Training Course
Course Overview
Engineers moving pipelines off local servers spend five days building, automating and securing them on the two largest public clouds. Day one covers elasticity and serverless design, prepares AWS and Azure accounts and manages S3 and Blob Storage. Day two works in AWS Glue with crawlers, the Data Catalog, ETL jobs and PySpark scripts. Day three builds pipelines and data flows in Azure Data Factory. Day four orchestrates multi-step pipelines with triggers, tunes cost and speed, and adds access controls and encryption. Day five reviews cases from finance, retail and healthcare, deploys a full pipeline and looks at new tools.
Format of Training
- Trainer-led builds that run the same task in AWS and then in Azure
- Storage lab on S3 buckets and Blob Storage, plus PySpark scripting in Glue
- Demos of an automated Glue pipeline and an end-to-end Data Factory flow
- Cost and security exercises, then a group business problem on Day 5
Course Objectives
- Explain how cloud data engineering is structured and why it works.
- Run ETL workflows in AWS Glue and Azure Data Factory.
- Store and manage data in AWS S3 and Azure Blob Storage.
- Automate and tune data pipelines.
- Connect cloud pipelines to analytics platforms.
- Apply good practice for security, scale and cost control in the cloud.
- Design and deploy data solutions in the cloud.
Prerequisites
- A basic grasp of data engineering ideas
- Some Python or SQL skills
- No AWS or Azure experience needed
- An interest in data workflows that run in the cloud
Course Outline
Day 1: Introduction to Cloud Data Engineering
Session 1: Fundamentals of Cloud-Based Data Engineering
- What cloud platforms are and why teams use them
- Main ideas: elasticity, scalability and serverless design
Session 2: AWS and Azure Ecosystems
- A first look at AWS Glue, S3 and Azure Data Factory
- Preparing AWS and Azure environments
Session 3: Data Storage in the Cloud
- How AWS S3 and Azure Blob Storage work
- Lab: managing cloud storage buckets and keeping them secure
Day 2: Building Data Pipelines with AWS Glue
Session 1: Introduction to AWS Glue
- The parts of Glue: Crawlers, Data Catalog and ETL Jobs
- Setting up AWS Glue for a simple ETL job
Session 2: Transforming Data with Glue
- Reshaping data with Python and PySpark
- Writing a transformation script and running it
Session 3: Automating Glue Workflows
- Scheduling Glue jobs and keeping an eye on them
- Demo: an automated data pipeline from start to end
Day 3: Designing Workflows with Azure Data Factory
Session 1: Introduction to Azure Data Factory
- The parts and design of Azure Data Factory
- Setting up a data pipeline in Azure Data Factory
Session 2: Data Transformation in Azure
- Building data flows and transformations
- Reshaping data in Azure Data Factory
Session 3: Integrating Azure Data Factory with Other Tools
- Linking to Azure Blob Storage, databases and APIs
- Demo: an end-to-end pipeline
Day 4: Advanced Cloud Data Engineering
Session 1: Orchestrating Complex Pipelines
- Triggers, event-based workflows and conditional logic
- Running a multi-step pipeline in AWS and Azure
Session 2: Optimizing Performance and Costs
- Ways to tune pipelines and keep costs down
- Demo: reading pipeline performance figures
Session 3: Securing Cloud Data Workflows
- Putting access controls, encryption and compliance in place
- Locking down a cloud data pipeline
Day 5: Real-World Applications and Deployment
Session 1: Real-World Use Cases
- Cases from finance, retail and healthcare
- Group task: solving a business problem with cloud data engineering
Session 2: Deploying Cloud Data Solutions
- Good practice for deploying and monitoring
- Putting a complete data pipeline live in AWS or Azure
Session 3: Future Trends in Cloud Data Engineering
- New ideas and tools
- Talk: keeping up as cloud platforms change
Book This Course
Customise This Course for Your Team
Teams committed to a single cloud can spend all five days in AWS or in Azure, with the freed time going to deeper labs. The Day 4 and Day 5 exercises can also use a pipeline your engineers are planning right now. Tell us which cloud you run on and we will set the schedule.
Need help with the right course to choose?
Explore more opportunities
- Duration: 3 Days
- 4.5 Ratings
Building Scalable Data Pipelines with Apache Airflow Training Course
- Duration: 3 Days
- 4.5 Ratings
Data Warehousing with Snowflake and Redshift Training Course
- Duration: 3 Days
- 4.5 Ratings
Real-Time Data Processing with Kafka Training Course