ETL Processes for Data Pipelines Training Course

Share this course

Duration

2 Days

Course Overview

Data staff who move records between systems learn over two days how to pull data out, reshape it and land it where analysts can use it. Day one explains where ETL sits in a data workflow and the common integration problems, connects to databases, APIs and file systems, and cleans, enriches and aggregates data in Apache NiFi or Talend. Day two loads results into relational databases and cloud platforms, schedules jobs with error handling and automates a full pipeline. Cases from finance, e-commerce and healthcare lead into a group task where teams design and present an ETL solution.

Format of Training

  • Trainer demos that build one pipeline in Apache NiFi or Talend, step by step
  • Exercises that connect to a database, an API and a set of flat files
  • Job scheduling and error handling practice on the shared pipeline
  • Team design task on a business case, presented back to the group

Course Objectives

  1. Explain how ETL works and why it matters in data engineering.
  2. Extract data from structured and unstructured sources.
  3. Transform data so it meets business and reporting needs.
  4. Load data into databases, data warehouses or cloud platforms.
  5. Automate ETL workflows with tools such as Apache NiFi or Talend.
  6. Solve practical data problems with ETL methods.
  7. Design and tune data pipelines that can grow with demand.

Prerequisites

Course Outline

Day 1: Fundamentals of ETL Processes

Session 1: Introduction to ETL Concepts

  • What ETL is and where it fits in a data workflow
  • Typical problems when joining data from many systems

Session 2: Extracting Data from Multiple Sources

  • Connecting to databases, APIs and file systems
  • Pulling data from structured and unstructured sources

Session 3: Data Transformation Techniques

  • Cleaning, enriching and aggregating data ready for analysis
  • Running transformations in Apache NiFi or Talend

Day 2: Building and Automating ETL Pipelines

Session 1: Loading Data into Target Systems

  • Ways to load data into relational databases and cloud platforms
  • Setting up an ETL pipeline to handle loading

Session 2: Automating and Monitoring ETL Workflows

  • Scheduling ETL jobs and catching errors
  • Automating a full ETL pipeline in Apache NiFi or Talend

Session 3: Real-World Applications and Optimization

  • Cases from finance, e-commerce and healthcare
  • Group task: designing an ETL solution for a business case and presenting it

Book This Course

Customise This Course for Your Team

We can point the extraction exercises at the kinds of systems your team already pulls from, such as your own databases or file drops. If you have chosen NiFi or Talend already, all labs can run in that one tool. Tell us about your sources and targets and we will build the labs around them.
Need help with the right course to choose?

Explore more opportunities

Introduction to Data Engineering Training Course

ETL Processes for Data Pipelines Training Course

Data Modeling for Analytics Training Course

Building Scalable Data Pipelines with Apache Airflow Training Course

Data Warehousing with Snowflake and Redshift Training Course

Real-Time Data Processing with Kafka Training Course

ETL Processes for Data Pipelines Training Course