AI-Powered Big Data Analytics: Working with Large Datasets Training Course

Share this course

Duration

5 Days

Course Overview

Data engineers and analysts whose datasets have outgrown a single machine learn to store, process and model them with Hadoop, Apache Spark and TensorFlow. Day one sets up HDFS, MapReduce and Kafka ingestion. Day two runs Spark SQL, MLlib and Spark Streaming, including a fraud detection case. Day three builds and serves deep learning models with TensorFlow and TensorFlow Serving. Day four links big data to Power BI and Tableau for churn and sales forecasts, and day five closes with privacy, governance and a group capstone. Graduates can design a pipeline from raw ingestion to a deployed model that answers a business question.

Format of Training

  • Trainer walkthroughs of cluster architecture, each followed by a build on Hadoop, Spark or TensorFlow
  • Five days of labs that grow one pipeline from Kafka ingestion to a served model
  • Industry cases covering healthcare analytics, fraud detection, predictive maintenance and supply chains
  • Team reviews of each lab result and a final capstone presentation

Course Objectives

  1. Explain how big data analytics works and where AI fits into it.
  2. Store and manage large datasets in the Hadoop Distributed File System (HDFS).
  3. Run distributed data processing jobs in Apache Spark.
  4. Apply AI and machine learning models to big data for predictive work.
  5. Use TensorFlow to process data and train AI models at scale.
  6. Analyze real-time data streams for business use cases.
  7. Tune big data workflows for speed, scale and efficient use of resources.

Prerequisites

Course Outline

Day 1: Introduction to Big Data and AI Integration

Session 1: Understanding Big Data and AI

  • Defining big data through the 5 V’s: Volume, Velocity, Variety, Veracity, Value
  • Where AI adds value in big data analytics
  • How big data is built: data lakes, data warehouses and distributed systems

Session 2: Big Data Ecosystem Overview

  • The Hadoop family of tools: HDFS, MapReduce and YARN
  • Apache Spark and its in-memory processing model
  • How AI, machine learning and big data frameworks support each other

Session 3: Hands-on Lab: Setting Up the Big Data Environment

  • Installing Hadoop and Spark on a laptop or in the cloud
  • First steps with HDFS for storing and managing large datasets
  • Running simple MapReduce jobs to process data

Session 4: Data Ingestion and ETL Processes

  • Batch ingestion compared with real-time ingestion
  • How ETL (Extract, Transform, Load) pipelines fit into big data work
  • Ingestion tools: Apache Kafka, Flume and Sqoop

Session 5: Hands-on Lab: Data Ingestion with Kafka and HDFS

  • Configuring Kafka to stream data in real time
  • Loading large datasets into HDFS ready for processing
  • Running simple ETL steps on big data

Day 2: Distributed Data Processing with Apache Spark

Session 1: Introduction to Apache Spark

  • How Spark is built: RDDs, DataFrames and Datasets
  • The main Spark parts: Spark SQL, Spark Streaming, MLlib and GraphX
  • Why Spark outperforms classic MapReduce

Session 2: Hands-on Lab: Working with Spark Core and Spark SQL

  • Preparing a Spark workspace with PySpark
  • Loading, reshaping and querying big datasets in Spark SQL
  • Distributed processing with RDDs and DataFrames

Session 3: Machine Learning with Spark MLlib

  • Spark MLlib as a tool for machine learning at scale
  • Supervised methods: linear regression and classification models
  • Unsupervised methods: clustering and dimensionality reduction

Session 4: Hands-on Lab: Building Machine Learning Models with Spark MLlib

  • Training regression and classification models on large datasets
  • Measuring model results across distributed data
  • Tuning hyperparameters and improving models in Spark

Session 5: Real-Time Data Processing with Spark Streaming

  • Real-time analytics and event-driven design explained
  • Configuring Spark Streaming to take in live data
  • Windowed operations, transformations and aggregations

Session 6: Hands-on Lab: Real-Time Analytics with Spark Streaming

  • Handling live data streams coming from Kafka
  • Real-time analysis shown on dashboard visuals
  • Case study: catching fraud in real time with Spark Streaming

Day 3: AI and Deep Learning for Big Data with TensorFlow

Session 1: Introduction to TensorFlow for Big Data Analytics

  • How TensorFlow is put together and its main components
  • TensorFlow compared with older machine learning libraries for big data
  • Running TensorFlow across distributed systems

Session 2: Hands-on Lab: Setting Up TensorFlow for Big Data Processing

  • Installing TensorFlow and TensorFlow Extended (TFX)
  • Loading big datasets and preprocessing them in TensorFlow
  • Simple neural network models for big data use cases

Session 3: Deep Learning Architectures for Big Data

  • CNNs, RNNs and the big data problems they suit
  • Designing deep neural networks for classification and regression
  • A first look at distributed training in TensorFlow

Session 4: Hands-on Lab: Deep Learning with TensorFlow

  • Deep learning models for image and text analysis
  • Training a model across a large distributed dataset
  • Checking results with TensorFlow metrics

Session 5: Transfer Learning and Model Deployment

  • Reusing trained models through transfer learning to save time
  • Serving AI models at scale with TensorFlow Serving
  • Getting models ready for production

Session 6: Hands-on Lab: Deploying AI Models with TensorFlow Serving

  • Wrapping AI models in RESTful APIs
  • Placing models inside real-time big data pipelines
  • Case study: predictive maintenance driven by big data models

Day 4: Big Data Analytics for Business Insights

Session 1: Predictive Analytics and Business Intelligence (BI)

  • Using big data to forecast business outcomes
  • Connecting big data to BI tools such as Power BI and Tableau
  • Case study: forecasting sales for an e-commerce platform

Session 2: Hands-on Lab: Predictive Analytics with Big Data

  • Models that predict customer churn and future sales
  • Showing big data findings on BI dashboards
  • Automating reports so decisions rest on current data

Session 3: Big Data Analytics in Real-World Industries

  • Case study 1: healthcare analytics with big data and AI
  • Case study 2: real-time fraud detection in financial services
  • Case study 3: supply chain planning with predictive analytics

Session 4: Hands-on Lab: End-to-End Big Data Analytics Project

  • Planning a data pipeline for a real analytics problem
  • Running AI models to produce business insights
  • Presenting the analysis and the recommendations that follow

Day 5: Advanced Topics, Challenges, and Capstone Project

Session 1: Challenges and Best Practices in Big Data Analytics

  • Dealing with privacy, security and scale problems in big data
  • Speeding up large AI models
  • Good practice for data governance and ethical AI

Session 2: Capstone Project: AI-Powered Big Data Solution

  • Team project: building a big data analytics solution with AI
  • Bringing together ingestion, processing, modeling and deployment
  • Presenting results, insights and strategic recommendations

Session 3: Key Takeaways and Future Trends in AI and Big Data

  • Trends to watch: AI-driven automation, edge computing and real-time analytics
  • Career routes and skills for big data professionals working with AI
  • Places to keep learning about AI, big data and data science

Session 4: Final Q&A and Course Wrap-Up

  • Open floor for participants’ own questions
  • Feedback on the course and talk about next learning steps

Book This Course

Customise This Course for Your Team

If your company already runs Hadoop, Spark or a cloud data platform, the labs can be set up to match it so people train on familiar tooling. A team focused on streaming could spend more of the week on Kafka and Spark Streaming and less on deep learning. Let us know what your data pipeline looks like now and we will plan the five days to fit.
Need help with the right course to choose?

Explore more opportunities

AI and Data Science Fundamentals: Concepts, Tools, and Techniques Training Course

Data Analysis with AI: Exploring Data Insights for Beginners Training Course

Getting Started with Python for AI and Data Science Training Course

Introduction to Machine Learning in Data Science with AI Training Course

Predictive Analytics Using AI and Machine Learning Training Course

AI for Data-Driven Decision Making: Business Intelligence Applications Training Course

AI-Powered Big Data Analytics: Working with Large Datasets Training Course