Introduction to Big Data and PySpark

LEVEL: BEGINNER

DURATION: 2-DAYS COURSE

DELIVERED: AT YOUR OFFICE

What you will learn

This two-days course will provide a hands-on introduction to the Big Data ecosystem, Hadoop and Apache Spark in practice.

Understand the challenges in the Big Data ecosystem

Describe the fundamentals of the Hadoop ecosystem

Use the core Spark RDD API to express data processing queries

Monitoring and tuning

Languages and libraries :

Python 3

Spark

PREREQUISITES

Fundamentals Python programming and use of the command line. You can acquire these skills at our Python bootcamp.

AUDIENCE

Those who are curious about the Big Data space and who want to feel comfortable getting their hands dirty with high volume, high velocity, diverse real-world datasets.

DAY ONE

Introduction to Big Data and Spark

Introduction to Big Data
- Context and Motivation
- Volume, Velocity, Variety
- Batch vs Streaming
Hadoop Ecosystem
- Distributed File System: HDFS
- Cluster Manager: YARN
- Distributed Processing Engine
- MapReduce Pattern
Spark RDD API
- Apache Spark Architecture
- Transformations and actions
- Aggregating Multiple RDDs
- Working with Key Value Pairs
Overview of AWS for Big Data Processing
- S3
- EC2
- EMR (live demo)

DAY TWO

Spark in Practice

Spark Dataframes and Spark SQL
- Spark vs. DBMS
- RDD vs Dataframes
- DataFrame API
- Further Dataframes Operations: Joins, Null values and UDFs
- Spark SQL
Internals, Shuffles, Data Skew and Tuning
- How does Pyspark work
- Partition
- Tasks, Stages and Jobs
- Data Skew
- Spark Web UI
Spark Testing
- The importance of testing
- Writing suitable Spark test cases
Big Data File Format and Parquet
- Context: CPU vs Memory
- Row vs Columnar layout
- Parquet File Layout
- Predicate pushdown and Projection pushdown
- Parquet encoding tricks
- PyArrow
- Parquet Tools
- Working with Parquet using Spark on AWS EMR

Get in Touch

CONTACT US

We will email you within the next 24 hours to arrange a quick call to help with any questions about the programme and recommend pre-course materials.

We look forward to speaking with you.

Dr. Raoul-Gabriel Urma

Director