This is a 6-week evening program providing a hands-on introduction to the Hadoop and Spark ecosystem of Big Data technologies. The course will cover these key components of Apache Hadoop: HDFS, MapReduce with streaming, Hive, and Spark. Programming will be done in Python. The course will begin with a review of Python concepts needed for our examples. The course format is interactive. Students will need to bring laptops to class. We will do our work on AWS (Amazon Web Services); instructions will be provided ahead of time on how to connect to AWS and obtain an account.
What is Hadoop?
Hadoop is a set of open-source programs running in computer clusters that simplify the handling of large amounts of data. Originally, Hadoop consisted of a distributed file system tuned for large data sets and an implementation of the MapReduce parallelism paradigm, but has expanded in many ways. It now includes database systems, languages for parallelism, libraries for machine learning, its own job scheduler, and much more. Furthermore, MapReduce is no longer the only parallelism framework; Spark is an increasingly popular alternative. In summary, Hadoop is a very popular and rapidly growing set of cluster computing solutions, which is becoming an essential tool for data scientists.
Prerequisites
To get the most out of the class, you need to be familiar with Linux file systems, Linux command line interface (CLI) and the basic linux commands such as cd, ls, cp, etc. You also need to have basic programming skills in Python, and are comfortable with functional programming style, for example, how to use map() function to split a list of strings into a nested list. Object oriented programming (OOP) in python is not required.
Syllabus
Unit 1: Introduction to Hadoop
1. Data Engineering Toolkits
- Running Linux using Docker containers
- Linux CLI command and bash scripts
- Python basics
2. Hadoop and MapReduce
- Big Data Overview
- HDFS
- YARN
- MapReduce
Unit 2 – MapReduce
3. MapReduce using MRJob 1
- Protocols for Input & Output
- Filtering
4. MapReduce using MRJob 2
- Top n
- Inverted Index
- Multi-step Jobs
Unit 3 – Apache Hive
5. Apache Hive 1
- Databases for Big Data
- HiveQL and Querying Data
- Windowing And Analytics Functions
- MapReduce Scripts
6. Apache Hive 2
- Tables in Hive
- Managed Tables and External Tables
- Storage Formats
- Partitions and Buckets
Unit 4 – Apache Pig
7. Apache Pig 1
- Overview
- Pig Latin: Data Types
- Pig Latin: Relational Operators
8. Apache Pig 2
- More Pig Latin: Relational operators
- More Pig Latin: Functions
- Compiling Pig to MapReduce
- The Parallel Clause
- Join Optimizations
Unit 5 – Apache Spark and AWS
9. Apache Spark – Spark Core
- Spark Overview
- Running Spark using Databricks Notebooks
- Working with PySpark: RDDs
- Transformations and Actions
10. Apache Spark – Spark SQL
- Spark DataFrame
- SQL Operations using Spark SQL
11. Apache Spark – Spark ML
- ML Pipeline using PySpark
12. Amazon Elastic MapReduce
- Overview
- Amazon Web Services: IAM, EC2, S3
- Creating EMR Cluster
- Submitting Jobs
- Intro to AWS CLI
Project: Data Engineering Project
This course is available for "remote" learning and will be available to anyone with access to an internet device with a microphone (this includes most models of computers, tablets). Classes will take place with a "Live" instructor at the date/times listed below.
Upon registration, the instructor will send along additional information about how to log-on and participate in the class.
School Notes: We offer a certification licensed by the NYS Board of Education.