Skip links

ApacheDev: Apache Spark Development

Because we know just how hard it is to get the size.

Training

ApacheDev: Apache Spark Development

Find Out More
Course Information

Course Name

ApacheDev: Apache Spark Development

Duration

4 Days

Overview

Spark is a fast and general cluster computing system for Big Data. It provides high-level APIs in Scala, Java, Python, and R, and an optimized engine that supports general computation graphs for data analysis. It also supports a rich set of higher-level tools including Spark SQL for SQL and DataFrames, Spark ML for machine learning, and Spark Streaming for live data stream processing. With Spark running on Apache Hadoop YARN, developers can create applications to derive actionable insights within a single, shared dataset in Hadoop.

This training course will teach you how to solve Big Data problems using the Apache Spark framework. The training will cover a wide range of Big Data use cases such as ETL, DWH and streaming. It will also demonstrate how Spark integrates with other well established Hadoop ecosystem products. You will learn the course curriculum through theory lectures, live demonstrations and lab exercises. This course will be taught in the Java programming language, and we will be using Cloudera 7.1.4 using Spark 3.x.

Overview

Overview

Spark is a fast and general cluster computing system for Big Data. It provides high-level APIs in Scala, Java, Python, and R, and an optimized engine that supports general computation graphs for data analysis. It also supports a rich set of higher-level tools including Spark SQL for SQL and DataFrames, Spark ML for machine learning, and Spark Streaming for live data stream processing. With Spark running on Apache Hadoop YARN, developers can create applications to derive actionable insights within a single, shared dataset in Hadoop.

This training course will teach you how to solve Big Data problems using the Apache Spark framework. The training will cover a wide range of Big Data use cases such as ETL, DWH and streaming. It will also demonstrate how Spark integrates with other well established Hadoop ecosystem products. You will learn the course curriculum through theory lectures, live demonstrations and lab exercises. This course will be taught in the Java programming language, and we will be using Cloudera 7.1.4 using Spark 3.x.

Audience Profile

  • Application developers, Data Engineers, DevOps engineers, Architects, QAs, Technical Managers

Prerequisites

Following are the prerequisites for the course.

  • Programming knowledge in Java is required
  • Basic Knowledge of big data use-cases.
  • Basic knowledge of databases, OLAP/OLTP use cases, SQL
  • Knowledge of Java stack – JVM is helpful

Course Outline

  • Brief overview of Java programming language
  • Benefits and features of Java
  • Variables
    • Declaration and initialization
    • Primitive data types (int, double, boolean, etc.)
    • Reference data types (String, arrays, objects)
  • Control Structures
    • Conditionals (if-else, switch)
    • Loops (for, while, do-while)
  • Functions/Methods
    • Declaring and defining methods
    • Passing parameters and returning values
  • Introduction to JAR files
  • Creating a JAR file using command-line tools
  • Including class files and resources in the JAR
  • Executing a JAR file
  • Overview of functional programming concepts
  • Lambda Expressions
    • Syntax and examples
    • Functional interfaces and lambda expressions
  • Functional Interfaces
    • Predefined functional interfaces (Predicate, Function, Consumer, etc.)
    • Creating custom functional interfaces
  • Streams
    • Introduction to streams
    • Stream operations (map, filter, reduce, etc.)
    • Chaining operations with streams
  • Working with Containers
  • Kubernetes as Cluster Manager
  • Understanding Pods/Deployments
  • Working with Spark Image
  • Starting Spark workloads on Kubernetes/OpenShift
  • What is Apache Spark: The evolution from Hadoop
  • Why Spark?
  • Advantages of Spark over Hadoop MapReduce
  • Logical Architecture of Spark
  • Programming Languages in Spark
  • Functional Programming with Spark
  • Data sources for Spark application
  • Introduction to S3
  • Spark with S3 as data source
  • RDD Basics
  • Spark Context and Spark Session
  • RDD Operations: Read from file, transforming, and saving persistent
  • Leveraging in-memory processing
  • Pair RDD operations
  • Working with semi-structured data formats using regex, JSON, XML libraries
  • MapReduce Operations
  • Joins
  • Hands-on Exercises
  • Understanding Execution plans
  • Spark literature: Narrow, wide operations, shuffle, DAG, Shuffle, Stages, and Tasks
  • Job metrics
  • Fault Tolerance
  • Hands-on Exercises
  • Spark Applications vs. Spark Shell
  • Developing a Spark Application
  • Modifying Spark Key Configurations
  • Submitting Spark Application.
  • Hands-on Exercises
  • Caching
  • Distributed Persistence
  • Storage Levels
  • Broadcast using RDD
  • Accumulator using RDD
  • Hands-on Exercises
  • Spark SQL Overview
  • Spark SQL Documentation
  • Sub-modules of Spark SQL
  • Difference between RDD and DataFrame
  • Loading and processing data into DataFrame
  • DataFrame API
  • DataFrame Operations
  • Saving DataFrame to file systems
  • DataFrame internals that make it fast: Catalyst Optimizer and Tungsten
  • Hands-on Exercises
  • Processing JSON data
  • Binary Data Processing
  • Why Binary Data Processing?
  • Comparison of various data formats
  • Working with Parquet
  • Working with Avro
  • Hands-on Exercise
  • Hive Context vs Spark SQL Context
  • Integrating Spark SQL with Hive
  • Working with Hive Tables
  • Working with JDBC data source
  • Data formats: text format such as CSV, JSON, XML, binary formats such as Parquet, ORC
  • UDF in Spark DataFrame
  • Spark SQL as JDBC service and its benefits and limitations
  • Analytical queries in Spark
  • Hands-on Exercises
  • What is Catalog API?
  • Catalog API Demo
  • Table, View
  • Key considerations for Performance Tuning
  • Hands-on Exercises
  • Capturing job metrics using Spark History Server
  • Benefits of shared variables: Accumulator, Broadcast var
  • Types of Spark caching and their use cases
  • Spark SQL Optimization
  • Understanding Catalyst Optimizer
  • Exploring and Interpreting Spark SQL Plan
  • Partitions
  • Partition Pruning
  • Broadcast Join
  • Hands-on Exercises
  • Explain the concepts necessary to read it
  • Illustrate how this UI can be used for debugging or performance tuning
  • Partition
  • Illustrate the RDD/Dataframe partitioning and formulate recommendations through different scenarios
    • Coalesce vs. repartition
    • Repartition on column
  • Illustrate and formulate recommendations on File System/Table partitioning
    • Benefits of FS/table partitioning
    • How to do it and when?
  • Driver Memory (including memory overhead) & number of Cores
  • Executor Memory (including memory overhead) & number of Cores
  • Quick tour on other relevant configuration parameters (shuffle partitions, parallelism, etc.)
  • How to structure your repository
  • Unit Testing
  • Functional approach vs classic OOP
  • Delta Lake (CRUD operations)
  • S3 buckets
  • Hive Tables
  • Oracle
  • Hands-on exercises including classic Data Manipulations
  • Broadcasting variables/Dataframe
Course Price

RM5,000.00 exc. 8% tax

Training Dates
Fill up the form for enquiry
Certification Details