Cart 0
Training
Certificate
Course Information
Course Name
DOPS-242: Ingesting with Cloudera DataFlow
Exam code
DOPS-242
Duration
4 Days
Certification
Overview
Audience Profile
Prerequisities
At Course Completion
Course Outline
Overview
Apache NiFi and this four-day course provides the fundamental concepts and experience necessary to automate the ingress, flow, transformation, and egress of data using NiFi. The course also covers tuning, troubleshooting, and monitoring the dataflow process as well as how to integrate a dataflow within the Cloudera CDP Hybrid ecosystem and external systems.
Overview
Overview
Apache NiFi and this four-day course provides the fundamental concepts and experience necessary to automate the ingress, flow, transformation, and egress of data using NiFi. The course also covers tuning, troubleshooting, and monitoring the dataflow process as well as how to integrate a dataflow within the Cloudera CDP Hybrid ecosystem and external systems.
Audience Profile
Audience Profile
This course is designed for developers, data engineers, administrators, and others with an interest in learning NiFi’s innovative no-code, graphical approach to data ingest.
Prerequisities
Prerequisities
Although programming experience is not required, basic experience with Linux is presumed, and previous exposure to big data concepts and applications is helpful.
At Course Completion
At Course Completion
During this course, you learn how to:
- Define, configure, organize, and manage dataflows
- Transform and trace data as it flows to its destination
- Track changes to dataflows with NiFi Registry
- Use the NiFi Expression Language to control dataflows
- Optimize dataflows for better performance and maintainability
- Connect dataflows with other systems, such as Apache Kafka, Apache Hive, and HDFS
- Utilize the Data Flow Service
Course Outline
Course Outline
Module 1: Introduction to Cloudera Flow Management
- Overview of Cloudera Data-in-Motion
- The NiFi User Interface
- DataFlow Catalog
- ReadyFlows
- Instructor-Led Demo: NiFi User Interface
- Hands-On Exercise: Build Your First Dataflow
Module 2: Processors
- Overview of Processors
- Processor Surface Panel
- Processor Configuration
- Hands-On Exercise: Start Building a Dataflow Using Processors
Module 3: Connections
- Overview of Connections
- Connection Configuration
- Connector Context Menu
- Hands-On Exercise: Connect Processors in a Dataflow
Module 4: Dataflows
- Command and Control of a Dataflow
- Processor Relationships
- Back Pressure
- Prioritizers
- Labels
- Hands-On Exercise: Build a More Complex Dataflow
- Hands-On Exercise: Creating a Fork Using Relationships
- Hands-On Exercise: Set Back Pressure Thresholds
Module 5: Process Groups
- Anatomy of Process Group
- Input and Output Ports
- Hands-On Exercise: Simplify Dataflows Using Process Groups
Module 6: FlowFile Provenance
- Data Provenance Events
- FlowFile Lineage
- Replaying a FlowFile
- Hands-On Exercise: Using Data Provenance
Module 7: Reading and Writing Record Data
- Querying Record Data
- QueryRecord Processor
- Writing Record Data
Module 8: Enriching Record Data
- ETL Operations
- Split and Join Processor
- Update Record Processors
- Wait and Notify Processors
Module 9: NiFi Architecture Overview
- NiFi Architecture Overview
- Public Cloud Architecture
- Private Cloud Architecture
Module 10: DataFlow Functions
- Overview
- Serverless functions
- Demo: Deploying a Flow Definition as a Function
Module 11: Parameters
- Parameter Contexts
- Referencing Parameters
- Managing Parameters
- Migrating from Variables
- Hands-On Exercise: Creating, Using, and Managing Parameters
Module 12: Flow Definitions and Templates
- Flow Definition Overview
- Creating a Flow Definition
- Importing and Deploying a Flow
- Using (migrating from) Templates
- Hands-On Exercise: Creating, Using, and Managing Flow Definitions
Module 13: Apache NiFi Registry
- Apache NiFi Registry Overview
- Using the Registry
- Hands-On Exercise: Versioning Flows Using NiFi Registry
Module 14: FlowFile Attributes
- FlowFile Attribute Overview
- Routing on Attributes
- Hands-On Exercise: Working with FlowFile Attributes
Module 15: NiFi Expression Language
- NiFi Expression Language Overview
- Syntax
- Expression Language Editor
- Setting Conditional Values
- Hands-On Exercise: Using the NiFi Expression Language
Module 16: Controller Services
- Controller Services Overview
- Common Controller Services
- Hands-On Exercise: Adding Apache Hive Controller
Module 17: Record-based Components
- Record-oriented data
- Record-based Processors
- Avro Schema Registry
- Schema Forma
Module 18: Dataflow Optimization
- Dataflow Optimization
- Control Rate
- Managing Compute
- Hands-On Exercise: Building an Optimized Dataflow
Module 19: Monitoring, Reporting, and Troubleshooting
- Monitoring from NiFi
- Reporting
- Examples of Common Reporting Tasks
- Hands-On Exercise: Monitoring and Reporting
Module 20: NiFi Security
- NiFi Security Overview
- Securing Access to the NiFi UI
- Metadata Management
Module 21: Integrating NiFi
- NiFi Integration Architecture
- Available ReadyFlows
- A Closer Look at NiFi and Apache Hive
Related Courses