Skip to main content

course

Introduction to Spark with sparklyr in R

Intermediate

Updated 12/2024

Learn how to run big data analysis using Spark and the sparklyr package in R, and explore Spark MLIb in just 4 hours.

Start course for free

Included for FreePremium or Teams

SparkData Engineering4 hours4 videos50 exercises4,600 XP19,191Statement of Accomplishment

Create Your Free Account

Google LinkedIn Facebook

or

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.

Training 2 or more people?

Try DataCamp for Business

Loved by learners at thousands of companies

Course Description

Explore the Advantages of R, Spark, and sparklyr

R is mostly optimized to help you write data analysis code quickly and readably. Apache Spark is designed to analyze huge datasets quickly. The sparklyr package lets you write dplyr R code that runs on a Spark cluster, giving you the best of both worlds. This 4-hour course teaches you how to manipulate Spark DataFrames using both the dplyr interface and the native interface to Spark, as well as trying machine learning techniques.

Load Data into Spark and Manipulate Spark DataFrames

You’ll start this Spark course by investigating how Spark and R work well together and practicing loading data, ready for cleaning, transformation, and analysis. You’ll use Spark frames and dplyr syntax to manipulate your data by filtering and arranging rows, and mutating and summarizing columns.

Delve into Big Data Analysis with Spark MLib

This course focuses on building your skills and confidence in analyzing huge datasets. The final chapters take you through Spark’s machine learning data transformation features and offer you the chance to practice sparklyr’s machine learning routines by using it to make predictions using gradient boosted trees and random forests. "

Prerequisites

Supervised Learning in R: Regression

1

Light My Fire: Starting To Use Spark With dplyr Syntax

Getting started

Made for each other

Here be dragons

The connect-work-disconnect pattern

Copying data into Spark

Big data, tiny tibble

Exploring the structure of tibbles

Selecting columns

Filtering rows

Arranging rows

Mutating columns

Summarizing columns

2

Tools of the Trade: Advanced dplyr Usage

Leveling up

Mother's little helper (1)

Mother's little helper (2)

Selecting unique rows

Common people

Collecting data back from Spark

Storing intermediate results

Groups: great for music, great for data

Groups of mutants

Advanced Selection II: The SQL

3

Going Native: Use The Native Interface to Manipulate Spark DataFrames

Two new interfaces

Popcorn double feature

Transforming continuous variables to logical

Transforming continuous variables into categorical (1)

Transforming continuous variables into categorical (2)

More than words: tokenization (1)

More than words: tokenization (2)

More than words: tokenization (3)

Sorting vs. arranging

Exploring Spark data types

Shrinking the data by sampling

Training/testing partitions

4

Case Study: Learning to be a Machine: Running Machine Learning Models on Spark

Machine learning on Spark

Machine learning functions

(Hey you) What's that sound?

Working with parquet files

Come together

Partitioning data with a group effect

Gradient boosted trees: modeling

Gradient boosted trees: prediction

Gradient boosted trees: visualization

Random Forest: modeling

Random Forest: prediction

Random Forest: visualization

Comparing model performance

Introduction to Spark with sparklyr in R

Course
Complete

Earn Statement of Accomplishment

Add this credential to your LinkedIn profile, resume, or CV
Share it on social media and in your performance review

Included withPremium or Teams

FAQs

Join over 15 million learners and start Introduction to Spark with sparklyr in R today!

Create Your Free Account

Google LinkedIn Facebook

or

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.