Skip to main content
HomeSpark

Feature Engineering with PySpark

Learn the gritty details that data scientists are spending 70-80% of their time on; data wrangling and feature engineering.

Start Course for Free
4 hours16 videos60 exercises14,806 learnersTrophyStatement of Accomplishment

Create Your Free Account

GoogleLinkedInFacebook

or

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.
Group

Training 2 or more people?

Try DataCamp for Business

Loved by learners at thousands of companies

Course Description

The real world is messy and your job is to make sense of it. Toy datasets like MTCars and Iris are the result of careful curation and cleaning, even so the data needs to be transformed for it to be useful for powerful machine learning algorithms to extract meaning, forecast, classify or cluster. This course will cover the gritty details that data scientists are spending 70-80% of their time on; data wrangling and feature engineering. With size of datasets now becoming ever larger, let's use PySpark to cut this Big Data problem down to size!
For Business

Training 2 or more people?

Get your team access to the full DataCamp platform, including all the features.
DataCamp for BusinessFor a bespoke solution book a demo.

In the following Tracks

Big Data with PySpark

Go To Track
  1. 1

    Exploratory Data Analysis

    Free

    Get to know a bit about your problem before you dive in! Then learn how to statistically and visually inspect your dataset!

    Play Chapter Now
    Where to Begin
    50 xp
    Where to begin?
    50 xp
    Check Version
    100 xp
    Load in the data
    100 xp
    Defining A Problem
    50 xp
    What are we predicting?
    100 xp
    Verifying Data Load
    100 xp
    Verifying DataTypes
    100 xp
    Visually Inspecting Data / EDA
    50 xp
    Using Corr()
    100 xp
    Using Visualizations: distplot
    100 xp
    Using Visualizations: lmplot
    100 xp
  2. 2

    Wrangling with Spark Functions

    Real data is rarely clean and ready for analysis. In this chapter learn to remove unneeded information, handle missing values and add additional data to your analysis.

    Play Chapter Now
    Dropping data
    50 xp
    Dropping a list of columns
    100 xp
    Using text filters to remove records
    100 xp
    Filtering numeric fields conditionally
    100 xp
    Adjusting Data
    50 xp
    Custom Percentage Scaling
    100 xp
    Scaling your scalers
    100 xp
    Correcting Right Skew Data
    100 xp
    Working with Missing Data
    50 xp
    Visualizing Missing Data
    100 xp
    Imputing Missing Data
    100 xp
    Calculate Missing Percents
    100 xp
    Getting More Data
    50 xp
    A Dangerous Join
    100 xp
    Spark SQL Join
    100 xp
    Checking for Bad Joins
    100 xp
  3. 3

    Feature Engineering

    In this chapter learn how to create new features for your machine learning model to learn from. We'll look at generating them by combining fields, extracting values from messy columns or encoding them for better results.

    Play Chapter Now
    Feature Generation
    50 xp
    Differences
    100 xp
    Ratios
    100 xp
    Deeper Features
    100 xp
    Time Features
    50 xp
    Time Components
    100 xp
    Joining On Time Components
    100 xp
    Date Math
    100 xp
    Extracting Features
    50 xp
    Extracting Text to New Features
    100 xp
    Splitting & Exploding
    100 xp
    Pivot & Join
    100 xp
    Binarizing, Bucketing & Encoding
    50 xp
    Binarizing Day of Week
    100 xp
    Bucketing
    100 xp
    One Hot Encoding
    100 xp
  4. 4

    Building a Model

    In this chapter we'll learn how to choose which type of model we want. Then we will learn how to apply our data to the model and evaluate it. Lastly, we'll learn how to interpret the results and save the model for later!

    Play Chapter Now
    Choosing the Algorithm
    50 xp
    Which MLlib Module?
    50 xp
    Creating Time Splits
    100 xp
    Adjusting Time Features
    100 xp
    Feature Engineering Assumptions for RFR
    50 xp
    Feature Engineering For Random Forests
    50 xp
    Dropping Columns with Low Observations
    100 xp
    Naively Handling Missing and Categorical Values
    100 xp
    Building a Model
    50 xp
    Building a Regression Model
    100 xp
    Evaluating & Comparing Algorithms
    100 xp
    Understanding Metrics
    50 xp
    Interpreting, Saving & Loading
    50 xp
    Interpreting Results
    100 xp
    Saving & Loading Models
    100 xp
    Final Thoughts
    50 xp
For Business

Training 2 or more people?

Get your team access to the full DataCamp platform, including all the features.

In the following Tracks

Big Data with PySpark

Go To Track

datasets

2017 St Paul MN Real Estate Dataset

collaborators

Collaborator's avatar
Adrián Soto
Collaborator's avatar
Nick Solomon

prerequisites

Introduction to PySparkSupervised Learning with scikit-learn
John Hogue HeadshotJohn Hogue

Lead Data Scientist, General Mills

See More

What do other learners have to say?

    Join over 15 million learners and start Feature Engineering with PySpark today!

    Create Your Free Account

    GoogleLinkedInFacebook

    or

    By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.