Skip to main content
HomeSpark

Introduction to Spark SQL in Python

Learn how to manipulate data and create machine learning feature sets in Spark using SQL in Python.

Start Course for Free
4 hours15 videos52 exercises17,205 learnersTrophyStatement of Accomplishment

Create Your Free Account

GoogleLinkedInFacebook

or

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.
Group

Training 2 or more people?

Try DataCamp for Business

Loved by learners at thousands of companies

Course Description

Learn Spark SQL

If you’re familiar with SQL and have heard great things about Apache Spark, this course is for you. Apache Spark is a computing framework for processing big data, and Spark SQL is a component of Apache Spark. This four-hour course will show you how to take Spark to a new level of usefulness, using advanced SQL features, such as window functions.

Over the course of four chapters, you’ll use Spark SQL to analyze time series data, extract the most common words from a text document, create feature sets from natural language text, and use them to predict the last word in a sentence using logistic regression.

Discover the Uses of Spark SQL

You’ll start by creating and querying an SQL table in Spark, as well as learning how to use SQL window functions to perform running sums, running differences, and other operations.

Next, you’ll explore how to use the window function in Spark SQL for natural language processing, including using a moving window analysis to find common word sequences.

In chapter 3, you’ll learn how to use the SQL Spark UI to properly cache DataFrames and SQL tables before exploring the best practices for logging in Spark.

Finally, you use all of the skills learned so far to load and tokenize raw text before extracting word sequences. You’ll then use logistic regression to classify the text, using raw natural language data to train a text classifier.

Gain a Thorough Introduction to Spark SQL

By the end of the course, you’ll have a firm understanding of Spark SQL and will understand how Spark combines the power of distributed computing with the ease of use of Python and SQL.
For Business

Training 2 or more people?

Get your team access to the full DataCamp platform, including all the features.
DataCamp for BusinessFor a bespoke solution book a demo.
  1. 1

    PySpark SQL

    Free

    In this chapter you will learn how to create and query a SQL table in Spark. Spark SQL brings the expressiveness of SQL to Spark. You will also learn how to use SQL window functions in Spark. Window functions perform a calculation across rows that are related to the current row. They greatly simplify achieving results that are difficult to express using only joins and traditional aggregations. We'll use window functions to perform running sums, running differences, and other operations that are challenging to perform in basic SQL.

    Play Chapter Now
    Creating and querying a SQL table in Spark
    50 xp
    Create a SQL table from a dataframe
    100 xp
    Determine the column names of a table
    100 xp
    Window function SQL
    50 xp
    Running sums using window function SQL
    100 xp
    Fix the broken query
    100 xp
    Dot notation and SQL
    50 xp
    Aggregation, step by step
    100 xp
    Aggregating the same column twice
    100 xp
    Aggregate dot SQL
    100 xp
    Convert window function from dot notation to SQL
    100 xp
  2. 2

    Using Window Function SQL for Natural Language Processing

    In this chapter, you will be loading natural language text. Then you will apply a moving window analysis to find frequent word sequences.

    Play Chapter Now
    Loading natural language text
    50 xp
    Loading a dataframe from a parquet file
    100 xp
    Split and explode a text column
    100 xp
    Using monotonically_increasing_id()
    50 xp
    Moving window analysis
    50 xp
    Creating context window feature data
    100 xp
    Repartitioning the data
    100 xp
    Common word sequences
    50 xp
    What type of data is this
    50 xp
    Finding common word sequences
    100 xp
    Unique 5-tuples in sorted order
    100 xp
    Most frequent 3-tuples per chapter
    100 xp
  3. 3

    Caching, Logging, and the Spark UI

    In the previous chapters you learned how to use the expressiveness of window function SQL. However, this expressiveness now makes it important that you understand how to properly cache dataframes and cache SQL tables. It is also important to know how to evaluate your application. You learn how to do do this using the Spark UI. You'll also learn a best practice for logging in Spark. Spark SQL brings with it another useful tool for tuning query performance issues, the query execution plan. You will learn how to use the execution plan for evaluating the provenance of a dataframe.

    Play Chapter Now
    Caching
    50 xp
    Practice query plans 2
    25 xp
    Practicing caching: part 1
    100 xp
    Practicing caching: the SQL
    100 xp
    Practicing caching: putting it all together
    50 xp
    Caching and uncaching tables
    100 xp
    The Spark UI
    50 xp
    Spark UI storage tab
    50 xp
    Inspecting cache in the Spark UI
    50 xp
    Logging
    50 xp
    Practice logging
    100 xp
    Practice logging 2
    100 xp
    Query plans
    50 xp
    Practice query plans
    100 xp
    Practice reading query plans 2
    100 xp
  4. 4

    Text Classification

    Previous chapters provided you with the tools for loading raw text, tokenizing it, and extracting word sequences. This is already very useful for analysis, but it is also useful for machine learning. What you've learned now comes together by using logistic regression to classify text. By the conclusion of this chapter, you will have loaded raw natural language text data and used it to train a text classifier.

    Play Chapter Now
    Extract Transform Select
    50 xp
    Practicing creating a UDF
    100 xp
    Practicing array column
    100 xp
    Creating feature data for classification
    50 xp
    Creating a UDF for vector data
    100 xp
    Applying a UDF to vector data
    100 xp
    Transforming text to vector format
    100 xp
    Text Classification
    50 xp
    Label the data
    100 xp
    Split the data
    100 xp
    Train the classifier
    100 xp
    Predicting and evaluating
    50 xp
    Evaluate the classifier
    100 xp
    Predict test data
    100 xp
    Recap
    50 xp
For Business

Training 2 or more people?

Get your team access to the full DataCamp platform, including all the features.

datasets

Sherlock (parquet file)Sherlock (txt file)Train schedule

collaborators

Collaborator's avatar
Hadrien Lacroix
Collaborator's avatar
Hillary Green-Lerman

prerequisites

Introduction to PySparkPython ToolboxPostgreSQL Summary Stats and Window Functions
Mark Plutowski HeadshotMark Plutowski

Big Data Architect & Scientist @ Flipboard

Mark Plutowski received his PhD in Computer Science from UCSD with emphasis in machine learning and econometrics. He holds 29 published patents, has been working with cloud apis and hadoop ecosystem for over ten years, and has been involved with hundreds of interviews.
See More

What do other learners have to say?

    FAQs

    Join over 15 million learners and start Introduction to Spark SQL in Python today!

    Create Your Free Account

    GoogleLinkedInFacebook

    or

    By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.