Direkt zum Inhalt

Startseite Spark

Kurs

Big Data Fundamentals with PySpark

Fortgeschritten

Updated 12.2024

Learn the fundamentals of working with big data with PySpark.

Kurs kostenlos starten

Kostenlos inbegriffenPremium or Teams

SparkData Engineering4 Stunden16 Videos55 Übungen4,600 XP52,938Leistungsnachweis

Kostenloses Konto erstellen

Google LinkedIn Facebook

oder

Durch Klick auf die Schaltfläche akzeptierst du unsere Nutzungsbedingungen, unsere Datenschutzrichtlinie und die Speicherung deiner Daten in den USA.

Trainierst du 2 oder mehr?

Versuchen DataCamp for Business

Beliebt bei Lernenden in Tausenden Unternehmen

Kursbeschreibung

There's been a lot of buzz about Big Data over the past few years, and it's finally become mainstream for many companies. But what is this Big Data? This course covers the fundamentals of Big Data via PySpark. Spark is a "lightning fast cluster computing" framework for Big Data. It provides a general data processing platform engine and lets you run programs up to 100x faster in memory, or 10x faster on disk, than Hadoop. You’ll use PySpark, a Python package for Spark programming and its powerful, higher-level libraries such as SparkSQL, MLlib (for machine learning), etc. You will explore the works of William Shakespeare, analyze Fifa 2018 data and perform clustering on genomic datasets. At the end of this course, you will have gained an in-depth understanding of PySpark and its application to general Big Data analysis.

Voraussetzungen

Introduction to Python

1

Introduction to Big Data analysis with Spark

Kapitel starten

What is Big Data?

The 3 V's of Big Data

PySpark: Spark with Python

Understanding SparkContext

Interactive Use of PySpark

Loading data in PySpark shell

Review of functional programming in Python

Use of lambda() with map()

Use of lambda() with filter()

2

Programming in PySpark RDD’s

Kapitel starten

Abstracting Data with RDDs

RDDs from Parallelized collections

RDDs from External Datasets

Partitions in your data

Basic RDD Transformations and Actions

Map and Collect

Filter and Count

Pair RDDs in PySpark

ReduceBykey and Collect

SortByKey and Collect

Advanced RDD Actions

CountingBykeys

Create a base RDD and transform it

Remove stop words and reduce the dataset

Print word frequencies

3

PySpark SQL & DataFrames

Kapitel starten

Abstracting Data with DataFrames

RDD to DataFrame

Loading CSV into DataFrame

Operating on DataFrames in PySpark

Inspecting data in PySpark DataFrame

PySpark DataFrame subsetting and cleaning

Filtering your DataFrame

Interacting with DataFrames using PySpark SQL

Running SQL Queries Programmatically

SQL queries for filtering Table

Data Visualization in PySpark using DataFrames

PySpark DataFrame visualization

Part 1: Create a DataFrame from CSV file

Part 2: SQL Queries on DataFrame

Part 3: Data visualization

4

Machine Learning with PySpark MLlib

Kapitel starten

Overview of PySpark MLlib

PySpark ML libraries

PySpark MLlib algorithms

Collaborative filtering

Loading Movie Lens dataset into RDDs

Model training and predictions

Model evaluation using MSE

Classification

Loading spam and non-spam data

Feature hashing and LabelPoint

Logistic Regression model training

Loading and parsing the 5000 points data

K-means training

Visualizing clusters

Congratulations!

Big Data Fundamentals with PySpark

Kurs
abgeschlossen

Leistungsnachweis verdienen

Fügen Sie diese Anmeldeinformationen zu Ihrem LinkedIn-Profil, Lebenslauf oder Lebenslauf hinzu
Teilen Sie es in den sozialen Medien und in Ihrer Leistungsbeurteilung

Im Lieferumfang enthaltenPremium or Teams

Machen Sie mit 15 Millionen Lernende und starten Sie Big Data Fundamentals with PySpark Heute!

Kostenloses Konto erstellen

Google LinkedIn Facebook

oder

Durch Klick auf die Schaltfläche akzeptierst du unsere Nutzungsbedingungen, unsere Datenschutzrichtlinie und die Speicherung deiner Daten in den USA.