Streaming Data with AWS Kinesis and Lambda
Learn how to work with streaming data using serverless technologies on AWS.
Start Course for Free4 hours22 videos56 exercises6,588 learnersStatement of Accomplishment
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.Training 2 or more people?
Try DataCamp for BusinessLoved by learners at thousands of companies
Course Description
What powers the systems that we use every day without realizing, like fraud detection to keep our transactions secure or traffic signals that keep traffic flowing smoothly? The answer is streaming data (data that is continuously generated by different sources) and serverless technologies—like Amazon Kinesis and AWS Lambda.
In this course, you’ll learn how to leverage these powerful technologies by helping a fictional data engineer named Cody. Your goal is to help her to collect real-time streaming data from city-owned vehicles, analyze the data, and send relevant alerts like speed warnings to drivers. Using Amazon Kinesis and Firehose, you’ll learn how to ingest data from millions of sources before using Kinesis Analytics to analyze data as it moves through the stream. You’ll also spin up serverless functions in AWS Lambda that will conditionally trigger actions based on the data received. By the end of this training you’ll know how to create live ElasticSearch dashboards with AWS QuickSight and CloudWatch—and hopefully helped Cody complete her ambitious project.
In this course, you’ll learn how to leverage these powerful technologies by helping a fictional data engineer named Cody. Your goal is to help her to collect real-time streaming data from city-owned vehicles, analyze the data, and send relevant alerts like speed warnings to drivers. Using Amazon Kinesis and Firehose, you’ll learn how to ingest data from millions of sources before using Kinesis Analytics to analyze data as it moves through the stream. You’ll also spin up serverless functions in AWS Lambda that will conditionally trigger actions based on the data received. By the end of this training you’ll know how to create live ElasticSearch dashboards with AWS QuickSight and CloudWatch—and hopefully helped Cody complete her ambitious project.
Training 2 or more people?
Get your team access to the full DataCamp platform, including all the features.- 1
Streaming in the cloud
FreeIn this first chapter, you will learn about the differences between batch and stream processing, create your first stream, manage its permissions, write to it and read from it.
What is streaming and why does it matter?50 xpBatch vs stream100 xpProducers and destinations100 xpManaging Firehose delivery streams100 xpGetting ready for the first stream50 xpCreating roles50 xpIAM users vs. roles100 xpPermissions practice100 xpS3 bucket creation100 xpWorking with the Firehose delivery stream50 xpCreate your first Firehose stream100 xpWriting to a Firehose stream100 xpReading Firehose data100 xp - 2
Going serverless
The next step in your streaming data journey is learning how to use transformational lambda functions to go serverless. Through hands-on exercises, you will add lambda layers and trigger lambda functions on specific conditions.
Going serverless50 xpCreating and running Lambda functions50 xpServerless vs servers100 xpLambda function components100 xpReading data from S3 based on an event100 xpYour first live lambda!50 xpAdding a lambda layer50 xpLambda flow100 xpUsing environment variables100 xpServerless data workflow50 xpFast vs slow lambdas100 xpA time-triggered lambda100 xpTime-triggered lambda code100 xpServerless APIs50 xpCreate an API lambda100 xpA serverless API100 xp - 3
Analyzing streaming data
You're now ready to encode and decode streaming data and analyze data directly in the stream. You will even use multiple streams to get daily vehicle top speeds.
A transformational Lambda50 xpTransforming data inside a stream50 xpEncoding and decoding base64100 xpCreate a transformational lambda100 xpA barebones transformational lambda100 xpAnalyzing data in the stream50 xpCreating a Kinesis data analytics application50 xpKinesis data analytics vs transformational Lambda100 xpBuilding a Kinesis data analytics application100 xpKinesis data analytics SQL components100 xpUsing multiple streams50 xpDelivering data from Kinesis Analytics50 xpGet the daily top speed100 xpUsing Kinesis data analytics for alerts100 xp - 4
Monitoring and visualizing streaming data
In this final chapter, you will discover how to monitor your stream's performance using logs, metrics, alarms and dashboards. You will use Elasticsearch and build your own Kibana dashboard.
Streaming data case study50 xpCreating an Elasticsearch cluster50 xpRedShift vs Elasticsearch100 xpMonitoring performance50 xpCloudwatch dasbhoards and alarms50 xpCloudwatch components100 xpCloudwatch monitoring flow100 xpVisualizing streaming data50 xpWorking with ElasticSearch using Kibana50 xpCloudwatch vs Elasticsearch100 xpElasticsearch visualization flow100 xpAn alternative approach100 xpAnother alternative approach100 xp
Training 2 or more people?
Get your team access to the full DataCamp platform, including all the features.collaborators
Maksim PecherskiyData Engineer
Maksim is a Data Engineer at the World Bank, and formerly the Chief Data Officer at the City of San Diego. He also works with tech companies to help them effectively connect technology to government. he came to the US from Ukraine when he was 10 years old, lived in 5 countries and speaks 3 languages.
What do other learners have to say?
Join over 15 million learners and start Streaming Data with AWS Kinesis and Lambda today!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.