Real-time Data Streaming

In this module, you’ll create an Amazon Kinesis stream to collect and store sensor data from our unicorn fleet. Using the provided command-line clients, you’ll produce sensor data from a unicorn on a Wild Ryde and read from the stream. Lastly, you’ll use the unicorn dashboard to plot our unicorns on a map and watch their status in real-time. In subsequent modules you’ll add functionality to analyze and persist this data using Amazon Kinesis Data Analytics, AWS Lambda, and Amazon DynamoDB.

Overview

The architecture for this module involves an Amazon Kinesis stream, a producer, and a consumer.

Our producer is a sensor attached to a unicorn currently taking a passenger on a ride. This sensor emits data every second including the unicorn’s current location, distance traveled in the previous second, and magic points and hit points so that our operations team can monitor the health of the unicorn fleet from Wild Rydes headquarters.

The Amazon Kinesis stream stores data sent by the producer and provides an interface to allow consumers to process and analyze those data. Our consumer is a simple command-line utility that tails the stream and outputs the data points from the stream in effectively real-time so we can see what data is being stored in the stream. Once we send and receive data from the stream, we can use the unicorn dashboard to view the current position and vitals of our unicorn fleet in real-time.

Implementation

❗ Ensure you’ve completed the setup guide before beginning the workshop.

1. Create an Amazon Kinesis stream

Use the Amazon Kinesis Data Streams console to create a new provisioned stream named wildrydes with 1 shard.

✅ Step-by-step Instructions

1. Go to the AWS Management Console, click Services then select Kinesis under Analytics.

2. Click Get started if prompted with an introductory screen.

3. Click Create data stream.

4. Enter wildrydes into Kinesis stream name

5. Select the Provisioned option and enter 1 into Number of shards, then click Create Kinesis stream.

6. Within 60 seconds, your Kinesis stream will be ACTIVE and ready to store real-time streaming data.

   

2. Produce messages into the stream

Use the command-line producer to produce messages into the stream.

✅ Step-by-step Instructions

1. Switch to the tab where you have your Cloud9 environment opened.

2. In the terminal, run the producer to start emitting sensor data to the stream.

   ./producer
   

   The producer emits a message a second to the stream and prints a period to the screen.

   $ ./producer
   ..................................................
   

3. In the Amazon Kinesis Streams console, click on wildrydes and click on the Monitoring tab.

4. After several minutes, you will see the Put Record Success (percent) - Average graph begin to record a single put a second. Keep the producer running till end of the Lab 2, so that you can see the unicorns flying in action.

3. Read messages from the stream

✅ Step-by-step Instructions

1. While the producer is running, switch to the tab where you have your Cloud9 environment opened.

2. Hit the (+) button and click New Terminal to open a new terminal tab.

3. Run the consumer to start reading sensor data from the stream.

   ./consumer
   

   The consumer will print the messages being sent by the producer:

   {
     "Name": "Shadowfax",
     "StatusTime": "2017-06-05 09:17:08.189",
     "Latitude": 42.264444250051326,
     "Longitude": -71.97582884770408,
     "Distance": 175,
     "MagicPoints": 110,
     "HealthPoints": 150
   }
   {
     "Name": "Shadowfax",
     "StatusTime": "2017-06-05 09:17:09.191",
     "Latitude": 42.265486935100476,
     "Longitude": -71.97442977859625,
     "Distance": 163,
     "MagicPoints": 110,
     "HealthPoints": 151
   }
   

4. Create an identity pool for the unicorn dashboard

Create an Amazon Cognito identity pool to grant unauthenticated users access to read from your Kinesis stream. Note the identity pool ID for use in the next step.

✅ Step-by-step directions

1. Go to the AWS Management Console, click Services then select Cognito under Security, Identity & Compliance.

2. Click Manage Identity Pools.

3. Click Create new identity pool. [This is not necessary if you do not have any identity pool yet.]

4. Enter wildrydes into Identity pool name.

5. Tick the Enable access to unauthenticated identities checkbox.

6. Click Create Pool.

7. Click Allow which will create authenticated and unauthenticated roles for your identity pool.

8. Click Go to Dashboard.

9. Click Edit identity pool in the upper right hand corner.

10. Note the Identity pool ID for use in a later step.

    

5. Grant the unauthenticated role access to the stream

Add a new policy to the unauthenticated role to allow the dashboard to read from the stream to plot the unicorns on the map.

✅ Step-by-step directions

1. Go to the AWS Management Console, click Services then select IAM under Security, Identity & Compliance.

2. Click on Roles in the left-hand navigation.

3. Click on the Cognito_wildrydesUnauth_Role.

4. Click Add inline policy (Add permissions -> Create inline policy).

5. Click on Choose a service and click Kinesis.

6. Click on Actions.

7. Tick the Read and List permissions checkboxes.

8. Under Resources you will limit the role to the wildrydes stream and consumer.

9. Click Add ARN next to consumer.

10. In the Add ARN(s) dialog box, enter the following information:

    • the region you’re using in Region (e.g. us-east-1)
    • your Account ID in Account
    • * in Stream type
    • wildrydes in Stream name
    • * in Consumer name
    • * in Consumer creation timestamp

11. Click Add.

12. Click Add ARN next to stream.

13. In the Add ARN(s) dialog box, enter the following information:

    • the region you’re using in Region (e.g. us-east-1)
    • your Account ID in Account
    • wildrydes in Stream name

14. Click Add.

    

15. Click Review policy.

16. Enter WildrydesDashboardPolicy in Name.

17. Click Create policy.

6. View unicorn status on the dashboard

Use the Unicorn Dashboard to see the unicorn on a real-time map. You may need to zoom out to find the unicorn. Please double check that producer and consumer are both running if you can’t find it.

✅ Step-by-step directions

1. Open the Unicorn Dashboard.

2. Enter the Cognito Identity Pool ID you noted in step 4 and click Start.

   

3. Validate that you can see the unicorn on the map.If you can not see the unicorn, please go back to Cloud9 and run ./producer.

   

4. Click on the unicorn to see more details from the stream and compare with the consumer output.

   

7. Experiment with the producer

Stop and start the producer while watching the dashboard and the consumer. Start multiple producers with different unicorn names.

1. Stop the producer by pressing Control + C and notice the messages stop and the unicorn disappear after 30 seconds.

2. Start the producer again and notice the messages resume and the unicorn reappear.

3. Hit the (+) button and click New Terminal to open a new terminal tab.

4. Start another instance of the producer in the new tab. Provide a specific unicorn name and notice data points for both unicorns in consumer’s output:

   ./producer -name Bucephalus
   

5. Check the dashboard and verify you see multiple unicorns.

   

⭐ Recap

🔑 Amazon Kinesis makes it easy to collect, process, and analyze real-time, streaming data so you can get timely insights and react quickly to new information.

🔧 In this module, you’ve created an Amazon Kinesis stream and used it to store and visualize data from a simulated fleet of unicorns.

Next

✅ Stop the producers. Depending on your time or interest you can proceed to either one of these modules

1. Stream Processing - You will process the streaming data using AWS Lambda and store in Amazon DynamoDB table and explore stream analytics with AWS Lambda, or
2. Stream Aggregation - You will store the summary data from the Kinesis Data Analytics application in Amazon DynamoDB.
3. Data Lake - You will deliver the raw stream data from the Kinesis stream to Kinesis Data Firehose for delivery to Amazon S3 and use Amazon Athena to query that raw data.

Alternatively you can follow the modules in sequence by going to Stream Processing.