In the EMnify User Interface (EUI), events provide details on the connectivity status of devices using EMnify SIMs. The Event Data Streamer provides a single interface for streaming platform events to application servers or other developer tools. In this tutorial, we will configure a basic but flexible notification service using the following AWS tools:
-
S3 to store platform events as
.csvfiles -
Lambda to read new CSV files, filter and route new events to SNS topics if we are interested in them. The two topics we use are:
-
A Slack Topic with a lambda subscriber for sending Slack notifications
-
An Email Topic with a lambda subscriber for sending Email notifications.
-
Finished source code for the above stack with more detailed examples is available in the following GitHub Repository.
Stream Platform Events to S3
The EMnify Data Streamer can be used to stream events directly to S3. The access credentials and the destination bucket should be configured in the EMnify User Interface. This has been covered in in detail in the previous EMnify webinar "How to store EMnify events and usage data in AWS S3".
CSV Read & Routing Lambda
The first part of the example stack is a Python lambda which parses and routes incoming messages whenever a new object is written to our S3 bucket. To create this, open the Lambda service in AWS and create a new Python lambda from scratch called s3tosns.py. Create a new role for the lambda which should have the following permissions:
-
ReadOnly Access for the S3 bucket with the event data
-
Write permissions to CloudWatch Logs
-
SNS Publish permissions
A basic working lambda which simply reads the event CSV file looks like the following:
Click + Add Trigger, select S3 and the bucket used for storing incoming EMnify events and select All object create events as the event type to trigger the lambda:
NOTE: Adding a trigger for this lambda can also be done on the S3 bucket directly at Properties → Events → +Add Notification and point the notification to the Lambda above.
At this point, the first lambda should be running and reporting incoming events to the console each time we have EMnify platform events.
Add an Email Notification
To get started with SNS, navigate to Simple Notification Service (SNS) in AWS. Create a topic by clicking on Topics → Create Topic. Name the topic with something like my-email-topic and in Access Policy, allow everyone to publish and subscribe to the topic (this will be restricted to only our Lambdas later). After the topic is created, make note of the ARN which should resemble the following format:
arn:aws:sns:{region}:my-email-topic
Lambda Subscriber
While it’s possible to add email addresses as direct subscribers to SNS topics, these emails are always sent in plain text without the possibility to add formatting. To receive an email with a user-friendly and readable HTML template, we will add a Lambda subscriber which reads the SNS message and populates a template with the message contents.
First, an email should be enabled in the SES service via "Email addresses" and Verify a New Email Address. After clicking the activation link sent via email, SES can now use this account to send emails:
Navigate to AWS Lambda and create a new Python lambda SNStoSES.py with permissions to access SES. Click on + Add trigger and select the SNS topic created earlier for email notifications (e.g. arn:aws:sns:{region}:my-email-topic). For getting this lambda to work, copy the following two files:
-
SNStoSES.py - the full example Python lambda
-
template.html - the template used by
SNStoSES.py(this should be saved in the same directory as theSNStoSES.pyPython lambda)
When SNStoSES.py lambda runs, it will replace template strings in the HTML and send a nicer-looking email:
To activate this trigger from real platform events, edit the s3tosns.py lambda to publish messages to our email topic. In the section where we log events to console, invoke the SNS publish method instead:
The email notification is now active for all incoming events!
Add a Slack Integration
In the same way that we created an email notification topic, create a Slack topic named my-slack-topic and make note of the ARN which should be something like arn:aws:sns:{region}:my-slack-topic and ensure that it has a relaxed access policy for the setup steps.
Lambda Subscriber
For the Slack lambda, an incoming webhook should be created for the destination Slack workspace. Create a new Slack app with the type Incoming Webhook and make note of the Webhook URL (these steps are documented in detail in the Slack Documentation).
Navigate to AWS Lambda and create a new Node.js file with permissions to access SES, this one can be called SNStoSlack.js. Paste the example code from the provided GitHub gist and change the {MY_WEBHOOK_URL} value to the URL from your newly-created Slack app.
Click + Add trigger and select the SNS topic created for Slack notifications (e.g. arn:aws:sns:{region}:my-slack-topic). It’s now possible to send SNS publish events from our main routing lambda. In the section where we are reading the CSV rows, change the SNS publish step to use our Slack topic:
The integration with your custom incoming webhook is now complete and when new events arrive from the EMnify platform Slack notifications will arrive in your workspace:
Cleaning Up and Adding Filtering
Now that we have two externally-publishing lambdas that post incoming SNS messages to either email or Slack channels, we can clean up our routing lambda and add some basic filtering. The final version of the example read & routing lambda looks like the following:
In this basic example, we are doing some routing to different SNS topics based on the text content of the event_type_description row of our event CSV entry.
Hints
- Test events
-
Mock test events when creating Lambdas by clicking Select a test event → Configure test events and adding some dummy data. For testing the email and Slack lambdas from this example, use an sns mock message.
- Better Templating
-
SES offers a service for handlebars-like HTML templates. The downside to using this feature is that the templates may only be created via the CLI. An example HTML template which can be uploaded to SES via a JSON file and can be invoked using the
.send_templated_mailfunction as demonstrated in this example lambda. - Restrict Access Policies
-
Some other steps for cleaning up would be restricting the access policies for the SNS topics so that only our newly-created Lambdas can publish and subscribe.
- Add More Filtering
-
Filtering by event type can be done in the EMnify Data Stream in the EUI, in our first 'routing' lambda and also in the SNS topic subscriptions. For examples with finer granularity on match criteria on incoming SNS messages, see the AWS documentation on SNS Subscription Filter Policies.
Comments
0 comments
Please sign in to leave a comment.