Chapter 1. Introduction
Purpose of This Document
This document describes the functionality of the EMnify Data Streamer and covers the procedures necessary to create and configure data streams. Data streams are detailed in terms of their potential uses with particular attention to the advantages of data streams in conjunction with third-party platforms. The typical use cases of data streams are described and examples of data streams in different domains are covered. This document also provides information on how to integrate data streams with selected third-party platforms via the EMnify User Interface (EUI).
Chapter 2. Data Streamer Overview
The EMnify Data Streamer (EDS) allows you to receive real-time data streams with event and usage data of endpoints and SIMs. Data streams created using this feature can be ingested by any third party analytics application or can be pushed to pre-integrated cloud services.
The following figure gives a high-level overview of the EDS system, showing which types of event data and usage metrics can be routed via data streams.
Chapter 3. Use Cases
The EMnify data streamer can be used for a wide range of use cases. The following section covers three domains.
3.1. Network Monitoring
Based on the rich network signaling events that the EMnify Data Streamer provides, users can get insights into which organizations and visited networks that signaling traffic is originating from. This enables network monitoring and service assurance of the underlying signaling traffic for SS7/SIGTRAN, Diameter and GTP traffic.
Using monitoring platforms updated in real time, network issues such as signaling storms or connectivity loss can easily be detected by means of alerts or notifications. When alerting is configured in this way, they can be used for issue escalation to connectivity carriers, VPLMNs or organizations that are causing issues. Customers can be informed on network issues before they realize that they are impacted.
3.2. Business Intelligence
The EMnify Data Streamer provides analytics for business decisions and rich information for reporting. Insight reports which show the largest customers by volume, region and country may provide deciding factors for roaming negotiations.
The impact of marketing campaigns on new sign-ups or increase of service usage can be analyzed. Finance departments can use this reporting to forecast revenue streams and validate billing. Service Operations can additionally analyze the usage of a system feature to plan for capacity or connectivity expansions.
3.3. Customer Relationship Management
Events generated by the Data Streamer can be used to enrich Customer Relationship Management (CRM) data with insights on how new and existing customers are using their system. Sales and Customer Success Management receive insights if customers are close to reaching traffic limits or if they have already depleted a free volume during an evaluation period.
With service usage information, Sales can prepare for a next round of negotiations as well as forecasting cost and revenue streams.
Chapter 4. Integrations
Customers who wish to use the RestAPI data stream must provide an API which itself consumes the data stream. If you choose to implement the EMnify API on your server, the Data Streamer will post data in the form of JSON objects as they occur.
This is the most flexible method of processing a data stream as it allows any implementation of analytics, reporting or a pipeline of tools to process usage and event data.
4.1.1. RestAPI (Bulk-Mode)
Using the EMnify API in bulk-mode, each HTTP POST will include an array of objects. The HTTP POSTs are sent at intervals and should be used if the receiving system should process multiple events in bulk instead of individual events as they occur.
4.2. AWS Kinesis
Amazon Kinesis allows for collecting and processing large streams of data records in real time. Applications created on Kinesis can run on Amazon EC2 instances and typical uses are to send processed records to dashboards, generate alerts, dynamically change pricing or advertising strategies, or send data to other AWS services.
Currently only region eu-west-1 (EU Ireland) is supported.
4.3. AWS S3
Amazon S3 allows for storage of the raw event and usage data as it arrives from a data stream. To receive a data stream in S3, access keys from an AWS account are used in the EUI for authorization. Shortly after the stream is created, CSV files are uploaded to the S3 bucket; one for event data and one for usage data. The CSV files can then be send to other Amazon services or consumed by a third-party analytics or BI tool for generating insights.
Currently only region eu-west-1 (EU Ireland) is supported.
With the data streamer Salesforce configuration, it’s possible to stream your event data directly into a Salesforce account. This is done by setting up a connected app in Salesforce which provides a client id and client secret. These credentials are used in the EMnify User Interface to grant access to the Salesforce platform event system.
Note: Consider configuring this data streamer type for selected event types only as there are limits on the number of events supported. Location Updates or PDP Context events can be extremely frequent and are probably not the most typical to be processed in Salesforce. More information on this feature is detailed in the section Filtering.
4.5. Additional Integrations
keen.io keen.io offers APIs for streaming, analyzing, and embedding rich data and integrates well with the EDS. The Access page of a Keen.io project displays a project ID and write key. These credentials are used in the EUI to grant the data streamer access to a Keen.io project.
DataDog provides real-time performance monitoring and analytics. In conjunction with EDS it allows you to collect and analyze metrics about the usage of your endpoints and SIM cards, you can create dashboards and trigger alerts on specific events using DataDog explorer.
Chapter 5. Features
One of the more dynamic features of the data streamer is the capability to apply filters to each data stream. By default, no filters are added to a data stream and all events are streamed. Multiple filters can be applied to each stream and this creates more granular and targeted data for analysis.
The following screenshot shows filters applied to a data stream via the EMnify User Interface. The data stream that the filters are applied to will only contain Update Location and User Authentication failed events.
5.2. Parallel Operation
Up to 10 data streams can be created and they will operate in parallel. Each data stream may feed data to a separate platform for consumption and processing. Different filtering rules can be applied to each of these data streams. More information about filters is detailed in the Filtering section.
5.3. Historical Data
When creating a new data stream, it’s possible to enable streaming historical data up to 20 days old. If this setting is enabled, it may take time for data to catch up to real-time events.
Chapter 6. Conclusion
The EMnify Data Streamer offers a real-time pipeline of event and usage data of endpoints and SIMs. The benefits of using data streams lies in the flexibility of how they are consumed, leading to a rich source of data that can feed multiple platforms for different purposes. The insights that can be gained from data streams can be fine-tuned by filtering and can range from a system-wide overview to fine-grained analytics by a specific event, topic or custom range.
The benefits of data streams are outlined below:
The usage of the system is fully transparent by creating and monitoring data streams that are directly relevant to a customer. This enables network monitoring and service assurance, Business Reporting and Customer Relationship Management.
Users can set in place a topic-based subscription to specific events and focus analysis on areas that matter.
Existing third-party platforms are pre-integrated into the EMnify system for the practicality of making use of data streams. These are easily and instantly applied via a setup wizard in the EMnify User Interface. Data streams use industry standard formats of JSON for RestAPI or CSV for consumption of the data.
Users of data streams can gain insights into the status of their networks without additional implementation of infrastructure between system components. Customers can manage and control the insights and analytics based on their interests and needs using any platform or tooling.
Chapter 7. Resources
7.1. Other Documentation
For additional resources, refer to the following resources:
EMnify Data Streamer User Manual
This document provides detailed instructions for activating and managing data streams with a step-by-step guide for activating third-party integrations. This document also contains a comprehensive reference of the data and objects contained within data streams for custom integrations.
More information can be found on the link below:
For questions and inquiries, contact support at email@example.com