Articles in this section

How to setup the EMnify Data Streamer (EDS)

Avatar
Brian Smith
  • Updated
Follow

The EMnify Data Streamer (EDS) allows you to receive real-time data streams with event and usage data of endpoints and SIMs. Data streams created using this feature can be ingested by any third party analytics application or can be pushed to pre-integrated cloud services.

1. Creating a Data Stream

To create a Data Stream, log in to an EUI account and navigate to the Technical Settings page by clicking on the "link" icon in the top-right navigation menu.
The Technical Settings page contains a panel dedicated to configuring and displaying Data Streams.

2018-11-02_15h48_46.png

1. Click on the action button "Add Data Stream".
The stream type can be any one of the following:

  • Usage Data
  • Event Data
  • Usage and Event Data combined in a single stream


2. Choose an "API Type". This can be any one of the following:

  • EMnify API
  • EMnify API (Bulk-mode)
  • Salesforce
  • keen.io
  • DataDog
  • AWS S3
  • AWS Kinesis

NOTE: Depending on the API type, API keys and configuration parameters may be required.
Details about the types of data streams available is covered in the <<Integrations>> section.

3. If past data is required, the checkbox "Stream historic data" should be checked.
The system will send historic data up to 20 days old. It may take time to catch up with live data if this option is enabled.

4. Click `Add Data Stream`

When created, new data streams are instantly active and you can monitor their status on the info panel on the *Technical Settings* page.
The column "Remote Status" displays the HTTP status code from the remote side of the stream.
This should be `200` (OK) under normal conditions.
If the configuration is invalid or the receiving component or server is down, it will show `5XX` errors.

2018-11-02_15h55_03.png

2. Modifying a Data Stream

Pausing a data stream:
A data stream can be paused and resumed at any time by using the action buttons on the right of the stream info. This is useful for actions such as maintenance of the receiving server.

Removing a data stream:
To permanently remove a data stream, click the icon:times[] *remove* icon on the right.
You will be asked to confirm this action.

Parallel data streams:
It is possible to add up to 10 data streams which run in parallel.
This means that one stream may be connected to a network monitoring system, another can connect to data analytics platform and another can be syncing with S3 for archiving at the same time, and so on.

Filtering:
It's possible to add filters to a data stream which will filter the entire stream to contain only a certain event type or usage data type. This is covered in more detail in the section Filtering below.

3. Integrations

3.1. EMnify RestAPI

Customers who wish to use the RestAPI data stream must provide an API which itself consumes the data stream. If you choose to implement the EMnify API on your server, the EDS will post data in the form of JSON objects as they occur.

This is the most flexible method of processing a data stream as it allows any implementation of analytics, reporting or a pipeline of tools to process usage and event data.

3.1.1. EMnify API (Bulk-Mode)

If you select EMnify API in bulk-mode, each HTTP POST will include an array of objects. The HTTP POST are sent at intervals and should be used if the receiving system should process multiple events in bulk instead of individual events as they occur.

3.2. AWS Kinesis

Amazon Kinesis allows for collecting and processing large streams of data records in real time. Applications created on Kinesis can run on Amazon EC2 instances and typical uses are to send processed records to dashboards, generate alerts, dynamically change pricing or advertising strategies, or send data to other AWS services.

Note: Currently only region "eu-west-1 is supported by EDS.

 

3.3. AWS S3

Amazon S3 allows for storage of the raw event and usage data as it arrives from a data stream. To receive a data stream in S3, access keys from an AWS account are used in the EUI for authorization. Shortly after the stream is created, CSV files are uploaded to the S3 bucket; one for event data and one for usage data. The CSV files can then be send to other Amazon services or consumed by a third-party analytics or BI tool for generating insights.

3.4. Salesforce

With the data streamer Salesforce configuration, it’s possible to stream your event data directly into a Salesforce account. This is done by setting up a connected app in Salesforce which provides a client id and client secret. These credentials are used in the EMnify User Interface to grant access to the Salesforce platform event system.

NOTE: Consider configuring this data streamer type for selected event types only as there are limits on the number of events supported. Location Updates or PDP Context events can be extremely frequent and are probably not the most typical to be processed in Salesforce. More information on this feature is detailed in the section Filtering below.

3.5. Additional Integrations

keen.io

keen.io offers APIs for streaming, analyzing, and embedding rich data and integrates well with the EDS. The Access page of a Keen.io project displays a project ID and write key. These credentials are used in the EUI to grant the data streamer access to a Keen.io project.

DataDog

DataDog provides real-time performance monitoring and analytics. In conjunction with EDS it allows you to collect and analyze metrics about the usage of your endpoints and SIM cards, you can create dashboards and trigger alerts on specific events using DataDog explorer.

4. Features

4.1. Filtering

One of the most powerful features of a data stream is the capability to apply filters to each individual stream. By default, no filters are added to a data stream and all events are streamed. Multiple filters can be applied to each stream and this creates more granular and targeted data for analysis.

The following screenshot shows filters applied to a data stream via the EMnify User Interface. The data stream that the filters are applied to will only contain Update Location and User Authentication failed events.

2018-11-02_16h05_28.png

5. Event Data Reference

5.1. Generic Event Data

The following table shows the properties in generic event data. The following properties are included in all events:

Property                Format                     Description

id

Numeric

A unique numerical identifier of this event. If multiple events with same id are received (e.g. due to transmission errors) these should be treated by the receiver as duplicates

timestamp

Timestamp

Date/time when this event happened

event_type

Nested Object

Type of the event, see Event Types for details

event_severity

Nested Object

Severity of the event, see Severity Levels for details

event_source

Nested Object

Source of the event, see Event Source for details

organisation

Nested Object

Organisation associated with this event, see Organisation Object for details

alert

Boolean

Event is a candidate to be alerted to an user

description

String

Human readable description of the event

 

5.2. Additional Properties

Event types relating specifically to SIMs, endpoints and users include the following additional properties:

Property Format                      Description

imsi

Nested Object

Details of IMSI, see IMSI Object for details (in case of multi-IMSI configuration, multiple different IMSIs may be reported for the same SIM)

sim

Nested Object

Details of SIM, see SIM Object for details

endpoint

Nested Object

Details of Endpoint, see Details Object for details

 

5.3. Additional Details by Event Type

The data streamer will send additional data when available depending on the event type. This data is added as a nested object called detail and contains information on the actual used Mobile Network Operator and country. See Details Object for more information.

5.4. Details Object

Property Format                                     Description

id

Numeric

Unique identifier of the actual used Mobile Network Operator

name

String

Name of the Mobile Network Operator

country

Nested Object

Country of Mobile Network Operator

country.id

Numeric

Unique identifier of the country

country.name

String

Name of country

country.country_code

String

Country code

country.mcc

String

Mobile Country Code (MCC)

country.iso_code

String

ISO code

pdp_context

Nested Object

PDP Context Details

volume

Nested Object

Volume consumed in PDP Context

volume.rx

Number (up to 6 decimal places)

Downstream Volume in MB

volume.tx

Number (up to 6 decimal places)

Upstream Volume in MB

volume.total

Number (up to 6 decimal places)

Total volume

 

5.5. PDP Context

Property                                            Format                  Description

pdp_context_id

String

Unique identifier of this PDP context

tunnel_created

Timestamp

Date/time when this PDP context was created

gtp_version

String

GTP Version (either 1 or 2)

ggsn_control_plane_ip_address

String

IP Address of GGSN/PGW Control Plane

ggsn_data_plane_ip_address

String

IP Address of GGSN/PGW Data Plane

sgsn_control_plane_ip_address

String

IP Address of SGSN/SGW Control Plane

sgsn_data_plane_ip_address

String

IP Address of SGSN/SGW Data Plane

region

String

Region where Data Plane is located

breakout_ip

String

IP Address used for Internet Breakout

apn

String

Access Point Name (APN)

nsapi

Integer

Network Service Access Point Identifier (NSAPI)

ue_ip_address

String

IP address assigned to Endpoint

imeisv

String

IMEISV

mcc

String

Mobile Country Code (MCC)

mnc

String

Mobile Network Code (MNC)

lac

Integer

Location Area Code (LAC)

sac

Integer

Service Area code (SAC)

rac

Integer

Routing Area code (RAC)

ci

Integer

Cell Identification (CI)

rat_type

Integer

Radio Access Type (RAT).
Integers [1-6] correspond to the following values:

1: 3G
2: 2G
3: WLAN
4: GAN
5:HSPA+
6:4G

 

5.6. Organisation Object

Property Format          Description

id

Numeric

Unique identifier of this organisation

name

String

Name of Organisation

 

5.7. User Object

Property Format         Description

id

Numeric

Unique identifier of this user

username

String

Username e.g. email address

name

String

Realname of user

 

5.8. IMSI Object

Property                         Format                         Description

id

Numeric

Unique identifier of this IMSI

imsi

String

International mobile subscriber identity (IMSI)

import_date

Timestamp

Date/Time this IMSI was provisioned

 

5.9. SIM Object

Property                           Format                         Description

id

Numeric

Unique identifier of this SIM

iccid

String

Integrated Circuit Card identifier (ICCID) without checksum digit

msisdn

String

MSISDN

production_date

Timestamp

Date/Time this SIM chip was produced

 

5.10. Endpoint Object

Property               Format                 Description                                                                   

id

Numeric

Unique identifier of this Endpoint

name

String

Configured name of this endpoint

ip_address

String

IP Address assigned to this Endpoint

tags

String

Tags assigned to this Endpoint

imei

String

International mobile equipment identity (IMEI)

 

5.11. Event Types

Id            Description

0

Generic

1

Update location

2

Update GPRS location

3

Create PDP Context

4

Update PDP Context

5

Delete PDP Context

6

User authentication failed

7

Application authentication failed

8

SIM activation

9

SIM suspension

10

SIM deletion

11

Endpoint blocked

12

Organisation blocked

13

Support Access

14

Multi-factor Authentication

15

Purge Location

16

Purge GPRS location

17

Self-Signup

18

Threshold reached

19

Quota used up

 

5.12. Severity Levels

Id            Description

0

INFO

1

WARN

2

CRITICAL

 

5.13. Event Source

Id            Description

0

Network

1

Policy Control

2

API

 

6. Usage Data Reference

6.1. Usage Data Properties

Property Format                   Description

id

Numeric

Unique identifier of this transaction

cost

Number

Cost calculation of reported traffic volume. May use up to 6 decimal places.

currency.id

Numeric

Unique identifier of currency of indicated cost

currency.code

ISO 4217

Currency Code

start_timestamp

UTC Timestamp

Start time of traffic measurement

end_timestamp

UTC Timestamp

End time of traffic measurement

volume.rx

Number

Downstream traffic (MB) received by the endpoint. May use up to 6 decimal places.

volume.tx

Number

Upstream traffic (MB) send by the endpoint. May use up to 6 decimal places.

volume.total

Number

Total traffic consumed. May use up to 6 decimal places.

imsi

15 digit numeric string

Currently used IMSI

endpoint.id

Numeric

Unique identifier of endpoint

sim.id

Numeric

Unique identifier of SIM

sim.iccid

19 digit numeric string

ICCID of SIM

organisation.id

Numeric

Unique identifier of organisation

organisation.name

String

Name of organisation

operator.id

Numeric

Unique identifier of visited operator

operator.name

String

Name of that mobile operator

operator.country.id

Numeric

Unique identifier of visited country

operator.country.name

String

Name of visited country

tariff.id

Numeric

Unique identifier of applied tariff

tariff.name

String

Name of Tariff

tariff.ratezone.id

Numeric

Unique identifier of applied ratezone

tariff.ratezone.name

String

Name of Ratezone

traffic_type.id

Numeric

Unique identifier of traffic type

traffic_type.name

String

Name of traffic type

 

6.2. Usage Data JSON Sample

{
    "cost": 0.00558275,
    "id": 174321498,
    "operator": {
            "id": 4,
        "name": "EPlus",
        "country": {
            "id": 74,
            "name": "Germany"
        }
    },
    "organisation": {
        "id": 399921,
        "name": "Test"
    },
    "tariff": {
        "ratezone": {
            "id": 83,
            "name": "Europe_I"
        },
        "id": 64,
        "name": "Global Pro I"
    },
    "traffic_type": {
        "id": 5,
        "name": "Data"
    },
    "endpoint": {
        "id": 8392037
    },
    "imsi": "901430999910777",
    "volume": {
        "rx": 0.0138,
        "tx": 0.008531,
        "total": 0.022331
    },
    "start_timestamp": "2017-03-19 21:06:33",
    "sim": {
        "iccid": "8988303000000011085",
        "id": 233746
    },
    "currency": {
        "symbol": "€",
        "code": "EUR",
        "id": 1
    },
    "end_timestamp": "2017-03-19 21:21:23"
}

6.3. Example Events

The following section describes the properties contained in events.

User Authentication Failed
{
    "id": 201388127,
    "alert": false,
    "description": "Failed authentication request from 'user@company.com', Reason: Invalid password from IP 9.9.9.9",
    "timestamp": "2017-10-26T07:42:00.000+0000",
    "event_type": {
        "id": 6,
        "description": "User authentication failed"
    },
    "event_source": {
        "id": 2,
        "description": "API"
    },
    "event_severity": {
        "id": 1,
        "description": "Warn"
    },
    "organisation": {
        "id": 839921,
        "name": "Demo Company"
    },
    "user": {
        "id": 84993,
        "username": "user@company.com",
        "name": "Scott Tiger"
    }
}
Update Location
{
    "id": 201370709,
    "alert": false,
    "description": "New location received from VLR for IMSI='90143012345678912345', now attached to VLR='491720013095'.",
    "timestamp": "2017-10-26T07:28:00.000+0000",
    "event_type": {
        "id": 1,
        "description": "Update location"
    },
    "event_source": {
        "id": 0,
        "description": "Network"
    },
    "event_severity": {
        "id": 0,
        "description": "Info"
    },
    "organisation": {
        "id": 839921,
        "name": "Demo Company"
    },
    "endpoint": {
        "id": 8638726,
        "name": "GPS Tracker",
        "ip_address": "100.96.234.249",
        "tags": null,
        "imei": "3577620833012201"
    },
    "imsi": {
        "id": 205672,
        "imsi": "90143012345678912345",
        "import_date": "2016-12-27T10:09:23.000+0000"
    },
    "sim": {
        "id": 274887,
        "iccid": "8988303001234567890",
        "production_date": "2016-12-27T10:09:23.000+0000"
    },
    "detail": {
        "id": 3,
        "name": "Vodafone",
        "country": {
            "id": 74,
            "name": "Germany",
            "country_code": "49",
            "mcc": "262",
            "iso_code": "de"
        },
        "tapcode": [{
            "id": 2,
            "tapcode": "DEUD2"
        }],
        "mnc": [{
            "id": 3,
            "mnc": "02"
        }]
    }
}

 

 

Related articles

Comments

0 comments

Please sign in to leave a comment.