Amazon Kinesis allows for collecting and processing large streams of data records in real time. Applications created on Kinesis can run on Amazon EC2 instances and typical uses are to send processed records to dashboards, generate alerts, dynamically change pricing or advertising strategies, or send data to other AWS services.

Note: Currently only region "eu-west-1 is supported by EDS.

Amazon S3 allows for storage of the raw event and usage data as it arrives from a data stream. To receive a data stream in S3, access keys from an AWS account are used in the EUI for authorization. Shortly after the stream is created, CSV files are uploaded to the S3 bucket; one for event data and one for usage data. The CSV files can then be send to other Amazon services or consumed by a third-party analytics or BI tool for generating insights.

With the data streamer Salesforce configuration, it’s possible to stream your event data directly into a Salesforce account. This is done by setting up a connected app in Salesforce which provides a client id and client secret. These credentials are used in the EMnify User Interface to grant access to the Salesforce platform event system.

NOTE: Consider configuring this data streamer type for selected event types only as there are limits on the number of events supported. Location Updates or PDP Context events can be extremely frequent and are probably not the most typical to be processed in Salesforce. More information on this feature is detailed in the section Filtering below.

4. Configuration by Integration

4.1. AWS S3 Setup

1. In your AWS account, navigate to S3 and create a new bucket.

Data Streamer Add S3 bucket

Currently only region eu-west-1 (EU Ireland) is supported.

Bucket names should be DNS-compliant, see AWS docs: Rules for Bucket Naming.

2. In IAM, select or create a user which has access to this S3 bucket, ideally access limited only to this bucket.

a. Navigate to Security Credentials and select Create access key. Securely store the Access ID and Secret Key

3. Log on to EUI and add an S3 Data Stream.

Data Streamer Add S3 bucket

Shortly after the stream is created, .csv files will be uploaded to the S3 bucket:

Data Streamer S3 bucket with data

Event Data

The event data filename follows the format events_YYYYMMDD_HHmmss.csv.

Usage Data

The usage data filename follows the format cdr_YYYYMMDD_HHmmss.csv.

4.1.1. S3 Security Guidelines

Event data that is sent via Data Streams may include usernames, email addresses and other data which can identify users or platform resources. The generated .csv files should therefore be treated as containing sensitive information. Precautions should be taken to ensure that the event and usage data in the destination S3 buckets are adequately secured.

The following three steps should be considered as the minimum security requirements for storing such data in S3:

1. Ensure that the S3 bucket is not publicly accessible. This can be applied in the Permissions tab of the S3 bucket:

Blocking Public Access on the S3 Bucket

2. Server-Side Encryption can be enabled per-bucket and S3 will encrypt objects before they are saved to disk. The decryption is then performed when downloading the objects. This can be enabled in the Properties tab of the S3 bucket:

Enabling Server-Side Encryption

3. The IAM user whose credentials (key ID & key secret) access the S3 bucket should have their permissions restricted to writing to the required bucket only. This can be done in the AWS console in Users → {Data Stream User} → Permissions → Add Permissions → Attach Policy Directly → Create Policy. An example JSON Policy would look like:

AWS provide an online JSON Policy Generator which can be used to create a policy like the example given above.

4.2. AWS Kinesis Setup

1. Log on to your AWS account

2. Navigate to Kinesis and create one or two new streams

                                    Currently only region eu-west-1 (EU Ireland) is supported.

3. Navigate to IAM, select or create a user which has access to this Kinesis server, ideally limited access to this server only.

a. Navigate to Security Credentials and select Create access key. Securely store the Access ID and Secret Key 

4. Log on to EUI and add a Kinesis Data Stream:

Add Kinesis Data Stream

4.3. Salesforce Setup

With the data streamer Salesforce configuration, it’s possible to stream your event data directly into a Salesforce account.

In Salesforce, we use events described as Platform Events which are intended for streaming data between external apps and Salesforce. More information about Platform Events and how to implement them can be found in the Salesforce Documentation - Platform Events.

Consider configuring this data streamer type for selected event types only as there are limits on

the number of events supported. Location Updates or PDP Context events can be extremely frequent and are probably not the most typical to be processed in Salesforce. Details of how to achieve this can be found in Filtering Data Streams.

To prepare Salesforce system to receive our event stream you need to perform the following steps:

1. Setup a connected app in Salesforce which will provide your client id and client secret
2. Obtain a refresh token via Oauth2 to grant the data streamer access to your Salesforce platform event stream
3. Create a new platform event in your Salesforce account

Afterwards, add the following custom fields and assign them to your previously created platform event:

Salesforce fields required for Data Streamer Events

Now you can configure your Salesforce event stream in EUI. Your platform event name will be reflected as event stream name.

EDS S3 add stream

4.4. keen.io Setup

keen.io offers APIs for streaming, analyzing, and embedding rich data and integrates with the EMnify Data Streamer. To get the EDS integrated with keen.io and run your first queries and build dashboards, the following steps are required:

1. Create a keen.io user account

2. Create a new project, e.g. EMnify Data Stream

3. In the project settings go to Access to find your project id and write key

Configuration for keen.io access

4. Log on to the EUI and add a keen.io stream using the Project ID and Write Key

EDS keen.io add stream

Shortly after the stream is created you will see first data arriving, which you can explore on keen.io on the streams tab. Depending on the stream type you selected there will one or two streams appear with the names EMnifyEventData and EMnifyUsageData

You choose the data collection, e.g. EMnifyUsageData and click on Last 10 Events Streamed to see the live event data coming in.

Streaming data arriving in keen.io

4.5. DataDog Setup

DataDog provides real-time performance monitoring (https://www.datadoghq.com). In conjunction with EDS it allows you to collect and analyze metrics about the usage of your endpoints and SIM cards, you can create dashboards and trigger alerts on certain situations.

1. Create a DataDog user account
2. Navigate to Integrations → APIs from the left navbar
3. Generate a new Application Key
4. Log on to the EUI and add a DataDog stream

EDS DataDog add stream

{
    "cost": 0.00558275,
    "id": 174321498,
    "operator": {
            "id": 4,
        "name": "EPlus",
        "country": {
            "id": 74,
            "name": "Germany"
        }
    },
    "organisation": {
        "id": 399921,
        "name": "Test"
    },
    "tariff": {
        "ratezone": {
            "id": 83,
            "name": "Europe_I"
        },
        "id": 64,
        "name": "Global Pro I"
    },
    "traffic_type": {
        "id": 5,
        "name": "Data"
    },
    "endpoint": {
        "id": 8392037
    },
    "imsi": "901430999910777",
    "volume": {
        "rx": 0.0138,
        "tx": 0.008531,
        "total": 0.022331
    },
    "start_timestamp": "2017-03-19 21:06:33",
    "sim": {
        "iccid": "8988303000000011085",
        "id": 233746
    },
    "currency": {
        "symbol": "€",
        "code": "EUR",
        "id": 1
    },
    "end_timestamp": "2017-03-19 21:21:23"
}

{
    "id": 201388127,
    "alert": false,
    "description": "Failed authentication request from 'user@company.com', Reason: Invalid password from IP 9.9.9.9",
    "timestamp": "2017-10-26T07:42:00.000+0000",
    "event_type": {
        "id": 6,
        "description": "User authentication failed"
    },
    "event_source": {
        "id": 2,
        "description": "API"
    },
    "event_severity": {
        "id": 1,
        "description": "Warn"
    },
    "organisation": {
        "id": 839921,
        "name": "Demo Company"
    },
    "user": {
        "id": 84993,
        "username": "user@company.com",
        "name": "Scott Tiger"
    }
}

{
    "id": 201370709,
    "alert": false,
    "description": "New location received from VLR for IMSI='90143012345678912345', now attached to VLR='491720013095'.",
    "timestamp": "2017-10-26T07:28:00.000+0000",
    "event_type": {
        "id": 1,
        "description": "Update location"
    },
    "event_source": {
        "id": 0,
        "description": "Network"
    },
    "event_severity": {
        "id": 0,
        "description": "Info"
    },
    "organisation": {
        "id": 839921,
        "name": "Demo Company"
    },
    "endpoint": {
        "id": 8638726,
        "name": "GPS Tracker",
        "ip_address": "100.96.234.249",
        "tags": null,
        "imei": "3577620833012201"
    },
    "imsi": {
        "id": 205672,
        "imsi": "90143012345678912345",
        "import_date": "2016-12-27T10:09:23.000+0000"
    },
    "sim": {
        "id": 274887,
        "iccid": "8988303001234567890",
        "production_date": "2016-12-27T10:09:23.000+0000"
    },
    "detail": {
        "id": 3,
        "name": "Vodafone",
        "country": {
            "id": 74,
            "name": "Germany",
            "country_code": "49",
            "mcc": "262",
            "iso_code": "de"
        },
        "tapcode": [{
            "id": 2,
            "tapcode": "DEUD2"
        }],
        "mnc": [{
            "id": 3,
            "mnc": "02"
        }]
    }
}