In the following article, we will describe how to set up the breakout connectivity between the EMnify Production Network and the customer's premises in Microsoft Azure.
- Problem Statement
EMnify has a constantly increasing number of customers who deploy their Application Servers on Azure infrastructure and require these Application Servers to be reachable from IoT devices located worldwide. To achieve this, breakout connectivity needs to be set up between the EMnify production network and the customer premises.
At the same time, in order to better address the reliability and QoS demands of the customers, EMnify is migrating its connectivity services to AWS native services, like the AWS VPN Connection and AWS Transit Gateway.
This document describes the steps an Azure customer should follow in order to successfully set up the breakout connectivity with the EMnify production network. In the more general sense, this is a guide on how to interconnect the native VPN services of AWS and Azure.
As a prerequisite, the basic infrastructure to host the customer’s Application Servers is supposed to have been set up in the Azure account. This includes the following resources:
- A VNet (awsintercon-vnet) – IP address range: 10.10.0.0/16;
- A subnet (awsintercon-subnet) – IP address range: 10.10.1.0/24;
- A VM (awsintercon-vm) – configured with the private IP address 10.10.1.4. Will be used to test the end-to-end connectivity. This VM basically represents an Application Server;
- One public IP address (awsintercon-vgw-ip) – used by the Virtual Network Gateway (may also be assigned during the Virtual Network Gateway configuration)
- A Route Table (awsintercon-rt) – this route table defines the routing for the VM and it may be initially empty. It shall be updated to enable the routing to the endpoints via the VPN, as subsequently explained;
- A Security Group (awsintercon-sg) – this security group defines the default firewall policies. It shall be updated to enable incoming traffic from the endpoints, as subsequently explained;
Note that the resource naming and IP address prefixes are customer specific and the ones used above are for exemplification purposes only.
3. VPN Connectivity Setup – Virtual Network Gateway
To set up breakout connectivity, a Virtual Network Gateway (awsintercon-vgw) has to be setup in the awsintercon-vnet VNet, as illustrated in Figure 1.
Figure 1 Virtual Network Gateway creation
The Virtual Network Gateway is associated to awsintercon-vnet, uses the awsintercon-vgw-ip public IP and a private subnet of awsintercon-vnet that is different from awsintercon-subnet.
Note that the VPN type shall be set to “Route-based”. Also, in the present scenario BGP as well as the “active-active” mode are disabled.
4. Configuration Parameter Exchange with EMnify
As soon as the public IP address of the customer’s Virtual Network Gateway is known, the following configuration parameters shall be exchanged with EMnify:
- The customer provides:
- The public IP address of the Virtual Network Gateway,
- Optionally, the customer chooses a PSK
- EMnify returns:
- Two public IP addresses representing the VPN gateways on the EMnify side,
- A PSK, in case the customer did not provide one
Note that for reliability purposes AWS creates two distinct VPN terminations. We recommend that the customers set up VPN connections to both public IP addresses. We also recommend the use of the same PSK for both VPN connections, in order to reduce the chance for configuration errors.
5. VPN Connectivity Setup – Local Network Gateway
The next step is to create two Local Network Gateways, awsintercon-vpn-conn-1 and awsintercon-vpn-conn-2. From Azure’s perspective, a Local Network Gateway represents the remote VPN peer, in this case the AWS VPN terminations. One Local Network Gateway has to be configured for each public IP address provided by EMnify, as illustrated in Figure 2.
Figure 2 Local Network Gateway creation
Make sure to configure the address space, which includes all the IP address range allocated to the endpoints. In order to avoid the necessity to update the Local Network Gateway configuration each time a new IP address range is allocated for the endpoints, a larger address space may be configured. In our example 100.64.0.0/16 would be a good choice, because it covers the current 100.64.1.0/24 and 100.64.3.0/24 as well as future /24 IP ranges within this address block. This is a safe practice and no interference with foreign endpoints will occur because on the EMnify side the routing to each individual IP address space is done on a per-customer basis.
6. VPN Connectivity Setup – Connections
So far we have configured a Virtual Network Gateway (the customer side termination of the VPN) and two Local Network Gateways (the EMnify side terminations of the VPN)1.
Next we basically need to connect the local side to the remote side and we do this by configuring two Connections, awsintercon-vgw-awsintercon-vpn-conn-1 and awsintercon-vgw-awsintercon-vpn-conn-2, as illustrated in Figure 3 and Figure 4.
Figure 3 Connection creation step 1
In the first step, we configure the connection type, which should be Site-to-site (IPsec).
Figure 4 Connection creation step 2
In the second step, we specify the two ends of the connection: the Virtual Network Gateway and the Local Network Gateway, as well as the Shared Key (PSK). Remember that we use the same PSK for both Connections.
We can also double-check the public IP addresses of the two ends of each Connection (see Figure 5).
Figure 5 Connection creation validation
Once the Connections have been configured, they have to be in the “Connected” state, as illustrated in Figure 6.
Figure 6 Connection creation completed
7. Routing and Security Groups Setup
In order to allow data traffic to flow, two more steps are required. First, a route table entry shall be added, to indicate that data traffic sent to the endpoints shall be routed via the Virtual Private Gateway (see Figure 7).
Figure 7 Routing via the Virtual Network Gateway
Note that the same considerations apply here with regard to the address prefix: a customer should add one route table entry for each address prefix currently allocated to his/her endpoints. However, in order to avoid repeated updates of the route table, the customer may actually use a shorter prefix length that covers the current and possible future endpoint address prefixes, as actually illustrated above.
Finally, the Security Group must also be updated to accept incoming traffic from the endpoints (see Figure 8).
Figure 8 Allowing Routing via the Virtual Network Gateway
The Security Group rules may be made more specific by narrowing down the port, protocol and destination to the specific applications and application servers used.
At this stage traffic between the endpoints and application servers can flow in both directions. The traffic can be monitored on the Virtual Network Gateway as illustrated in Figure 9.
Figure 9 Traffic through the Virtual Network Gateway
Note that the traffic through the two Connections may be asymmetrical, in the sense that incoming traffic may be received over one connection while outgoing traffic may be sent over the other connection. This happens because the two VPN terminations created by AWS are set up for reliability purposes and not for load sharing. By removing one Connection, traffic in both directions will be exchanged over the one remaining Connection.
9. Important Notes
The only limitation is that currently EMnify does not support overlapping IP address ranges across customer premises. Therefore EMnify may require a customer to use a different IP address range for his/her Application Servers if the desired IP address range is already taken. In order to reduce the incidence of overlapping we encourage customers to use for their premises IP address ranges with a long prefix length (e.g. /24 or longer) or, whenever possible, public IP addresses.
On the other hand, it is safe to use a wider range for the endpoints address space in order to avoid repeated updates to the routing, security groups and Local Network Gateway address space.
For the breakout connectivity setup it is important to configure the Virtual Network Gateway as “Route-based” (see Section 3) and don’t forget to specify an address space for each Local Network Gateway (see Section 5).