Secure Global Networks with OpenVPN

OpenVPN is a powerful TLS/SSL based VPN solution. OpenVPN is typically used for providing clients with access to a corporate network. This article goes outside the box, and looks at using OpenVPN to build secure server to server links in an effort to build a Global Private Network for production services such as MySQL, HTTP Back-ends etc.

by  John Buswell     

OpenVPN is a powerful SSL/TLS based VPN solution. OpenVPN supports clients under Linux, Windows and MacOS X. So it is a good option if you need to support a wide variety of clients. That however, is not the focus of this article. Instead, we are looking at OpenVPN as a solution to build a private global network between geographically diverse sites over the Internet.

Why do I need this?

In today's global economy, it's no longer sufficient to simply throw up a server at some hosting provider in the United States, and serve clients around the globe. Customers simply don't like "web wait". If your site isn't sufficiently fast enough, they are going to look elsewhere. This is particularly important in today's web 2.0 world of highly interactive and dynamic web applications. The solution is to place servers at multiple locations, which we will call sites, around the world. Place enough sites, and you will be relatively close to almost any user around the world. However, for the experience to be seamless to the user, a user for example posting in a forum in the United States, should have their post seen in real-time or near real-time by a user on the other side of the globe. This means that you'll somehow need to synchronize the data between each site. So you will need a private encrypted global network, to exchange back-end data securely between sites.

Traffic Types

Before looking at the design and deployment of such a network, we need to look at the type of traffic that will run over our private network. In most cases, you will have management, monitoring and production traffic. Depending on the number of sites you have, and how your network is deployed, you might want to separate the production traffic from the monitoring and management traffic, and then prioritize the tunnel that the production traffic is running through using normal QoS techniques.

Management traffic will usually be pretty low and will primarily include SSH connections to the remote servers to perform maintenance tasks and debugging. Monitoring traffic, on the other hand, would involve service checks every few seconds. Most importantly though is the production traffic that really depends on your network. If your network is fast enough, you might deploy a web accelerator and static web server at out the edge, and pass dynamic requests back to a central datacenter location, where something like Mongrel would handle the dynamic work, and pass the data back to the user. While this has the advantage of saving some effort with SQL replication, as the network usage grows, its not going to be an optimal solution.

In general, you will probably want to push MySQL or whatever database you are using for your dynamic web applications out to the edge. This means setting up a complex master multi-slave setup, and possibly modifying your web applications to pass SQL writes to a different SQL server. As you can see the majority of the back-end traffic over your private production network is going to be SQL (database) or HTTP traffic. You can expect some rsync or svn / cvs traffic for distributing content between the servers.

IP Addressing

The private back-end network over OpenVPN is going to need a number of private addresses. You will need to use the non-routable reserved blocks that we're all familiar with - 10.0.0.0/8, 192.168.0.0/16 etc. With this type of deployment you will need two different blocks of IPs - point to point, and production services. The point to point subnets should be /30 (two host) subnets. To avoid confusion, use 192.168.0.0/16 as the source of your point to point, and 10.0.0.0/8 networks for your production services. For our example, we will use 192.168.192.0/24 as the source subnet for our point to points, and 10.43.43.0/24 as our production services subnet.

Caching vs Replication

Database replication is tricky business, and keeping data integrity preserved can be a challenge. In a multi-master situation, if one of the multi-masters drops out for awhile, it can cause all sorts of problems. The ideal solution is to only replicate the database out to the edge when it is absolutely necessary. In most cases, using some creative Caching techniques, the only data that needs to be sent back and forth over the VPN to the central web application servers might be a HTTP request and HTTP response, with the graphics, css, javascript etc, all being served locally. If you get complaints from customers, you might need to throw some additional sites into the mix, or perhaps replicate to a central location in different regions (eg. one central European site with the database, then multiple European cache / http servers). Typically as your network grows, you'll end up with some form of hybrid solution.

EasyRSA

OpenVPN comes with a key management system, which is basically a set of scripts called EasyRSA. This is the best way to generate keys and manage certificates for OpenVPN. You should install EasyRSA on a secure system, ideally at your corporate headquarters, and limit access to the system. We have covered OpenVPN and EasyRSA in the past, but simply build your local

Certificate Authority (CA) with the script in EasyRSA, and then you simply use build-key to build client side keys, and build-key-server when you need to generate a key for a new server.

Come up with an easy to remember naming scheme for your keys, something like vpn-sX-location, where X is the server name or number, and location is the name of the location (for example atlanta, houston, london). Then simply build your keys: ./build-key vpn-s2-london for example, will produce vpn-s2-london.crt, vpn-s2-london.key and vpn-s2-london.csr. You need to transfer the .key and the .crt files along with the ca.crt from EasyRSA to your server.

Client side Configuration

Configuring OpenVPN on the client side, which would be the remote "edge" servers, is relatively simple. You need to specify the OpenVPN server's public IP, the point to point IP address information for the local and the remote, and the port number. Here is an example configuration file:

It's relatively straight forward. The ifconfig line tells OpenVPN that we are 192.168.192.2 and the remote side of the tunnel is 192.168.192.1. Port 1921, tells OpenVPN we are running on port 1921 on the server side, comp-lzo specifies lzo compression, persistent keys and persistent tunnel.

Server side Configuration

The server side has its own certificate and key files, along with ca.crt. The server side also has a dh1024.pem file, which is generated by EasyRSA. Each remote site is going to have its own instance of OpenVPN running on the centralized VPN server. For each remote server (or site) you will have a separate configuration file. Here is an example configuration file to go with our client configuration:

As you can see, the configuration is very similar to the client side, with just a few minor changes.

Starting the VPN

Make sure you bunch a hole in your iptables configuration to allow the source IP of the remote site to access to TCP port we have configured for OpenVPN, in this case its 1921. If you are using a Red Hat based system (CentOS, Fedora, RHEL), you can edit /etc/sysconfig/iptables and add :

where 172.16.55.20 is the example public IP (its not really a public IP but we don't publish real public IPs in examples) of the server, and 172.16.56.99 is the example public IP of the VPN client. First, start openvpn on the server side:

You would spawn this with the respective configuration files and log redirects for each remote site. Then on the client side:

If you look at the logs, you should see:

Testing the Link

The quickest way to test the link is to ping the remote side's point to point IP. In our case, the client side should be 192.168.192.2 and the server side 192.168.192.1. Bounce a few ICMP packets around with ping. Using scp, you can also test the data rate by transferring a dummy file to and from the server. This will give you a basic idea of how well the link is going to perform.

Adding Routes

Right now you just have a point to point connection, but you want to be able to access resources on the remote side. This is where things will vary depending on what you want to do. If you are doing MySQL replication, you will want a production subnet on both sides (remote "edge" site and the central data-center location). In that case, you'll need to setup a private subnet on both ends. If however you are simply using the remote "edge" site as a cache and static HTML server, then you'd probably only need a subnet on the remote end, unless you had multiple servers at the remote edge site, which is often the case.

For our example, we are going to use 10.43.43.0/24 on the central data-center location, and 10.43.100.0/24 on the remote site. On the VPN server, you just need to add an interface and a route:

Notice that we haven't added a dev statement to the route. The reason for this is that the tunnel can respawn and perhaps not end up on the same device each time. This works, and is safe. We've added 10.43.43.1 to lo, so that when packets from the remote side come in, we have something to test with.

On the client side:

Again, the same thing. The final piece is that you will need to add statements to your /etc/sysconfig/iptables file to enable the VPN traffic to pass back and forth.

Conclusion

Rather than bouncing production back-end traffic, along with management and monitoring traffic over the Internet, we've created our own managed private network of inter-connected VPNs. OpenVPN is ideal for this type of solution, there are other solutions out there such as IPsec.