nginx aDeploying Globally Distributed Web Applications
Apache, HTTP, Nginx

Deploying Globally Distributed Web Applications

Global Server Load Balancing is all about controlling traffic and distributing it among geographically distributed servers. Global Server Load Balancing aims to direct the user to the fastest and closest server. Normally Global Server Load Balancing is an expensive deployment, until now. Find out how to deploy GSLB with Nginx.

 

Nginx, called engine-x is a high performance HTTP server and reverse proxy, with proxy capabilities for IMAP/POP3/SMTP. Nginx is the creation of Russian developer, Igor Sysoev, and has been running in production for over two years. The latest stable release at the time of writing is Nginx 0.5.30, and is the focus of this article. While Nginx is capable of proxying non-HTTP protocols, we’re going to focus on HTTP and HTTPS.

 

High Performance, Yet Lightweight

Nginx uses a master process and N+1 worker process model. The number of workers is controlled by the configuration, yet the memory footprint and resources used by Nginx are several orders of magnitude less than Apache. Nginx uses epoll() in Linux. In our lab, Nginx was handling hundreds of requests per second, while using about 16MB of ram and a consistent load average of about 1.00. This is considerably better than Apache 2.2, and Pound doesn’t scale well with this type of usage (high memory usage, lots of threads). In general, Nginx offers a very cost effective solution.

 

Lighttpd

Lighttpd is a great lightweight option, but it has a couple of drawbacks. Nginx has very good reverse proxy capabilities with integrated basic load balancing. This makes it a very good option as a front end to dynamic web applications, such as those running under Rails and using Mongrel. Lighttpd on the other hand, has an old and unmaintained proxy module. Now it does have a new proxy module with Lighttpd 1.5.x, but that is the other problem with Lighttpd, where its going. Lighttpd 1.4 is lightweight, relies on very few external libraries and is fast. Lighttpd 1.5.x on
the other hand requires many more external libraries, including glib, now I don’t know about you but anything using glibc is far from “lightweight”.

 

Basic Configuration

The basic configuration of Nginx specifies the unprivileged user to run as, the number of worker processes, error log, pid and events block. After this basic configuration block, you have per protocol blocks (http for example).

 

  • user nobody;
  • worker_processes 4;
  • error_log logs/error.log;
  • pid logs/nginx.pid;
  • events {
  • worker_connections 1024;
  • }

 

Basic HTTP server

Nginx is relatively easy to configure as a basic web server, it supports IP and Name based virtual hosts, and it uses a pcre based URI processing system. Configuring static hosting is very easy, you just specify a new server block:

 

  • server {
  • listen 10.10.10.100:80;
  • server_name www.foocorp.com foocorp.com;
  • access_log logs/foocorp.com.log main;
  • location / {
  • index index.html index.htm;
  • root /var/www/static/foocorp.com/htdocs;
  • }
  • }

 

Here we are listening on port 80 on 10.10.10.100, with name virtual hosting using www.foocorp.com and foocorp.com. The server_name option also supports wildcards, so you can specify *.foocorp.com and have it handled by the configuration. The usual access logs, and root specifies htdocs. If you have a large number of name virtual hosts, you’ll need to increase the size of the hash bucket with server_names_hash_bucket_size 128;

 

Gzip compression

Nginx like many other web servers, can compress content using gzip.

 

  • gzip on;
  • gzip_min_length 1100;
  • gzip_buffers 4 8k;
  • gzip_types text/plain text/html text/css text/js;

 

Here Nginx allows you to enable gzip, specify a minimum length to compress, buffers and the mime types that Nginx will compress. Gzip compression is supported by all modern browsers.

 

HTTP Load Balancing

Nginx can be used a simple HTTP load balancer, in this configuration, you would place Nginx in front of your existing web servers. The existing web servers can be running Nginx as well. In HTTP load balancer mode, you simply need to add an upstream block to the configuration :

 

  • upstream a.serverpool.foocorp.com {
  • server 10.80.10.10:80;
  • server 10.80.10.20:80;
  • server 10.80.10.30:80;
  • }
  • upstream b.serverpool.foocorp.com {
  • server 10.80.20.10:80;
  • server 10.80.20.20:80;
  • server 10.80.20.30:80;
  • }

 

Then in the server block, you add the line:

 

  • proxy_pass http://a.serverpool.foocorp.com;

 

Health Check Limitations

Nginx has only simple load balancing capabilities. It doesn’t have health checking capabilities and it uses a simple load balancing algorithm. However, Nginx is a relatively new project, so one would expect to see various load balancing algorithms and health checking support added over time. While it might not be wise to replace your commercial load balancer with Nginx anytime soon, Nginx is almost there in terms of a very competitive solution. Monit, and other monitoring applications offer good options to compensate for a lack of health checking capabilities in Nginx.

 

Global Server Load Balancing

Nginx has a very interesting capability. With a little configuration can provide Global Server Load Balancing. Now Global Server Load Balancing (GSLB) is a feature you’ll find on high-end load balancing switches such as those from F5, Radware, Nortel, Cisco etc. Typically GSLB is an additional license you have to purchase for a few thousand dollars, on top of a switch that typically start around US$10,000.

 

GSLB works by having multiple sites distributed around the world, so you might have a site in Europe, a site in Asia and a site in North America. Normally, you would direct traffic by region by using different top level domains (TLD). So www.foocorp.com might go to North America, www.foocorp.co.uk to Europe, www.foocorp.com.cn to the server in Asia. This isn’t a very effective solution because it relies on the user to visit the proper domain. A user in Asia, might see a print advertisement for the North American market, hitting the .com address means they aren’t visiting the closest and fastest server.

 

GSLB works by looking at the source IP address of the request, and then determines which site is closest to that source address. The simplest method is to break the Internet address space down per region, then to route
traffic to the local site in that region. When we say region, we mean – North America, South America, EMEA (Europe, Middle East and Africa) and APAC (Asia-Pacific).

 

Configuring Nginx for GSLB

The geo {} block is used to configure GSLB in Nginx, the geo block causes Nginx to look at the source IP, and set a variable based on the configuration. The nice thing with Nginx is that you can set a default.

 

  • geo $gslb {
  • default na;
  • include conf/gslb.conf
  • }

 

Here in our configuration, we’re setting the default to na (North America) and then including the gslb.conf. The configuration file gslb.conf is a basic file consisting of subnet variable. Here is an excerpt from gslb.conf:

 

  • 32.0.0.0/8 emea;
  • 41.0.0.0/8 emea;
  • 43.0.0.0/8 apac;

 

When Nginx receives a request from a source IP in 32.0.0.0/8 (for those of you unfamiliar with slash notation, this is the entire Class A, 32.0.0.0 thru 32.255.255.255), it sets the variable $gslb to emea. We then use that later in the configuration to redirect.

 

Inside the location block of our server configuration in Nginx, we add a number of if statements before the proxy_pass (if used) statement. These instruct the server to do a HTTP 302 Redirect (temporary redirect).

 

  • if ($gslb = emea) {
  • rewrite ^(.*) http://europe.foocorp.com$1 redirect;
  • }
  • if ($gslb = apac) {
  • rewrite ^(.*) http://asia.foocorp.com$1 redirect;
  • }

 

These are configured under the www.foocorp.com named virtual server, if someone from North America hits www.foocorp.com, it hits the default and simply loads from the same server. If the user is from Europe, the request should match one of the subnets listed in gslb.conf, and sets the gslb variable to emea. This request causes the North American site hosting the .com domain to redirect the client to the server(s) at the site in Europe.

 

On the European server, the configuration is slightly different. Instead of the emea check, you check for NA and redirect to the US site. This is to handle the situation when someone in North America hits the .eu or .co.uk site.

 

  • if ($gslb = na) {
  • rewrite ^(.*) http://www.foocorp.com$1 redirect;
  • }

 

Traffic Control: In-region not always faster

The problem with commercial solutions is that they are too generalized. In our example configurations so far, we make some pretty wild assumptions. The problem with the Internet is that a user in Asia, might not for example, have a faster connection to servers in Asia. A good example of this is India and Pakistan. A server hosted in Hong Kong or Singapore, is in Asia, and would be considered “in region” for customers in India and Pakistan. The reality though is that traffic from those countries to Hong Kong, is actually routed through Europe, so packets from India to Hong Kong, go from India thru Europe, across the United States and hit Hong Kong from the Pacific. However, in the same subnet, customers in Australia are only a few hops away from Hong Kong.

 

In such a situation, with commercial solutions, you are just out of luck, but with Nginx you can fine tune how traffic is directed. Here we know 120.0.0.0/6 is mainly APAC, but 122.162.0.0/16 and 122.163.0.0/16 have faster connections to Europe. So, we simply add these subnets to the configuration. Nginx will use the closest match to the source IP. So 122.162.0.0/16 is
finer grained than 120.0.0.0/6, so Nginx will use it.

 

Manual Tuning

The initial tuning can be done by using the whois command, for example whois 120.0.0.0 will give you an idea which region it belongs to – ARIN, RIPE, etc. ARIN, RIP, APNIC, AFRINIC, and LACNIC are regional internet registries or RIR. An RIR is an organization overseeing the allocation and registration of Internet number resources within a particular region of the world. IP addresses both IPv4 and IPv6 are managed by these RIRs. However, as in our previous example, you’re going to need to fine tune the gslb configuration with traceroute and ping information. Probably the best approach is to do a general configuration and then fine tune the configuration based on feedback from customers.

 

Cost Savings vs. Features

Looking at a well known Layer 4-7 switching solution, you would need a minimum of $15k per site to purchase the necessary equipment and licensing. Commercial solutions do have some additional fault tolerant measures, such as the ability to measure load and availability of servers at remote sites. However, with Nginx offering a very close solution which is available for FREE with the source code, it is only a matter of time before such features are part of Nginx or available thru other projects.

 

gslb.conf

The following is an initial example of gslb.conf, it should be sufficient for most users.

 

  • 25.0.0.0/8 uk;
  • 32.0.0.0/8 emea;
  • 41.0.0.0/8 emea;
  • 43.0.0.0/8 apac;
  • 51.0.0.0/8 uk;
  • 53.0.0.0/8 emea;
  • 57.0.0.0/8 emea;
  • 58.0.0.0/8 apac;
  • 59.0.0.0/8 apac;
  • 60.0.0.0/8 apac;
  • 61.0.0.0/8 apac;
  • 62.0.0.0/8 emea;
  • 77.0.0.0/8 emea;
  • 78.0.0.0/7 emea;
  • 80.0.0.0/5 emea;
  • 88.0.0.0/6 emea;
  • 90.192.0.0/11 uk;
  • 91.104.0.0/13 uk;
  • 91.125.0.0/16 uk;
  • 92.0.0.0/8 emea;
  • 93.0.0.0/8 emea;
  • 116.0.0.0/6 apac;
  • 120.0.0.0/6 apac;
  • 122.162.0.0/16 uk;
  • 122.163.0.0/16 uk;
  • 124.0.0.0/7 apac;
  • 126.0.0.0/8 apac;
  • 129.0.0.0/8 emea;
  • 130.0.0.0/8 emea;
  • 131.0.0.0/8 emea;
  • 133.0.0.0/8 apac;
  • 134.0.0.0/8 emea;
  • 139.0.0.0/8 emea;
  • 141.0.0.0/8 emea;
  • 145.0.0.0/8 emea;
  • 150.0.0.0/8 apac;
  • 151.0.0.0/8 emea;
  • 157.0.0.0/8 apac;
  • 162.0.0.0/8 emea;
  • 163.0.0.0/8 emea;
  • 164.0.0.0/8 emea;
  • 171.0.0.0/8 emea;
  • 188.0.0.0/8 emea;
  • 193.0.0.0/8 emea;
  • 194.0.0.0/8 emea;
  • 195.0.0.0/8 emea;
  • 196.0.0.0/8 emea;
  • 202.0.0.0/7 apac;
  • 210.0.0.0/7 apac;
  • 212.0.0.0/7 emea;
  • 217.0.0.0/8 emea;
  • 218.0.0.0/6 apac;
  • 219.0.0.0/8 apac;
  • 220.0.0.0/7 apac;
  • 222.0.0.0/8 apac;

 

WiFi tower and logo
Networks, WiFi

Enterprise WiFi – Thin Access Points

Thin Access Point solutions place very little 802.11 intelligence on the Access Point, instead these Thin Access Points pass the 802.11 packets to a centralized controller or switch through some form of IP tunneling or encapsulation. Thin APs have many advantages including centralized management, lower cost APs and a wide variety of Enterprise grade features.

The wireless access point (AP), is the device which connects the wireless network to the wired network. It is a key component of the 802.11 wireless LAN. Unless you are in a very small office space, most businesses will need to deploy multiple access points, enabling mobile users to roam around the office. Traditional access points, often called "FAT", "smart" or "thick" access points, are self-contained solutions that have full support for transitioning packets between the 802.11 and wired ethernet network that they are connected to.

The down side to "Thick" APs is that they do not scale very well. Managing a single access point in your home office, or even 5 access points in a small office, isn’t a big problem. However when you have a large multi-building campus, managing hundreds or even thousands of access points becomes a serious problem. It just isn’t practical to have many individually managed devices trying to provide a common wireless network. Thick APs simply don’t scale just from a manageability stand-point.

Multiple SSIDs

Very few "Thick" APs support more than one SSID. An SSID is basically the public name for the wireless network. In many cases you might have a shared area, where you need to provide access to different groups who have their own SSID (for example sales, engineering, guests). Unless your "Thick" AP supports multiple SSIDs, you will need to deploy more and more access points. The more access points you deploy, the more problems you run into with RF interference and manageability issues.

Thin AP

The Thin AP is typically just a device which has one or more 802.11 radio(s), and a wired network port (typically Fast Ethernet). Thin APs generally support Power over Ethernet (PoE), so you don’t need to worry about running power bricks to the devices. Very little, if any of the 802.11 stack is handled by the Thin AP. What happens is that the Thin AP is used with a centralized switch or controller. The 802.11 frames are taken from the radio, encapsulated and sent down the wire to the controller. All of the 802.11 intelligence is handled at the centralized controller. One key advantage to this is that you can place a Thin AP physically on a network that the actual wireless users never have access to. The 802.11 frame is passed over that network, and then on to the centralized controller, where the controller typically has one or more uplinks to various networks the wireless users are supposed to have access to. All of the authentication and encryption is handled by the controller.

Centralized Control

Immediately it becomes clear that the Thin AP with a centralized controller is a much better solution than trying to manage hundreds of devices individually. Even on a relatively small network of maybe 10 to 30 APs, it makes sense, if for no other reason than to reduce human error to switch to a centrally managed solution. Configuration and monitoring are managed from the controller. Some commercial products will help tune and avoid RF interference problems, and support multiple SSIDs. Some products, such as those from Aruba Networks, will help you select the number and type of Thin access points you need
based on parameters about your buildings that you give it. The Thin AP solution from Aruba also supports an Air Monitor (AM) mode, where an access point can be converted into an air monitor, where it will monitor traffic on the wireless side for wireless IDS and other security applications. Centralized control provides a means to define and enforce network access control policies from a central location, rather than attempting to synchronize policies across many devices.

Change in standards

If you deploy "thick" access points, what happens when there is a change in 802.11 standards or a new feature is made available? In some cases, the "thick" access points will not be powerful enough or lack memory to implement the feature. However, for "thin" access points, the 802.11 frames are handled by the controller. So a simple software upgrade on the controller can be done to implement the new standard. In some cases, such as 802.11n, the access points use different types of radios. So to support 802.11n, new access points would have to be deployed, but 802.11n is a different situation.

New encryption methods

Encryption methods are created and defeated, over the past few years we’ve seen WEP, TKIP, WPA, and WPA2. With thick access points, the encryption technology is typically handled by a chip. This warrants a hardware replacement to switch to a new encryption method. On the other hand, with the "Thin" AP solution, the encryption is handled on the controller, worse case you may need to replace a blade or a controller. Even then this is a far better situation than having to replace hundreds of APs, some of which maybe hanging in less than accessible locations.

Lower cost of ownership

When deploying a wireless network, especially over a large area such as a multi-building campus, the most expensive part of that network is going to be the access points.
In order to get the coverage to support the users over the coverage area, many APs are going to be necessary. The "thin" access point solution offers a lower cost of ownership per AP, than a "thick" AP solution, as there isnt too much to a "Thin" AP. All of the intelligence is handled back at the controller. While the controller may initially seem like a higher cost, when you factor in the lower cost per AP, over a large deployment, the cost savings are obvious.

Remote Access Points

One of the neatest features that centralized control can provide is a feature called Remote AP. Remote AP enables an administrator to provision an AP, and then place it at a home or remote office. The Remote AP communicates over a secure VPN connection to the centralized controller, and provides the same wireless network(s) at the remote location that you would have at the office. This enables remote employees perhaps thousands of miles away, to have the same SSIDs available to them that they would have if they were located in a cube at their headquarters! Remote AP is one of many cool and interesting capabilities thanks to the Thin AP architecture thats available from Aruba Networks.

Advanced Services and Features

The "Thin" AP architecture makes a number of other features possible. Aruba Networks offers real-time location services where the Thin APs are used to report signal measurements for the tracked device, and then the coordinates are calculated. For details on the advanced services available, visit Aruba Networks.

Mesh

Several of the leading wireless mobility companies are offering Mesh solutions using Thin APs. A wireless MESH is a wireless network where packets are relayed over wireless links, as it is too difficult or costly to run wired networks to the APs. This is an ideal solution for outdoor venues or indoor venues where wiring is extremely difficult. The general concept with MESH is that
instead of using a wired back haul, the back haul is wireless.

802.11n

Many large enterprise deployments exist out there with VoIP and data on the same 802.11g, 802.11a networks using the Thin AP solution. The upcoming introduction of 802.11n offers new challenges to the Thin AP architecture, as increased bandwidth is needed on the back haul. However, some companies have suggested that 802.11n will mark the end of the Thin AP. One such company is Xirrus who offer a sort of hybrid solution for 802.11n, which is more or less a fancy Thick AP. It is highly unlikely that 802.11n will be the death of Thin AP architectures. In fact, 802.11b/g/a is more than capable of handling the demands of the corporate WLAN. As the 802.11n standard is ratified, we will see 802.11n Thin APs, and with many networks already supporting gigabit ethernet, as well as 10GbE, the wired side of the network will have plenty of bandwidth to keep up with 802.11n and support the Thin AP model. In many cases, the businesses themselves will simply not have the bandwidth to the Internet to support 802.11n, which is why 802.11b/g/a Thin APs will still be around for the foreseeable future.

Conclusion

At the time of writing, there are no open source Thin AP solutions. If you are interested in open Thin AP solutions, Aruba Networks has an Aruba Labs site (here) that is worth checking out. Other Thin AP vendors include Trapeze Networks, Meru Networks, and Cisco Systems (formally Airspace).

nginx aDeploying Globally Distributed Web Applications
Apache, Nginx

Deploying Globally Distributed Web Applications

Global Server Load Balancing is all about controlling traffic and distributing it among geographically distributed servers. Global Server Load Balancing aims to direct the user to the fastest and closest server. Normally Global Server Load Balancing is an expensive deployment, until now. Find out how to deploy GSLB with Nginx.

Nginx, called engine-x is a high performance HTTP server and reverse proxy, with proxy capabilities for IMAP/POP3/SMTP. Nginx is the creation of Russian developer, Igor Sysoev, and has been running in production for over two years. The latest stable release at the time of writing is Nginx 0.5.30, and is the focus of this article. While Nginx is capable of proxying non-HTTP protocols, we’re going to focus on HTTP and HTTPS.

High Performance, Yet Lightweight

Nginx uses a master process and N+1 worker process model. The number of workers is controlled by the configuration, yet the memory footprint and resources used by Nginx are several orders of magnitude less than Apache. Nginx uses epoll() in Linux. In our lab, Nginx was handling hundreds of requests per second, while using about 16MB of ram and a consistent load average of about 1.00. This is considerably better than Apache 2.2, and Pound doesn’t scale well with this type of usage (high memory usage, lots of threads). In general, Nginx offers a very cost effective solution.

Lighttpd

Lighttpd is a great lightweight option, but it has a couple of drawbacks. Nginx has very good reverse proxy capabilities with integrated basic load balancing. This makes it a very good option as a front end to dynamic web applications, such as those running under Rails and using Mongrel. Lighttpd on the other hand, has an old and unmaintained proxy module. Now it does have a new proxy module with Lighttpd 1.5.x, but that is the other problem with Lighttpd, where its going. Lighttpd 1.4 is lightweight, relies on very few external libraries and is fast. Lighttpd 1.5.x on
the other hand requires many more external libraries, including glib, now I don’t know about you but anything using glibc is far from “lightweight”.

Basic Configuration

The basic configuration of Nginx specifies the unprivileged user to run as, the number of worker processes, error log, pid and events block. After this basic configuration block, you have per protocol blocks (http for example).

  • user nobody;
  • worker_processes 4;
  • error_log logs/error.log;
  • pid logs/nginx.pid;
  • events {
  • worker_connections 1024;
  • }

Basic HTTP server

Nginx is relatively easy to configure as a basic web server, it supports IP and Name based virtual hosts, and it uses a pcre based URI processing system. Configuring static hosting is very easy, you just specify a new server block:

  • server {
  • listen 10.10.10.100:80;
  • server_name www.foocorp.com foocorp.com;
  • access_log logs/foocorp.com.log main;
  • location / {
  • index index.html index.htm;
  • root /var/www/static/foocorp.com/htdocs;
  • }
  • }

Here we are listening on port 80 on 10.10.10.100, with name virtual hosting using www.foocorp.com and foocorp.com. The server_name option also supports wildcards, so you can specify *.foocorp.com and have it handled by the configuration. The usual access logs, and root specifies htdocs. If you have a large number of name virtual hosts, you’ll need to increase the size of the hash bucket with server_names_hash_bucket_size 128;

Gzip compression

Nginx like many other web servers, can compress content using gzip.

  • gzip on;
  • gzip_min_length 1100;
  • gzip_buffers 4 8k;
  • gzip_types text/plain text/html text/css text/js;

Here Nginx allows you to enable gzip, specify a minimum length to compress, buffers and the mime types that Nginx will compress. Gzip compression is supported by all modern browsers.

HTTP Load Balancing

Nginx can be used a simple HTTP load balancer, in this configuration, you would place Nginx in front of your existing web servers. The existing web servers can be running Nginx as well. In HTTP load balancer mode, you simply need to add an upstream block to the configuration :

  • upstream a.serverpool.foocorp.com {
  • server 10.80.10.10:80;
  • server 10.80.10.20:80;
  • server 10.80.10.30:80;
  • }
  • upstream b.serverpool.foocorp.com {
  • server 10.80.20.10:80;
  • server 10.80.20.20:80;
  • server 10.80.20.30:80;
  • }

Then in the server block, you add the line:

  • proxy_pass http://a.serverpool.foocorp.com;

Health Check Limitations

Nginx has only simple load balancing capabilities. It doesn’t have health checking capabilities and it uses a simple load balancing algorithm. However, Nginx is a relatively new project, so one would expect to see various load balancing algorithms and health checking support added over time. While it might not be wise to replace your commercial load balancer with Nginx anytime soon, Nginx is almost there in terms of a very competitive solution. Monit, and other monitoring applications offer good options to compensate for a lack of health checking capabilities in Nginx.

Global Server Load Balancing

Nginx has a very interesting capability. With a little configuration can provide Global Server Load Balancing. Now Global Server Load Balancing (GSLB) is a feature you’ll find on high-end load balancing switches such as those from F5, Radware, Nortel, Cisco etc. Typically GSLB is an additional license you have to purchase for a few thousand dollars, on top of a switch that typically start around US$10,000.

GSLB works by having multiple sites distributed around the world, so you might have a site in Europe, a site in Asia and a site in North America. Normally, you would direct traffic by region by using different top level domains (TLD). So www.foocorp.com might go to North America, www.foocorp.co.uk to Europe, www.foocorp.com.cn to the server in Asia. This isn’t a very effective solution because it relies on the user to visit the proper domain. A user in Asia, might see a print advertisement for the North American market, hitting the .com address means they aren’t visiting the closest and fastest server.

GSLB works by looking at the source IP address of the request, and then determines which site is closest to that source address. The simplest method is to break the Internet address space down per region, then to route
traffic to the local site in that region. When we say region, we mean – North America, South America, EMEA (Europe, Middle East and Africa) and APAC (Asia-Pacific).

Configuring Nginx for GSLB

The geo {} block is used to configure GSLB in Nginx, the geo block causes Nginx to look at the source IP, and set a variable based on the configuration. The nice thing with Nginx is that you can set a default.

  • geo $gslb {
  • default na;
  • include conf/gslb.conf
  • }

Here in our configuration, we’re setting the default to na (North America) and then including the gslb.conf. The configuration file gslb.conf is a basic file consisting of subnet variable. Here is an excerpt from gslb.conf:

  • 32.0.0.0/8 emea;
  • 41.0.0.0/8 emea;
  • 43.0.0.0/8 apac;

When Nginx receives a request from a source IP in 32.0.0.0/8 (for those of you unfamiliar with slash notation, this is the entire Class A, 32.0.0.0 thru 32.255.255.255), it sets the variable $gslb to emea. We then use that later in the configuration to redirect.

Inside the location block of our server configuration in Nginx, we add a number of if statements before the proxy_pass (if used) statement. These instruct the server to do a HTTP 302 Redirect (temporary redirect).

  • if ($gslb = emea) {
  • rewrite ^(.*) http://europe.foocorp.com$1 redirect;
  • }
  • if ($gslb = apac) {
  • rewrite ^(.*) http://asia.foocorp.com$1 redirect;
  • }

 These are configured under the www.foocorp.com named virtual server, if someone from North America hits www.foocorp.com, it hits the default and simply loads from the same server. If the user is from Europe, the request should match one of the subnets listed in gslb.conf, and sets the gslb variable to emea. This request causes the North American site hosting the .com domain to redirect the client to the server(s) at the site in Europe.

On the European server, the configuration is slightly different. Instead of the emea check, you check for NA and redirect to the US site. This is to handle the situation when someone in North America hits the .eu or .co.uk site.

  • if ($gslb = na) {
  • rewrite ^(.*) http://www.foocorp.com$1 redirect;
  • }

Traffic Control: In-region not always faster

The problem with commercial solutions is that they are too generalized. In our example configurations so far, we make some pretty wild assumptions. The problem with the Internet is that a user in Asia, might not for example, have a faster connection to servers in Asia. A good example of this is India and Pakistan. A server hosted in Hong Kong or Singapore, is in Asia, and would be considered “in region” for customers in India and Pakistan. The reality though is that traffic from those countries to Hong Kong, is actually routed through Europe, so packets from India to Hong Kong, go from India thru Europe, across the United States and hit Hong Kong from the Pacific. However, in the same subnet, customers in Australia are only a few hops away from Hong Kong.

In such a situation, with commercial solutions, you are just out of luck, but with Nginx you can fine tune how traffic is directed. Here we know 120.0.0.0/6 is mainly APAC, but 122.162.0.0/16 and 122.163.0.0/16 have faster connections to Europe. So, we simply add these subnets to the configuration. Nginx will use the closest match to the source IP. So 122.162.0.0/16 is
finer grained than 120.0.0.0/6, so Nginx will use it.

Manual Tuning

The initial tuning can be done by using the whois command, for example whois 120.0.0.0 will give you an idea which region it belongs to – ARIN, RIPE, etc. ARIN, RIP, APNIC, AFRINIC, and LACNIC are regional internet registries or RIR. An RIR is an organization overseeing the allocation and registration of Internet number resources within a particular region of the world. IP addresses both IPv4 and IPv6 are managed by these RIRs. However, as in our previous example, you’re going to need to fine tune the gslb configuration with traceroute and ping information. Probably the best approach is to do a general configuration and then fine tune the configuration based on feedback from customers.

Cost Savings vs. Features

Looking at a well known Layer 4-7 switching solution, you would need a minimum of $15k per site to purchase the necessary equipment and licensing. Commercial solutions do have some additional fault tolerant measures, such as the ability to measure load and availability of servers at remote sites. However, with Nginx offering a very close solution which is available for FREE with the source code, it is only a matter of time before such features are part of Nginx or available thru other projects.

gslb.conf

The following is an initial example of gslb.conf, it should be sufficient for most users.

  • 25.0.0.0/8 uk;
  • 32.0.0.0/8 emea;
  • 41.0.0.0/8 emea;
  • 43.0.0.0/8 apac;
  • 51.0.0.0/8 uk;
  • 53.0.0.0/8 emea;
  • 57.0.0.0/8 emea;
  • 58.0.0.0/8 apac;
  • 59.0.0.0/8 apac;
  • 60.0.0.0/8 apac;
  • 61.0.0.0/8 apac;
  • 62.0.0.0/8 emea;
  • 77.0.0.0/8 emea;
  • 78.0.0.0/7 emea;
  • 80.0.0.0/5 emea;
  • 88.0.0.0/6 emea;
  • 90.192.0.0/11 uk;
  • 91.104.0.0/13 uk;
  • 91.125.0.0/16 uk;
  • 92.0.0.0/8 emea;
  • 93.0.0.0/8 emea;
  • 116.0.0.0/6 apac;
  • 120.0.0.0/6 apac;
  • 122.162.0.0/16 uk;
  • 122.163.0.0/16 uk;
  • 124.0.0.0/7 apac;
  • 126.0.0.0/8 apac;
  • 129.0.0.0/8 emea;
  • 130.0.0.0/8 emea;
  • 131.0.0.0/8 emea;
  • 133.0.0.0/8 apac;
  • 134.0.0.0/8 emea;
  • 139.0.0.0/8 emea;
  • 141.0.0.0/8 emea;
  • 145.0.0.0/8 emea;
  • 150.0.0.0/8 apac;
  • 151.0.0.0/8 emea;
  • 157.0.0.0/8 apac;
  • 162.0.0.0/8 emea;
  • 163.0.0.0/8 emea;
  • 164.0.0.0/8 emea;
  • 171.0.0.0/8 emea;
  • 188.0.0.0/8 emea;
  • 193.0.0.0/8 emea;
  • 194.0.0.0/8 emea;
  • 195.0.0.0/8 emea;
  • 196.0.0.0/8 emea;
  • 202.0.0.0/7 apac;
  • 210.0.0.0/7 apac;
  • 212.0.0.0/7 emea;
  • 217.0.0.0/8 emea;
  • 218.0.0.0/6 apac;
  • 219.0.0.0/8 apac;
  • 220.0.0.0/7 apac;
  • 222.0.0.0/8 apac;

tux linux penguin
Linux, Security

Linux Zero Day Attack Protection

 

What is a Zero Day Attack?

A Zero Day attack is a security vulnerability for which there is no patch or work around to counter-act the problem. The threat of Zero Day attacks is the time from which the vulnerability can be exploited until the IT administrator has patched the problem. Zero Day attack scenarios vary from a publicly available exploit for which the vendor has not yet provided a fix, to the IT administrator who simply has not patched the system or is unaware of the problem.

 

What is Zero Day Attack Protection?

There are a number of different ways to provide Zero Day Attack Protection, most companies offer complex pattern and protocol analysis on real-time traffic in an attempt to detect unusual activity, and then block it. The problem with this approach is that its not really Zero Day Attack Protection, its more like fancy Intrusion Prevention. The solution provided in this series of articles is TRUE Zero Day Attack Protection. It is a series of technologies and techniques that when combined together counter-act the tools used by malicious users to enter the system.

 

Who developed this technique?

The technique was developed by a small engineering team at Spliced Networks, consisting of John Buswell and Frank Boyd. Spliced Networks had a working prototype as early as March 2003, the solution evolved to provide full Zero Day Attack Protection by early 2005. Spliced Networks was an Open Source Appliance company that built a general purpose Linux based operating system that ran on compact flash. Spliced Networks took a minimal approach and focused on providing image based deployments for servers, similar to those you see used in virtualized environments today. The technique enables application partitioning, so if security between applications running on the same server is the goal, this technique provides a more secure option to virtualization.

 

Buffer OverFlow Attacks

One of the most common attacks used in computer systems are buffer overflows. This is not intended to be a detailed discussion of those attacks, but for those unfamiliar with such attacks, a buffer is typically a string that holds data. Often the buffer is populated with data from user input, so badly written code will not take into account that the user may attempt to overload the buffer by entering too much data. For example, code might take a source string thats 20 bytes long and attempt to copy it into a string variable that only holds 16 bytes. Those additional 4 bytes could potentially overrun into other areas of memory, as its exceeded the boundary of the variable. Some of the security features below provide protection against those scenarios and others, but on a system-wide protection basis rather than code changes that a programmer needs to make to current code.

 

Stack Smashing Protector

As of GCC 4.1, the Stack Smashing Protector is integrated into GCC. This is an extension for the GCC compiler used for protecting applications from stack-smashing attacks. The solution works by automatically inserting code that protects the C application at compilation time. The protection works by providing buffer overflow detection and reordering of variables to avoid the corruption of pointers. The reordering works by placing buffers after pointers to avoid the corruption of pointers that could be used to corrupt random memory locations. It also copies pointers in function arguments to an area preceding local variable buffers to prevent the corruption of pointers and it omits instrumentation code from some functions to decrease the performance overhead. By using the stack smashing protector, one of the basic tools used by malicious users, and one of the most common mistakes in application coding is taken off the table.

 

Getting Random Numbers Faster

The next component that is used in this solution is frandom, which is a faster and better random number generator than the typical /dev/urandom used in Linux. The frandom component is pretty trivial but since its several orders of magnitude faster than urandom, it provides a performance boost that can counter-act some of the additional measures used for security.

 

FORTIFY_SOURCE

This is another set of patches for GCC, these provide runtime buffer overflow protection for applications that have been compiled with GCC. It provides lightweight buffer overflow protection to some memory and string functions. It offers advantages over mudflap, a similar project as it has a much smaller runtime overhead.

 

strlcat and strlcpy

Everyone knows that string manipulation functions are prime targets for finding security vulnerabilities, especially in applications written in C. To further protect against buffer overflow attacks, strlcpy and strlcat functions are used to replace strcpy and strcat functions that are typically used. These functions come from the OpenBSD project. These functions provide the developer with an easy way to write bullet proof code, with both functions guaranteeing to NUL-terminate the destination string for all strings. Both functions also take into consideration the size of the destination string, preventing overloading of the destination string. These functions will truncate the strings appropriately if you attempt to copy a larger source string into a smaller destination string.

 

Position Independent Executable Support (PIE/PIC)

Position Independent Code can be copied to any memory location without modification and be executed there. The memory location is typically determined by the compiler / linker when the code is built, with a particular offset. This is how malicious users can determine how and where to execute code in order to take advantage of a particular exploit. This is one of the reasons why its important to hide version numbers of applications, operating system types and to avoid using pre-built binaries from package management systems. If an attacker can determine your running versions and obtain the same copy as you did, they now have a roadmap for breaking into your system. Position Independent Executables get around this problem by allowing the memory location to be determined at run-time rather than compile time. This makes it a lot more difficult for an attack to determine the memory location, instead of having a map, they now have to try to win the lottery. Position Independent Executables require the entire platform to be rebuilt, later on in the series we will demonstrate how to disable relocatable code, making it impossible to run statically linked executables and statically linked libraries. This makes the system extremely secure, as the system-wide policy prevents non-PIE code from executing on the system.

 

In GCC, the options -fpie / -fPIE options are available and function similar to -fpic/-fPIC. There are also some patches to binutils that need to be done. Now rebuilding the entire Linux system is a bit of a task, however this solution calls for a minimal approach. Spliced Networks built a custom build environment, however today, a similar environment can be found at the Hardened Linux From Scratch project.

 

Hardened Linux From Scratch

For step by step instructions on how to build a completely Position Independent Linux system, follow the Kernel 2.6 and GLIBC version of the Hardened Linux From Scratch project. The book is well written, while it does not contain all the components used by Spliced Networks, for the purpose of this technique it is sufficient enough to provide the initial base system. Follow along the instructions to the end of Chapter 6. Do not follow the instructions beyond chapter 6 as our solution takes a different approach.

 

Next Article

The next article will use the resulting skeleton Linux system produced in this article as the foundation for the Unhackable Linux platform.

linux vs windows Open Source vs Closed Source
BSD, Linux, Microsoft, Unix, Windows

Open Source vs Closed Source — Its about investing in People

Investing in Open Source is about investing in People instead of investing in a Vendor. For the Enterprise considering the leap to Open Source, those people have to be on the payroll for a successful migration.

There has been an interesting debate going on between bloggers Mike Dailey and Hans Bezemer. Mike is of the opinion that Linux has not won in the Enterprise because CIOs have not selected it, and therefore it is not better than Microsoft's product offerings. All the usual points and counter points have been played, these debates typically end up in a stalemate, but perhaps this time the issue can be settle once and for all by stating the obvious. To quote Monty Python..

And Now For Something Completely Different

Everyone is tired of the usual rhetoric that is associated with these kinds of debates, one solution can be hardened, so can the other, one solution has high uptimes, so does the other. The reality of the situation is that IT executives make business decisions about technology, they do not necessarily choose the best technology, they choose the technology that makes the most business sense for their company. Mike holds the concept that IT executives continue to look towards Microsoft, this is not because Microsoft has a better product offering, its due to the fact that IT executives look at open source as a product. This article is intended to help IT executives see past the rhetoric, the warped statistics (from both sides) and the dogma, and get down to the fact that Open Source is a strategy.

1. Open Source is a strategy not a product

Open Source is an Information Technology strategy, it is not a product. When evaluating Open Source, comparing licensing, pricing, and support options are important, but the key to a successful evaluation of Open Source is to recognize that it is strategy. A good Open Source strategy reduces the software licensing costs to near-zero, develops in-house engineering talent, integrates as transparently as possible to the end-users and improves scalability on like hardware. The cost involved is the amount of outside help the strategy needs, and how much of it can be done in-house.

2. Open Source is an investment in People instead of Vendors

When an IT executive chooses a Closed Source solution such as the Windows product line of Data center software solutions, they are choosing to invest a large portion of their budget in a vendor, in this case Microsoft. When selecting an Open Source solution, to maximize the return on that investment, the IT executives must invest in People. Open Source provides the user with access to the source code, the user in this case is the business. This is an incredibly powerful feature if it is managed properly, as you can fully support yourself with the source code. The talent must be available to help the business leverage this advantage, otherwise moving to Open Source makes a lot less sense. To use Open Source successfully, the IT executive must have someone on staff who can take a requirement, and using open source components, produce a solution within a reasonable timeframe. This could mean adding dedicated developers to the IT team, or training individuals to have software capability. Every organization is different, how they get the right people on the team is their business decision, but to attempt a switch to Open Source without the investment in people is a recipe for disaster.

3. Open Source is about long-term exponential savings

Everyone has seen the total cost of ownership calculators, which can be skewed in either direction depending on who is providing the calculator. Open Source provides long-term exponential cost savings because it is a flat investment. Microsoft licenses its software on a per user or per processor basis, and for a finite term. Microsoft does not support software forever, they release new versions which require new licenses, essentially requiring the customer to purchase the software again. The cost is recurring and as the organization grows, the cost increases as more users and more processors are required. The cost of a Microsoft solution increases exponentially but the cost of an Open Source solution remains flat regardless of how the organization scales. A successful Open Source strategy would continue the investment in People, so that over time, the employees involved would know the open source components inside and out, not just from an operational level but from a code level as well.

4. Open Source is modular and minimalist

The more moving parts something has the higher the likelyhood something will break. The same adage can be applied to computer software and operating systems. The Linux kernel is highly modular and Open Source. The end user can select the components they need, customize the kernel for their hardware, their requirements, so that the kernel itself only has the minimal support needed to get the job done. The libraries and applications work in a similar manner, features can be compiled in or left out, providing a great deal of flexibility. There are many alternatives which are compatible with each other, for example glibc which is a core library can be replaced with the smaller uclibc or klibc, depending on the desired application. The kernel, application and support libraries can be deployed in a very minimal fashion.

The security debate is always taken out of context, both platforms have their experts, and its just common sense to lock down services, firewall ports and remove unnecessary software components. The problem with Microsoft Windows is that its capability in this area is very limited because it is closed source. Microsoft have to have a base level of support, which means lots of libraries, lots of applications and lots of built-in APIs are retained even in a hardened system. Windows is less secure simply because it has a much larger menu of potentially exploitable components left on a hardened system, than a highly minimal deployment of Linux.

Linux can be stripped down to just the necessary hardware support, networking support and I/O support in the kernel, libc, libraries needed for the server application and the server application itself. Linux applications can be compiled position independent, placed within hardened chroot (unbreakable) and in special file systems, making the system extremely difficult to utilize even if it is running exploitable code. Windows on the other hand, you can do registry hacks but ultimately you cannot disable some API calls you might never need, but could be exploited through shell code passed into a valid service.

It is this minimalist approach that provides Linux with its great stability and insane uptimes. Even though Open Source advocates throw the Blue Screen of Death card once in a awhile, Windows has good stability, but a lot of that has to do with the higher hardware requirements and deployment checks that Microsoft use to steer customers in a particular direction.

5. Open Source has better performance

The minimalist nature of Linux means that more resources are available to the system for running the server application that it needs to, rather than eating up resources with GUI and processes that are part of the minimal operating requirements of Windows. Windows by design, has a resource tax on it. Microsoft have made efforts to reduce the resource tax with their Core Server offshoot, but ultimately Core Server still retains the GUI components they have simply changed the shell and cut down some of the applications. On like for like hardware, Linux will use considerably less resources for the base operating system. This is why Linux is used on supercomputing projects because supercomputer projects have thousands of nodes, and with Linux there is a minimal operating system cost per node, leaving more resources for actual work. As more applications are loaded onto the Windows platform, more and more resources get shuffled away from work and into management. Thus the hidden cost of the Microsoft platform is that it has less resources to do work, meaning more servers, more facilities costs to get like for like capacity when comparing it to Open Source.

6. Open Source is about choice

A key advantage to Open Source is choice. There are a lot of very smart people out there working on Open Source, many projects come from research groups at Universities, students working on Masters Degrees and advanced research labs at large companies such as IBM, Sun, Google, Apple, Red Hat, Hewlett Packard and Dell. Take the Web server for example, the industry standard server is Apache, but there is a better solution out there called Nginx (Engine-X). Written by a developer in Russia who had a need for a more scalable web server, took a different approach to how low level i/o was done and produced an innovative piece of software. In an Open Source deployment, this new choice can be tested in the lab and migrated to over time. In a closed source environment, such a good idea would have to get past product managers, pushed into a roadmap and perhaps end up with a hybrid approach that protects the legacy product. Open Source provides choice and freedom of movement, so the organization can select the best solution that offers the feature, performance and scalability mix that is best for that business.

7. Open Source is about saving money not spending money

When a Microsoft shop needs to do something new, such as deploy iSCSI or a new database server. The IT administrator will look at the budget, determine what they need to order and spend money. When an Open Source shop needs to do something new, they research the options, determine the best configuration, and test the solution. The difference here is the Open Source shop is looking for a solution from a technical standpoint, the Microsoft shop is looking for a solution that meets their basic needs and fits within their budget. The Open Source shop has the flexibility to deploy technologies now, rather than perhaps having to postpone projects for a quarter or two because of budget limitations.

Open Source makes sense so why do IT executives continue to look towards Microsoft

The simple answer to this is People. Microsoft have done a great job at making things easy, including System Administration. Anyone can pickup a Microsoft product, and by clicking enough options on or off, will eventually stumble upon a working solution. There are plenty of so called IT consultants out there who started off life as some kind of Business Manager who was tasked with installing Windows and setting up some application. It took them some time, but through trial and error, they could click through the options and make the solution work. Several years pass by, they are familiar with the Microsoft product line, know what to click to make things work, and know what to change when things break. There are lots of people in the industry like this, nice people but have far less of a grasp of how the actual technology works. These are the people that the “Elitist Linux People” have a problem with. The people who claim to be experts, promote Microsoft's solution but really lack the fundamental knowledge of how the technology works to be competent enough to evaluate one solution over another, technically. They are purely promoting Microsoft because it makes business sense to them. This me too concept of selecting technology is not the best way to go about it.

Contrast that to someone with an Open Source background, they will have some code level experience because sooner or later they would have had to patch, or compile a project from source, including the kernel. Many open source projects require you to edit or build configuration files, no fancy GUI of options to select from. The configuration files require the engineer to know what needs to be set, what function it provides and what impact it has on the deployment. The open source engineer will also have a grasp of how to pull low-level data from the system, from looking at /proc to using tools like lspci. The very nature of Open Source builds out a richer skill set overtime.

When an IT executive is looking at Open Source, existing non-Open Source staff are bound to get defensive. Open Source means looking at source code, employees realize they are not programmers, will they get replaced if they cannot learn to program? Depending on the employees, some may look at it as an opportunity to grow their skills and further their careers, but many of them will fear there is too much to learn and it is easier to simply replace them. If the IT executive is not entirely behind Open Source, a quick revolt from the rank and file is often enough to kill the idea.

Migration to Open Source starts with Integration

A successful Open Source strategy will begin with an entry point, such as a new project or replacement of an existing system. The best time to replace an existing system is at the end of a lease or when new software licensing is required, such as an upgrade. People are key to a good strategy, so whether you bring in a new one or train an existing one is a key business decision. To avoid concerns or revolts, adding basic software capabilities to the existing IT administrators is often a good idea. It will show the management team which employees are capable of learning new skills and which ones may need to be replaced. A good strategy will continue to leverage this new and replace strategy, until the organization has eventually migrated the majority of systems that it can to Open Source. Solutions such as VDI and Terminal Services enable applications which may not be readily available as Open Source to existing within an Open Source environment. The mix, or total replacement is something that needs to be handled on a case by case basis by the IT executives.

Conclusion

Open Source is an IT strategy and not a product. Effective migration to Open Source requires having the right people on the team. Third party services and consultants can be used to speed up the development of the in-house team, but ultimately the in-house team will need to be open source capable. The long term savings and gains for the business are clear, as Open Source costs remain flat as the solution scales, whereas Microsoft solutions tend to increase exponentially in cost as the solution scales up.

web-traffic
Bandwidth

Thinking outside the box – Filenames and Web Optimization

Overlooking the obvious, web developers and content management systems using long file and path names are wasting bandwidth. While these are handy to identify and manage objects, they increase the length of HTTP requests, force extra processing on Application Delivery Controllers and waste bandwidth. We look at simple and yet creative ways to reduce bandwidth costs for high traffic sites.

There are many techniques a web developer can use to improve the performance of their high traffic web site. However one that is often overlooked, despite being blatantly obvious is the length of directories and filenames that are used to create URLs. This article addresses the problem of descriptive yet wasteful path names, points out the pitfalls and demonstrates some solutions to address the problem.

Basics: The HTTP GET Request

When a client visits a web site, regardless of whether they click on a link or type the URL directly into their browser, they generate what is called a GET request.
The initial GET request pulls down the HTML file, which is then processed, and subsequent links generate new HTTP GET requests, such as CSS, Javascript or Images. The longer the PATH to those links, the larger the HTTP GET request will be. Using telnet we can simulate a HTTP GET request. Here is an example to o3magazine.com.

  • telnet www.o3magazine.com 80
  • Trying 38.106.106.237…
  • Connected to www.o3magazine.com (38.106.106.237).
  • Escape character is ‘^]’.
  • GET /index.html HTTP/1.1
  • host: www.o3magazine.com
  •  
  • HTTP/1.1 200 OK
  • Server: nginx/0.6.35
  • Date: Fri, 27 Mar 2009 00:30:37 GMT
  • Content-Type: text/html
  • Content-Length: 15954
  • Last-Modified: Fri, 27 March 2009 00:30:06 GMT
  • Connection: keep-alive
  • Accept-Ranges: bytes
  •  
  • …. HTML document is returned ….

 So the GET request consists of GET /URL HTTP/1.1. For o3magazine, the next GET request would be for /c/0.css. o3magazine already optimized its filenames, so it will produce a smaller GET request. Compare that to say techcrunch.com, the next GET request would be for /wp-content/themes/techcrunchmu/style.1238108540.css. Assuming UTF-8 encoding, and standard ASCII characters then each character is represented by one byte. The shorter o3magazine GET request will use four bytes for the GET and whitespace, then eight bytes for the /c/0.css and nine bytes for the trailing information. This is a grand total of 21 bytes. The longer techcrunch request will share the GET and trailer, so thats thirteen bytes, then it has 52 bytes for its request. This is a grand total of 65 bytes, over three times the size of the shorter request.

But its just a few bytes

So by now you are wondering why we care about this 44 byte difference. Going through the rest of the techcrunch source file, you see that similarly long requests are done all over the place thanks to the way Word Press is designed. There is roughly 57 such requests on the techcrunch page, from javascript to images, most of the URL paths are actually longer than the CSS one but for arguments sake, lets average it down to just 44 bytes extra per request. So on a single page load, we have 44 bytes x 57 requests for a total of 2508 bytes.

Factoring in the 2508 bytes extra is just on the GET request from the client to the server. The server has sent 2508 bytes of extra information that we really don’t need in the initial HTTP response, looking at the source file even closer, we can make the same assumption about the HREF code pointing to other links. The techcrunch page has around 182 HREF references, assuming our 44 byte different on the longer URLs, and thats being generous since Word Press URLs are basically sentences, gives us another 8,008 bytes of waste. So downstream we have a total of 10.5k in waste, and 2.5k upstream in waste.

Factoring in the masses

Using data from compete.com, techcrunch gets at least 7,650,594 visits a month. This is roughly 246,793 per day. So in a single day, techcrunch has wasted 2,531MB downstream, and 603MB of upstream. Over the space of a month, Techcrunch has wasted 78,461MB and 18,693MB in unnecessary data transfer. That is approximately 232k/sec of bandwidth (sustained). No big deal right? Its just 232k/sec.

Perhaps a valid point until the concept is applied to the 100+ character URL happy Facebook home.php page. There are roughly 150 source file references on this page, and rounding down to about 100 HREF requests for arguments sake. Being generous, assuming 80 bytes of waste per URL. Thats 12000 bytes of upstream, and 20000 bytes of downstream waste. So using data again from compete.com, facebook has 1,273,004,274 visits per month. This is roughly 41,064,654 requests per day. So on a single day, the folks over at facebook have wasted roughly 783GB downstream and 469GB upstream. This works out to be 74Mbit/sec downstream and 44MBit/sec upstream of bandwidth.

The Questionable Math

So to calculate the bandwidth utilization we took the visits per month (1,273,0004,274) and divided it by 31. Giving us 41,064,654. We then multiplied that by 20, to give us the transfer in kilobytes per day of downstream waste, based on 20k of waste per visit. This gave us 821293080, which we then divided by 86400 which is the number of seconds in a day. This gives us 9505 kilobytes per second, but we want it in kilobits, so we multiply it by 8. Giving us 76040, finally we divide that by 1024 to give us the value in MBits/sec. Giving us 74Mbit/sec. One caveat with these calculations is that we do not factor in gzip compression. Using gzip compression, we could safely divide the bandwidth wasting figures by about 50%. Browser caching does not factor in the downstream values, as we are calculating the waste just on the HTML file. It could impact the upstream usage as not all objects maybe requested with every HTML request.

CSS Identifiers

A similar approach to optimization can be taken with CSS identifiers. These are the arbitrary names given to classes and identifiers in div blocks to denote styles. Most web developers, especially the folks over at facebook like to make these very descriptive, such as sound_player_holder, presence_preload or presence_menu_opts_wrapper, while these are easy to identify they are a terrible waste of bandwidth. These could shortened to say pr_m_op_w and be relatively easy to figure out. A simple script could be used to minify these identifiers, leaving the developers with their descriptive names but the production solution running a shorter, optimized name.

Bandwidth Savings

So far we have shown that descriptive but excessively long URL paths can waste a considerable amount of bandwidth, and just acting with some common sense in this area can yield dramatic cost savings. As service providers look to cut costs, looking at the bandwidth savings from smaller GET requests alone could save service providers some money, especially if such optimizations were done to many different high-volume sites. However there are more advantages than just saving bandwidth.

Mobile Users

Shorter URLs and shorter CSS identifiers lead to friendlier content for mobile-device users. Taking a device such as an iPhone running on AT&Ts EDGE network, which typically gets between 75 and 135Kbps, you are looking at dramatically faster content and happier users by reducing the size of the URLs. Reducing the size of the GET requests, as well as reducing the amount of processing the mobile device has to do itself in parsing the HTML, will improve the overall experience for the user, as well as reduce their data-usage rates. Everyone wins.

Layer 7 Processing

Many providers and content delivery networks deploy deep packet inspection devices, sometimes called Layer 7 switches or Application Delivery Controllers. These devices are typically embedded network switches or appliances that perform what is known as deep packet inspection. They look inside at the content and requests for content, to perform additional processing and traffic management. There are dozens of different features, but popular ones include URL load balancing, URL rewriting, Cookie Processing and HTTP Header processing. These all typically look at the HTTP request and make some kind of routing decision based on its content. For example, URL load balancing may look for an image file extension in the request, such as .jpg, and forward it to a different set of servers than those that process .php or .html.

These devices have to typically buffer the HTTP client request. This requires storing the HTTP request packet in memory, perform processing on it, perhaps altering the request, then forwarding it on to the server. The longer the URL or Cookie contained within the request, the more memory and processor resources it will consume on one of these devices. So while the device is capable of processing the packet and handling the request, it is far from optimal. In fact, by using very long URLs, the web developer is inadvertently putting undue load on the system. The shorter the URL, the less buffer space it will take up, and the faster the device can process it. This results in lower latency times, increased capacity and lower loads on such systems. It just makes common sense to use shorter URLs.

CMS Developers

Developers of CMS, Blogging and other popular Web based applications should take this kind of optimization to heart. It is a relatively simple design change, that they can implement within their web application, enabling their end users to reap the benefits of shorter URLs and pathnames. The best solution would be one that allows developers to utilize descriptive and easy to follow names, while the published solution uses short and highly optimized URLs. Combined with Javascript and CSS minification (which is stripping out unnecessary characters, removing duplication from such files) and gzip compression, you would have a highly optimized solution.

Web Developers

While CMS developers such as WordPress can foot the blame for some things like Techcrunchs numbers. Web developers should also take these techniques to heart when developing custom applications. While a low-bandwidth site probably wouldnt benefit from short URLs, any site that has the potential to become a high-traffic site, or a widely used web application would benefit considerably. It is only responsible to invest some time and effort into using optimally sized URLs. For example, shortening /images to /i, /javascript to /j and /css to /c, you would save a few bytes everywhere. Using numbers for filenames for images, javascript and css files, and perhaps documenting in a separate file what they are for, could also save some bandwidth. For example, 0.jpg could have a note indicating its the company logo instead of logo.jpg.

URL Rewrite

Changing back ends, especially third party back ends can be a time consuming effort and is often difficult to support. So instead of waiting for the web application provider to fix their application, one possible work around is to use URL rewrite capabilities on some web servers to use short URLs and map them to the longer ones used in the CMS or other application. This obviously is not an ideal solution, but its one possible stop-gap measure to utilize shorter URLs without having to rewrite the back end. However it will still use some additional processing on the server side.

Proxies

Many content providers place reverse proxies in front of web server farms to perform what is known as Application Acceleration. This is basically a fancy marketing term for using a reverse proxy so that not all requests are passed back to the server farm. It should be possible with most modern reverse proxies to perform the URL rewrite function there. It would require the proxy to be capable of altering the HTML content thats passed back to the user and mapping it back to the actual real value when passing requests to the server. However it could be an easier option than changing massive web applications.

Conclusion

For high-bandwidth web-sites the highly descriptive URLs which make development a lot easier are not necessary a cost-effective mechanism or a necessary one. As shown with the facebook example, there are considerable savings that can be made by just reducing the size of the URLs. While not necessary, it is one extra tool in the efforts to cut costs and produce more efficient websites.

 

OpenVPN
Open VPN

Secure Global Networks with OpenVPN

OpenVPN is a powerful TLS/SSL based VPN solution. OpenVPN is typically used for providing clients with access to a corporate network. This article goes outside the box, and looks at using OpenVPN to build secure server to server links in an effort to build a Global Private Network for production services such as MySQL, HTTP Back-ends etc.

 

OpenVPN is a powerful SSL/TLS based VPN solution. OpenVPN supports clients under Linux, Windows and MacOS X. So it is a good option if you need to support a wide variety of clients. That however, is not the focus of this article. Instead, we are looking at OpenVPN as a solution to build a private global network between geographically diverse sites over the Internet.

 

Why do I need this?

In today’s global economy, it’s no longer sufficient to simply throw up a server at some hosting provider in the United States, and serve clients around the globe. Customers simply don’t like “web wait”. If your site isn’t sufficiently fast enough, they are going to look elsewhere. This is particularly important in today’s web 2.0 world of highly interactive and dynamic web applications. The solution is to place servers at multiple locations, which we will call sites, around the world. Place enough sites, and you will be relatively close to almost any user around the world. However, for the experience to be seamless to the user, a user for example posting in a forum in the United States, should have their post seen in real-time or near real-time by a user on the other side of the globe. This means that you’ll somehow need to synchronize the data between each site. So you will need a private encrypted global network, to exchange back-end data securely between sites.

 

Traffic Types

Before looking at the design and deployment of such a network, we need to look at the type of traffic that will run over our private network. In most cases, you will have management, monitoring and production traffic. Depending on the number of sites you have, and how your network is deployed, you might want to separate the production traffic from the monitoring and management traffic, and then prioritize the tunnel that the production traffic is running through using normal QoS techniques.

Management traffic will usually be pretty low and will primarily include SSH connections to the remote servers to perform maintenance tasks and debugging. Monitoring traffic, on the other hand, would involve service checks every few seconds. Most importantly though is the production traffic that really depends on your network. If your network is fast enough, you might deploy a web accelerator and static web server at out the edge, and pass dynamic requests back to a central datacenter location, where something like Mongrel would handle the dynamic work, and pass the data back to the user. While this has the advantage of saving some effort with SQL replication, as the network usage grows, its not going to be an optimal solution.

In general, you will probably want to push MySQL or whatever database you are using for your dynamic web applications out to the edge. This means setting up a complex master multi-slave setup, and possibly modifying your web applications to pass SQL writes to a different SQL server. As you can see the majority of the back-end traffic over your private production network is going to be SQL (database) or HTTP traffic. You can expect some rsync or svn / cvs traffic for distributing content between the servers.

 

IP Addressing

The private back-end network over OpenVPN is going to need a number of private addresses. You will need to use the non-routable reserved blocks that we’re all familiar with – 10.0.0.0/8, 192.168.0.0/16 etc. With this type of deployment you will need two different blocks of IPs – point to point, and production services. The point to point subnets should be /30 (two host) subnets. To avoid confusion, use 192.168.0.0/16 as the source of your point to point, and 10.0.0.0/8 networks for your production services. For our example, we will use 192.168.192.0/24 as the source subnet for our point to points, and 10.43.43.0/24 as our production services subnet.

 

Caching vs Replication

Database replication is tricky business, and keeping data integrity preserved can be a challenge. In a multi-master situation, if one of the multi-masters drops out for awhile, it can cause all sorts of problems. The ideal solution is to only replicate the database out to the edge when it is absolutely necessary. In most cases, using some creative Caching techniques, the only data that needs to be sent back and forth over the VPN to the central web application servers might be a HTTP request and HTTP response, with the graphics, css, javascript etc, all being served locally. If you get complaints from customers, you might need to throw some additional sites into the mix, or perhaps replicate to a central location in different regions (eg. one central European site with the database, then multiple European cache / http servers). Typically as your network grows, you’ll end up with some form of hybrid solution.

 

EasyRSA

OpenVPN comes with a key management system, which is basically a set of scripts called EasyRSA. This is the best way to generate keys and manage certificates for OpenVPN. You should install EasyRSA on a secure system, ideally at your corporate headquarters, and limit access to the system. We have covered OpenVPN and EasyRSA in the past, but simply build your local

Certificate Authority (CA) with the script in EasyRSA, and then you simply use build-key to build client side keys, and build-key-server when you need to generate a key for a new server.

Come up with an easy to remember naming scheme for your keys, something like vpn-sX-location, where X is the server name or number, and location is the name of the location (for example atlanta, houston, london). Then simply build your keys: ./build-key vpn-s2-london for example, will produce vpn-s2-london.crt, vpn-s2-london.key and vpn-s2-london.csr. You need to transfer the .key and the .crt files along with the ca.crt from EasyRSA to your server.

 

Client side Configuration

onfiguring OpenVPN on the client side, which would be the remote “edge” servers, is relatively simple. You need to specify the OpenVPN server’s public IP, the point to point IP address information for the local and the remote, and the port number. Here is an example configuration file:

 

  • dev tun
  • remote 172.16.55.20
  • ifconfig 192.168.192.2 192.168.192.1
  • tls-client
  • ca /etc/sslvpn/cert/ca.crt
  • cert /etc/sslvpn/cert/vpn-s2-london.crt
  • key /etc/sslvpn/keys/vpn-s2-london.key
  • port 1921
  • proto tcp-client
  • user nobody
  • group nogroup
  • comp-lzo
  • persist-tun
  • persist-key
  • verb 3

It’s relatively straight forward. The ifconfig line tells OpenVPN that we are 192.168.192.2 and the remote side of the tunnel is 192.168.192.1. Port 1921, tells OpenVPN we are running on port 1921 on the server side, comp-lzo specifies lzo compression, persistent keys and persistent tunnel.

 

Server side Configuration

The server side has its own certificate and key files, along with ca.crt. The server side also has a dh1024.pem file, which is generated by EasyRSA. Each remote site is going to have its own instance of OpenVPN running on the centralized VPN server. For each remote server (or site) you will have a separate configuration file. Here is an example configuration file to go with our client configuration:

 

  • local 172.16.55.20
  • dev tun
  • ifconfig 192.168.192.1 192.168.192.2
  • tls-server
  • dh /etc/sslvpn/crypto/dh1024.pem
  • ca /etc/sslvpn/cert/ca.crt
  • key /etc/sslvpn/keys/vpn-server.key
  • port 1921
  • proto tcp-server
  • user nobody
  • group nogroup
  • comp-lzo
  • persist-tun
  • persist-key
  • verb 3

 

As you can see, the configuration is very similar to the client side, with just a few minor changes.

 

Starting the VPN

Make sure you bunch a hole in your iptables configuration to allow the source IP of the remote site to access to TCP port we have configured for OpenVPN, in this case its 1921. If you are using a Red Hat based system (CentOS, Fedora, RHEL), you can edit /etc/sysconfig/iptables and add :

 

  • -A RH-Firewall-1-INPUT -p tcp –dport 1921 -s 172.16.56.99 -d 172.16.55.20 -j ACCEPT

 

where 172.16.55.20 is the example public IP (its not really a public IP but we don’t publish real public IPs in examples) of the server, and 172.16.56.99 is the example public IP of the VPN client. First, start openvpn on the server side:

  • openvpn –config /etc/sslvpn/vpn-s2-london.conf 1>>/var/log/vpn/vpn-s2-london.log 2>>/var/log/vpn/vpn-s2-london_err.log &

You would spawn this with the respective configuration files and log redirects for each remote site. Then on the client side:

 

  • openvpn –config /etc/sslvpn/vpn-atlanta-server.conf 1>>/var/log/vpn/vpn-atlanta.log 2>>/var/log/vpn/vpn-atlanta_err.log &

 

If you look at the logs, you should see:

 

  • Attempting to establish TCP connection with 172.16.55.20:1921
  • TCP connection established with 172.16.55.20:1921
  • …
  • Peer Connection Initiated with 172.16.55.20:1921
  • Initialization Sequence Completed

 

 

Testing the Link

The quickest way to test the link is to ping the remote side’s point to point IP. In our case, the client side should be 192.168.192.2 and the server side 192.168.192.1. Bounce a few ICMP packets around with ping. Using scp, you can also test the data rate by transferring a dummy file to and from the server. This will give you a basic idea of how well the link is going to perform.

 

Adding Routes

Right now you just have a point to point connection, but you want to be able to access resources on the remote side. This is where things will vary depending on what you want to do. If you are doing MySQL replication, you will want a production subnet on both sides (remote “edge” site and the central data-center location). In that case, you’ll need to setup a private subnet on both ends. If however you are simply using the remote “edge” site as a cache and static HTML server, then you’d probably only need a subnet on the remote end, unless you had multiple servers at the remote edge site, which is often the case.

For our example, we are going to use 10.43.43.0/24 on the central data-center location, and 10.43.100.0/24 on the remote site. On the VPN server, you just need to add an interface and a route:

 

  • ip addr add 10.43.43.1/24 dev lo
  • ip route add 10.43.100.0/24 via 192.168.192.2

 

Notice that we haven’t added a dev statement to the route. The reason for this is that the tunnel can respawn and perhaps not end up on the same device each time. This works, and is safe. We’ve added 10.43.43.1 to lo, so that when packets from the remote side come in, we have something to test with.

On the client side:

 

  • ip addr add 10.43.100.1/24 dev lo
  • ip route add 10.43.43.0/24 via 192.168.192.1

 

Again, the same thing. The final piece is that you will need to add statements to your /etc/sysconfig/iptables file to enable the VPN traffic to pass back and forth.

 

Conclusion

Rather than bouncing production back-end traffic, along with management and monitoring traffic over the Internet, we’ve created our own managed private network of inter-connected VPNs. OpenVPN is ideal for this type of solution, there are other solutions out there such as IPsec.

 

ruby on rails
Ruby on Rails

Rapid Web Development with Rails

Ruby on rails is a rapid web application development framework. learn how rails can help you keep your web projects on track and on budget with record delivery times.

 

Ruby on Rails is an open Source web framework that is optimized for rapid web application development. Rails provides a
structured framework that makes development feel natural and easy to maintain. Ruby is a relatively easy programming language to learn; any programmer with even just a vague idea of Java, Python or another object orientated programming language will pickup Ruby fairly quickly.
With a little effort, Ruby on Rails will allow you to quickly develop and modify web applications in an efficient and cost-effective manner. Projects typically go over budget due to unrealistic time frames, unanticipated problems or the varying skills of developers on the team. Ruby on Rails will help mitigate these traditional problems by providing a fast and easy to follow framework for building web applications. Ruby on Rails applications are database centric, and Rails provides an object-relational management layer called ActiveRecord that significantly reduces the headaches caused trying to map object based programming languages with data contained in relational databases.

Rails provides instant gratification – you make a change, you point your browser, and you see the change in effect. It’s instant. This has a key benefit when demonstrating an application or proof of concept application to a client. A lot of problems around customer web applications center around the ability of the engineering team to communicate with the customer and many companies have a management “translator” who interacts between the customer and the engineers.
Rails eliminates this need, as the customer can see the application in real time. If the developer misinterpreted the customer’s feature request, the developer can quickly load up an editor, modify the feature and show the customer the change instantly.
This allows for faster and easier communication with the customer and will increase the customer’s atisfaction with your business. In fact, many non-technical customers who have dealt with developers using traditional means, such as the document, develop, demonstrate, document, develop, and demonstrate cycles, will be highly impressed by the instant nature of rails. I had one businesswomen refer to it as “magic”.

If you are still not convinced about the speed and simplicity of Rails, then I suggest you take a look at the screencasts that are available over at http://www.rubyonrails.com/screencasts.

 

Getting started

Ruby is an interpreted, high-level object orientated programming language, and in some situations it may not perform as fast as lower-level languages such as C. However, there are a number of things you can do to improve performance. Slow running programs typically have a few locations where the processes are heavily hit. You can use the Benchmark module and code profiler that come with Ruby to help locate and fix these types of problems. Since most web developers write code these days in high level languages such as Python, Perl and Java anyway, and Ruby stacks up very well in performance compared to these languages, so the performance hit, if any for a specific application is well worth the cost and time saving benefits of the language. If you would like to learn more about the Ruby Language itself you can look at http://www.ruby-lang.org/. If you would prefer a book, I would highly recommend Programming Ruby: The Pragmatic Programmer’s Guide by Dave Thomas.

RubyGems is the package management system for Ruby. The command was shortened to gem. Gems interacts over the Internet with rubyforge.com, a huge repository of software code maintained by the rapidly growing Ruby community. RubyForge provides access to over 1,100 hosted projects, so you will often find something you are looking for without having to code it yourself.

 

Building the environment

Rails can be integrated with Apache, Lighttpd and many other web servers that support SCGI or FastCGI. Lighttpd is a good option, and was reviewed in last month’s issue of O3. To get started you will need Ruby and RubyGems. The recommend builds are listed on the Rails download page (http://www.rubyonrails.com/down).

 

Building ruby

The recommended release for Rails is currently 1.8.2. Untar it, and go through the usual POSIX motions (./configure && make && make install). If you want to make use of the ruby documentation tool (ri) then you will also want to run make install-doc.

 

Building rubygems

Installing RubyGems is trivial – simply untar and run ruby setup.rb from the rubygems-0.8.11 directory.

 

Installing rails

Now that you have both Ruby and RubyGems installed, you can use the gem command to grab rails. You will need to make sure that you have Internet access on the system you are using Ruby on before running the following command:

 

  • gem install rails –include-dependencies

 

As simple as that, rails is now on your system. Keep in mind that Rails is database centric, meaning you will need to have some kind of database available to you in order to utilize Rails. A wide range of databases are supported including MySQL and PostgreSQL. For this article, I used PostgreSQL 8.1.

 

Architecture

The Rails architecture is built around MVC, ActiveRecord (Object-Relational database management) and URL mapping. These are core concepts for using Rails so we shall look at them briefly.

 

MVC

MVC is an architecture designed by Trygve Reenskaug back in 1979. It is a Model, View, Controller architecture. The model manages the state of the application. It is not just a representation of data since it enforces the business rules that are applicable to that data. Such rules might be that shipping within the state should always be UPS Ground because it will get there overnight regardless, that sales tax is charged based on the destination within the state or that customers outside the European Union are not charged VAT.

The view is the visual presentation of the data to the user. This is the user interface, and from a rails point of view, is usually the HTML that is displayed to the user’s browser. The view might be different for different perspectives, so the administrator might see all the users, while a single normal user may only see their preferences.
As the name suggests, a controller maintains control of the application. It takes events from the user, interacts with the model (data) and provides a new view to the user.

 

Activerecord

Anyone familiar with injecting SQL commands into their PHP or C code is engaged in whats called Database-centric programming. If you know SQL, then this is a relatively easy and pain free method of obtaining data from a database quickly and easily. If you are a good programmer, you probably write centralized functions that are called throughout your code, so that if you need to modify how your application interacts with the database or to change databases, you have a relatively pain free method of doing so. Unfortunately, not everyone is a good programmer, and maintaining code over time with hundreds of SQL statements throughout thousands of lines of code can become a very painful task.

Rails utilizes a method called Object/Relational Mapping where database tables are mapped to object classes. This is not an easy thing to achieve and often requires considerable amounts of XML configuration files. ActiveRecord solves this problem by providing an ORM layer within Rails. ActiveRecord, like many aspects of Rails, relies on convention and sensible defaults, making it easy to modify and customize to your specific needs. ActiveRecord integrates seamlessly with the rest of the Rails framework.

 

URL mapping

The last important concept to grasp is that Rails utilizes URLs to map requests to a specific controller and action.

 

  • http://192.168.99.202/rails/helloworld/greet/hello

 

In the URL above the http://192.168.99.202/rails/helloworld/ identifies the application, the greet/ selects the controller (greet) and the hello provides the action to invoke. However, Rails is reasonably flexible about how it parses the URLs, so it is possible to parse them differently should you wish to do so.

 

Building a simple application

We will now walk you through the creation of a simple “hello world” application in rails. The steps we will take will be to create the application, add a controller, add a view, add some dynamic content and then test the application.

 

Creating the application

Creating applications with rails is trivial. With rails installed earlier, we simply change directory over to where we want to store our new project then run rails, for our application we will run rails helloworld. This will create a new directory called helloworld, and populate it with the default files for rails. To start the test server (assuming your not running it via FastCGI under another web server) simply run helloworld/scripts/server and then go open up a new terminal.

 

Adding a controller

Next we will need to add a controller. We will add one called Greet as it will manage the greeting to the user.

 

  • ruby script/generate controller Greet
  • exists app/controllers/
  • exists app/helpers/
  • create app/views/greet
  • exists test/functional/
  • create app/controllers/greet_controller.rb
  • create test/functional/greet_controller_test.rb
  • create app/helpers/greet_helper.rb

 

Now we edit apps/controllers/greet_controller.rb. As you can see, it already has some code in there for us:

 

  • class GreetController < ApplicationController
  • end

 

We will add the action “hello” to our controller:

 

  • class GreetController < ApplicationController
  • def hello
  • end
  • end

 

If you point the browser at the server http://192.168.99.202:3000/greet/hello/, you will get a message about a missing template. This is produced because we haven’t added the view yet. The view is added under app/views/greet/.rhtml. In our case hello.rhtml.

 

Creating the view

Adding the view is simple. We just create hello.rhtml with some html in it:

  • <html>
  • <head>
  • <title>Hello World!</title>
  • </head>
  • <body>
  • <b>Hello World!</b>
  • </body>
  • </html>

 

Point the browser at Rails again, and now we get Hello World!

 

Adding some dynamic content

Just to demonstrate how easy it is to add dynamic content in Rails we will modify our code now to display the time. To do this, we need to update the controller (greet_controller.rb) to capture the time by modifying the hello action to:

 

  • def hello
  • @time = Time.now
  • end

 

Then you simply edit the hello.rhtml file to display the time:

  • <body>
  • <b>Hello World!</b><p />
  • The time is now <%= @time %>.
  • </body>

 

It’s as simple as that. We now get the following in our browser window:

 

  • Hello World!
  • The time is now Mon Dec 19 12:57:43 EST 2005.

 

Conclusion

Overall, Ruby is an intuitive and easy language to learn and Ruby on Rails provides a fast and optimized framework for developing web applications quickly. Not only is it great for prototyping and proof of concept applications, but you can simply expand upon the original proof of concept code and utilize it as a base for the actual application. Ruby on Rails is well worth a look whether you’re looking for a simple in-house project or trying to keep unrealistic project deadlines with traditional web solutions. The bottom line for businesses, especially web development businesses, is that Ruby can save you time and resources regardless of the size of the project.

 

smtp
Server, SMTP

Designing Scalable SMTP Networks

This article looks at designing scalable SMTP networks. The main principle behind the design used in this article is a frontline group of SMTP relay servers that act as the public MX servers. Then a core set of servers on the backend that provide the local delivery and more in-depth processing. The relay servers skim the bulk of the junk, leaving less processing to be done on the core.

The SMTP protocol is used for transferring email from clients to servers, and between servers. SMTP stands for Simple Mail Transport Protocol. This article looks at building scalable SMTP networks. While many small businesses and projects shy away from solutions with terms such as “scalable” and “enterprise”, this is not the case here. Small businesses often equate scalable and enterprise with costly, due to the way expensive, and often less useful products are marketed. Before going in-depth, let’s cover some of the SMTP basics.

 

DNS and MX records

DNS plays an important part in how email is delivered with SMTP. The SMTP servers are defined in A records. These are those DNS entries that assign an IP directly to a host name. DNS has a special record type called MX or Mail Exchange. These MX records tell a remote mail server how an email should be delivered to that domain. MX records consist of a preference number and a host name. The host name must point to an A record, not a CNAME (alias).

 

  • IN MX 10 mx2.smtp.splicednetworks.com.
  • IN MX 10 mx3.smtp.splicednetworks.com.
  • IN MX 10 mx4.smtp.splicednetworks.com.
  • IN MX 10 mx6.smtp.splicednetworks.com.

 

In the example above from o3magazine.com, you can see the IN MX defining the MX type.
The preference number set here as 10 for each server. The final element is the host name. Also note the full stop that is required to terminate the host name in the zone file. Here the preference number is the same, so the DNS server will randomize the response list with each request. The remote server will then go down the list one by one. The server will stop when it successfully delivers the mail, or it receives an unreachable host or unknown user response from one of the servers in the mx list. If the preference numbers were different, then the highest priority (lowest MX preference value) would be tried first.

 

Looking up MX values

 

The host command (or nslookup) can be used to find the MX values for a particular domain. For example, host -t mx yahoo.com will yield:

 

  • yahoo.com mail is handled by 1 a.mx.mail.yahoo.com.
  • yahoo.com mail is handled by 1 b.mx.mail.yahoo.com.
  • yahoo.com mail is handled by 1 c.mx.mail.yahoo.com.
  • yahoo.com mail is handled by 1 g.mx.mail.yahoo.com.

 

 

Malicious Users

 

Spammers and those attempting to propagate viruses sometimes target the lowest priority servers on an MX list. These are often backup servers, sometimes at an ISP, which may not have the same anti-virus or anti-spam techniques employed as the primary servers. With modern high availability solutions, and the nature of MX records, it is not really necessary to utilize
higher MX values. It is advisable to simply use the techniques employed above by o3 magazine and Yahoo, thus avoiding targeting of specific servers.

 

Sender Policy Framework

 

The experimental RFC 4408 defines SPF. SPF allows SMTP servers to identify and reject forged addresses in the MAIL FROM header during the SMTP exchange of mail. The forged addresses are often used in spam, as an attempt to get around filtering systems. SPF works by having the owner of a domain define a special DNS entry. The SPF DNS entry is a TXT entry which contains information about how email should be sent from that domain. The SPF entry contains the version and some other information.

 

  • mydomain.com. IN TXT “v=spf1 mx a:mx0.mydomain.com -all”

 

In the example above, the zone file for the domain is configured with an SPF version of 1, if it comes from one of the MX servers for that domain, or if it comes from mx0.mydomain.com. Anything received from any other sources will be rejected (-all). There is also a PTR entry which will check the reverse DNS of the senders IP address, to see if it resolves to the same domain as the sender domain.

 

SMTP Load Balancing

 

In addition to the round robin MX load balancing that DNS does by default, you could use a hardware load balancer to load balance each MX. There are several open source and commercial options available to do this. The advantage of using load balancing is scalability. Instead of having a large list of MX records, you can have a small number of MX records. Each one of those MX records would point to a virtual server on the load balancer. Each virtual server could have hundreds of servers behind it. If the deployment has only a handful of servers, using individual MX records will work perfectly fine, regardless of whether a server is down or not. Load balancing SMTP may also enable a finer level of control over load
balancing, but the advantages would be marginal.

 

Relay and Core Servers

 

The scalable solution discussed in this article uses two type of servers, relay and core. The relay servers exchange mail with other SMTP servers on the Internet. Then the relay servers forward valid email to the core servers. This design enables N+1 relay servers, which are defined by the MX records for the domain. These relay servers will privately deliver mail to the core server(s). The core servers are only known to the relay servers, and only accept mail from the relay servers. The idea here is that the N+1 relay servers receive all email, whether it is legitimate or spam. The relay servers weed out as much spam as possible, before forwarding the mail on to the core servers. The core servers have more costly anti-spam and anti-virus measures, but as the relay servers have already trimmed the mail down substantially, there is less load to deal with. The MTA (Mail Transfer Agent) of choice for the rest of this article will be Postfix.

 

Relay Servers

 

The relay servers have no local delivery mechanism and they do not deliver mail to any users. Instead the relay servers perform specific anti-spam and other checks, before delivering mail to the core servers. The solution uses RBL (Realtime Blackhole Lists), Relay Domains, Relay Recipient Maps and header checks. There are no advance anti-spam measures on the relay servers, no dspam, no spam assassin, no postgrey. Those solutions need to gather data from multiple emails. The problem is that with N+1 relay servers, each relay server only receives 1/N of the traffic. The goal with this solution is to keep it simple, so the advanced processing occurs on the core servers, with reduced levels of mail.

 

Relay Servers: RBL

 

Realtime Blackhole Lists use DNS to determine if a particular SMTP server is on the list or not. There are several RBLs out there, we recommend Spamhaus (zen.spamhaus.org), Distributed Sender Blackhole List (list.dsbl.org) and RFC Ignorant (dsn.rfc-ignorant.org). Spamhaus deals with spammers, spam gangs and other parties that just spew junk mail in all directions. Spamhaus is a good way to block all those source IPs that are ripe for spam, and unlikely to contain serious business customers. Spamhaus references all the IP address blocks that are used for dynamic IP allocations for residential cable, dsl and dial up users at all the major ISPs. The distributed sender blackhole list, sends out test messages to see if a particular mail server will relay for something it shouldn’t. For example, if you try to send email for someone@yahoo.com through the Google Mail servers, from someone@hotmail.com, it won’t work. However, many users who setup things like Microsoft Exchange on their corporate networks, some of those users being IT consultants, create open relays. Spammers love open relays, and open relays today really should not exist, as they are simply a sign of incompetence. Finally, RFC Ignorant checks to see if a server is RFC compliant using some tests. Today, if a server is not RFC compliant, good chance it’s some off the wall junk setup by someone who shouldn’t be running a mail server. The RBL is a first line of defense, and will cut spam down considerably.

 

To configure RBL with Postfix, use the smtpd_recipient_restrictions command, along with the reject_rbl_client or reject_rhsbl_sender keywords.

 

  • smtpd_recipient_restrictions = reject_rbl_client zen.spamhaus.org reject_rbl_client list.dsbl.org reject_rhsbl_sender dsn.rfc-ignorant.org permit

 

This is just an example, a full smtpd_recipient_restrictions entry would reject on additional parameters, see the full configuration listed later in the article.

 

Relay Servers: Relay Domains

 

Postfix enables the configuration of a relay domain map. A relay domain is simply a list of domains that you will relay for, your domains. If a remote system tries to send to a domain not configured in the map, it won’t work. This is easily configured with the relay_domains configuration option. For this article, we used mostly hashes, which are generated by running postmap against a text file in /etc/postfix. For example, postmap relay_domains will produce relay_domains.db from the relay_domains configuration text file. In the main.cf configuration file for Postfix, you would configure:

 

SSL
SSL

Open Source SSL Acceleration

A look at utilizing Open Source projects to build an SSL Accelerator for Web Servers that rivals even the most expensive commercial solutions.
Advanced SSL Acceleration, Layer 7 URL Processing and Web Acceleration through gzip off-loading.

SSL acceleration is a technique that off-loads the processor intensive public key encryption algorithms used in SSL transactions to a hardware accelerator. These solutions often involve a considerable up front investment as the specialized equipment is rather costly. This article though looks at using off the shelf server hardware and open source software to build a cost effective SSL accelerator.

Types of SSL Acceleration

Ultimately there are two ways to do SSL Acceleration or SSL off-load. It can be done on the server side by installing an SSL Accelerator card, which has special custom processors designed to perform the public key encryption algorithms in hardware rather than software. A quick search on Google will provide a number of guides on how to use those cards with projects such as Apache. This solution has the server performing the SSL transaction but the SSL transaction is processed on the card rather than using server resources. In fact, adding one of the higher end versions of these cards to the solution in this article would further increase its performance. However, it may simply be cheaper to add more horsepower to the server itself than to invest in expensive SSL off-load cards.

The other way to do SSL acceleration is to install a device in front of the web servers, this is typically an appliance or switch with comparable hardware to the SSL accelerator card. These devices often provide other features such as load balancing. They typically have higher transactions per second and thruputcapacity than a single server with an SSL accelerator card. The SSL accelerator in front of the servers takes the incoming SSL transactions, decrypts them, and then forwards them on to the servers as HTTP. This is still secure as the connection between the SSL accelerator and the servers is a private local network, there is no unsecured transaction going over the public Internet. This is the type of solution provided in this article, but instead of using an expensive SSL accelerator, it leverages the power of open source and off the shelf server processors.

Advantages of SSL Acceleration

SSL Acceleration reduces the amount of server processing considerably. The transaction coming into the web servers is simply a HTTP request, so there is no SSL encryption or decryption that needs to be done. The cost of managing SSL certificates and the number of SSL certificates is drastically reduced, now those SSL certificates reside on one or two SSL Accelerators, instead of many web servers. Improved security, as the SSL Accelerator is the only device that needs to be accessible to the Internet, the back-end web servers can reside on private IP subnets. The solution hides the back-end topology and the only IPs that need to access the web server ports are the SSL Accelerators.

Overview of the Solution

The Open Source SSL Accelerator requires a dedicated server running Linux. Which Linux distribution does not matter, Ubuntu Server works just as well as CentOS or Fedora Core. A multi-core or multi-processor system is highly recommended, with an emphasis on processing power and to a lesser degree RAM. This would be a good opportunity to leverage new hardware options such as Solid State Drives for added performance. The only software requirement is Nginx (Engine-X) which is an Open Source web server project. Nginx is designed to handle a large number of transactions per second, and has very well designed I/O subsystem code, which is what gives it a serious advantage over other options such as Lighttpd and Apache. The solution can be extended by combining a balancer such as HAproxy and a cache solution such as Varnish. These could be placed on the Accelerator in the path between the Nginx and the back-end web servers. Adding such options to the mix requires some trickery on the back-end to preserve the original client IP request.

In this solution, Nginx will handle the inbound SSL connections just like a regular HTTPS web server, but it will pass the request off to another web server rather than handling it by itself.

Configuring Nginx

Nginx can be built from source or installed using your favorite package management tool. Building it from source has been discussed in previous articles, and is well documented on the Nginx site itself. So to avoid duplication, this article will dive straight into the configuration. Within the http {} block, enable gzip compression.

  • gzip on;
  • gzip_min_length 1100;
  • gzip_buffers 4 8k;
  • gzip_types text/plain text/html text/css text/js;

With gzip compression enabled on the SSL Accelerator, there is no need to enable gzip compression on the web servers behind it. Thus further reducing the load on the servers. The upstream {} command is used to define which back-end set of servers to send requests to. For this example, the configuration is for ssl.b.o3magazine.com with three servers defined by private IPs. The SSL Accelerator in the test example had two NICs, one facing the public side and one facing the back-end server network, with an IP of 192.168.77.1/24.

  • upstream ssl.b.o3magazine.com {
  •     server 192.168.77.10:80;
  •     server 192.168.77.20:80;
  •     server 192.168.77.30:80;
  • }

Finally the magic is configured within the server block.

  • server {
  •     listen 10.77.4.43:443;
  •     server_name ssl.o3magazine.com ssl.b.o3magazine.com;
  •     access_log logs/sslacc.log main;
  •     root /sslacc/www/o3magazine/htdocs;
  •     index index.html index.htm;
  •     ssl on;
  •     ssl_certificate /certs/sslo3.crt;
  •     ssl_certificate_key /keys/sslo3.key;
  •    
  •     location / {
  •      proxy_set_header X-Real-IP $remote_addr;
  •      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  •      proxy_set_header Host $http_host;
  •      proxy_set_header X-FORWARDED_PROTO https;
  •      proxy_redirect false;
  •      if (!-f $request_filename) {
  •       proxy_pass http://ssl.b.o3magazine.com;
  •      }
  •     }
  • }

So this looks like a regular configuration entry for an SSL virtual server. It will listen on 10.77.4.43 port 443, the site is called ssl.o3magazine.com, has the usual ssl certificate and key configuration. The accelerator part is handled by the configuration within the location block. The X-Real-IP header is set to pass back the remote client IP, add the X-Forwarded-For, and set the HTTP host header. The X-FORWARDED_PROTO is there for web applications such as those running under Ruby on Rails. This command tells the web application to map its URLs to the HTTPS protocol even though the web application is receiving the request via HTTP. This is important because without it, the web application will generate HTTP rather than HTTPS type URLs, which will break links for the user.

The proxy_pass command passes the request via HTTP to the upstream block configured as ssl.b.o3magazine.com. This is what generates the HTTP request rather than a HTTPS request to the back-end servers. That is it, entries can be stacked in the server_name group, so if the same IP is used to bind to multiple sites and they use the same certificate. The block can be duplicated with the appropriate certificate lines changed and server_name lines changed to handle sites bound to the same servers but using different certificates.

Fine Tuning

The worker processes and worker connections determine the number of simultaneous connections Nginx can handle. Typically the number of worker processes should match the number of CPU cores on the system, then tweak the worker connections to meet the project requirements. It may also make sense to run multiple instances on Nginx, with the worker processes defined as 2, and the connections tweaked as needed. Then utilize the CPU affinity capabilities of the Linux distribution in use to assign each Nginx instance (the worker processes) to a particular CPU core. This is particular useful if one particular site is heavily hit, that site could be handled by a dedicated Nginx instance and assigned to a dedicated CPU core.

Storing Files on the SSL Accelerator

One thing that this solution can do that many commercial SSL accelerators cannot, is the ability to store files locally. In the example above, the conditional if statement only passes requests back to the web servers if the file is not found locally first. The ! -f in the if statement means if not found. To have files served by the SSL accelerator instead of back passing the request to the server, simply store the file in the document root within the appropriate path. In the example above, placing images in /sslacc/www/o3magazine/htdocs/images/ would have the SSL accelerator using local storage to serve up https://ssl.o3magazine.com/images/. On the flip side, if nothing is stored locally, simply remove the if statement completely to avoid the extra processing. On a high traffic site, it might make sense to store small heavily used files such as CSS, Javascript and Images on the SSL accelerator instead. This avoids passing traffic back to the web servers.

Offloading to different server farms

The conditional statements within Nginx are very powerful, its possible to perform a wide variety of Layer 7 packet processing on the HTTP or HTTPS request itself. This feature enables the solution to be deployed in non-traditional ways. A customer web site may consist of PHP and Rails components, along with static content. It might make sense to deploy the static content directly on the SSL Accelerator, then define two upstream blocks one for PHP and one for Rails. For example, php.ssl.o3magazine.com and rails.ssl.o3magazine.com. These would point to different IPs in the same format as the upstream block above. The conditional statement would simply redirect requests that matched the PHP application (perhaps by the .php and .inc extensions), using $request_uri and regular regex matching to http://php.ssl.o3magazine.com. Place that conditional above the existing one. Anything not PHP and not stored locally will get passed on to the rails back end. This makes the configuration simple and able to handle the URL based routing used in Rails. Further conditionals could be used to map the URLs used by the Rails App only to further secure the solution but depending on the application that maybe tedious.

Limiting Ciphers

There are many different reasons for limiting the types of ciphers that can be used. Perhaps a commerce solution where only high-end ciphers are desired. In that case, the follow lines added to any server block will restrict the ciphers and SSL version that can be used. The SSL Accelerator makes implementing site-wide policies like this very easy to do.

  • ssl_ciphers HIGH:!ADH;
  • ssl_perfer_server_ciphers on;
  • ssl_protocols SSLv3;

Offloading HTTP Compression

This solution can also be used to off-load gzip compression to the SSL Accelerator. Perhaps a HTTP device does not support gzip compression or the system administrator wants to reduce the processing overhead on the entire web server farm. To achieve gzip compression offloading, simply place the SSL accelerator in front of the web servers, configure the web servers in an upstream block, and add regular HTTP server entries to the nginx configuration. This would be the same process as above, just listening on port 80, without the ssl commands and without the proxy_set_header options. HTTP requests would be passed through the SSL Accelerator to the web servers, the return traffic being compressed on its way back to the client.

Performance

The lab test system was a dual processor AMD Opteron 2380 2.5GHz Quad-Core system, for a total of 8 processor cores, L2 cache was 4 x 512k, and L3 cache was 6MB. Standard Intel Server Gigabit NICs, and 32GB of RAM. On a Tyan Transport platform, the system cost was under $5k. The system had no problems handling over 26,590 TPS, the test lab ran out of capacity to generate additional transactions. Compare that to the F5 Networks Big-IP 6900 which handles a maximum of 25,000 TPS but carries a starting price tag of $55,000. That starting price only includes 500 SSL TPS, so expect to pay a lot more to get up to the 25,000 level. It should be possible using this solution and even better hardware (8xxx series Opterons and more RAM), and perhaps some 10GbE adapters from Intel to come close, if not beat the performance of the BIG-IP 8900. Which has a maximum rate of 58,000 TPS. While obviously not as polished as the F5 solution, this Open Source solution does get the job done for a fraction of the price.

Conclusion

Nginx once again has shown that it is a versatile open source project. For the cost of a server and a few hours work, any system administrator can increase the capacity of their existing server farm by building an Open Source SSL Accelerator. Reducing the complexity of certificate management, reducing the number of certificates needed and reducing the overall load per request on the existing server farm, this solution offers a cost-effective way of breathing new life into an existing server farm.