July 15, 2014

Load Balancing with HAProxy

Setting up and using HAProxy 1.5 on Ubuntu 14.04.

Proxies

SnapShooter Backups Server, Database, Application and Laravel Backups - Get fully protected with SnapShooter AD

While there are quite a few good options for load balancers, HAProxy has become the go-to Open Source solution.

It's used by many large companies, including GitHub, Stack Overflow, Reddit, Tumblr and Twitter.

HAProxy (High Availability Proxy) is able to handle a lot of traffic. Similar to Nginx, it uses a single-process, event-driven model. This uses a low (and stable) amount of memory, enabling HAProxy to handle a large number of concurrent requests.

Setting it up is pretty easy as well! We'll cover installing and setting up HAProxy to load balance between three sample NodeJS HTTP servers.

Common Setups

In this example, I'll show using HAProxy to proxy requests between three NodeJS "web servers" (NodeJS applications using Node's HTTP library). This is just for example - in reality, you'll likely see HAProxy used to distribute requests across other "real" web servers, such as Nginx or Apache.

I try to give examples that are as good for production as they are for examples, but in this case, it's not overly important - from HAProxy's stand-point, a web server, is a web server, is a web server.

In a more "real" setup, web servers such as Apache or Nginx will stand between HAProxy and a web application. These web servers will typically either respond with static files or proxy requests they receive off to a Node, PHP, Ruby, Python, Go, Java or other dynamic application that might be in place.

You can see examples of Apache/Nginx proxying requests off to an application in the SFH editions/articles on Apache and Nginx.

HAProxy can balance requests between any application that can handle HTTP or even TCP requests. In this example, setting up three NodeJS web servers is just a convenient way to show load balancing between three web servers. How HAProxy sends requests to a web server or TCP end point doesn't end up changing how HAProxy works!

Read up on how your application might be affected by using a Load Balancer here.

Installation

We'll install the latest HAProxy (1.5.1 as of this writing) on Ubuntu 14.04. To do so, we can use the ppa:vbernat/haproxy-1.5 repository, which will get us a recent stable release:

sudo add-apt-repository -y ppa:vbernat/haproxy-1.5
sudo apt-get update
sudo apt-get install -y haproxy

If you're missing the add-apt-repository command on Ubuntu 14.04, installing the software-properties-common package will retrieve it. This is different from previous Ubuntu's, who used the python-software-properties package to install add-apt-repository.

Sample NodeJS Web Server

Now that HAProxy is installed, we need a few web servers to load balance between. To keep this example simple, we'll use a previously mentioned NodeJS application, which just opens up three HTTP listeners on separate ports:

// File /srv/server.js
var http = require('http');

function serve(ip, port)
{
        http.createServer(function (req, res) {
            res.writeHead(200, {'Content-Type': 'text/plain'});
            res.write(JSON.stringify(req.headers));
            res.end("\nThere's no place like "+ip+":"+port+"\n");
        }).listen(port, ip);
        console.log('Server running at http://'+ip+':'+port+'/');
}

// Create three servers for
// the load balancer, listening on any
// network on the following three ports
serve('0.0.0.0', 9000);
serve('0.0.0.0', 9001);
serve('0.0.0.0', 9002);

On my server, I saved this at location /srv/server.js

We'll bounce between these three web servers with HAProxy. These will simply respond to any request with the IP address/port of the server, along with the request headers received in the HTTP request to these listeners.

HAProxy Configuration

HAProxy configuration can be found at /etc/haproxy/haproxy.cfg. Here's what we'll likely see by default:

global
    log /dev/log    local0
    log /dev/log    local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

    # Default SSL material locations
    ca-base /etc/ssl/certs
    crt-base /etc/ssl/private

    # Default ciphers to use on SSL-enabled listening sockets.
    # For more information, see ciphers(1SSL).
    ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    timeout connect 5000
    timeout client  50000
    timeout server  50000
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http

Here we have some global configuration, and then some defaults (which we can override as needed for each server setup).

Within the global section, we likely won't need to make any changes. Here we see that HAProxy is run as the user/group haproxy, which is created during install. Running as a separate system user/group provides some extra avenues for increasing security through user/group permissions.

Furthermore, the master process is run as root - that process then uses chroot to separate HAProxy from other system areas, almost like running with its own container. It also sets itself as running as a daemon (in the background).

Within defaults section, we see some logging and timeout options. HAProxy can log all web requests, giving you the option to turn off access logs in each web node, or conversely, turning logs off at the load balancer while having them on within each web server (or any combination thereof). Where you want your logs to be generated/saved/aggregated is a decision you should make based on your needs.

If you want to turn off logging regular (successful) HTTP requests within HAProxy, add the line option dontlog-normal. The dontlog-normal directive will tell HAProxy to only log error responses from the web nodes. Alternatively, you can simply separate error logs from the regular access logs via the option log-separate-errors option.

Load Balancing Configuration

To get started balancing traffic between our three HTTP listeners, we need to set some options within HAProxy:

  • frontend - where HAProxy listens to connections
  • backend - Where HAPoxy sends incoming connections
  • stats - Optionally, setup HAProxy web tool for monitoring the load balancer and its nodes

Here's an example frontend:

frontend localnodes
    bind *:80
    mode http
    default_backend nodes

This is a frontend, which I have named 'localnodes'. I named it 'localnodes' because the NodeJS app, used to simulate three web servers, is just being run locally. The name of the frontend is arbitrary.

  • bind *:80 - I've bound this frontend to all network interfaces on port 80. HAProxy will listen on port 80 on each available network for new HTTP connections
  • mode http - This is listening for HTTP connections. HAProxy can handle lower-level TCP connections as well, which is useful for load balancing things like MySQL read databases, if you setup database replication
  • default_backend nodes - This frontend should use the backend named nodes, which we'll see next.

TCP is "lower level" than HTTP. HTTP is actually built on top of TCP, so every HTTP connection uses TCP, but not every TCP connection is an HTTP request.

Next let's see an example backend configuration:

backend nodes
    mode http
    balance roundrobin
    option forwardfor
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request add-header X-Forwarded-Proto https if { ssl_fc }
    option httpchk HEAD / HTTP/1.1\r\nHost:localhost
    server web01 127.0.0.1:9000 check
    server web02 127.0.0.1:9001 check
    server web03 127.0.0.1:9002 check

This is where we configure the servers to distribute traffic between. I've named it "nodes". Similar to the frontend, the name is arbitrary. Let's go through the options seen there:

  • mode http - This will pass HTTP requests to the servers listed
  • balance roundrobin - Use the roundrobin strategy for distributing load amongst the servers
  • option forwardfor - Adds the X-Forwarded-For header so our applications can get the clients actually IP address. Without this, our application would instead see every incoming request as coming from the load balancer's IP address
  • http-request set-header X-Forwarded-Port %[dst_port] - We manually add the X-Forwarded-Port header so that our applications knows what port to use when redirecting/generating URLs.
    • Note that we use the dst_port "destination port" variable, which is the destination port of the client HTTP request.
  • option httpchk HEAD / HTTP/1.1\r\nHost:localhost - Set the health check HAProxy uses to test if the web servers are still responding. If these fail to respond without error, the server is removed from HAProxy as one to load balance between. This sends a HEAD request with the HTTP/1.1 and Host header set, which might be needed if your web server uses virtualhosts to detect which site to send traffic to
  • http-request add-header X-Forwarded-Proto https if { ssl_fc } - We add the X-Forwarded-Proto header and set it to "https" if the "https" scheme is used over "http" (via ssl_fc). Similar to the forwarded-port header, this can help our web applications determine which scheme to use when building URL's and sending redirects (Location headers).
  • server web01-3 127.0.0.0:9000-2 check - These three lines add the web servers for HAProxy to balance traffic between. It arbitrarily names each one web01-web03, set's their IP address and port, and adds the directive check to tell HAProxy to health check the server

Load Balancing Algorithms

Let's take a quick minute to go over something important to load balancing - deciding how to distribute traffic amongst the servers. The following are a few of the options HAProxy offers in version 1.5:

Roundrobin: In the above configuration, we used the pretty basic roundrobin algorithm to distribute traffic amongst our three servers. With roundrobin, each server is used in turn (although you can add weights to each server). It is limited by design to 4095 (!) servers.

Weights default to 1, and can be as high as 256. Since we didn't set one above, all have a weight of 1, and roundrobin simply goes from one server to the next

We can accomplish sticky sessions with this algorithms. Sticky sessions means user sessions, usually identified by a cookie, will tell HAProxy to always send requests from a client to a same server. This is useful for web applications that use default session handling, which likely saves session data on the server, rather than within a cookie on the clients browser or in a centralized session store, such as redis or memcached.

To do so, you can add a cookie SOME-COOKIE-NAME prefix directive into the backend. Then simply add the cookie directive within each server. Then HAProxy will append a cookie (or add onto an existing one) a identifier for each server. This cookie will be sent back in subsequent requests from the client, letting HAProxy know which server to send the request to. This looks like the following:

backend nodes
    # Other options above omitted for brevity
    cookie SRV_ID prefix
    server web01 127.0.0.1:9000 cookie check
    server web02 127.0.0.1:9001 cookie check
    server web03 127.0.0.1:9002 cookie check

I suggest using cookie-based sessions or a central session store rather if you have the option to do so within your web applications. Don't rely on requiring clients to always connect to the same web server to stay logged into your application.

static-rr: This is similar to the roundrobin method, except you can't adjust server weights on the fly. In return, it has no design limitation on the number of servers, like roundrobin does.

leastconn: The server with the lowest number of connections receives the connection. This is better for servers with long-running connections (LDAP, SQL, TSE), but not necessarily for short-lived connections (HTTP).

uri: This takes a set portion of the URI used in a request, hashes it, divides it by the total weights of the running servers and uses the result to decide which server to send traffic to. This effectively sets it so the same server handles the same URI end points.

This is often used with proxy caches and anti-virus proxies in order to maximize the cache hit rate.

Not mentioned, but worth checking out are the remaining balancing algorithms:

  • rdp-cookie - Session stickiness for the RDP protocol
  • first
  • source
  • url_param
  • hdr

Test the Load Balancer

Putting all those directives inside of the /etc/haproxy/haproxy.cfg file gives us a load balancer!

Here's the complete configuration file at /etc/haproxy/haproxy.cfg:

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # Default ciphers to use on SSL-enabled listening sockets.
        # For more information, see ciphers(1SSL).
        ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

frontend localnodes
    bind *:80
    mode http
    default_backend nodes

backend nodes
    mode http
    balance roundrobin
    option forwardfor
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request add-header X-Forwarded-Proto https if { ssl_fc }
    option httpchk HEAD / HTTP/1.1\r\nHost:localhost
    server web01 127.0.0.1:9000 check
    server web02 127.0.0.1:9001 check
    server web03 127.0.0.1:9002 check

listen stats *:1936
    stats enable
    stats uri /
    stats hide-version
    stats auth someuser:password

You start/restart/reload start HAProxy with these settings. Below I restart HAProxy just because if you've been following line by line, you may not have started HAProxy yet:

# You can reload if HAProxy is already started
$ sudo service haproxy restart

Then start the Node server:

$ node /srv/server.js

Note that I'm assuming the Node server is being run on the same server has HAProxy for this demo - that's why all the IP addresses used are referencing localhost 127.0.0.1.

Then head to your servers IP address or hostname and see it balance traffic between the three Node servers. I broke out the first request's headers a bit so we can see the added X-Forwarded-* headers:

{"host":"192.169.22.10",
"cache-control":"max-age=0",
"accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36",
"accept-encoding":"gzip,deflate,
sdch","accept-language":"en-US,en;q=0.8",
"x-forwarded-port":"80",             // Look, our x-forwarded-port header!
"x-forwarded-for":"172.17.42.1"}     // Look, our x-forwarded-for header!
There's no place like 0.0.0.0:9000   // Our first server, on port 9000

{"host":"192.169.22.10","cache-control":"max-age=0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36","accept-encoding":"gzip,deflate,sdch","accept-language":"en-US,en;q=0.8","x-forwarded-port":"80","x-forwarded-for":"172.17.42.1"}
There's no place like 0.0.0.0:9001  // Our second server, on port 9001

{"host":"192.169.22.10","cache-control":"max-age=0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36","accept-encoding":"gzip,deflate,sdch","accept-language":"en-US,en;q=0.8","x-forwarded-port":"80","x-forwarded-for":"172.17.42.1"}
There's no place like 0.0.0.0:9002  // Our third server, on port 9002

See how it round-robins between the three servers in the order they are defined! We also have the x-forwared-for and x-forwarded-port headers available to us, which our application can use

Monitoring HAProxy

You may have noticed the following directives which I haven't discussed yet:

listen stats *:1936
    stats enable
    stats uri /
    stats hide-version
    stats auth someuser:password

HAProxy comes with a web interface for monitoring the load balancer and the servers it is setup to use. Let's go over the above options:

  • listen stats *:1936 - Use the listen directive, name it stats and have it listen on port 1936.
  • stats enable - Enable the stats monitoring dashboard
  • stats uri / - The URI to reach it is just / (on port 1936)
  • stats hide-version - Hide the version of HAProxy used
  • stats auth someuser:password - Use HTTP basic authentication, with the set username and password. In this example, the username is someuser and the password is just password. Don't use that in production - in fact, make sure your firewall blocks external HTTP access to that port.

When you head to your server and port in your web browser, here's what the dashboard will look like:

For me, I reached this at http://192.168.22.10:1936 in my browser. The IP address 192.168.22.10 happened to be the IP address of my test server.

We can see the Frontend we defined under localhost, which I named the frontend. Actually when this screenshot was taken, it was called "localhost", but I changed it to localnodes after to be less confusing. This section shows the status of incoming requests.

There is also the nodes section (again, the name I chose for the defined backend section), our defined backend servers. Each server here is green, which shows that they are "healthy". If a health check fails on any of the three servers, then it will display as red and it won't be included in the rotation of the servers.

Finally there is the stats section, which just gives information about the stats page that shows this very information.

Resources

All Topics

Beginners

Development

Containers

PHP

Security

Proxies

Continuous Integration

Configuration Management

Databases