HLB - HTTP Load Balancer - Implementation

Base Class

Most any AppLogic appliance image can be used as the base class for HLB. The simplest one is LUX and it should install over it without the need to resize its disk image. NOTE: the actual HLB 1.0.0 as released is bult on a clone of LUX, with its /usr directory separated into a read-only volume and quite a few bugs in the original LUX image fixed. Some of these bugs are documented in Bug 564.

Class Volumes

The volume configuration for HLB is determined by the base class image from which it is created. HLB itself is not sensitive to the volume configuration and will work on most any setup. Besides the 'pound' binary that implements the load balancing, it requires only 'bash' and 'perl' to be present to be able to start. The setup chosen for the 1.0 release is as follows: - boot volume, 60MB size, mounted as "/", writeable, instantiable. - usr volume, 200MB size, mounted as "/usr", read-only, common. The two volumes contain a basic Linux installation as the one present on the LUX appliance. The HLB software is intstalled under the "/usr" sub-tree, except for the init script, which has to live in /etc/init.d/.

The following modifications are made compared to Pound 2.0.1:

• addition of code to modify or add session cookies, to support the synch and insert modes.
• minor performance optimizations.

Packages

• Main: 3tapp-hlb-1.0.0-1.noarch.rpm. This contains the initscripts to configure the Pound load balancer from the properties defined in /etc/applogic.sh (as generated by the AppLogic build system) and to run it as a background daemon.
• Dependencies: pound-2.0.2-1.i386.rpm. The Pound load balancer and all patches and additions.

contents of the release directory:

Exported appliance image:

• HLB-1.0.0exp.tar.bz2

"Generator" RPMs: install these in the order listed to re-create the appliance:

• pound-2.0.2-1.i386.rpm
• 3tapp-hlb-1.0.0-1.noarch.rpm

Source RPMs:

• pound-2.0.2-1.src.rpm
• 3tapp-hlb-1.0.0-1.src.rpm

Tests (see the test plan CatSwitchesHlbTp and the release notes CatSwitchesHlbRelNotes):

• hlb-tst.tar.bz2 - test scripts
• hlb-tst-app.tar.bz2 - test application

Theory of Operation

HLB has a single background service running - the Pound load balancer (http://www.apsis.ch/pound), modified and improved to provide the functionality documented in the HLB data sheet.

Here's a short summary of how Pound works. For more details - may the source be with you (it is only 3000 lines of code or so). It has a number of 'front end' and 'back end' objects defined in a configuration file ('listeners' and 'back-ends'). In the HLB setup, only a single 'listener' object is used, set up to port 80 on the 'in' terminal of the appliance. The back-ends are configured to the IP addresses of the connected 'outX' terminals. The 'listener' receives HTTP requests and starts a new task for each, running in its own thread. In the modified version of Pound, no actual thread creation takes place at runtime, a number of worker threads running in a loop is used instead. Each request is processed sequentially in its thread, going through these steps:

• analyse the request headers and check for errors
• identify the back-end to which the request is to be sent, using a cookie in the headers or selecting a random back-and as appropriate
• re-write the headers that need to be changed (e.g., modify or remove the cookie, in the 'synch' or 'insert' modes), add the originator's IP address as a custom header (back-end scripts could use this to idenitfy the client, instead of the IP address, which for them is always the IP address of HLB itself).
• send the request on
• wait for the response
• analyse the response headers, if a cookie is found that matches the session cookie, associate its value with a 'session object' later used to match new requests to the same back-end server
• re-write headers as needed
• forward the response back to the client

Configuration

The configuration file for Pound is generated at startup time by the initscript, specially built for the HLB appliance. It uses the properties defined in /etc/applogic.sh and uses them to fill in a template (written in Perl), which is then fed to Pound as 'stdin', with a command line like this:

perl pcfg.pl pcfg.tmpl | pound -f /proc/self/fd/0

A few of the properties related to new functionality which is not part of the Pound config file syntax are not written into the config file, they are accessed by the patched code directly from the environment (to avoid having to dig into the Pound's config file parser, which is rather messy): mode - the cookie mode (passive, synch, insert). The native mode in which the un-modified Pound operates is 'passive', the other two modes are supported by the patches added specially for HLB. cookie_domain, cookie_path - the path and domain properties for the 'insert' mode. max_connections - this is translated to the number of worker threads created for the thread pool. If it is set to 0, the thread pool code checks the available memory and creates 4 threads per megabyte (256K per thread, that is), setting the stack size to 128K, and assuming that the two network sockets used by each worker thread will use 64K of buffer memory.

Log Files

The request logging is disabled in Pound. It logs only errors and warnings into the system log. At the request rates that HLB is expected to be able to serve, the writing of request logs into the appliance's boot volume will oveflow it in a matter of hours (a request log entry is about 100 bytes in size, and HLB can easily handle a few million requests in one hour).

-- LeoKalev - 02 Mar 2006