r2 - 17 Nov 2006 - 15:23:00 - BeckyHYou are here: Wiki >  AppLogic23 Web > CatSwitchesHlbTp
ALERT! AppLogic 2.3 Beta Documentation The latest production release is AppLogic 3.0.30

HTTP Load Balancer - Test Plan

Preparation

Host platform

The automated tests described here for the HLB appliance are designed to run on a Linux system and are verified to work on Fedora Core 3. To the best of my knowledge, they don't need anything Fedora-specific and should work on other Linux distros.

3rd-party tools

Both of the 3-rd party utilities used in the test are included in binary form in the test archive, for easy running of the test. Below is a description of these utilities and where the sources for them can be found.

thttpd

the test for HLB uses three instances of the THTTPD web server as back-ends for the load-balancer (GPL, from http://www.acme.com/software/thttpd/, version used in the tests: 2.25b http://www.acme.com/software/thttpd/thttpd-2.25b.tar.gz, but really any version will work fine).

http_load

a simple load-generator tool, for performance testing (GPL, from http://www.acme.com/software/http_load/, version used in the tests: http://www.acme.com/software/http_load/http_load-29jun2005.tar.gz).

Tests Summary

Automatic tests (mandatory)

  • round-robin scheduling, no session cookies (cookie_name = '')
  • failover test - kill back-end server (get connection-refused errors)
  • failover test - kill back-end network subsys (or the entire OS) (get timeout)
  • failover test - recovery on restored back-end server
  • session persistence - passive mode
  • session persistence - revert to round-robin on server death
  • session persistence - insert mode
  • session persistence - synch mode, revert to rr on server death
  • session persistence - check cookie path and cookie domain is set correctly ('insert' mode)
  • session persistence - timeout kills old sessions, timeout is reset on active session
  • sess. persistence - timeout=0 disables sess. timing out
  • protocol compliance - HLB serves both 1.0 and 1.1 clients (nb: emulate 1.0 clt using telnet)
  • performance - throughput test, reqs/second
  • performance - throughput test, bytes/second (small & large file: 100 bytes, 6K)
  • performance - concurrent sessions test - opening up to the max number of connections at the same time

Manual / special tests (optional)

These tests either require operator intervention or special test conditions that are not available on the AppLogic grids that are currently set up.

  • session persistence with timeout=0. The automated test checks this with a short delay, but a delay of 1hr or more can be tested manually, or ran overnight.

  • performance tests with multiple back-end servers. This is to try and see the max performance (req/sec and bytes/sec) that can be achieved with the power of more than one HTTP server on the back end (2-4, each having a full CPU at its disposal).

Running the Tests

To run the tests on the controller:

Copy and un-compress the tests archive file.

Install the test harness application, or create one by connecting 4 instances of the LUX appliance (or anything similar), named clt1, srv1, srv2 and srv3 as follows:

clt1.out -> HLB.in
HLB.out1 -> srv1.in
HLB.out3 -> srv1.in
HLB.out8 -> srv1.in

(Note: any three outputs can be picked, it should work regardless)

Your test application should look something like this:

hlb-tst.png

Note that the names of the instances are important, they have to be HLB (the testee), clt1, srv1, srv2 and srv3. The test scripts will access them all by name.

Change to the directory where the tests were un-archived.

Prepare the test setup, by running the following command (this also verifies SSH connectivity to the test appliance instances, needed to control the test automatically):

bash tst-prep.sh localhost app-name

where app-name is the name of the test harness application.

The script will take a minute or so to run, and will start the test harness and copy all necessary files into it.

If the setup completes without errors, run the tests:

bash runall.sh localhost app-name</var

Preparing the 3rd Party tools

If re-building the test utilities (thttpd and http_load) is needed, do the following:

  • download and un-archive the sources
  • build the sources (./configure ; make ), but don't do 'make install'
  • copy the thttpd binary into the test root directory, copy the http_load bindary into the ./remote sub-directory in the test root directory
  • run the tests, as described above

Design

Structure

The test application is built from from the following:

  • an AppLogic application (test harness) containing the component under test (HLB) and several instances of the LUX appliance - a bare-bones Linux installation with only SSH running as an externally-accessible service.
  • test utilities and a remote test script, intended to run on the AppLogic controller. The test may also be run from a remote console that has SSH access to the AppLogic controller. Note: the remote access to the controller uses the experimental access macros that are being developed for use with a "closed" controller. Their use requires the setup of a "restricted" key, instead of or in addition to the normal (full access) SSH key in the .ssh/authorized_keys file on the controller.

The test script if fully automated and includes the ability to install the necessary test software on the LUX instances comprising the test harness.

Back-end Setup - Web space contents

The test script configures each of the LUX instances that serve as back-end HTTP servers for the test with the following structure of files (shown here relative to the HTTP namespace root of the server):

/index.html tiny web page (<100 bytes). The contents of this page are different for each back-end server.
/bugz.html snapshot of a Bugzilla front page, 6K HTML code. A larger file for data throughput tests.
/cgi/* cgi script directory
/cgi/cookie-tst bash shell script, sets and checks cookies
These are optional, for testing with a browser:
/sub1/index.html empty page in a sub-dir, for testing the cookie path setting (insert mode only)
/sub1/sub2/index.html empty page in a sub-dir, for testing cookie path (insert mode only)
/cgi/sub1/cookies_tst copy of cookies_tst above - use to test cookie path (passive & synch modes)

Test Details

round-robin scheduling, no session cookies (cookie_name = '')

This test submits multiple requests for 'index.html' (which is different on each back-end) and counts the number of responses from each back-end to verify even distribution of requests.

failover test - kill back-end server (get connection-refused errors)

The round-robin test described above is repeated, but one of the back-ends has its HTTP server stopped, while the back-end is still alive. This will cause the network stack on the back-end to reply with RST to connection attempts (seen as "connection refused" error by the load-balancer). HLB should *instantly* re-distribute traffic to the remaining back-ends.

failover test - kill back-end network subsys (or the entire OS) (get timeout)

Same as above, but the network stack on the back-end is disabled completely, resulting in time-out errors seen by HLB. The traffic should be re-balanced to the remaining back-ends after a *short* initial delay on the first connection that hits the dead back-end. [the network on the back-end is disabled using 'ifdown', which should have the same effect as stopping the OS altogether, as far as HLB is concerned].

failover test - recovery on restored back-end server

While a round-robin test is running with one of the back-ends disabled, the back-end is brought back into operation. HLB should discover this within 15 seconds and start directing traffic to it as well. The test must verify that once the first request goes through to the restored server, the balance between all the back-ends is even.

This test is combined with each of the two tests above in the test script implementation.

session persistence - passive mode

For this test, HLB is set to "passive" mode with cookie_name=sess_tst2. The /cgi/cookie-tst file is requested by the test, which should return randomly one of four different cookies (all unique to that back-end), imitating the existence of 4 independent sessions to that back-end.

The test first continues submitting new requests for /cgi/cookie-tst, without providing cookies in the requests, resulting in each request being directed to a different back-end (same as in the RR test above). The test records the cookies and the back-end server ID contained in each response.

After having collected a few responses, the test proceeds to submit new requests, this time using one of the cookies returned. Multiple such requests should result in an identical response coming from the same back-end server. Repeat this with several different cookies, each time the response should match the one received when the cookie was first recorded by the test.

The test code for this test should also verify that the cookies sent to a back end are received there intact (i.e., HLB does not 'eat' them or modify them). This is done by having the back-end script (/cgi/cookie-tst) send back the cookie values in the response body.

session persistence - insert mode

For this test, HLB is set to "insert" mode, with the cookie name set to 'sess_tst3'. Similar to the above test for "passive" mode, the /cgi/cookie-tst is requested and all checks performed for that test should also pass in this test, except that the 'sess_tst3' cookie must not reach the back-end (/cgi/cookie-tst sends the 'sess_tst1' and 'sess_tst2' cookies only, while 'sess_tst3' is inserted by HLB and should be removed on forwarding requests to the back-ends).

The above is repeated, but the test script does not send all cookies returned, only 'sess_tst3'. This is to verify that the cookie is removed correctly, when it is the only cookie in the request - the reply generated by /cgi/cookie-tst in this case will contain no cookie data in the body, indicating that the entire "cookie: " header was removed from the request.

session persistence - synch mode

This test is the same as in "passive" mode, except the value of the cookie that the client-side sees will be different from the cookie sent by the back-end. The 'sess_tst1' cookie is chosen as the 'session cookie'. The /cgi/cookie-tst script on all back-ends returns the same value for this cookie ("1"). The test should receive different values for this cookie each time it sends a new request to HLB. The expected cookie value seen by the test client is "?1" where ? is a character identifying the back-end (generated by HLB, different for each back-end).

When it sends a request with the cookie set, the test should verify that: (a) the response always comes from the same back-end, and (b) the back-end sees the cookie value as "1", not as "A1" or something like that.

session persistence - revert to round-robin on server death

The client first sends multiple requests and collects cookie values, to get one from each back-end. Then, one of the back-ends is disabled and requests are sent using the collected cookies. All but one should be served normally, by the same back-end that issued the cookie. The request for the dead back-end is expected to be directed to one of the living back-ends with no modification (i.e., the cookie is not stripped by HLB, even though it may be "invalid" from the perspective of the new back-end).

session persistence - check cookie path and domail is set correctly ('insert' mode)

The cookie path is set to a non-default value. Verify that the value is returned in the 'set-cookie' header generated by HLB when the cookie mode is 'insert'. This setting should not affect cookies in the other modes.

Repeat with setting the cookie domain. Note that HLB should return a cookie header with "domain=.." only if the property is set to a non-empty value.

session persistence - timeout kills old sessions, timeout is reset on active session

(same conn and re-opened connection alike)

Set the timeout to a small value (say 15 sec.), set cookie_name='sess_tst2'.

Request the /cgi/cookie-tst page multiple times, re-sending the cookie returned on the 1st request. All responses should come from the same server. Run this for 3 times over the timeout value, the session should be retained, since it is active.

Sleep long enough to exceed the timeout, re-send the request with the same cookie. The request should reach a random back-end (may be the same server as when the session was active - repeat the test, after a few re-runs it should get a reply from a different back-end).

sess. persistence - timeout=0 disables sess. timing out

Verify that a session is retained for 1min. if timeout=0.

* Manual test:* verify a session is retained for over 1hr, or however much you can wait if timeout=0. This can't be conveniently made part of the automated test.

protocol compliance - HLB serves both 1.0 and 1.1 clients

HTTP 1.1 client operation is verified by all the above tests (done with 'wget'). Use the /dev/tcp pseudo-file in bash to submit a 'get /cgi/cookie-tst HTTP/1.0' (with no headers, etc. - just the request line). This should be accepted by HLB and a valid page should be returned.

performance - throughput test, reqs/second (3 back-ends)

IMPORTANT: The net.ipv4.tcp_tw_recycle option must be set to 1 for this test (sysctl -w net.ipv4.tcp_tw_recycle=1). Without this, the client is unlikely to be able to supply the necessary load due to slow timing-out of closed TCP sockets.

This test may require a system with 3 or more CPUs present (one for HLB and one for the client and back-end each), to provide the necessary traffic load on both ends of HLB. In all cases, no less than 2 CPUs are needed, with HLB occupying 100% of one CPU. Verify that HLB is on a dedicated CPU not used for the back-end or the test client (TBD: need to see that this is correctly handled by the AppLogic scheduler).

Use the 'http-load' utility to measure the request rate that HLB can handle. 'http-load' should be used with '-parallel X' option, to submit requests as fast as it can, up to X pending requests in parallel (use X=3..10 - when requesting a static page from the THTTPD back-ends, the response should be fast enough to saturate the CPU on which HLB is running with very few pending requests). The /index.html file is requested for this test (<100 bytes size).

performance - throughput test, bytes/second (small & large file: 100 bytes, 6K)

The setup is the same as for the req/s test, but this time a larger file is requested. HLB should provide no less than 100Mbit rate on the 6K file. The bytes/s rate is also recorded for the small file (from the req/s rate test above) and published in the rel. notes, but it is not verified against a minimum limit.

performance - concurrent sessions test - opening up to the max number of connections at the same time

NOTE: HLB must be given its max. memory for this test.

In this test, the http_load application is set to open a large number of simultaneous connections. This should complete without delay for up to the max # of connections supported by HLB.

Once the max request limit is reached, the test drops all connections and after a short delay repeats the procedure, to verify that HLB cleans up properly and can continue accepting new requests.

-- LeoKalev - 01 Mar 2006

 
Copyright © CA 2005-2011. All Rights Reserved.
%