AppLogic 2.4 Documentation The latest production release is AppLogic 3.0.30
L3LB Load Balancer - Test Plan
Preparation
Host platform
The automated tests described here for the L3LB appliance are designed to run on a Linux system and are verified to work on Fedora Core 5. To the best of my knowledge, they don't need anything Fedora-specific and should work on other Linux distributions. These tests must be launched from a remote host, that has an SSH access to the AppLogic controller. All performance tests are highly dependant on the resources available on the grid.
3rd-party tools
Several 3rd-party utilities are used for the tests. These tools are included in binary form in the test package and are deployed automatically. You need SSH installed and ssh keyfile for your AppLogic grid to perform the tests. Below is a description of these utilities and where the sources for them can be found.
thttpd
The test for L3LB uses two to eight instances of the THTTPD web server as back-ends for the load-balancer (GPL, from http://www.acme.com/software/thttpd/, version used in the tests: 2.25b http://www.acme.com/software/thttpd/thttpd-2.25b.tar.gz).
http_load
A simple load-generator tool, for performance testing (GPL, from http://www.acme.com/software/http_load/, version used in the tests: http://www.acme.com/software/http_load/http_load-12mar2006.tar.gz).
Tests Summary
All tests are completely automated, there is no need to manually configure the environment.
Configuration Tests
- Configuration propagation test: verify the HAProxy conf file is updated according to L3LB configuration.
- CPU resource test: verify proper updates to the HAProxy config file when more than one CPU is available.
- Memory resource test: verify that the maximum connection limit is properly calculated.
- Network subsystem configuration test: verify that network subsystem is configured properly by L3LB.
-
in terminal address. * outX server addresses. * ctl terminal address and port. * network subsystem tuning.
Operation tests
-
roundrobin mode of operation.
- Round-robin balancing: verify proper round-robin balancing.
-
source mode of operation. * Session persistence: verify proper client <-> server session persistence for different source IP addressses. * Failover: verify the transfer of the session to another server on server death.
-
udp mode of operation. * Session persistence: verify proper client <-> server session persistence for different source IP addressses. * Failover: verify the transfer of the session to another server on server death.
- healthcheck.
- Verify automatic disabling/enabling of failed backend server (unavailable server). * Verify automatic disabling/enabling of failed backend server (unavailable service). * Verify that only tcp connection is made for health check.
Endurance Tests
- Start-Stop test of the L3LB appliance: verifies that appliance can be started and shut down multiple times.
- Persistent load test: verifies that L3LB operates as designed under consistent load for long period of time.
Benchmarks
- Benchmark: verifies the maximum L3LB performance under various CPU/Memory/Bandwidth resources available.
- Throughput test, requests/second, 50-byte object.
- Throughput test, requests/second, 1Kb object.
- Throughput test, requests/second, 32Kb object.
- Throughput test, requests/second, 100Kb object.
- Throughput test, bytes/second, 8Kb object.
- Throughput test, bytes/second, 32Kb object.
- Throughput test, bytes/second, 100Kb object.
- Concurrent sessions test.
Statistics
- Statistics: verifies correctness of reported statistics through the
mon terminal.
Web control api tests (ctl terminal)
- /api/disable test: verifies disabling of output terminals specifying the output terminal name (/api/disable?channle=outX).
- /api/disable test: verifies disabling of output terminals specifying the backend server ip address (/api/disable?xx.xx.xx.xx).
- /api/enable test: verifies enabling of output terminals specifying the output terminal name (/api/enable?channle=outX).
- /api/enable test: verifies enabling of output terminals specifying the backend server ip address (/api/enable?xx.xx.xx.xx).
- /api/status test: verifies correct output terminal state reporting.
in terminal
- Statistics URL: verifies availability of statistics at the specified URL.
Running the Tests
To run the tests:
- Note the location of your ssh keyfile, or verify that the key is supplied by ssh-agent.
- Uncompress the tests archive file.
- Run the 'test.sh' script.
- Either create a new test application, or select an existing application. If you choose to create a new test application, the test suite will use a L3LB appliance, found in the /proto/ catalog.
- Run the required test(s), or run all tests in one batch.
Design
Structure
The test application is built from from the following:
- an AppLogic24? application (test harness) containing the component under test (L3LB) and several instances of the LUX5 appliance - a bare-bones Linux installation with only SSH running as an externally-accessible service.
- test utilities (http_load).
- test scripts for particular tests.
- control scripts that create, modify and start/stop test application and separate components.
The test script is fully automated and includes the ability to install the necessary test software on the LUX5 instances comprising the test harness.
Back-end Setup - Web space contents
The test suite configures each of the LUX5 instances that serve as back-end HTTP servers for the test with the following structure of files (shown here relative to the HTTP namespace root of the server):
/index.html | tiny web page (100 bytes). The contents of this page are different for each back-end server. |
/50.html | 50-byte test file for performance tests. |
/1024.html | 1kb test file for performance tests. |
/8192.html | 8Kb test file for performance tests. |
/32768.html | 32Kb test file for performance tests. |
/102400.html | 100Kb test file for performance tests. |
| =/cgi/* = | cgi script directory |
Test Details
Configuration Tests
These tests verify propagation of properties from application boundary to the appliance and proper modification of the haproxy.cfg file.
Configuration propagation test
This test modifies various properties of L3LB and verifies that changes are properly propagated and that the haproxy.cfg file is modified appropriately.
CPU resource test
This test verifies proper detection of multiple CPUs and proper updates to the haproxy.cfg file.
Memory resource test
This test verifies proper detection of memory available for the L3LB and updates of the maximum connection limit in the haproxy.cfg file and haproxy startup options.
Network subsystem configuration test
Verifies proper detection of in, ctl, out1 - out8 addresses and ports and appropriate updates of the haproxy.cfg file. It also verified that the necessary tweaks of the networking subsystem are performed.
Operation tests
These tests verify proper operation of L3LB in different modes.
roundrobin mode of operation.
Verifies L3LB operation in roundrobin mode.
Round-robin balancing
This test submits multiple requests for 'index.html' (which is different on each back-end) and counts the number of responses from each back-end to verify even distribution of requests.
Failover
This test verifies proper session transfer to another server upon server unavailability:
Failover test - shutdown http server on a back-end server (get connection-refused errors)
The test described above is repeated, but one of the back-ends has its HTTP server stopped, while other back-ends are kept alive. This will cause the network stack on the back-end to reply with RST to connection attempts (seen as a "connection refused" error by the load-balancer). L3LB should re-distribute the traffic to the remaining back-ends.
Failover test - kill networking on a back-end server
Same as above, but the network stack on the back-end is disabled completely, resulting in time-out errors seen by L3LB. The traffic should be re-balanced to the remaining back-ends after a failed healthcheck. [the network on the back-end is disabled using 'ifdown'].
Failover test - recovery on restored back-end server
While a round-robin test is running with one of the back-ends downed, the back-end is brought back into operation. L3LB should discover this within several seconds and start directing traffic to it as well. The test must verify that once the first request goes through to the restored server, the balance between all the back-ends is even.
source mode of operation.
Verifies L3LB operation in source mode.
Session persistence
This test submits multiple requests for 'index.html' file, and uses a cookie to verify session 'stickness' to the backend server.
Failover
This test verifies proper session transfer to another server upon server unavailability:
Failover test - shutdown http server on a back-end server (get connection-refused errors)
The session-persistense test described above is repeated, but one of the back-ends has its HTTP server stopped, while other back-end is kept alive. This will cause the network stack on the back-end to reply with RST to connection attempts (seen as "connection refused" error by the load-balancer). L3LB should re-distribute traffic to the remaining back-ends.
Failover test - kill networking on a back-end server
Same as above, but the network stack on the back-end is disabled completely, resulting in time-out errors seen by L3LB. The traffic should be re-balanced to the remaining back-ends after a failed healthcheck. [the network on the back-end is disabled using 'ifdown'].
Failover test - recovery on restored back-end server
While a round-robin test is running with one of the back-ends downed, the back-end is brought back into operation. L3LB should discover this within several seconds and start directing traffic to it as well. The test must verify that once the first request goes through to the restored server, the balance between all the back-ends is even.
udp mode of operation.
Verifies L3LB operation in udp mode.
Session persistence
This test sends multiple datagrams to configured port from different clients and verifies session 'stickness' to the backend server.
Failover
This test verifies proper session transfer to another server upon server unavailability:
Failover test - kill networking on a back-end server
Same as above, but the network stack on the back-end is disabled completely, resulting in time-out errors seen by L3LB. The traffic should be re-balanced to the remaining back-ends after a failed healthcheck. [the network on the back-end is disabled using 'ifdown'].
Failover test - recovery on restored back-end server
While a round-robin test is running with one of the back-ends downed, the back-end is brought back into operation. L3LB should discover this within several seconds and start directing traffic to it as well. The test must verify that once the first request goes through to the restored server, the balance between all the back-ends is even.
Health check.
Verifies operation of health check. This test verifies proper detection of failed and/or recovered backend server.
Endurance Tests
These tests verify proper operation of L3LB under load and in extreme conditions.
Start-Stop test
Verifies that L3LB may be started and stopped multiple times.
Persistent load test
Verifies successful L3LB operation under various load for prolonged time. Objects of different a sizes are requested at a high rates.
Benchmarks
These tests verify performance of L3LB and are highly dependent of resource availability on the grid and current load. It is advised to stop all other application that are running on the grid for the time of these tests.
Throughput tests
These tests verify maximum achievable requests/second and bytes/second rates using test objects of a different sizes.
Concurrent sessions test
This test tests maximum achievable concurrent sessions count. It opens up to the maximum number of connections at the same time.
Statistics
Verifies correctness of reported statistics through mon terminal.
Web control api tests
These tests use the web control api to disable and enable separate output terminals.
/api/disable tests
These tests use the '/api/disable?' call to disable the selected output terminal using it's name (?channel=outX) or backend server address (?xx.xx.xx.xx). Active terminals should be disabled, already disabled terminals should be left disabled, and an error returned, non-existent terminals should be ignored and an error returned.
/api/enable tests
These tests use the '/api/enable?' call to enable the selected output terminal using it's name (?channel=outX) or backend server address (?xx.xx.xx.xx). Disabled but connected terminals should be enabled, already enabled terminals should be left enabled, and an error returned, non-existent terminals should be ignored and an error returned.
/api/status test
A status page should be returned by L3LB, with the current satus of all connected terminals (up/down).
Copyright © CA 2005-2011. All Rights Reserved.
AppLogic24 Web