SLA - Controller - Implementation

Base Class

SLA is an assembly containing two appliances.

• PS8 from the system catalog is used unmodified.
• SLACTL is branched from the WEB5 base class.

Class Volumes

The volume configuration for SLA is determined by the base class images from which it is created.

See PS8 - Scalable Port Switch? for relevant data on the port switch.

Volumes for release 1.0 of SLACTL include:

• boot volume, 160MB size, mounted as "/", writeable, instantiable.
• usr volume, 500MB size, mounted as "/usr", read-only, common.
• config volume, mounted as "/mnt/config", writeable, placeholder.

The boot volume is unmodified from the WEB5 appliance. The usr volume is enlarged to 500MB. The content volume is removed and /etc/fstab is modified to additionally mount the config volume. All software specific to SLACTL is installed in /usr/SLA and requires apache, PHP, ssh and bash in order to function.

Packages

• Main: 3tapp-sla-1.0.0-1.noarch.rpm. This contains the scripts and web interface which perform the basic operations of SLA. All files are installed in /usr/SLA.

Contents of the release directory:

Exported appliance images:

• SLA-1.0.0exp.tar.bz2
• SLACTL-1.0.0exp.tar.bz2

"Generator" RPMs: install this on the /usr volume of SLACTL to re-create the appliance:

• 3tapp-sla-1.0.0-1.noarch.rpm

Source RPMs:

• None. All sources are PHP or Javascript and are available in the "Generator" RPM.

Appliance related customization files:

• 3tapp-sla_custom-1.0.0.tar.bz2

Tests (see the test plan CatDynSlaTp and the release notes CatDynSlaRelNotes):

• sla-tst-app.tar.bz2 the test application

Re-creating the Appliance

To recreate the SLA and SLACTL appliances:

• Drag a blank class definition assembly onto the canvas. Set its interfaces, volumes and properties according to the SLA Data Sheet
• Edit the interior of the assembly and insert a PS8 and WEB5 appliance.
• Set the PS8 properties as follows:
  • out1_protocol = tcp
  • out1_in_port = port_no (redirected to assembly)
  • out1_out_port = 80
• Set the PS8 maximum resources equal to the default resources.
• Branch the WEB5 appliance and name the new class SLACTL. Name the instance in the assembly ctl
• Edit the SLACTL class:
  • Add the mtr terminal limited to the http protocol.
  • Rename the content volume to config and mark it rw, not shared, required
  • Temporarily mark the usr volume as rw, not shared
  • Add properties according to theta SLA Data Sheet (logs_base_dir is already present)
  • Set min, max and default resources to 0.10 CPU, 192M memory, 1M bandwidth
• Edit the SLACTL instance settings:
  • Re-direct the volume config to the assembly volume config
  • Re-direct the new properties to the assembly properties
• Connect the appliances according to the diagram at the top of the page
• Set mandatory properties, connect mandatory terminals, re-size the usr volume to 500 MB and start in debug mode
• Install the rpm 3tapp-sla-1.0.0-1.noarch.rpm on SLACTL
• Install the appliance related customization files
• Set the config volume to rw, shared and move the singleton into the dynamic catalog

Installation of Appliance Related Customization Files

These appliance related scripts have been added to, or modified from, those on the original WEB5 base class:

• /appliance/appliance.sh modified to call sla_setup.sh
• /appliance/httpd_prerun.sh modified to link /mnt/content to /usr/SLA/html
• /appliance/sla_setup.sh created to handle directories, mounts and the log terminal on boot
• /etc/ccad.conf custom counter definitions
• /etc/fstab modified to mount the config volume and not any content volume

To install these scripts, copy the file 3tapp-sla_custom-1.0.0.tar.bz2 to / on the appliance, and:

           cd /
           tar -jxf 3tapp-sla_custom-1.0.0.tar.bz2
           rm 3tapp-sla_custom-1.0.0.tar.bz2

Third Party Software

None in addition to the base class images.

Theory of Operation

Basic operations performed by SLA are:

• Collect data from MON
• Enforce the policy
• Expose a GUI allowing users to:
  • edit the policy
  • stop or start enforcement of the policy
  • manually initiate the stop or start of an appliance within the collection

Policy

• The policy is represented using a text file in XML format on the config volume. The first release of SLA supports a single policy which reacts to the trailing average of a single counter. The XML format is used because it is easily extensible to multiple policies which react to multiple counters.
• Each policy records the following information:
  • Whether the policy is stopped or running (enforceable or not)
  • The collection of appliances which can be started and stopped.
  • For each of start and stop, the entity and counter examined. The GUI constrains these to be the same counter for both stop and start.
  • For the counter examined, the conditional evaluation to be performed. Each conditional evaluation comprises:
    • trigger_operator either < or >
    • trigger_value a number
    • trigger_period number of seconds. This is the period over which the trailing average is calculated
• Example policy:

                    <sla_root>
                       <sla name="user" desc="description" status="stopped">
                          <app name="app1" />
                          <app name="app2" />
                          <app name="app3" />
                          <app name="app4" />
                          <start>
                             <condition ent="ent1" cnt="cnt1" op="G" val="2.8" period="180" />
                          </start>
                          <stop>
                             <condition ent="ent1" cnt="cnt1" op="L" val="0.5" period="180" />
                          </stop>
                       </sla>
                    </sla_root>

sla_daemon

• sla_daemon performs the following operations:
  • Every collection_period get the counter values for all running appliances in the collection. Calculate the current current average and current trailing average.
  • Policy conditional evaluation. Each condition relates to a single counter and is defined by trigger_operator, trigger_value and trigger_period. A counter relates to a single operation stop or start over a collection of appliances. Evaluations are performed by:
    • Getting the the collected value of the counter for each running appliance in the collection for each collection_period during the most recent trigger_period. These values are averaged to form trailing_average.
    • if trailing_average trigger_operator trigger_value then condition is true
  • Every enforcer_period, evaluate the policy if it is running:
    • Evaluate the start condition. If the condition is true and at least one appliance in the collection is stopped, then an appliance in the collection is started. If all appliances are already started, a message is logged to the AppLogic dashboard.
    • Evaluate the stop condition. If the condition is true and at least two appliances in the collection are running, then an appliance in the collection is stopped.
    • If stop or start occurs, no further policy evaluations are performed until the stop or start operation is complete and another 6 minutes have passed.
    • Because a stop or start can also be initiated through the GUI, a lock file is used.
  • Counter dump to file. Trailing average dump to file. These are needed for the GUI (see below).
  • Logging of operations (including start/stop of appliances).

Logs

• Stop or start of an appliance:
  • date & time
  • operation start or stop
  • the entity and counter name and trailing_average which triggered the operation
• Failure to stop or start (3 total tries)
• Re-starting sla_daemon
• Failure to re-start sla_daemon

Cron

• The root crontab, created on start, periodically calls a script which:
  • Checks sla_daemon is running. If not, then sla_daemon is re-started. On fail log message to the AppLogic Dashboard.
  • Rotates the logs based on size. 256 KB each, keep 4.

On-start Script

• Start the daemon.
• Create the root crontab.

GUI

• The GUI performs the follow operations upon initial access:
  • authentication
  • verify access to the grid controller
  • verify daemon is running.
  • verify MON is responsive
• Aside from authentication, there is a single screen which contains two areas, top and bottom:
  • The top area allows for policy definition
  • The bottom area includes a graph which is generated using GD and includes:
    • horizontal scale of 24 hrs, vertical scale within scope of counter variation in that time.
    • a graph for the counter average
    • a graph for the counter trailing average
    • a graph for the number of running appliances in the group
    • a vertical bar at the length of the trailing average
• Both the user area and the graph are updated using AJAX.

Configuration

Configuration is performed by scripts in /appliance as well as by the start-up script /var/www/html/.rc.local, all of which use properties in /etc/applogic.sh.

Log Files

The log of SLA operations is created either in the configured logs_base_dir of the share accessed through the log terminal, or in /mnt/config/3tera/log/. The log is rotated and three historical copies are kept at 256KB each.

-- StephenQ - 13 Dec 2007