BCK - Backup Enabler - Implementation

Base Class

BCK is an assembly containing two appliances.

• PS8 from the system catalog is used unmodified.
• BCKCTL is branched from the WEB5 base class.

Class Volumes

The volume configuration for BCK is determined by the base class images from which it is created.

See PS8 - Scalable Port Switch? for relevant data on the port switch.

Volumes for release 1.0 of BCKCTL include:

• boot volume, 160MB size, mounted as "/", writeable, instantiable.
• usr volume,400MB size, mounted as "/usr", read-only, common.
• content volume 5 MB size, mounted as "/mnt/content", read-only, common
• config volume, mounted as "/mnt/config", writeable, placeholder

The boot and usr volumes are unmodified from the WEB5 appliance except for /etc/fstab which additionally mounts the config volume. All software specific to BCKCTL is installed on the content volume and requires apache, PHP, ssh and bash in order to function.

Packages

• Main: 3tapp-bck-1.0.0-1.noarch.rpm. This contains the scripts and web interface which perform the basic operations of BCK.

Contents of the release directory:

Exported appliance images:

• BCK-1.0.0exp.tar.bz2
• BCKCTL-1.0.0exp.tar.bz2

Exported BackupHelper application:

• BackupHelper-1.0.0exp.tar.bz2

"Generator" RPMs: install these in the order listed on the content volume of BCKCTL to re-create the appliance:

• 3tapp-bck-1.0.0-1.noarch.rpm

Source RPMs:

• None. All sources are PHP or Javascript and are available in the "Generator" RPM.

Tests (see the test plan CatDynBckTp and the release notes CatDynBckRelNotes):

• bck-tst-app.tar.bz2 - test application

Re-creating the Appliance

To recreate the BCK and BCKCTL appliances:

• Drag a blank class definition assembly onto the canvas. Set its interfaces, volumes and properties according to the BCK Data Sheet
• Edit the interior of the assembly and connect a PS8 and WEB5 appliance according to diagram at the top of this page.
• Set the PS8 properties as follows:
  • out1_protocol = tcp
  • out1_in_port = gui_port_no (redirected to assembly)
  • out1_out_port = 80
• Branch the WEB5 appliance and name the new class BCKCTL. Name the instance in the assembly ctl
• Edit the BCKCTL class:
  • Delete the content volume. Add a new content volume on /dev/hda3 Common, read-only, shared with a size of 5M and filesystem ext3.
  • Add a new volume config on /dev/hda4 Placeholder, mandatory
  • Add properties according to the BCK Data Sheet
  • Set max resources equal to min resources.
• Edit the BCKCTL instance settings:
  • Re-direct the volume config to the assembly volume config.
  • Re-direct the new properties to the assembly properties.
• Edit /etc/fstab on BCKCTL to mount the config volume with defaults.
• Install the rpm 3tapp-bck-1.0.0-1.noarch.rpm on BCKCTL (temporarily change the content volume to writeable).

Third Party Software

BCK makes use of third party software comprised of PHP files installed and configured as outlined below. These files are included in the "Generator" RPM and are all installed in the phps3tk directory.

• phps3tk includes a "library" libs3.php which exposes an API for using S3. phps3tk is available at http://edoceo.com/dl/phps3tk.tgz and is released under the GNU GPL Version 3. This library was modified to use the curl progress meter during get operations by including, as indicated in the source, this line of code:

           if (! $this->_s3_stats['ls']) curl_setopt($ch, CURLOPT_NOPROGRESS, false);

• phps3tk requires HMAC.php from the PEAR Crypt_HMAC package available at http://download.pear.php.net/package/Crypt_HMAC-1.0.1.tgz and released under v3 of the PHP license available here.
• The PHP code for creating S3 buckets relies on four additional PEAR packages: Net_Socket, Net_URL, PEAR, and HTTP_Request.
  • Socket.php is released under v2 of the PHP license available here. The package is available at http://download.pear.php.net/package/Net_Socket-1.0.8.tgz
  • URL.php is released under the BSD license availabe here. The package is available at http://download.pear.php.net/package/Net_URL2-0.1.0.tgz
  • PEAR.php is released under v3 of the PHP license. The package is available at http://download.pear.php.net/package/PEAR-1.7.1.tgz
  • Request.php is released under the BSD license. The package is available at http://download.pear.php.net/package/HTTP_Request-1.4.2.tgz

Theory of Operation

BCK enables the containing application to backup and restore itself to a remote service. These operation can be initiated through a web interface and work in conjunction with the reference application BackupHelper.

Basic operations performed by BCK are:

• manually initiated backup
• manually initiated restore
• scheduled backup
• cancel of backup in progress
• cancel of restore in progress
• delete existing backup
• monitor operation in progress

Start-up Script on BCK

• Log activity to file (for GUI)
• Create a crontab for user root according to the schedule properties
• Get some info:
  • check for a lock file indicating:
    • backup in progress - a normal result during backup
    • restore in progress - an abnormal result - the application stopped and started during restore
    • cancel in progress - an abnormal result - the application stopped and started during cancel (this will have aborted the cancel as cancel is done by BCK, unlike backup and restore which are done by BackupHelper)
  • check for an existing BackupHelper_app_name on the grid; wget it's status (working, success, failure, nonresponsive) and operation (backup, restore, nonresponsive)
• If no lock file and no BackupHelper - normal condition; exit.
• If no lock file and BackupHelper exists - abnormal condition. Log msg to Applogic dashboard. exit. BackupHelper remains. This needs a human. A backup or restore attempt during this condition will not proceed. This could happen if someone manually provisions BackupHelper_app_name and then re-starts the app.
• If lock file but no BackupHelper - abnormal condition. Log msg. Delete lock file. exit. This could happen if someone manually stops the application during backup, manually destroys BackupHelper, and then re-starts the app.
• If both lock file and BackupHelper exist:
  • If lock file indicates cancel in progress, delete lock file. Cancel is cancelled by abnormal application re-start. BackupHelper remains. Log msg. exit.
  • Polling loop, wget status from BackupHelper every 5 seconds until:
    • BackupHelper status is success or failure:
      • Stop and destroy BackupHelper. On fail to destroy log msg, as it's left running.
      • Delete lock file
      • exit
    • BackupHelper nonresponsive for some time then timeout: log msg and delete lock file. BackupHelper remains. exit.
    • If lock file indicates backup and BackupHelper indicates restore, or vice versa, sanity failure: log msg and delete lock file. BackupHelper remains. exit.

Backup Script on BCK (called by crond or through GUI)

• Log activity to file (for GUI)
• Get some info:
  • check for a lock file indicating backup, restore or cancel in progress
  • check for an existing BackupHelper_app_name on the grid
• If there is no access to remote service, log msg, exit.
• If lock file or BackupHelper exists, abort backup, log msg, exit. Note: lock file from manual backup could exist prior to provisioning of BackupHelper; BackupHelper could exist without lock file from a variety of failure conditions in the start-up script.
• Compare size of application volumes against _impex. If insufficient space, log msg, exit.
• Create lock file. On fail log msg, exit.
• Provision an instance of BackupHelper with correct volume size and env property to perform backup. On fail log msg; delete lock file; exit.

BackupHelper During Backup (Start-up Script)

• Log activity to file (for GUI)
• Provide status: working, success, failure
• Stop the app. On fail retry once. On fail again, log msg, set status to failure, exit (cleanup is done by BCK when status is set to failure - a complete state)
• Export application to _impex. On fail, log msg, set status to failure, exit
• Re-start the app. On fail retry. On fail, log msg. Continue the backup.
• Copy the exported app to an application volume. On fail, log msg, set status to failure, exit.
• Delete the _impex files. On fail log msg.
• Create tarball of app export files. On fail log msg, set status to failure, exit
• Delete app export files.
• Split tarball into 1GB pieces. On fail, log msg, set status to failure, exit
• Store split files on remove service. On fail, retry. On fail, log msg, set status to failure, exit
• Status success. Log msg. exit.

Manual Backup (initiatied through GUI)

• Get some info:
  • check for a lock file indicating backup, restore or cancel in progress
  • check for an existing BackupHelper_app_name on the grid
  • check for grid access
  • check for remote service access
• If lock file or BackupHelper exists, or if access fails, prevent backup.
• Compare size of application volumes against _impex. If insufficient space, prevent backup
• Exec backup script

BCK GUI

• Do these things first:
  • Authentication
  • grid and remote service access verification
  • Check for lock file - if it exists, go to the monitor.
  • Check existing BackupHelper
  • Determine _impex size and app size
• If BackupHelper exists or access fails or _impex has insufficient space, prevent backup and restore
• If BackupHelper exists or access fails prevent delete
• User selection from: manual backup, manual restore, manually delete backups.
• If lock file exists, or the user manually initiates a backup or restore, then the GUI displays two monitors:
  • BCK: provisioning BackupHelper, stopping and destroying BackupHelper (during cancel or upon completion)
  • BackupHelper: all logged operations
• During backup and restore a cancel button is available when both BCK and BackupHelper are running

Manual Restore (GUI)

• List backups on remote service and their sizes.
• Prevent backups larger then _impex from being restored
• Provide for user selection of restored app name. Verify this name is not in use.
• Exec restore script

Restore Script on BCK

• Log activity to file (for GUI)
• Get some info:
  • check for a lock file indicating backup, restore or cancel in progress
  • check for an existing BackupHelper_app_name on the grid
• If there is no access to remote service, log msg, exit.
• If lock file or BackupHelper exists, log msg. exit.
• Create lock file. On fail log msg; exit.
• Provision an instance of BackupHelper with correct volume size and env property. On fail log msg; exit.
• Polling loop, wget status from BackupHelper every 5 seconds until:
  • BackupHelper status is success or failure:
    • Stop and destroy BackupHelper. On fail log msg, as it's left running.
    • Delete lock file
    • exit
  • BackupHelper nonresponsive for some time timeout. Log msg and delete lock file. BackupHelper remains. exit.
  • If lock file indicates backup and BackupHelper indicates restore, or vice versa, sanity failure: log msg and delete lock file. BackupHelper remains. exit.

BackupHelper During Restore (Start-up Script)

• Log activity to file (for GUI)
• Provide status: working, success, failure
• Copy the backup from the remote service to the application volume. On fail retry. On fail log msg; set status to failure; exit. BCK will perform cleanup.
• Join split files to create tarball. On fail log msg; set status to failure; exit
• Delete original pieces.
• Un-tar. On fail log msg; set status to failure; exit
• Copy files from application volume to _impex. On fail log msg; set status to failure; exit
• Import application using new name. On fail delete the _impex files; log msg; set status to failure; exit
• Delete the _impex files. On fail log msg.
• Status success. Log msg. exit.

Manual Cancel

• Log activity to file (for GUI)
• Check for a lock file indicating backup, restore or cancel in progress
• Check for an existing BackupHelper_app_name on the grid
• If no BackupHelper, or no lock file or lock file indicates cancel in progress, then no action, exit.
• If lock file indicates backup or restore and BackupHelper exists:
  • Get info from BackupHelper indicating temp files in use on _impex
  • stop and destroy BackupHelper. On fail log msg.
  • delete any temp files on _impex.
  • delete lock file
  • exit

Configuration

Configuration is performed by the start-up script which uses properties in /etc/applogic.sh.

Log Files

Logs specific to a given operation are created in the /3tera directory of the config volume and are viewed by the web monitor during backup and restore. These files are overwritten for each operation.

-- StephenQ - 26 Oct 2007