Appliance Kit User Manual

Overview

APK is a package of tools and runtime for creating AppLogic appliances out of OS installations. It contains scripts and utilities that help implement the appliance boundary, as well as a set of tools to assist appliance developers in preparing appliances.

Description

APK includes:

• user manual (this document):
  • appliance boundary definition (what makes a VM an AppLogic appliance)
  • appliance cookbook (how to build and customize appliances)

• installation guide (separate documents, one per supported target OS)
  • Installation Guide for RedHat and CentOS Linux Distributions
  • Installation guide for Ubuntu Linux Distributions
  • Installation guide for Open Solaris and Solaris 10 Distributions

• binary archive (one per supported target), including an install script. The binary archives can be found within the AppLogic release folder on the ALDO server.

• The APK installation supports:
  • creating appliances from physical server installs or from VMware/Xen images,
  • upgrading old-style (volfix) AppLogic appliances to APK
  • upgrading new-style (APK) appliances to a newer version of APK

APK can be installed either on a boot volume image mounted into a subdirectory, or on a live appliance. The following software is installed by APK:

• All platforms:
  • initialization script - sets up the network as necessary for the OS to operate as an appliance and starts the AppLogic-specific services (VMA and CCA).
  • Virtual Machine Agent (VMA), including a command to send events to the controller
  • Counter Collection Agent (CCA). CCA is started only if the appliance has a configured 'mon' terminal and the terminal is connected.
• for Linux, a pre-compiled para-virutalized kernel is also provided as a separate archive.

Appliance Boundary

This chapter explains in detail what is an AppLogic appliance.

Overview / General

An AppLogic virtual appliance (or simply, appliance, in the text below) is an instance of an OS installation or another self-contained executable entity that can run as an i386 (or x86_64) virtual machine and has the following characteristics:

• the executable code is contained on one or more volume images, formatted either as a PC x86 bootable hard disk or as single disk partitions;
• configures itself (network, properties, etc) using the dhcp-based protocol described below;
• upon startup, reports success/failure to the AppLogic grid, using a utility provided for this purpose by 3tera;
• if used as part of an AppLogic application, communicates with other appliances using peer-to-peer IPv4 connections - the AppLogic equivalent of a direct link between two Ethernet ports;
• all configurable settings of the appliance are in the form of properties - that is, named parameters, each one with a defined type chosen from the types supported by AppLogic and optionally a defined range of valid values. The property types currently supported by AppLogic are: string, integer, IP address.

(Note: AppLogic also supports a 'legacy' appliance, which is not configured over the network and requires its volumes to be modified off line prior to start. This type of appliance isn't covered by the present document).

Exterior Boundary

This section defines how the appliance looks to the "outside world" - that is, to the grid controller and to other appliances on the grid. The "exterior boundary" is implemented mostly by APK, the appliance designer does not need to do much beyond installing APK in order to make the OS behave like an AppLogic appliance.

Boot File Locations (PVM appliances only)

A para-virtualized (PVM) appliance is one that has been configured with a virtualization-aware OS kernel, and uses the XEN hypervisor APIs for access to all hardware (console, memory, disk, network).

A para-virtualized appliance must have its initial executable image in the form of a statically-linked ELF file, optionally compressed with gzip. The initial executable is usually the OS kernel. The appliance may also use one or more additional binaries loaded into memory at boot time (e.g., an initial ramdisk).

The locations of the initial executable and any other optional binaries must be defined in a file stored on the appliance's boot volume, and named either /boot/grub/grub.conf or /boot/grub/menu.lst. The file format is that of the GRUB configuration file (e.g., see man 8 grub). The files referred to in the grub configuration file are relative to the boot disk's file system and must be on the boot disk image itself. Note that the GRUB binary itself need not be present - it is not used to boot a para-virtualized appliance.

Booting para-virtualized appliances is limited to the same file system formats as the ones supported by GRUB (I.e., ext2/3, fat, ufs, reiserfs, minix, xfs, jfs).

Network Configuration

An appliance uses its last network interface (the one appearing last in a network device discovery scan) as the service interface, used to configure itself and to communicate with the AppLogic grid. This interface is referred to in AppLogic documentation as the 'default' interface.

Upon startup, an appliance configures itself as follows:

• set up the default interface and request DHCP configuration for it
• the DHCP response will contain the network parameters for the interface, plus a file name (in the filename DHCP option).
• retrieve the following files using HTTP requests to port 8080 sent to the address of the server that responded to the DHCP request ($filename is the name received in the DHCP response):
  $filename.net
  $filename.desc
  $filename.ext
• use the data contained in the retrieved files to configure itself and begin normal operation. Note that except for the DHCP-configured default interface and the local loopback interface, the appliance's network is set up entirely according to data provided in the $filename.net configuration file (and not using any OS-specific network configuration data stored on the appliance itself).

The format of the configuration files that the appliance receives from the AppLogic grid is described in the Appendix. Note that in most cases the appliance designer need not be concerned with the format of these files, as the APK initialization scripts take care to interpret them and configure the appliance accordingly. For appliances that have configurable properties, APK provides their values in various easy to use 'cooked' formats which can be read by the appliance-specific code. Also, in many cases it is possible to instruct APK to modify appliance-specific configuration files on startup and insert the values of the appliance's properties into them at the right places. See the Interior Boundary section further below.

VM Agent and Boot Completion Report

After it has configured its network, the appliance starts the VM Agent (a binary executable provided by APK), which opens a service connection to the AppLogic infrastructure, using a 3tera proprietary binary protocol.

When the boot is completed, the appliance uses the service connection to report success or failure. APK provides the necessary tools to the appliance designer to add customized code to check if the appliance started OK and report status to the grid. If failure is reported, or if the appliance fails to contact the grid altogether, the appliance start is considered un-successful and the virtual machine (and possibly the entire application in which this appliance participates) is shut down.

Interior Boundary

This section describes the AppLogic-specific services that APK provides for use within the appliance OS itself.

An appliance is not required to use any of these services. Only the "exterior" boundary (implemented by APK and described above) is what defines an appliance as far as AppLogic is concerned. The ";interior boundary" services are provided to help adapt an existing generic service (e.g., firewall, a mail server, or Web server program) for use as a self-contained virtual appliance running in AppLogic.

Boot Status Check/Report

By default, when the appliance boot process completes, the APK initialization code sends a "boot completed" notification using the service connection established by the VM Agent.

This is used by the grid as a signal that the appliance started successfully.

The appliance can be equipped with an optional "boot check" script, which is run on completion of the OS startup and can check (in some appliance-specific way) whether the appliance is initialized correctly and is ready to operate. For example, if the appliance is an HTTP server, the script might attempt to make a request to the HTTP port on the local host to verify that the HTTP daemon is operational.

The boot check script is a shell script named applogic_appliance (stored in an OS-specific location - see the install guides), ran as "include files" (e.g., with the "." command for a Posix shell). It is expected to return a success / failure status and optionally print an error message to the standard error stream (stderr).

Log Messages

Code on the appliance may report info, warning or error messages to be posted in the grid system log.

APK provides the 'vme' command for this purpose. It should be invoked with the following command line syntax.

   vme id=log msg='text'

Note that if the message text contains spaces or other meta-characters, it has to be quoted appropriately.

Network Access - Terminals

Communication between appliances in AppLogic is over peer-to-peer IP connections. The endpoints of these connections are referred to as terminals. This is somewhat different from the "usual" setup of a multi-device network application, where most devices may be connected to a common bus (e.g., an Ethernet switch) and each appliance may connect (or mis-connect) to any other appliance on the same bus.

In contrast, AppLogic uses a separate "virtual wire" for each connection (equivalent to a port-to-port Ethernet cable between two network devices) and ensures that traffic goes only along those wires. This is made possible by the fact that the AppLogic virtual appliances are not limited by a fixed number of physical connection ports and can therefore dedicate a separate "virtual terminal" for each connection.

There are two logical types of terminals:

• input terminal : a terminal on which the appliance provides a specific service (I.e, acts as a server).
• output terminal : a terminal that an appliance uses to request a service from another appliance (I.e, acts as a client).

An input terminal can have any number of output terminals connected to it. An output terminal can be connected to a single input terminal. Either kind of terminal may remain unconnected. An application designer may specify that an output terminal is "mandatory", i.e., the appliance requires the service provided on it and will not work without it being connected.

APK automatically configures the network settings for the virtual network interfaces uses as terminals. It makes the terminals visible to the appliance code by name (providing the name resolution and the necessary IP routes), as follows:

• a connected output's name resolves to the address of the appliance that is connected on the other side, that is, the address to which the terminal name resolves can be used directly to request a connection. For example, if the appliance has an output terminal named 'web', connectible to an http server, one might do the equivalent of a command like this: wget http://web/some-page.html.
• an input terminal's name resolves to the IP address of the terminal itself, thus allowing one to use the name when configuring a server, e.g., if the appliance has an input terminal named 'in' that is supposed to respond to HTTP requests, one might add this directive to the Apache configuration file: Listen in:80.

An output terminal that is not connected will have its name resolve to an invalid IP address: 0.255.255.255. An appliance that is designed to have optional outputs (which may be left unconnected) should recognize this, or be prepared to handle gracefully the "invalid address" error on an attempt to connect.

Network Access - Gateway Terminal

A gateway terminal is a special case of an output terminal (as described above). An appliance can have at most one gateway terminal. It is configured as regular output terminal in that the name resolves to the remote peer. In addition:

• the default route is set to be via the peer address to which the terminal is connected, so that connections to any address other than those belonging to regular terminals or those used by the 'default' interface (the AppLogic internal service network) are routed via the gateway output.
• the DNS nameserver is set to the gateway terminal's peer address

Thus, an appliance with a gateway terminal can operate like a device connected to a "regular" LAN, intranet or WAN, within the limits of whatever name resolution and routing is provided by the "gateway" appliance connected to the terminal. (AppLogic provides ready-made appliances that act as gateways, including one that provides un-restricted outgoing access to the Internet).

Properties

AppLogic provides configuration for appliances in the form of properties (named configuration strings). An appliance receives its configured properties on startup in the configuration files retrieved from the DHCP server (see the Exterior Boundary section above).

APK is responsible to retrieve the configuration files and it provides the appliance properties in several formats. Any or all of these formats can be used by the appliance-specific code to read its configuration data. In addition, APK can be instructed (during installation) to use the received property values to update one or more files on the appliance (see the next sub-section - Automatic Configuration File Update).

APK provides the property values in the following formats. The file names for each format are provided in parentheses. The location of the files in the appliance file system is OS-specific (see the install guide for each OS).

• raw (appliance.conf) : a text file, containing one property value per line. Each line starts with the property name, followed by a single space character and the property value extending to the end of the line. The values are provided verbatim, with no meta-character quoting of any kind. The "raw" format is suitable for reading with the Posix shell's 'read' command, or with read() or scanf() from binary code.
• Posix shell (appliance.sh) : a text file, suitable for being included by a Bourne or Posix compatible shell script, with the "." command. Each property is defined as an exported shell variable, prefixed with the underscore character (to avoid name conflicts with common shell variables).
• C shell (appliance.csh) : a text file, suitable for being included by a a C-shell script. Each property is defined as an exported shell variable, prefixed with the underscore character.

Automatic Configuration File Update

In many cases, the appliance properties correspond directly to strings that need to be inserted into an appliance-specific configuration file. In such cases, rather than writing code to read one of the APK-provided property files (described above) and generate the appropriate configuration files on each boot, it is possible to instruct APK to apply the property values directly to the configuration file(s). This can be done in all cases where the following conditions are met:

• the appliance has a writable disk volume (i.e., this method is not suitable for an appliance whose volumes are all read-only)
• the property value needs to be simply inserted into the configuration file
• the configuration file is a text file
• the configuration file format allows for adding comments to it, and the program(s) using it either do not modify the file, or it re-write it keeping the comments intact.

To set up APK to do automatic config file updates on boot:

• the configuration file(s) are "instrumented" with special comments that specify exactly where the property values are to be inserted. The instrumentation syntax is can be found here: http://doc.3tera.net/AppLogic2/AdvADLPropertyMarkup.html
• The list of files to be updated is configured in the APK_CONFIG_FILES setting in the applogic_init config file.

Appliance Cookbook

This chapter has step-by-step recipes for creating several kinds of appliances and is intended to serve as a tutorial and as a source of examples to be emulated when building new appliances.

The following recipes are included:

• building a "blank" appliance template (basic OS installation, with no specific functionality) from scratch. This is primarily useful for creating a few non-specialized templates which can later be re-used and configured for a specific purpose. Since a few such ready-made basic templates are provided in the AppLogic system catalog, this recipe is intended mostly for use when porting APK to a new OS type for which there is no ready appliance in the catalog.
• Creating a "virtual private server" (VPS) appliance - a virtual machine that is not intended to interact with other appliances and be part of a multi-tier application, but rather serve as the equivalent of a private hosted server, with a single Internet-accessible network interface.
• Building a fast-boot mini-appliance, based entirely on a read-only volume. This use case is suitable for creating various "filter" type virtual devices that do not store persistently any data, like WAN routers, firewalls, load balancers, etc. This recipe is specific to the Linux OS, which is generally well-suited to operate in an embedded environment with a read-only boot volume.

Blank Appliance

OS Install

The first step of this recipe is to prepare a vanilla OS installation. This has to be done outside of AppLogic, because (for now), AppLogic does not provide access to the system console of a virtual machine under its control.

There are two options for the OS install platform:

1. use a real server and install the OS on a blank hard disk
2. use a XEN virtual machine in hardware-emulation mode, to run the OS install.

Only the second option will be covered here, as being more flexible, requiring less invasive actions (e.g., swapping hard disks in a real machine) and providing more hardware independence - e.g., a relatively old OS like RedHat 9 might be set up this way and be used successfully as an AppLogic appliance, on hardware that is not natively supported by the OS.

A running host with a CPU that supports HVM (I.e., newer Intel Xeon or AMD Opteron processors), and installed XEN 3.0.4 or later will be needed. Create one or more zero-filled files that will serve as the "boot disks" for the new OS. Choose their sizes according to the expected OS configuration that will be installed. the sizes must be an exact multiple of 1M (1024*1024). In most cases, a single "disk" is enough, but for OS-es that allow some of their files to be stored on a read-only partition (e.g., the /usr file space on the various Unix-ish OS-es) might benefit from having two disks: AppLogic has to copy the writable boot volumes of each instantiable appliance, so making them as small as possible is desirable.

Have the OS installation media (CD or DVD image) ready, in the form of an ISO file on the XEN host.

Have the APK archive (and DomU kernel, for Linux) ready and available on the local network, over a simple protocol like FTP or HTTP (we will need to get them from within the new OS, once it is up and running).

Set up and run a new XEN virtual machine (in hardware-emulation mode) with the OS install ISO file as the boot disk. Here is an example XEN config file with one emulated HDD and a CDROM (used as the boot device):

import os, re 
arch = os.uname()[4] 
arch_libdir = 'lib' (lib64 for 64-bit OSes)
device_model = '/usr/' + arch_libdir + '/xen/bin/qemu-dm' 

kernel = "/usr/lib/xen/boot/hvmloader" 
builder='hvm' 
memory = 1024 
name = "hvm1" 
acpi=1 
apic=0 
vif = [ 'type=ioemu, mac=00:16:3e:00:00:94, bridge=xenbr1' ] 
disk = [ 'file:/tmp/boot-disk1,ioemu:hda,w', 'file:/root/sol-nv-b76-x86-dvd.iso,hdc:cdrom,r' ] 

on_poweroff='destroy' 
on_crash='destroy' 
on_reboot='destroy' 

boot="d" 
vnc=1 
vncviewer=0 
vncpasswd='' 
vnclisten='192.168.0.122' 
ne2000=0 

Once the VM goes up, a VNC connection can be opened to see its console. Even though the console is an emulated VGA with graphical capabilities, it is advisable to choose text-based install, if the OS has one, especially if the GUI install is overly dependent on mouse usage. The latter doesn't work very well over VNC.

Do the OS install according to taste, taking care not to install more than could fit on the small volumes that were prepared for it. Keep the following in mind:

• ( Important, verify this after the OS install!) SSH server must be installed and set up to start on boot. it should have public key authentication enabled and root logins allowed. To keep your OS safe, disable password logins for root (or disable it altogether). A version of the SSH server that is compatible with the OpenSSH format of authentication keys is necessary. See the appropriate APK install guide for any OS-specific notes regarding SSH setup.
• do not set up more than one partition per "disk". Use of swap partitions is not recommended for appliances, but if desired, leave that step for later (when the image is ready and proven to run under AppLogic). Do not set up swap partitions during install.
• The network setup should be the bare minimum necessary, so as to be able to transfer the APK files to the new OS for installation. The network setup will not be kept beyond the install phase - once the image is ready, its network will be configured by APK.
• Do not install graphical tools / GUI, if at all possible. This will be a headless appliance, not a desktop

Once the install is done, let the installer shut down the VM and reset it. It will not auto-reboot by itself (we configured XEN to destroy the VM on attempt to reboot).

Now, modify the VM config file to use the disk as the boot device, rather than the CD/DVD image (set the boot variable to "c"), and re-start it. Alternatively, if the OS installer has the option to chain-boot from hard disk, use that instead - saves the modification of the VM config file.

Log in via VNC and check that your new OS is running OK.

Download the APK files and do the install as described in the OS-specific install guide (including any listed steps outside of running the install script - most importantly, delete all of the network setup that was put in during the install and disable any services that need real hardware to work - e.g., 'kudzu' in RedHat, smartd, etc.).

Shut down the OS.

Linux only: strip the master boot record and any padding that comes with it from the disk image files, leaving only the actual FS partition in each image file. This can be done using 'fdisk -lu' to see the starting sector of the partition (usually 63) and then dd to grab data from the start sector to end of file. Ship the installed OS image files to an AppLogic grid, e.g, as follows:

1. Create a new application and add a new blank component to it. Edit the new component class (right-click Edit Class) and add un-formatted volumes with the same exact sizes as the ones created for the OS install above. Always start with the boot volume, then add the other volumes in the order in which they were set up during OS installation. Note: do not specify a mount point for the OS volumes.
2. For each of the volumes, do the following (the example assumes that the appliance was named my-app and the component was named newos:
   3t vol manage my-app:newos.volname --rw ip=avail-ip-addr netmask=mask gateway=gw
   (use an available external IP address for your grid, with the corresponding netmask and gateway, to configure the vol manager instance with a working external network access).
   1. from the vol manager shell, download the volume image onto the disk (commands may vary with OS and the method used to transfer the volume image, also see the OS-specific APK install guide for the exact location of the appliance.desc file on the vol manager - it is needed to find the device name):
      devname=`sed -n -e '/volume dst/s/ *volume dst: *dev=//p' /var/run/applogic/appliance.desc`
      wget -O $devname ftp://myftphost/osimage.boot
   2. exit from the vol manager shell (and repeat the above steps for other volumes, if the OS was installed on more than one volume).
3. Once all volumes have been copied, start the application and log into the appliance (from the applogic shell):
   app start my-app
   ssh my-app:main.newos
   (or if using the remote-access macros: ca my-app ; astart; assh newos ; )
4. Check the log file for any unusual error messages. If needed correct any glitches found (e.g., disable any useless hardware-related service that wasn't disabled during the initial setup and now fails, because it can't see its favorite piece of metal).

The new blank appliance is now almost ready for being moved into a catalog. Final cleanups:

1. delete all log files, clean out the temp file directory
2. zap all free space on the disks, e.g., using commands like these (repeat for each volume, e.g., once for / and once for /usr, if these are your two volume mount points):
   cat /dev/zero >/path/to/somefile where /path/to is a directory somewhere on the volume
   wait for the cat command to fail (with disk full)
   rm /path/to/somefile
3. If one or more of the appliance volumes were designated to become read-only: stop the application, and edit the appliance class to set them as read-only. Restart and log back in to see if it still works.
4. Stop the application, move the singleton into a catalog of your choice (e.g., 'user', which is always present on a grid and writable).

Creating a VPS Template

This recipe builds on the "Blank Appliance" one, and adds the following customizations, appropriate for using the appliance as a general purpose virtual server:

• an internet-accessible network interface
• secondary SSH login with its own configuration file and access key setup, independent of the automatic setup performed by APK for the benefit of the 'ssh' command of the AppLogic shell.
• configuration properties to facilitate the use of the VPS appliance class for easy provisioning (I.e., simple/automated making of copies of the VPS to rent out to clients).

The first step is to create the OS image for the future VPS template. The "Blank Appliance" recipe described above can be used, or one can start from an existing blank appliance class. Here is what needs to be set up differently:

• larger disk volume, with more software pre-installed
• the appliance class will not become a catalog class, instead the entire application containing the VPS component is saved as a template.
• the root password is set to empty before saving the application as a template - this is used as an indication to the startup script to initialize the password (see the next section for more on root password configuration).
• the 'external' interface is enabled in the component descriptor (Edit Class -> Interfaces, External interface).

VPS Server Configuration

The following standard properties should be defined for any VPS server appliance and exported to the application boundary, so that they can be configured with the 'app configure' command or the 'app provision' command:

(VPS templates provided with AppLogic also have these properties, which allow creating an initial user account: user, user_pw)

The appliance should be outfitted with a startup script that runs immediately after the APK startup script (applogic_init) and before the ssh server starts (important to keep the initial password setup safe). This script is responsible for applying the VPS configuration properties in an OS-specific manner, as outlined in the above table.

Setting up two SSH server instances

In this step we set up a secondary SSH sever, so that the VPS is accessible both to the client that rents it, and to the provider of the VPS (as well as to maintainers of the AppLogic grid itself), while each SSH instance maintains its own configuration and changes to it do not interfere with the operation of the other instance. The "primary" SSH instance, as configured by the OS install is left for use by the VPS user(s) and is configurable by them. The secondary instance is dedicated to access from the grid, for the VPS provider / maintainter.

Note that the actual setup may vary with OS type and SSH server version. The examples here assume the OS is unix-style and the server is OpenSSH.

The two instances of the SSH server can be configured with different security settings, to match specific needs of the VPS user.

1. Copy the configuration of sshd into a new sub-directory, e.g.:
   mkdir /etc/ssh_grid
   cp /etc/ssh/sshd_config /etc/ssh_grid
2. Edit the original configuration file (/etc/ssh/sshd_config) and:
   • remove all ListenAddress directives, if any
   • add the following lines to the configuration file (with no empty lines between them):
     # $$propN: 1111:primary_ip
     ListenAddress 1111
3. Edit the copied configuration file (/etc/ssh_grid/sshd_config) and make the following changes:
   • disable reverse DNS lookups:
     UseDNS no (or LookupClientHostnames no on some SSH implementations)
   • disable password logins:
     PasswordAuthentication no
   • allow root login (with key only):
     PermitRootLogin without-password
   • set a custom key file name, which works only for root:
     AuthorizedKeysFile /root/.ssh/alt_authorized_keys
   • disable GSSAPI authentication (cannot be used from the AppLogic controller):
     GSSAPIAuthentication no
   • Remove all 'ListenAddress' options
4. Create a new auto-start script for the second ssh server, with the following commands in it:
   # get service IP address
   f=/var/run/applogic/appliance.desc
   p=instance:`udlparse elst $f instance`/interface:default
   addr=`udlparse get $f $p/ip`
   
   # start ssh daemon
   sshd -f /etc/ssh_grid/sshd_config -o ListenAddress=$addr

Final Steps

The following settings need to be added to the applogic_init configuration file (usually /etc/sysconfig/applogic_init - see the APK install guide for exact location).

NOTE: modify the file paths to match the settings in the sshd config file for the secondary SSH instance, as configured above.

   # apply configuration to these files:
   APK_CONFIG_FILES="/etc/ssh/sshd_config"
   # make AppLogic configure this file as the grid ssh access key
   APK_AUTH_KEY_PATH=/root/.ssh/alt_authorized_keys

Building a mini-appliance

This section describes how to build a fast-booting mini-appliance, based entirely on a read-only volume. A small footprint appliance like this can be the basis of a NAT gateway, load balancer and anything else that does not need to keep persistent state.

(coming soon)

-- BeckyH - 25 Apr 2008