Sep 7 - AppLogic 2.1 has been officially released! See the release notes for details.

The AppLogic Application Model

The AppLogic Application Model

The AppLogic application model is a set of abstractions that make it simple to express the structure and behavior of a distributed web application without requiring any modifications to already exising code and data. These abstractions can be grouped into the following categories:

virtual resources, including virtual machines, volumes and network interfaces
virtual appliances, which are instantiable, connectable and configurable component objects
composite appliances (assemblies) that encapsulate structures of interconnected appliances
catalogs of appliances that provide an easy way to create libraries of appliances
applications

The application model is a distributed component object model in which each component encapsulates a complete software stack, including OS, application software, configuration data and scripts. It makes it easy to assemble a wide range of distributed transactional applications and deploy them on grids of commodity hardware.

Virtual resources

AppLogic abstracts the underlying hardware system by virtualizing hardware resources. It is the first system to make distributed web applications portable and hardware-independent. AppLogic achieves this by abstracting hardware into three distinct types of virtual resources: virtual machines, virtual volumes and virtual network interfaces.

AppLogic sees the hardware system as a grid of computing and/or storage nodes connected with a Gigabit or faster network, where at least one node is acting as a grid controller. Each node provides up to three pools of virtual resources, one for each resource type. The controller aggregates the discrete resource pools into a single, scalable distributed resource pool. As a result, for each type of virtual resource there is one scalable, system-wide resource pool.

AppLogic allocates or creates virtual resources from their respective system pools. Each resource carries a system-wide identification. This enables AppLogic to access resources in a uniform fashion, no matter where on the grid they are actually located and to migrate resources transparently from one node to another without disrupting the running applications.

Virtual machines

AppLogic implements virtual machines by integrating the Xen virtual machine manager. Xen partitions a physical server into multiple virtual machines (VM). Each VM boots a separate operating system (e.g. Linux), and runs any other software it may be configured with.

AppLogic virtualizes the access to two types of peripheral devices - network interface cards (NIC) and block storage devices. It also has the ability to migrate live virtual machines from one server to another, transparently to the software that runs inside each virtual machine.

You can still access other peripherals, such as serial ports, tape drives, etc. from within a virtual machine, but you will have to configure that virtual machine to run on the particular server that has access to the device you want to access and disable its migration.

Virtual volumes

In AppLogic, a virtual storage volume (a.k.a. "virtual volume") is a logical disk exposed by one of the servers in the grid and accessible from virtual machines running on any server.

AppLogic virtual volumes are persistent, named objects. Their size is defined at the time they are created. They reside on the system until explicitly destroyed. A virtual volume defined on one of the servers is accessible from any other server within the same AppLogic grid. This allows a VM that uses the volume to be migrated freely to any server. In the current version of AppLogic, each individual virtual volume is stored as a file on one of the servers, shared on the network as a logical volume and accessible to the other servers.

A typical virtual volume is accessed by a single virtual machine. Whenever a volume is shared by multiple VMs, the access to that volume is usually read-only. This enables AppLogic to cache most volumes aggressively using the local disks of the server, where the VM accessing the volume happens to run.

AppLogic also makes it easy to create multiple instances of the same virtual volume. Those are useful whenever there is a need to share a large set of data among multiple VMs in a way that permits each VM to make relatively small modifications to the common set of data, such as configuration settings, bindings, etc.

Virtual network interfaces

AppLogic uses virtual network interfaces to abstract the structure of the interconnect in the application.

A virtual network interface is a unique connection point within the AppLogic system. An instance of a virtual network interface can be attached to a virtual network interface card (vNIC) on the boundary of a virtual machine, effectively terminating all traffic through that vNIC.

A pair of virtual networks interfaces can be connected to form a virtual wire - a point-to-point connection that carries IP traffic between its endpoints independently of the underlying network technology. The virtual wire then becomes a logical equivalent of a crossover cable that connects two network interface cards directly: it transfers packets between the two vNICs.

Depending on the physical network used, AppLogic implements virtual wires by tunneling traffic through IP connections, InfiniBand reliable connections, or as direct memory-to-memory transfer whenever both network interfaces happen to be on the same server. All of this is completely transparent to the communicating VMs.

Virtual wires make it possible to migrate network connections live from one medium to another. For example, AppLogic can migrate a connection to your database server from Gigabit Ethernet to 10 Gigabit InfiniBand without disrupting the flow of transactions.

Virtual appliances

Virtual appliances are the key concept in the AppLogic application model.

An AppLogic virtual appliance is an instantiable object that consists of a boundary and an interior. The boundary includes everything necessary to configure the appliance, bind it to data on external storage volumes and connect it to other appliances. The interior consists of a virtual machine and a boot volume that contains the OS, configuration files and application software that runs inside the appliance.

Figure 1. The anatomy of a virtual appliance.

Figure 1 shows in more detail the anatomy of a typical virtual appliance. The boundary of the appliance consists of the following elements:

default interface, through which you can log onto a running appliance using SSH
terminals, including inputs like IN and outputs such as DBASE and LOG
properties, which are appliance-specific configuration paramenters
add-on volumes, such as the Content volume
hardware resources (CPU, memory and bandwidth)
execution attributes which control the starting and scheduling of the specific appliance.

The interior of the appliance consists of a virtual machine and a boot volume. The boot volume is configured with a suitable version of Linux, and all other software packages and scripts that are needed for the appliance to work. The virtual machine is configured to boot from the boot volume and has several virtual network interface cards.

The eth0 vNIC is always set up as a default interface, which gives you the ability to log onto the appliance, install software and troubleshoot it, the same way you would do with any remote server.

In addition, AppLogic creates a separate vNIC for each terminal (input or output) defined on the boundary of the appliance. While it is possible to direct all traffic in and out of the appliance through a single vNIC, having a separate vNIC per terminal makes it easy for AppLogic to shape and monitor network traffic, enforce connection protocols and improve security.

The boundary contains a set of appliance-specific properties, or parameters which you can use to configure the appliance. When you start the appliance, AppLogic automatically creates an environment variable for each property and initializes it with the value that you have assigned to that property. In addition, AppLogic propagates the property values into one or more of the configuration files contained on the boot volume. This makes it easy to use properties as the primary mechanism for configuring appliances.

The boundary may also contain one or more volume placeholders. A placeholder is a predefined slot for a storage volume. You fill up the slot by configuring the appliance with the name of a volume to mount. This gives you the virtual equivalent of a removable disk. Many of the appliances use this mechanism to access content such as HTML pages, custom code, or databases.

AppLogic also assigns to each appliance a hardware resource budget that includes a set of three ranges that define minimum and maximum CPU use, memory use, and bandwidth use for the appliance.

Finally, Applogic associates with each appliance a set of execution attributes that affect the way the system schedules and runs the appliance. You are likely to use frequently two of those attributes - the start order value, and the migrateable flag. The start order allows you to specify and change the order in which appliances from the same application start, accounting for any dependencies among them. The migrateable flag is true by default and indicates to AppLogic that it is OK to migrate this appliance from one server to another at the scheduler's discretion. Setting this flag to false will allow you to keep the appliance on a specific server.

Classes and Instances

AppLogic treats virtual appliances as first-class component objects. When you create a new type of appliance, what you are really creating is a class of appliances, letting AppLogic know how to manufacture such appliances (they are called instances of the class) on demand. Since appliance classes are templates for making instances, they cannot be executed. An actual running application contains only instances, configured and interconnected to work together.

Figure 2. How AppLogic manufactures instances from classes

Figure 2 shows how AppLogic instantiates classes. Every class has an associated class descriptor, which defines:

the name of the class and a template from which instance names will be generated
one or more class volumes, one of which is designated as a boot volume
the boundary of the class, including properties, terminals and placeholder volumes
default values for properties, hardware resources and execution attributes

The complete definition of a class consists of its class descriptor and the full set of class volumes identified in the descriptor, including their content - the OS, application software and scripts which together define the behavior of the appliance.

Ordinary appliance classes can produce any number of instances. Singletons are classes that are limited to a single instance at a time. Singletons are useful when you need to edit or troubleshoot a class. Since only instances can run on the system, you always need an instance to test and troubleshoot the class. Singletons make this process simpler. They allow you to test and troubleshoot the instance, and then generate a new class automatically from the working instance.

The class definition is not sufficient to create a usable running instance. To do this, AppLogic also needs to know the instance settings - a set of parameters that define how the instance is to be connected, configured and executed. The instance settings consist of specific values for hardware resources, execution attributes, terminal connections, appliance properties and names of external volumes.

To create an instance, AppLogic interprets the class descriptor and creates a virtual machine with one virtual network adapter for each terminal and a virtual block device for each volume. It then creates an instance of a virtual network interface for each of the adapters and binds it to the appropriate adapter.

Next, the system creates a virtual volume instance for each volume specified in the descriptor by replicating the appropriate class volume and binds it to the corresponding block device.

The newly created instance is configured using the property values, usually by modifying the configuration files identified in the class definition and located on one or more of the instantiated volumes. Since each instance volume is a copy of the respective class volume, these modifications are private to the instance.

AppLogic then starts the VM which results in booting and starting the various services configured in the appliance. It uses the hardware resources and execution attributes to control the starting, placement, scheduling and migration of the newly minted instance.

AppLogic separates the configuration and connection data which is specific for each appliance instance from the information common to all appliances of a given class, including OS code, application engine code, and the configuration parameters needed to make them work together.

Properties

Unlike execution attributes which can be applied to any appliance, in AppLogic you can define configuration parameters specific for each appliance class. AppLogic provides a property mechanism for defining and editing a set of such configuration parameters through a universal property interface.

To take advantage of the AppLogic property mechanism, you define the properties by specifying their names, data types and default values as part of the class definition. You can also select one or more of the configuration files located on the volumes of the appliance as targets for the property values.

See more details in the Editor Reference guide - Instance Settings.

Inside an appliance instance, you can access a property value in one of two ways:

as an environment variable
as a parameter value in one or more configuration files

For each property on the boundary of the appliance, AppLogic defines an environment variable named after the property. At boot time, the value of the variable is initialized to the value asigned to the property. This makes properties easily accessible to daemons, utilities and scripts that execute within the appliance.

A property defined on an appliance can be used to set values of one or more parameters located in text-based configuration files. To map a property of the appliance to a parameter in a configuration file, you tag the value in the file with a special tagged comment, identifying the name of the property you are trying to map. When the property is set, AppLogic will replace the value found in the file with the value assigned to the property.

Terminals

AppLogic terminals are connection points for logical interactions between appliances. The terminal abstraction is designed so that existing software packages inside virtual appliances can communicate through terminals without requiring modification.

A terminal can be an input or an output. Inputs are terminals for accepting network connections. Outputs are terminals for originating network connections. With respect to the flows of requests and data, both types of terminals are bi-directional. A terminal consists of a network name, a virtual network adapter and a virtual network interface.

When an output of one appliance is connected to an input of another appliance, AppLogic creates a virtual wire between their respective virtual network interfaces and assigns virtual IP addresses to both ends of the connection.

Virtual IP addresses are only there for the sake of the software running inside the VM. They are not actual, routable network addresses. The actual traffic on the connection is delivered through the virtual wire.

To the software running in a virtual appliance, terminals appear as named network hosts. An input defines a host name on which a server such as Apache can listen for incoming connection requests. When left unconnected, an output is visible inside the virtual machine as an unreachable host. When connected to an input of another appliance, the same output acts as a valid network host, to which connections are established. If you logon to the appliance and ping a connected output, you will see that the name of the output resolves to the IP address of the input of the other appliance to which the output is connected.

Terminals eliminate references to other appliances and servers.

The code running inside an appliance will see a "network" that consists only of few hosts, one for each terminal defined on the boundary of the appliance. This makes it possible to connect different instances of the same appliance in different structures without modifying the configuration of the appliance.

For example, if you are building an appliance that needs access to a database server, you may define an output for accessing that database and call it DBASE. When configuring the JDBC driver inside your appliance, you simply set the name of the output "DBASE" as the host name of the target database server. At runtime, each instance of your appliance may be connected to a different database server without you having to change the target host name for the JDBC driver - the same host name "DBASE" will automatically resolve to the correct IP address of the database server, to which the particular instance is connected.

For each terminal you can specify what protocol(s) are allowed on it. This has the effect of creating a virtual firewall on the terminal, letting in or out only the allowed protocols. As an added bonus, when you are setting up the application, AppLogic makes sure that only terminals with matching protocols are connected to each other. So, if the software inside your appliance is configured to work with MySQL through the DBASE output, you can declate DBASE's protocol as MYSQL to prevent someone from accidentally connecting it to an Oracle server.

See more details in the Editor Reference guide - Appliance Class Editor.

You can declare any terminal as mandatory, letting AppLogic know that your appliance cannot work without being connected to this kind of external service. AppLogic will refuse to start an application, if a mandatory terminal is left unconnected, and it will let you know which terminal you have forgotten to connect.

Volumes

Each appliance needs at least one storage volume - the one from which it boots. These volumes are provided as part of the class definition of the appliance and instantiated whenever an instance of the apliance is created. It is often useful to define your appliances with placeholders for additional volumes. When you define a placeholder, it becomes a "slot", into which you can later plug in an external volume.

The volumes you can "plug" into a placehoder slot are known as application volumes. You create an application volume explicitly, by executing the appropriate command and specifying a name, storage size and the type of the desired file system. You can mount application volumes on the AppLogic controller to transfer data in and out of them.

For example, it is useful to define a web server appliance with two volumes - a boot volume and a content volume. The boot volume contains the software and configuration files needed to boot Linux and run an Apache web server. This volume becomes part of the class definition and is instantiated for each instance of the appliance.

The content volume, however, is a placeholder for an application volume that will contain the HTML files, static images and scripts for a specific web site. Configuring an instance of the web server appliance with a reference to the content volume produces an instance of an Apache web server that will serve the particular web site. The pages and other content for this web server will be placed on the content volume.

See more details in the Editor Reference guide - Appliance Class Editor.

You can use a single application volume to deploy content for multiple web servers. A simple way to do this is to add a property that specifies the base directory on the volume, from which the given appliance would access the content. This makes it easy to configure several web servers with the same content volume, while setting up each instance to serve content from a different directory.

The same pattern can be used to design a generic J2EE server configured with a volume (and base path on it) that contains the EJB packages for a particular application function, or a generic database server configured with a volume and path containing a specific database.

In fact, using this combination of application volume plus a directory path property makes it possible to combine the user interface, static content, code and data of the application on a single volume, making it easier to deploy, modify and maintain.

Structures of appliances

AppLogic makes it easy to combine appliances into structures that perform advanced application functions. Assuming that all required appliance classes already exist, building a structure involves three general steps:

define the instances
configure attributes, properties and placeholder volumes (if any) on each instance
connect the instances together by connecting the appropriate terminals.

Figure 3. Shared file storage implemented as a structure of appliances

Figure 3 shows shared file storage implemented as a structure of appliances. The structure consists of two application servers, app1 and app2 which are instances of the same class APP, and a network attached storage (NAS) appliance nas1 of class NAS. The outputs of both application servers are connected to the cifs input of the nas1 box. The NAS appliance is configured with a suitably large volume, sufficient to meet the storage needs of app1 and app2. Each appliance is configured with additional properties and attributes as needed.

AppLogic makes it easy to describe arbitrarily complex structures of virtual appliances in a uniform way by capturing the set of instances that participate in them, the configuration parameters for each instance and the connections between their terminals. This allows AppLogic to instantiate such structures automatically, by interpreting structure descriptions, instantiating appliances, configuring them with the provided values and establishing virtual wires, through which the appliances will interact.

Assemblies

The ability to easily capture structures of appliances and instantiate them on demand enables AppLogic to define a new type of appliances which we call composite appliances , or assemblies . An assembly is an appliance that consists of a boundary and an interior. The boundary of the assembly is defined in a manner similar to the boundary of a virtual appliance, and its interior consists of a structure of virtual appliances.

Figure 4. A web switch appliance

Figure 4 shows the boundary of a web switch appliance. The web switch is a useful appliance that accepts HTTP requests on its in input and forwards each request to one of its five HTTP outputs, namely img, out1, out2, out3, and out4. It also has an output named log through which it generates messages that need to be logged in a system log.

The web switch has three main functions:

it recognizes incoming HTTP requests for images and forwards them through img output
it caches static images in memory
it forwards all other HTTP requests through out1 ... out4 in a load-balanced fashion

To build an appliance like this the old-fashioned way, you have to install and configure properly at least three software packages on a server, write some scripts to glue them together, and test the resulting image. Depending on the complexity of the software, your familiarity with the specific packages and the thoroughness of your testing, this process will take from a few days to 2+ weeks of work. Worse, you will have to do most of the work again every time you upgrade one or more of the software packages.

With AppLogic, there is a better way.

Figure 5. The web switch implemented as an assembly of appliances

Figure 5 shows the same web switch built as an assembly of appliances. More specifically, the figure shows the interior of the assembly as the AppLogic editor will visualize it. Each terminal defined on the boundary of the assembly is represented by a "terminal object" that carries the name of the respective terminal (e.g. in, img, log, etc.).

The web switch is assembled from three appliances - a URL switch url of class urlsw, an HTTP load balancer lb of class web_lb, and a content caching appliance cache of class squid.

To make the design of structures easier to understand, we recommend that each instance in a structure is given a name that identifies the instance's role within the given structure.

The URL switch is a simple appliance. It can be configured with up to four regular expressions which it applies to the URLs included in all incoming HTTP requests received on its in terminal. If the URL maches one of the expressions, the switch forwards the whole request through the respective output out1 ... out4. If there is no mach, the request is forwarded through the aux terminal.

The load balancer accepts HTTP requests on its in input and forwards them in a load-balanced fashion through its out1 ... out4 outputs. In addition, it has a log output through which it generates log messages that can be collected in a system-wide log.

The cache is a content cache in memory. It accepts incoming HTTP requests on in and tries to satisfy them from the cache. If the requested object is not found in the cache, the appliance forwards the request through its out terminal and optionally caches the object when the request completes.

In the web switch appliance, the URL switch is configured to recognize the patch for static images within incoming URLs, and forward whose requests on out1 which is connected to the input of the cache. All other HTTp requests leave url through the aux output and are fed into the load balancer. The rest of the web switch behavior should be fairly obvious.

Catalogs

AppLogic catalogs are an easy way to package multiple appliance classes into a library. A catalog can be used in multiple applications, or it may contain appliances that are used in a single large application.

Each appliance class consists of a class descriptor and one or more class volume images referenced by that descriptor. Each assembly class consists of a class descriptor similar to the class descriptor of the regular appliance classes, and an interior descriptor that captures the structure inside the assembly.

Figure 6. The structure of an AppLogic catalog.

An AppLogic catalog consists of a catalog descriptor that lists all classes included in the catalog, and a set of class descriptors. Each class descriptor, in turn, refers to one or more class volumes and/or interior descriptors. In the current release of AppLogic, catalogs are implemented as shared directories, in which all descriptors reside. The associated volumes may reside in the same directory (as volume image files), or in the AppLogic volume store.

Placing the volume images in files makes it easy to publish a catalog through a web or ftp interface.

The name of a class included in a catalog is unique within the catalog. When a class makes a reference to another class contained within the same catalog, the name of that class is sufficient to resolve the reference. Whenever a class has a reference to a class from another catalog, the name of the catalog is pre-pended to the name of the class to form a name that is unique within the AppLogic class namespace.

See more details in the Editor Reference guide - Application Editor.

Applications

An AppLogic application consists of an application descriptor, one or more application volumes, two singletons called Top and Main respectively, and a local catalog that contains classes used only in this application. The application may contain additional singletons if you define them, and those will be referenced in the application descriptor the same way as Top and Main are.

Figure 7. The structure of an AppLogic application.

Most applications define one or more aplication volumes. These volumes are named objects, and their names are unique within the application. Whenever you use an appliance that defines a volume placeholder, you configure it with the name of one of your application volumes.

The Main singleton is an assembly that contains the top-level structure of the application. For obvious reasons, each application has one of those. The Top singleton is the topmost assembly in the application. It contains the single instance of Main, and configures it with the application settings - instance settings that apply to the application as a whole.

The application structure makes it possible to treat the whole application as a single appliance.

When you define appliance classes in your application, their place in the application structure depends on the type of the class. Singletons are included only in the application descriptor; they cannot exist outside of an application, and cannot be included in catalogs. All other classes specific to your application are placed in the application catalog (sometimes called also the local catalog). Note that every application has an application catalog, even if it is empty.

Singletons do not exist outside of an application and cannot be included in catalogs.

The application description language (ADL)

The AppLogic application model defines a cohesive set of abstractions that is sufficient to describe the structure of an arbitrary distributed application without references to the hardware system on which it is to execute, and without explicit dependencies on the actual software functionality encapsulated in each of the appliances.

The model makes it easy to express the structure and configuration of the application through a set of static descriptors using a structure descriptor language.

AppLogic defines an Application Descriptor Language called ADL. The ADL language is semantically equivalent to XML, but is less verbose and much more suitable for direct editing by humans. Using this language, you can describe an arbitrarily complex distributed web application as a set of text files (descriptors) including virtual appliance descriptors, composite appliance boundary descriptors, assembly interior descriptors and package descriptors.

This set of descriptors, together with the images of class volumes and application volumes, is sufficient to instantiate and execute the application on any hardware grid that runs AppLogic.

-- Main.VladM - 05 Jan 2006

-- VladM - 23 May 2006

Topic AdvApplicationModel . { Edit | WYSIWYG | Attach | Backlinks: Web All webs | Printable | History: %REVISIONS% More }

	Wiki > AppLogic1 > AdvApplicationModel	Wiki webs: Main \| TWiki \| Sandbox
	AppLogic1 . { Changes \| Index \| Search \| Go }	Hello TWiki::User=HASH(0x 8b 4def 0)? !