INSSLR - Redundant HTTP Input Gateway with SSL Support

	At a Glance
	Catalog	System
	Category	Gateways
	User volumes	yes
	Min. memory	64M
	OS	Linux
	Constraints	no
	Questions/Comments	Ask Forum

The INSSLR appliance is a layer-7 gateway for secure HTTP requests. It converts the requests to unencoded HTTP requests. This can be used whenever it is necessary to support secure HTTP on the client's side, but the back-end processing infrastructure does not or cannot support SSL, including:

• using a fast and light-weight HTTP server that does not have SSL support
• using multiple back-end servers, for performance or for redundancy, connected through a load-balancing switch
• using multiple back-end servers for unrelated functions, connected through a URL switch
• offloading the SSL encryption/decryption to a separate server to improve throughput

If configured, INSSLR also supports client certificate authentication. In the case of SSL mutual authentication, both the client and server exchange their certificates and corresponding public keys. The client may contact the certificate authority (CA) which issued the server's certificate and confirm the certificate is authentic before proceeding. The server may do likewise.

In its default configuration INSSLR acts as a layer-7 gateway for secure HTTP requests, also providing firewalled entry point for network traffic into an AppLogic application, which can be configured with an Internet-accessible static IP address.

When configured, two INSSLR appliances can be used in failover mode to provide a redundant service. In this case, INSSLR IP address (configured via ip_addr) is running on only one of the nodes and is automatically transferred to the other INSSLR appliance in case of failure. At any given moment, only one of the INSSLR appliance is active. When running in failover mode INSSLR can be configured to run in several modes:

• asymmetric - one of the appliances is primary and whenever it is online it takes over the IP address and becomes active. The backup appliance becomes active only if the primary node fails (or is stopped). Whenever the primary appliance becomes available again, it takes over the IP address and becomes active. The primary appliance is the one with the lowest IP address (string comparison) as set in the fover_local_ip property.
• symmetric - both appliances are primary and whichever appliance starts first takes over the IP address and becomes active. The other appliance becomes active only if the first appliance fails (or is stopped). Whenever the first appliance becomes available again, it does not automatically take over the IP address.

When running in redundant mode, INSSLR provides an interface on its ctl terminal for:

• forcing INSSLR to become primary
• forcing INSSLR to become backup
• check the state of INSSLR (primary/backup)

INSSLR constantly monitors the health state of the backend appliance connected to its http terminal. The health state checks conducted by INSSLR may include a simple TCP connect check or a more complex HTTP request (specified on INSSLR's boundary). In the case of a failure of the connected appliance, INSSLR reports an error to the grid dashboard or, if in redundant mode and configured to do so, it does a failover to the backup INSSLR appliance.

To support applications that need to appear at a single IP address for more than one service, INSSLR can be configured to direct non-HTTP traffic transparently to a separate output terminal. For such connections, the appliance acts as a layer-3 firewall/NAT router.

Boundary

Resources

The default memory for the AppLogic 2.7 release was changed to 128MB. The minimum memory remains 64MB.

Notes

• When using INSSLR in failover mode ( fover_mode is not none), the minimum memory is 100M.
• The amount of memory given to INSSLR does not affect its throughput. Use more memory only to support more concurrent requests where the back-end servers may hold the requests for excessively long periods of time.

Terminals

Properties

name	type	description
ip_addr	IP addr	external IP address of the gateway. This property has no default value and must be set. Default: (empty)
netmask	IP addr	Netmask. This property has no default value and must be set. Default: (empty)
gateway	IP addr	Default gateway for outgoing traffic. Default: (empty)
l7_accept	enum	This specifies what kinds of HTTP traffic to accept for forwarding to the http terminal. Valid values: https, http, both , none. If set to none all traffic is redirected only according to the l3_accept_* properties. Default: both.
l3_accept_proto	enum	Specifies which protocols should be forwarded to the aux terminal. Valid values: none, tcp, udp, raw, all. If set to tcp or udp, the l3_accept_port property may be used to specify the port. If set to raw the l3_accept_port property specifies the protocol number. If set to all all incoming traffic on the external interface is forwarded to the aux terminal. Note that the l7_accept property takes precedence over this one - if you set l7_accept to value different from none all http(s) is forwarded to the http terminal, the rest of the traffic goes to aux as specified by this property. Default: none.
l3_accept_port	string	A comma (or space) separated list of protocols to accept and route at the protocol specified by l3_accept_proto to the aux terminal; Protocols in the list may be specified either as port numbers or as standard protocol names (e.g., ftp, smtp etc. when specifying tcp/udp ports or gre, tcp, etc. when using raw protocols). Port ranges can also be specified (1024:10000, 0:1024). If left empty all ports of the specified protocol are forwarded. Note: If you set l3_accept_proto to raw you must specify this property which in this case specifies the protocol number (more than one raw protocols may be specified but no proto range (e.g. 20:30) is allowed) Default: all
allowed_hosts	String	List of hosts and/or subnets allowed to connect. Separate multiple entries with spaces or commas. Supported format example: 192.168.1.2 192.168.1.0/24 192.168.2.0/255.255.255.0. Default: 0.0.0.0/0 (all allowed)
key_on_fs	string	Indicates whether keys are stored on an nfs mount through the fs terminal (on) or on the local key volume (off). Valid values: on and off. Default: off.
cert_file	string	File name (relative to the data volume root) of the server certificate that this gateway instance should present to the client. Note that a valid certificate must be present on the configured data volume (see Volumes below) at the location specified by this property if you set l7_accept to https or both, otherwise INSSLR fails to start. Default: server.pem.
unsafe_ssl	string	Enable the use of 'unsafe' ssl ciphers for compatibility with legacy browsers. The default value of disabled disables SSLv2 ciphers as well as some other SSLv3 and TLSv1 ciphers that are not considered secure. It is recommended to leave this property set to disabled unless you need to support https sessions for legacy browsers which only work with SSLv2. When set to enabled, all SSL ciphers available on the system (including SSLv2) may be used for https sessions. Default: disabled. This property was added in version 1.2.1.
keepalive	int	Specify the maximum keepalive time between INSSLR and the client (specified in seconds). Default: 15.
timeout	int	Specifies how many seconds INSSLR waits for output from the backend server. If the backend server does not send output for timeout seconds, the connection is closed. Default: 300
max_request_size	int	Maximum size (in MB) of the client request. Increase if your application needs to handle large client uploads. Default: 10.
adv_config	string	Add raw configuration to be inserted in nginx conf inside location blocks for both http and https listeners (whichever is enabled). For example, to add custom headers, set adv_config to proxy_set_header myport $proxy_port;. This adds a myport: 80 to the request sent to the backend server. adv_config may be set to any valid statement for a location block (refer to nginx documentation for more details). You can add multiple configuration lines separated by ;. If set, this property must have a valid nginx syntax (i.e. end with ;) or the appliance fails to start. Default: (empty)
client_cert	string	Enables client certificate authentication. Valid values: on, ask and off. If set to on, client certificate authentication is forced and only clients with valid certificates can make a successful request to INSSLR. When set to ask, clients are asked to present certificates, but the connection is established even if a valid certificate is not presented. Default: off.
client_depth	int	The depth of verification to pursue in a chained client certificate. This property has no affect if client_cert is not set. Valid values: 1-9. Default: 1
ca_list_client	string	A file containing a sequence of CA certificates in PEM format. The names of the listed CA certificates are sent to the client on connection. This informs the client which client certificate it should send. The same list is used for verifying the client certificate. The file name is relative to the root of the mounted key volume or the root of the nfs mount made via the fs terminal and may contain a path, e.g. path/to/keys/ca_list_client.pem. Default: ca_list_client.pem.

Health check properties

Advanced Properties (used in failover scenarios)

fover_mode	String	Failover mode. Possible values are none (no failover), symmetric and assymetric. When set to none INSSLR acts just like an INSSL appliance and does not provide failover capabilities. When set to symmetric, INSSLR runs in symmetric failover mode (you need two INSSLR appliances, both running in symmetric failover mode). When set to asymmetric, INSSLR runs in asymmetric failover mode (you need two INSSLR appliances, both running in asymmetric failover mode). When running in failover mode, both appliances must have fover_mode set to the same value. Default: none
fover_local_ip	IP addr	Local IP address to be used in failover mode for communicating with the other INSSLR appliance. This can be any available IP, including any reserved private address (as defined by rfc1918). This address is used only for monitoring the status of the other INSLLR appliance. This should be set to the same IP as the fover_remote_ip property on the other INSSLR appliance. Leave this empty if you have set fover_mode to none. Default: (empty)
fover_remote_ip	IP addr	Remote IP address of the other INSSLR appliance used in failover mode. This should be set to the same IP as the fover_local_ip property on the other INSSLR appliance. Leave this empty if you have set fover_mode to none. Default: (empty)
fover_netmask	IP addr	Netmask for fover_local_ip. Leave this empty if you have set fover_mode to none. Default: (empty)
fover_nfy_prefix	String	Url prefix that is requested whenever a failover occurs. The requested URL is http://nfy/fover_nfy_prefixfover_mode=fover_mode&state=<start|stop>&ip_addr=ip_addr&fover_local_ip=fover_local_ip&fover_remote_ip=fover_remote_ip&fover_netmask=fover_netmask and goes through the nfy terminal. Default: ?
fover_on_healthcheck	Int	Specify if INSSLR should do a failover to the backup node if a health check on the http terminal fails. If set to non-zero value, INSSLR does a failover after this many subsequent health check failures. Don't set too low to avoid false positives when starting/stopping your application. See also healthcheck_alert if you just need notifications for the failures. Default: 0 (disable).

Volumes

Error Messages

In case of appliance startup failure, the following errors may be logged to the system log:

Performance

Application Failover

INSSLR detects the failure of the other INSSLR in about 10 seconds. It takes an additional 10 seconds to failover to the other application. The total time from when the application failure first occurs to when the other application takes over the traffic is about 20 seconds.

Request Rate

INSSLR routes no less than 3000 transactions (request/response pairs) per second, subject to document size and network bandwidth available.

Data Throughput

INSSLR routes no less than 7 MBytes/second, subject to document size and network bandwidth available

Concurrent Connections

With its default memory INSSLR supports no less than 500 concurrently pending requests. (A "pending request" being an open TCP connection from the client, on which there is one or more un-completed HTTP request in progress). Maximum amount of concurrent connections depends of available free memory. Each 16 MB additional memory increases the number of concurrent http transactions by 500. For example, if you run in redundant mode and you want to be able to serve 2000 concurrent connections, you need 100M + 3*16M = 148M (minimum memory when running in redundant mode is 100M).

Notifications

INSSLR send two notifications:
* A notification to the grid dashboard: "INSSLR appliance APP_NAME:COMP_NAME <acquired|gave up> resource ip_addr"
* An http request via the nfy terminal. It is up to the receiving end to take some action based on the notification. The requested URL is:

http://nfy/fover_nfy_prefixfover_mode=fover_mode&state=<start|stop>&ip_addr=ip_addr&fover_local_ip=fover_local_ip&fover_remote_ip=fover_remote_ip&fover_netmask=fover_netmask

When using fover_nfy_prefix different than the default make sure it ends with ? if fover_nfy_prefix is just the path the the script or & if fover_nfy_prefix includes additional parameters.
If a node becomes unavailable it may not be able to send a notification that it became passive and only the node that became active sends notification.
The timeout for the http request is 5 seconds so that the failover is not slowed.

Health Check

INSSLR does a basic check on its http terminal. Errors are reported to to grid dashboard (at maximum rate 1 per 10 minutes). If INSSLR is running in redundant mode and fover_on_healthcheck is set to non-zero value, INSSLR tries to do a failover to its backup appliance after fover_on_healthcheck number of subsequent health check failures. The failover may not succeed if the backup INSSLR appliance is not running or not configured properly.
Setting fover_on_healthcheck to 1 makes INSSLR failover on each failed healthcheck, which may not be always desired. Using a higher value helps avoid false positives (like when stopping the application).

Server Certificates

In order to use SSL you need both the signed certificate and the private key it was encrypted with. The key and the certificate should be in PEM format and put in a single file specified by the cert_file property.

Generating a server certificate

First, you need a private key. You can generate one by executing:

• openssl genrsa -out privkey.pem 2048

To generate a pass protected key, use the following (note that in order to use the key with INSSLR you need a passwordless key, if you create a pass protected key you need to remove the password before using it in INSSLR)

• openssl genrsa -des3 -out privkey.pem 2048

Next you need a certificate. You have two options here - create a certificate request and have it signed by a trusted CA (for which they will charge you), or create a self-signed certificate for test purposes (in this case browsers requesting your site will issue warnings that the certificate is not signed by a trusted CA).

To generate a certificate request, execute the following:

• openssl req -new -key privkey.pem -out server.csr

After you send the .csr file to your trusted CA, it will give you back a signed certificate ( .crt file) which you can use.

To generate a self signed certificate, execute the following:

• openssl req -new -x509 -key privkey.pem -out server.crt -days 1095

Using the server certificate

You can now put the certificate and the key in a file and use it in INSSLR:

• cat privkey.pem  server.crt > server.pem

If your key is password protected, you can remove the password by executing the following:

• openssl rsa -in key_with_pass.pem -out privkey.pem

Using existing apache+mod_ssl server certificate

If you have an existing certificate that you use in Apache, you can use it in INSSLR as well. Just make sure the key is not password protected (see above) and put the private key and the certificate in one file in that order. Example:

• cat privkey.pem  server.csr > server.pem

If you are using a chained certificate, you should also include the intermediate certificate provided by the issuer. Put the private key, the certificate and the intermediate certificate in one file in that order. Example:

• cat privkey.pem server.csr sf_issuing.crt > server.pem

The server signing key is your Web site's "proof of identity". It is also vulnerable, since it is not password-encrypted (so that the appliance can read it without your help). Take the necessary measures to protect the key file, when installing it on the data volume. Do not use the same data volume for other purposes and don't make it visible to a Web server, e.g., to host Web-accessible data (HTML pages, scripts and such).

Client certificates

The following example outlines how to create your own CA and use it to sign your own certificates using OpenSSL. This was tested using OpenSSL 0.9.8b, the same version as is installed in INSSLR.

Create a Certificate Authority

If you already have your own CA used to create self signed server sertificate, you can skip this step and use that CA. When creating the instruments of the trusted authority for your application, it is important that the environment be secure. Create a working directory on a secure host and within that directory:

• mkdir CA
• mkdir CA/private
• Create a password protected RSA key for the CA with a 2048 bit key length:
  • openssl genrsa -des3 -out CA/private/CA_key.pem 2048
• Create, from the private key, the public-key certificate for the CA:
  • openssl req -new -key CA/private/CA_key.pem -x509 -days 365 -out CA/CA_cert.pem

The CA's private key is the basis of trust for the application, so do not misplace it or provide it to other parties.

Create Client Certificates Signed by the CA

• Generate an RSA private key (to create a password protected key, use the -des3 switch):
  • openssl genrsa -out clientA_privkey.pem 2048
• Generate a certificate signing request (CSR) from the private key:
  • openssl req -new -key clientA_privkey.pem -out clientA_request.csr
• Sign the certificate contained in the CSR using the certificate generated for the CA:
  • openssl x509 -req -days 365 -in clientA_request.csr -CA CA/CA_cert.pem -CAkey CA/private/CA_key.pem -CAcreateserial -out clientA.cer

The -CAcreateserial switch enables the assignment of unique serial numbers to issued certificates. Since this is the first certificate issued by the CA, the file CA/CA_cert.srl is created to track serial number assignments. In creating susbsequent client certificates, the -CAserial switch is used in place of the -CAcreateserial switch. For example:

• openssl genrsa -out clientB_privkey.pem 2048
• openssl req -new -key clientB_privkey.pem -out clientB_request.csr
• openssl x509 -req -days 365 -in clientB_request.csr -CA CA/CA_cert.pem -CAkey CA/private/CA_key.pem -CAserial CA/CA_cert.srl -out clientB.cer

To create a single PEM formatted file which includes both the client certificate and key:

• cat clientA_privkey.pem clientA.cer > clientA.pem

To create the equivalent file in pkcs12 format (usable by most browsers):

• openssl pkcs12 -export -in clientA.cer -inkey clientA_privkey.pem -out clientA.p12

Create the CA List Files Used by INSSLR

To create the file identified by the ca_list_client property:

• cat CA/private/CA_key.pem CA/CA_cert.pem > ca_list_client.pem

Place this file either on the key volume or the volume which is nfs mounted via the fs terminal.

If desired, INSSL's server certificate, e.g. server.pem, can also be created in the same manner as the client certificates and likewise signed by the created CA. Do not use a password for this certificate.

Once server.pem and ca_list_client.pem are in place, client certificate authentication can be tested as follows:

• To test from a browser, import a pkcs12 formatted client certificate into FireFox using Tools=>Options=>View Certificates=>Your Certificates=>Import. Point the browser at the IP address of INSSLR.
• To test the SSL handshake from a remote host:
  • openssl s_client -host IP-address -port 443 -showcerts -ssl3 -cert clientA.cer -key clientA_privkey.pem -state

Client certificate headers

If a client connects to INSSLR via HTTPS, and if it presents a client certificate, INSSLR adds the following headers to the request it issues to the backend server:

• X-SSL-Subject details about the certificate owner.
• X-SSL-Issuer details about the certificate issuer (Certificate Authority).
• X-SSL-serial certificate serial number (decimal).
• X-SSL-cipher the cipher currently in use.
• X-SSL-certificate the full client certificate (PEM-format multi-line).

It is the application's responsibility to make use of these headers or not. INSSLR just passes this information without checking it in any way (except for signature and encryption correctness).

Typical Usage

Web applications

To provide http(s) access to your application, connect the http terminal directly to the WEB appliance.


If you need a scalable web application hook the http terminal to a HLB appliance.

Configuration for web applications

Note: the configuration examples list only properties set to non-default values and lack the network configuration (ip_addr, netmask, gateway).

Web application with additional services

If you have more services than just http in your application, you can use INSSLR to pass http traffic to its http terminal and use the aux terminal for other services.

Web applications with > 1 additional services

If you have multiple tcp/udp services and http, you can use INSSLR to pass http traffic to its http terminal and use the aux to feed the rest of the traffic to PS8, which feeds the desired traffic to the backend servers.

Redundant Web applications

If you need to provide a redundant web application, you can copy the application and use INSSLR in order to provide failover capabilities for the external IP address.

Backup Web application

If you just want a backup application that will notify users for the downtime, you can use INSSLR to build a backup application that requires a minimum resources.

Appliances in use:

• user - input gateway for user requests
• web - web server displaying maintenance message

INSSLR on primary application:

INSSLR on backup application:

Redundant Web application (single application)

If you want to have your application run in redundant mode without creating a new application, you can just double the appliances in it and run them in redundant mode. Since this involves using (at least) two web servers and two database appliances, in normal mode they are all used (providing scalability), but each of them is able to serve the application alone in case the other appliance fails. If you need additinal scalability you can add more web and database appliances. In this example half of the appliances (in1, sw1, web1, db1) are running in one failover group and the rest (in2, sw2, web2, db2) are in another failover group so that the application can survive a server crash. This application provides redundancy at a very low cost as all of the appliances (except for one INSSLR and one HLB appliance which require very low resources) are in active state and no resources are spent for a backup application that runs only when the primary fails.

in1

in2

db1

db2

Redundant Web application (two identical applications)

You can run two identical applications on the same grid (but on separate servers so that failure of a server can only affect one of the application), or on different grids, provided that the IP address you use is accessible from both grids. The example below shows an application that uses a database appliance which is also running in redundant mode so in case of failure of one application, the other application can take over without data loss.

Appliances in use:

• in1 - redundant input gateway for user requests
• admin - input gateway for log files access
• sw - redirect port 8080 from admin to ui on db
• repl_in - input for the remote application to connect to the db appliance in order to replicate the database
• web_lb - web load-balancer for user requests
• web1, web2 - web servers with active content (e.g., CGI scripts)
• db - MYSQLR64 configured to be both a master and a slave at the same time
• content - storage for database error log files, web content and web logs
• logs - storage for database error log files
• repl_out - output gateway for the db appliance to connect to the remote application in order to replicate the database
• mon - MON appliance

Client request arrive on the in1 gateway. The gateway forwards the requests to the web_lb load balancer, which directs the request to one of the web servers web1 and web2. The web servers access the db database. The db appliance connects to the remote application (which is an identical copy, the only difference being the server_id of db and the network setup) in order to replicate the database. The remote application connects to the db appliance via the repl_in gateway which is configured to allow connection only from the repl_out gateway of the remote application. The db appliances in the two applications are running in master-master setup so they always have identical data.

Example property configuration (properties that are not listed should be left to their default values):

Web access to db is available via admin gateway on port 8080.

in1

db

Only one of the applications is active at any time, the other one is running just for failover and is used when the active application fails.

Notes and Links

Notes

• INSSLR supports HTTP 1.0 and HTTP 1.1, but if the client identifies itself as MSIE, HTTP 1.1 support is turned off for that connection (to avoid bugs in some versions of MSIE).
• If the client is not MSIE, INSSLR supports HTTP 1.1 for the client (including the multiple requests per TCP session ability), even if the back-end server uses supports only HTTP 1.0.
• INSSLR supports a single external IP and therefore only a single SSL certificate may be used.

Open source and 3rd party software used inside the appliance

INSSLR uses the following 3rd party open source packages in addition to the 3rd party open source packages used by its base class LUX5.

To see the full list of open source packages used in this appliance, please see its implementation design.

Related Documents

• Implementation Design
• Test Plans and Results
• Release Notes?

Questions and Comments

To post a question or comment on this appliance, visit our forum.

-- PavelGeorgiev - 25 Jan 2010