corosync/README.amf

AMF B.01.01 Implementation
--------------------------
This patch contains the basis of the AMF B.01.01 service targeted for release
in Wilson (1.0).  It is a work in progress and incomplete at this time.

What does AMF do?
-----------------
The AMF has many major duties:
 * issue instantiate, terminate, and cleanup operations for components
 * assignment of component service instances to components
 * detection of component faults and executing recovery actions

The AMF starts and stops processes that are part of the component.  A SU
contains multiple components.  A service group contains multiple SUs.
A SU is the unit of redundancy used to implement high availability.

The process of starting and stopping components takes place using the CLC
operations.  The AMF specification is exceedingly clear about which CLC
operations occur for which component types and openais implements the full
CLC operations for all of the various component types.

If a component is not sa-aware, the only level of high availability that
can be applied to the application is through execution of the CLC interfaces.

A special component, called a proxy component, can be used to present an
sa-aware component to AMF to manage a non-sa-aware component.  This would be 
useful, for example, to implement a healthcheck operation which runs some
operation of the unmodified application service.

Components that are sa-aware have been written specifically to the AMF
interfaces.  These components provide the most support for high availability
for application developers.

When an sa-aware component is registered, service instances are assigned
to the component once the service unit is available to take service.  This
service instance specifies whether the component is ACTIVE or STANDBY.  The
component is directed by the AMF to enter either ACTIVE or STANDBY states
and then executes its assigned operational mode.  The number of CSIs assigned
to a component is determined by a reduction process with 6 levels of
reduction.  The AMF provides a very clear definition of what is required
with several examples for each reduction level.

The AMF detects faults through the use of a healthcheck operation.  The user
specifies in a configuration file healthcheck keys and timing parameters.
This configuration is then used by the application developer to register
a healthcheck operation in the AMF.  The healthcheck operation can be started
or stopped.  Once started, the AMF will periodically send a request to the
component to determine its level of health.  The AMF reacts to negative
healthchecks or failed healthchecks by executing a recovery policy.

The recovery policy attempts to restart components first.  When components
are restarted and fail a certain number of times within a timeout period, the
entire service unit is failed over.  When SUs on one node are restarted and fail
a certain number of times within a timeout period, the service unit is failed
over to a standby service unit.

Currently openais implements most of what is described above.

How to configure AMF
--------------------
The AMF doesn't specify a configuration file format.  It does specify many
configuration options, which are mostly implemented in openais.  The
configuration file specifies the service groups, service units, service
instances, recovery configuration options, and information describing where
components and CLI (command line interface) tools are located.

There are several configuration options which are used to control the component
life cycle (CLC) of the component.  These configuration options are:

in the group section:
clccli_path=/home/sdake/amfb-dec/test
  The path to the CLC CLI applications.

binary_path=/home/sdake/amfb-dec/test
  The path to the components.

in the unit section:
bn=testamf1
  The bn parameter specifies the binary name of the application that should be
  run by the instantion script.  Note instantiate may already know this
  information and hence, this is optional.

instantiate=clc_cli_script
  The instantiate parameter specifies the CLC-CLI binary program to be run to
  instantiate a component.  An instantiation starts the processes representing
  the component.

terminate=clc_cli_script
  The terminate parameter specifies the CLC-CLI binary program to be run to
  terminate a component.  A terminate CLC terminates the processes representing
  the component nicely by properly shutting down.

cleanup=clc_cli_script
  The cleanup parameter specifies the CLC-CLI binary program to be run to
  cleanup a component.  A cleanup CLC terminates the processes representing
  the component abruptly.

There are several options to describe the component recovery escalation
policies.  These are:

component_restart_probation=100000
  This specifies the number of milliseconds that a component can be restarted
  in escalation level 0 (only restart components) before escalating to level 1.

component_restart_max=4
  This specifies the number of times within component_restart_probation period
  before escalating from level 0 to level 1.

unit_restart_probation=200000
  This specifies the number of milliseconds that a unit can be restarted
  in escalation level 1 (restart entire SU) before escalating to level 2.

unit_restart_max=6
  This specifies the number of times within unit_restart_probation period
  before escalating from level 1 to level 2.

The AMF will execute a N+M reduction process based upon the number of service
instances specified in the configuration file and 4 configuration options
at the groups level:

preferred-active-units=3
  This is the preferred number of active units that should be active.

maximum-active-instances=3
  This is the naximum number of active CSIs that can be assigned to a component.

preferred-standby-units=2
  This is the preferred number of standby units that should be active.

maximum-standby-instances=4
  This is the naximum number of standby CSIs that can be assigned to a component.

A service instance is specified only as a name.  If there are 4 SIs, the
reduction process will execute as per the AMF specification to assign the proper
number of active and standby CSIs to components currently registered.  This
is a little buggy at the moment.

serviceinstance {
	name = siaa
}

Failure detection occurs through the healthcheck option.  The healthcheck
options are
key
  The name of the healthcheck parameter

period
  The number of milliseconds to wait before issueing a new healthcheck.

maximum_duration
  The maximum amount of time to wait for a healthcheck to complete before
  declaring a failure.


The example programs
--------------------
First the openais test programs should be installed.  When compiling openais
in the exec directory a file called openais-instantiate is created.  Copy this
to the test directory

exec# cp openais-instantiate ../test

Set execute permissions for the clc_cli_script

exec# cd ../test
test# chmod +x ../clc_cli_script

IMPORTANT NOTE:
Within the amf stanza, the mode variable should be set to enabled.  This option
defaults to off and the default configuration file turns this off as well.
This is configured off by default to keep from confusing openais users
interested in using AIS without the alpha-AMF.

example openais.conf:
amf {
	mode: enabled
}

The following two paths must be set in the groups.conf file:
       clccli_path=/home/sdake/amfb-l/test
       binary_path=/home/sdake/amfb-l/test

If these are not set, the path to the clc_cli_script and component binaries
cannot be determined and AMF will not institate the testamf1 binary.

Once aisexec is run using the default configuration file, 5 service units
will be instantiated.  The testamf1 C code will be used for all 5 SUs
and both comp_a and comp_b.  The testamf1 program determines its component
name at start time from the saAmfComponentNameGet api call.  The result is
that 10 processes will be started by AMF.

The testamf1 will be assigned CSIs after they execute a saAmfComponentRegister
operation.  Note this operation causes the presence state of the testamf1
component to be set to INSTANTIATED as required by the AMF specification.  The
service instances and their names are defined within the configuration file.

The testamf1 program reports an error via saAmfErrorReport after 10
healthchecks.  This results in openais calling the cleanup handler, which for
an sa-aware component, is the CLC_CLI_CLEANUP command.  This causes the cleanup
operation of the clc_cli_script to be run.  This cleanup command then reads the
pid of the process that was stored to /var/run at startup of the testamf1
program.  It then executes a kill -9 on the PID.  Custom cleanup operations can
be executed by modifying the clc_cli_script script program.

After this is done 4 times (configurable) the entire service
unit is terminated and restarted. Once this happens 6 times, the code
escalates to level 2, which is currently unimplemented.

Currently working:
component register, healthcheck start and stop, csi assignment, n+m with
all 6 reduction levels, error report, amf response, terminate, cleanup and
restart escalation levels 0-1, single node (multinode not tested),
setting presence and operational state of components internally, initial
assignment of n+m csis based upon configuration options and fully
following AIS AMF B spec.

Not working or tested:
escalation levels 2-3 (switchover/failover), protection group tracking,
protection groups in general, any other model besides n+m, amf B
specified reassignment of csis to terminated and restarted components,
support for proxied or non-sa aware components, state machine for n+m
needs alot of work after initial start.  Timeout periods to reduce
escalation level for escalation policies are unimplemented.

Any feedback appreciated.

Keep in mind this is very early code and may have many bugs which I'd
be happy to have reported :).