corosync/QUICKSTART

Application Interface Specification Quckstart Guide
---------------------------------------------------

***
All cryptographic software in this package is subject to the following legal
notice:
This package includes publicly available encryption source code which,
together with object code resulting from the compiling of publicly
available source code, may be exported from the United States under License
Exception TSU prsuant to 15 C.F.R Section 740.13(e).
***

This corosync package is broken into four parts.  The exec directory contains
all of the code responsible for serving the APIs.  The lib directory contains
APIs the to which the user may link.  The test directory contains some simple
test programs which exercise the APIs.  The directory conf contains example
configuration files which can be copied directly onto the target system.

The API implements SA Forum APIs for Cluster Membership (CLM), Availabilty
Management Framework (AMF), Checkpointing (CKPT), and Eventing (EVT).

The API also contains an extended virtual synchrony API which can be used
in distributed applications.

Configuring the corosync executive:
---------------------------------
The corosync executive will automatically determine cluster membership by
communicating on a specified multicast address and port.

The directory conf contains the file corosync.conf

totem {
	bindnetaddr: 192.168.1.0
	mcastaddr: 226.94.1.1
	mcastport: 5405
}

logging {
	logoutput: file
	logoutput: stderr
	logoutput: syslog
	logfile: /tmp/ais
	debug: on
	timestamp: on
}

timeout {
	token: 200
	token_retransmit: 50
	hold: 30
	retransmits_before_loss: 4
	join: 100
	consensus: 200
	merge: 200
	downcheck: 1000
	fail_recv_const: 250
}

The totem section contains three values.  All three values must be set
or the corosync executive wll exit with an error.

bindnetaddr specifies the address which the corosync Executive should bind to.
This address should always end in zero.  If the local interface taffic
should routed over is 192.168.5.92, set bindnetaddr to 192.168.5.0.

mcastaddr is a multicast address.  The default should work but you may have
a different network configuration.  Avoid 224.x.x.x because this is a "config"
multicast address.

mcastport specifies the UDP port number.  It is possible to use the same
multicast address on a network with the corosync services configured for different
UDP ports.

The logging section contains values.  These values do not have to be set in which
case the system defaults to logging to syslog and stderr with timestamping and debug.

It is possible to select 3 destinations for logs: files, stderr, and syslog.  One or
more may be selected at the same time.  If file is selected as a destination, the file
name must be specified via the logfile option or the corosync executive will exit.

The debug option prints out internal debugging information during runtime which may
be helpful for developers.

The timestamp option prints the date and time on each log message.

The timeout section contains seven values.  This section is not normally used, but
rather used to override the program defaults for the purposes of fine tuning for
a given networking/processor combination or for debugging purposes. Be careful to
use the same timeout values on each of the nodes in the cluster or unpredictable
results may occur.

All timeout values except fail_recv_const are miliseconds. fail_recv_const is 
a message count. Until the man page is done you'll have to check the code and the 
totem spec for the function and usage of the timeouts.


The directory conf contains the file amf.conf which specifies the failover
groups, service units, components, and policies to be used by the AMF.  The
configuration file matches the testamf1-6 programs in the test directory and
can be copied directly.

These two files should be placed in the /etc/ais directory.

A few notes about the config files:
1) Do not use DOS style termination.  This breaks the parser.
2) Do not have blank lines in the amf.conf file.  This breaks the amf configuration
   file parser.  We are working on fixing these bugs, but for the moment, it is
   easy to simply avoid them.

Building corosync
----------------
corosync requires GCC, LD, and a Linux 2.4/2.6 kernel.  corosync has been tested on
Debian Sarge(i386), Redhat 9(i386), Fedora Core 2(i386), Fedora Core
4(i386,x86_64) and MontaVista Carrier Grade Edition 3.1(i386, x86_64,
classic ppc, ppc970, xscale).

Compile corosync by running make in the root directory.  Make can also be run
in the individual directories.  Nothing is installed by make.  If install
is desired, the files must be copied manually.

Configure Host
--------------
For security reasons, the corosync only allows a process that had the EGID/GID
of "ais" to connect to it.  To make development easier, it is recommended to
create an "ais" user with the "ais" group.

[root@slickdeal root]# adduser ais -g ais

Set the ais user's password:

[root@slickdeal root]# passwd ais
Changing password for user ais.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.

Generate a private key
----------------------
corosync uses cryptographic techniques to ensure authenticity and privacy of
messages.  A private key must be generated and shared by all processors for
correct operation.

First generate the key on one of the nodes:

unix# exec/keygen
Opencorosync Authentication key generator.
Gathering 1024 bits for key from /dev/random.
Writing corosync key to /etc/ais/authkey.


After this is complete, a private key will be in the file /etc/ais/authkey.
This private key must be copied to every processor that will be a member of
the cluster.  If the private key isn't the same for every node, those nodes
with nonmatching private keys will not be able to join the same configuration.

Copy the key to some transportable storage or use ssh to transmit the key
from node to node.  Then install the key with the command:

unix# install -D --group=0 --owner=0 --mode=0400 /path_to_authkey/authkey /etc/ais/authkey

If the message invalid digest appears, the keys are not the same on each node.

Run the corosync executive
-------------------------
Get one or more nodes and run the corosync executive on each node.  A list of
node IPs should be logged when the nodes join a configuration.  Run the
aisexec program after following the previous directions.

A final note on permissions:
It is not absolutely required that corosync executive runs as root.  If
it runs as root, it schedules at the highest round robin realtime
priority and locks all of it's pages into ram in case a swap would cause a
delay in the real-time nature of the protocol.  The warning "not  
able to lock pages" is simply a warning and can be ignored if you choose
to run as a non root user.

The ais user/group is required because applications are authenticated
against the ais user and group.  If an application(/library) is not root
or ais, then the application cannot connect to the ais executive.

please read SECURITY to understand the threat model assumed by corosync
and the techniques corosync use to overcome these threats.

Before running any of the test programs
---------------------------------------
The corosync executive will ensure security by only allowing the ais group (or
uid root) to connect to the service.  Switch to the ais group before
running any applications linked to the ais apis, or the applications will
not be authenticated and won't be able to access services.

[sdake@slickdeal sdake]$ su ais
Password:
[ais@slickdeal sdake]$ id
uid=501(ais) gid=502(ais) groups=502(ais)

Try out the corosync CLM functionality
-------------------------------------
After aisexec is running

su to ais user

Run test/testclm on one node.  Then kill and add nodes.  This will cause
callbacks to be called in the testclm application which will print out
the node state changes.  The testclm program will not print any output
after it is started and has printed the current configuration until nodes
are added to or deleted from the configuration by starting and stopping
aisexec on other nodes.

Killing aisexec on the node the testclm is connected will cause the 
API to return error codes indicating the system has failed.

Try out the corosync AMF functionality
-------------------------------------
After aisexec is running

su to ais user

The test/testamf{1-6} implement three seperate service units (SU).  SU #1
consists of testamf1, testamf2.  SU #2 consists of testamf3, testamf4.
SU #3 consists of testamf5, testamf6.  The active and backup directives 
in amf.conf define how many SU's become active and how many
become standby in the service group (SG).

To test the corosync AMF, run testamf3 and testamf4 on one node.  Both
components become in service and active.  Then run testamf1.  Nothing
appears to happen, because testamf1 is not placed in service (and made
standby) until testamf2 is registered.  Running testamf2 will show 
a variety of state changes.  testamf1 will match these state changes.
testamf2 is special because is reports an error, and later cancels
the error, causing the entire SU to go out of service, then back in
service.  This behavior is expected by the corosync specification and the
code in testamf2.c can be read for a clearer understanding of what
is happening.

Pressing ctrl-z to background the task (which causes the healthcheck to
timeout) on a component will cause the remaining component to go
out of service.  If ctrl-z is pressed on the active SU, the standby
SU will become active.  CTRL-C on these tests behaves the same way.
A crash behaves the same way.

Try out the corosync CKPT functionality
--------------------------------------
su to ais user

run testckpt.  This will execute various checkpoint API operations.

run ckptbench.  This will execute non-threaded write benchmarks.

run ckptbenchth.  This will execute threaded write benchmarks.

The benchmark configuration (how many threads to run, how many writes
per benchmark run, and data write size are specified in the ckptbench.c
and ckptbenchth.c programs.

Two node clusters should approach 8.5 MB/sec on 100 mbit networks for 
larger checkpoint sizes with encryption and authentication.  If you are not
seeing these results, please report to the mailing list.

Try out the corosync EVT functionality
-------------------------------------
su to ais user

run testevt.  This will execute various eventing API operations.

Try out the corosync EVS functionality
-------------------------------------
su to ais user
run testevs.  This will generate multicast messages and self deliver them

run evsbench.  This will display the benchmark performance of the evs service.

Write your own applications
---------------------------
Without real applications, finding the hard bugs will be difficult.  Please
port or write apps and let us know of the progress!

Contribute!
-----------
Code, examples, documentation, bug reports, testing are all appreciated.
Read the TODO or the ask on the mailing lists for ways to contribute.