Remove old README.devmap

Signed-off-by: Angus Salkeld <asalkeld@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com>

README.devmap

@@ -1,1220 +0,0 @@
-Copyright (c) 2002-2004 MontaVista Software, Inc.
-Copyright (c) 2006, 2009 Red Hat, Inc.
-
-All rights reserved.
-
-This software licensed under BSD license, the text of which follows:
-
-Redistribution and use in source and binary forms, with or without
-modification, are permitted provided that the following conditions are met:
-
-- Redistributions of source code must retain the above copyright notice,
-  this list of conditions and the following disclaimer.
-- Redistributions in binary form must reproduce the above copyright notice,
-  this list of conditions and the following disclaimer in the documentation
-  and/or other materials provided with the distribution.
-- Neither the name of the MontaVista Software, Inc. nor the names of its
-  contributors may be used to endorse or promote products derived from this
-  software without specific prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
-AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
-IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
-ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
-LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
-CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
-SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
-INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
-CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
-ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
-THE POSSIBILITY OF SUCH DAMAGE.
-
--------------------------------------------------------------------------------
-This file provides a map for developers to understand how to contribute
-to the corosync project.  The purpose of this document is to prepare a
-developer to write a service for corosync, or understand the architecture
-of corosync.
-
-The following is described in this document:
-
- * all files, purpose, and dependencies
- * architecture of corosync
- * taking advantage of virtual synchrony
- * adding libraries
- * adding services
-
--------------------------------------------------------------------------------
- all files, purpose, and dependencies.
--------------------------------------------------------------------------------
-
-*----------------*
-*- AIS INCLUDES -*
-*----------------*
-
-include/saAmf.h
------------------
-	Definitions for AMF interface.
-
-include/saCkpt.h
-------------------
-	Definitions for CKPT interface.
-
-include/saClm.h
------------------
-	Definitions for CLM interface.
-
-include/saAmf.h
------------------
-	Definitions for the AMF interface.
-
-include/saEvt.h
------------------
-	Defintiions for the EVT interface.
-
-include/saLck.h
------------------
-	Definitions for the LCK interface.
-
-include/cfg.h
-	Definitions for the CFG interface.
-
-include/cpg.h
-	Definitions for the CPG interface.
-
-include/evs.h
-	Definitions for the EVS interface.
-
-include/ipc_amf.h
-	IPC interface between client and server for AMF service.
-
-include/ipc_cfg.h
-	IPC interface between client and server for CFG service.
-
-include/ipc_ckpt.h
-	IPC interface between client and server for CKPT service.
-
-include/ipc_clm.h
-	IPC interface between client and server for CLM service.
-
-include/ipc_cpg.h
-	IPC interface between client and server for CPG service.
-
-include/ipc_evs.h
-	IPC interface between client and server for EVS service.
-
-include/ipc_evt.h
-	IPC interface between client and server for EVT service.
-
-include/ipc_gen.h
-	IPC interface for generic operations.
-
-include/ipc_lck.h
-	IPC interface between client and server for LCK service.
-
-include/ipc_msg.h
-	IPC interface between client and server for MSG service.
-
-include/hdb.h
-	Handle database implementation.
-
-include/list.h
-	Linked list implementation.
-
-include/swab.h
-	Byte swapping implementation.
-
-include/queue.h
-	FIFO queue implementation.
-
-include/sq.h
-	Sort queue where items are sorted according to a sequence number.  Avoids
-	Sort, hence, install of a new element takes is O(1).  Inline implementation.
-
-	depends on list.
-
-*---------------*
-* AIS LIBRARIES *
-*---------------*
-lib/amf.c
----------
-	AMF user library linked into user application.
-
-lib/cfg.c
----------
-	CFG user library linked into user application.
-
-lib/ckpt.c
----------
-	CKPT user library linked into user application.
-
-lib/clm.c
----------
-	CLM user library linked into user application.
-
-lib/cpg.c
----------
-	CPG user library linked into user application.
-
-lib/evs.c
----------
-	EVS user library linked into user application.
-
-lib/evt.c
----------
-	EVT user library linked into user application.
-
-lib/lck.c
----------
-	LCK user library linked into user application.
-
-lib/msg.c
----------
-	MSG user library linked into uer application.
-
-lib/amf.c
----------
-	AMF user library linked into user application.
-
-lib/ckpt.c
-----------
-	CKPT user library linked into user application.
-
-lib/evt.c
-----------
-	EVT user library linked into user application.
-
-lib/util.c
-----------
-	Utility functions used by all libraries.
-
-*-----------------*
-*- AIS EXECUTIVE -*
-*-----------------*
-
-exec/aisparser.{h|c}
-	Parser plugin for default configuration file format.
-
-exec/aispoll.{h|c}
-	Poll abstraction interface.
-
-exec/amfapp.c
-	AMF application handling.
-
-exec/amfcluster.c
-	AMF cluster handling.
-
-exec/amfcomp.c
-	AMF component level handling.
-
-exec/amf.h
-	Defines all AMF symbol names.
-
-exec/amfnode.c
-	AMF node level handling.
-
-exec/amfsg.c
-	AMF service group handling.
-
-exec/amfsi.c
-	AMF Service instance handling.
-
-exec/amfsu.c
-	AMF service unit handling.
-
-exec/amfutil.c
-	AMF utility functions.
-
-exec/cfg.c
-	Server side implementation of CFG service which is used to display
-	redundant ring status and reenabling redundant rings.
-
-exec/ckpt.c
-	Server side implementation of Checkpointing (CKPT API).
-
-exec/clm.c
-	Server side implementation of Cluster Membership (CLM API).
-
-exec/cpg.c
-	Server side implementation of closed procss groups (CPG API).
-
-exec/crypto.{c|h}
-	Cryptography functions used by corosync.
-
-exec/evs.c
-	Server side implementation of extended virtual synchrony passthrough
-	(EVS API).
-
-exec/evt.c
-	Server side implementation of Event Service (EVT API).
-
-exec/ipc.{c|h}
-	All IPC operations used by corosync.
-
-exec/jhash.h
-	A hash routine.
-
-exec/keygen.c
-	Secret key generator used by corosync encryption tools.
-
-exec/lck.c
-	Server side implementation of the distributed lock service (LCK API).
-
-exec/main.{c|h}
-	Main function which connects all components together.
-
-exec/mainconfig.{c|h}
-	Reads main configuration that is set in the configuration parser.
-
-exec/mempool.{c|h}
-	Currently unused.
-
-exec/msg.c
-	Server side implementation of message service (MSG API).
-
-exec/objdb.{c|h}
-	Object database used to configure services.
-
-exec/corosync-instantiate.c
-	instantiates a component by forking and exec'ing it and writing its
-	pid to a pid file.
-
-exec/print.{c|h}
-	Non-blocking thread-based logging service with overflow protection.
-
-exec/service.{c|h}
-	Service handling routines including the default service handler
-	description.
-
-exec/sync.{c|h}
-	The synchronization service implementation.
-
-exec/timer.{c|h}
-	Threaded based timer service (deprecated - use qb_loop_timer).
-
-exec/totemconfig.{c.h}
-	The totem configuration configurator from data parsed with aisparser
-	in the configuration file.
-
-exec/totem.h
-	General definitions for the totem protocol used by the totem stack.
-
-exec/totemip.{c.h}
-	IP handling functions for totem - lowest on stack.
-
-exec/{totemrrp.{c.h}
-	The totem multi ring protocool and currently unimplemented.  Between
-	totemsrp and totempg.
-
-exec/totemnet.{c.h}
-	Network handling functions for totem - between totemip and totemrrp.
-
-exec/totempg.{c|h}
-	Process groups interface which is used by all applications - highest on
-	stack.
-
-exec/totemrrp.{c.h}
-	Redundant ring functions for totem - between totemnet and totemsrp.
-
-exec/util.{c|h}
-	Utility functions used by corosync executive.
-
-exec/version.h
-	Defines build version.
-
-exec/vsf.h
-	Virtual Synchrony plugin API.
-
-exec/vsf_ykd.c
-	Virtual Synchrony YKD Dynamic Linear Voting algorithm.
-
-exec/wthread.{c|h}
-	Worker threads API.
-
-loc
----
-Counts the lines of code in the AIS implementation.
-
--------------------------------------------------------------------------------
- architecture of corosync
--------------------------------------------------------------------------------
-
-The corosync standards based cluster framework is a generic cluster plugin
-architecture used to create cluster APIs and services.  Usually there are
-libraries which implement APIs and are linked into the end user application.
-The libraries request services from the aisexec process, called the AIS
-executive.  The AIS executive uses the Totem protocol stack to communicate
-within the cluster and execute operations on behalf of the user.  Finally the
-response of the API is delivered once the operation has completed.
-
-
-           --------------------------------------------------
-           |   AMF and more services libraries              |
-           --------------------------------------------------
-           |                      IPC API                   |
-           --------------------------------------------------
-           |                 corosync Executive              |
-           |                                                |
-           |     +---------+ +--------+ +---------+         |
-           |     | Object  | |  AIS   | | Service |         |
-           |     | Datbase | | Config | | Handler |         |
-           |     | Service | | Parser | | Manager |         |
-           |     +---------+ +--------+ +---------+         |
-           |     +-------+ +-------+                        |
-           |     |  AMF  | | more  |                        |
-           |     |Service| |svcs...|                        |
-           |     +-------+ +-------+                        |
-           |                 +---------+                    |
-           |                 |  Sync   |                    |
-           |                 | Service |                    |
-           |                 +---------+                    |
-	   |                 +---------+                    |
-           |                 |   VSF   |                    |
-           |                 | Service |                    |
-           |                 +---------+                    |
-           | +--------------------------------+ +--------+  |
-           | |                 Totem          | | Timers |  |
-           | |                 Stack          | |  API   |  |
-           | +--------------------------------+ +--------+  |
-           |                +-----------+                   |
-           |                |   Poll    |                   |
-           |                | Interface |                   |
-           |                +-----------+                   |
-           |                                                |
-           -------------------------------------------------
-
-                    Figure 1: corosync Architecture
-
-Every application that intends to use corosync links with the libais library.
-This library uses IPC, or more specifically BSD unix sockets, to communicate
-with the executive.  The library is a small program responsible only for
-packaging the request into a message.  This message is sent, using IPC, to
-the executive which then processes it.  The library then waits for a response.
-
-The library itself contains very little intelligence.  Some utility services
-are provided:
-
- * create a connection to the executive
- * send messages to the executive
- * retrieve messages from the executive
- * Poll on a fd
- * create a handle instance
- * destroy a handle instance
- * get a reference to a handle instance
- * release a reference to a handle instance
-
-When a library connects, it sends via a message, the service type.  The
-service type is stored and used later to reference the message handlers
-for both the library message handlers and executive message handlers.
-Every message sent contains an integer identifier, which is used to index
-into an array of message handlers to determine the correct message handler
-to execute For the library.  Hence a message is uniquely identified by the
-message handler ID number and the service handler ID number.
-
-When a library sends a message via IPC, the delivery of the message occurs
-to the proper library message handler.  The library message handler is
-responsible for sending the message via the totem process groups API to all
-nodes in the system.
-
-This simplifies the library handler significantly.  The main purpose of the
-library handler should be to package the library request into a message that
-can be sent to all nodes.
-
-The totem process groups API sends the message according to the extended
-virtual synchrony model.  The group messaging interface also delivers the
-message according to the extended virtual synchrony model.  This has several
-advantages which are described in the virtual synchrony section.  One
-advantage that must be described now is that messages are self-delivered;
-if a node sends a message, that same message is delivered back to that
-node.
-
-When the executive message is delivered, it is processed by the executive
-message handler.  The executive message handler contains the brains of
-AIS and is responsible for making all decisions relating to the request
-from the libais library user.
-
--------------------------------------------------------------------------------
- taking advantage of virtual synchrony
--------------------------------------------------------------------------------
-
-definitions:
-processor: a system responsible for executing the virtual synchrony model
-configuration: the list of processors under which messages are delivered
-partition: one or more processors leave the configuration
-merge: one or more processors join the configuration
-group messaging: sending a message from one sender to many receivers
-
-Virtual synchrony is a model for group messaging.  This is often confused
-with particular implementations of virtual synchrony.  Try to focus on
-what virtual syncrhony provides, not how it provides it, unless interested
-in working on the group messaging interface of corosync.
-
-Virtual synchrony provides several advantages:
-
- * integrated membership
- * strong membership guarantees
- * agreed ordering of delivered messages
- * same delivery of configuration changes and messages on every node
- * self-delivery
- * reliable communication in the face of unreliable networks
- * recovery of messages sent within a configuration where possible
- * use of network multicast using standard UDP/IP
-
-Integrated membership allows the group messaging interface to give
-configuration change events to the API services.  This is obviously beneficial
-to the cluster membership service (and its respective API0, but is helpful
-to other services as described later.
-
-Strong membership guarantees allow a distributed application to make decisions
-based upon the configuration (membership).  Every service in corosync registers
-a configuration change function.  This function is called whenever a
-configuration change occurs.  The information passed is the current processors,
-the processors that have left the configuration, and the processors that have
-joined the configuration.  This information is then used to make decisions
-within a distributed state machine.  One example usage is that an AMF component
-running a specific processor has left the configuration, so failover actions
-must now be taken with the new configuration (and known components).
-
-Virtual synchrony requires that messages may be delivered in agreed order.
-FIFO order indicates that one sender and one receiver agree on the order of
-messages sent.  Agreed ordering takes this requirement to groups, requiring that
-one sender and all receivers agree on the order of messages sent.
-
-Consider a lock service.  The service is responsible for arbitrating locks
-between multiple processors in the system.  With fifo ordering, this is very
-difficult because a request at about the same time for a lock from two seperate
-processors may arrive at all the receivers in different order.  Agreed ordering
-ensures that all the processors are delivered the message in the same order.
-In this case the first lock message will always be from processor X, while the
-second lock message will always be from processor Y.   Hence the first request
-is always honored by all processors, and the second request is rejected (since
-the lock is taken).  This is how race conditions are avoided in distributed
-systems.
-
-Every processor is delivered a configuration change and messages within a
-configuration in the same order.  This ensures that any distributed state
-machine will make the same decisions on every processor within the
-configuration.  This also allows the configuration and the messages to be
-considered when making decisions.
-
-Virtual synchrony requires that every node is delivered messages that it
-sends.  This enables the logic to be placed in one location (the handler
-for the delivery of the group message) instead of two seperate places.  This
-also allows messages that are sent to be ordered in the stream of other
-messages within the configuration.
-
-Certain guarantees are required by virtual synchrony.  If a message is sent,
-it must be delivered by every processor unless that processor fails.  If a
-particular processor fails, a configuration change occurs creating a new
-configuration under which a new set of decisions may be made.  This implies
-that even unreliable networks must reliably deliver messages.   The
-mplementation in corosync works on unreliable as well as reliable networks.
-
-Every message sent must be delivered, unless a configuration change occurs.
-In the case of a configuration change, every message that can be recovered
-must be recovered before the new configuration is installed.  Some systems
-during partition won't continue to recover messages within the old
-configuration even though those messages can be recovered.  Virtual synchrony
-makes that impossible, except for those members that are no longer part
-of a configuration.
-
-Finally virtual syncrhony takes advantage of hardware multicast to avoid
-duplicated packets and scale to large transmit rates.  On 100mbit network,
-corosync can approach wire speeds depending on the number of messages queued
-for a particular processor.
-
-What does all of this mean for the developer?
-
- * messages are delivered reliably
- * messages are delivered in the same order to all nodes
- * configuration and messages can both be used to make decisions
-
--------------------------------------------------------------------------------
- adding libraries
--------------------------------------------------------------------------------
-
-The first stage in adding a library to the system is to develop the library.
-
-Library code should follow these guidelines:
-
- * use SA Forum coding style for SA Forum APIs to aid in debugging
- * use corosync coding guidelines for APIs that are not SA Forum that
-   are to be merged into the corosync tree.
- * implement all library code within one file named after the api.
-   examples are ckpt.c, clm.c, amf.c.
- * use parallel structure as much as possible between different APIs
- * make use of utility services provided by util.c.
- * if something is needed that is generic and useful by all services,
-   submit patches for other libraries to use these services.
- * use the reference counting handle manager for handle management.
-
-------------------
- Version checking
-------------------
-
-struct saVersionDatabase {
-	int versionCount;
-	SaVersionT *versionsSupported;
-};
-
-The versionCount number describes how many entries are in the version database.
-The versionsSupported member is an array of SaVersionT describing the acceptable
-versions this API supports.
-
-An api developer specifies versions supported by adding the following C
-code to the library file:
-
-/*
- * Versions supported
- */
-static SaVersionT clmVersionsSupported[] = {
-	{ 'B', 1, 1 },
-	{ 'b', 1, 1 }
-};
-
-static struct saVersionDatabase clmVersionDatabase = {
-	sizeof (clmVersionsSupported) / sizeof (SaVersionT),
-	clmVersionsSupported
-};
-
-After this is specified, the following API is used to check versions:
-
-SaErrorT
-saVersionVerify (
-	struct saVersionDatabase *versionDatabase,
-	const SaVersionT *version);
-
-An example usage of this is
-	SaErrorT error;
-
-	error = saVersioNVerify (&clmVersionDatabase, version);
-
-	where version is a pointer to an SaVersionT passed into the API.
-
-error will return SA_OK if the version is valid as specified in the
-version database.
-
-------------------
- Handle Instances
-------------------
-
-Every handle instance is stored in a handle database.  The handle database
-stores instance information for every handle used by libraries.  The system
-includes reference counting and is safe for use in threaded applications.
-
-The handle database structure is:
-
-struct saHandleDatabase {
-	unsigned int handleCount;
-	struct saHandle *handles;
-	pthread_mutex_t mutex;
-	void (*handleInstanceDestructor) (void *);
-};
-
-handleCount is the number of handles
-handles is an array of handles
-mutex is a pthread mutex used to mutually exclude access to the handle db
-handleInstanceDestructor is a callback that is called when the handle
-	should be freed because its reference count as dropped to zero.
-
-The handle database is defined in a library as follows:
-
-static void clmHandleInstanceDestructor (void *);
-
-static struct saHandleDatabase clmHandleDatabase = {
-	.handleCount			= 0,
-	.handles			= 0,
-	.mutex		 		=  PTHREAD_MUTEX_INITIALIZER,
-	.handleInstanceDestructor	= clmHandleInstanceDestructor
-};
-
-There are several APIs to access the handle database:
-
-SaErrorT
-saHandleCreate (
-	struct saHandleDatabase *handleDatabase,
-	int instanceSize,
-	int *handleOut);
-
-Creates an instance of size instanceSize in the handleDatabase paraemter
-returning the handle number in handleOut.  The handle instance reference
-count starts at the value 1.
-
-SaErrorT
-saHandleDestroy (
-	struct saHandleDatabase *handleDatabase,
-	unsigned int handle);
-
-Destroys further access to the handle.  Once the handle reference count
-drops to zero, the database destructor is called for the handle.  The handle
-instance reference count is decremented by 1.
-
-SaErrorT
-saHandleInstanceGet (
-	struct saHandleDatabase *handleDatabase,
-	unsigned int handle,
-	void **instance);
-
-Gets an instance specified handle from the handleDatabase and returns
-it in the instance member.  If the handle is valid SA_OK is returned
-otherwise an error is returned.  This is used to ensure a handle is
-valid.  Eveyr get call increases the reference count on a handle instance
-by one.
-
-SaErrorT
-saHandleInstancePut (
-	struct saHandleDatabase *handleDatabase,
-	unsigned int handle);
-
-Decrements the reference count by 1.  If the reference count indicates
-the handle has been destroyed, it will then be removed from the database
-and the destructor called on the instance data.  The put call takes care
-of freeing the handle instance data.
-
-Create a data structure for the instance, and use it within the libraries
-to store state information about the instance.  This information can be
-the handle, a mutex for protecting I/O, a queue for queueing async messages
-or whatever is needed by the API.
-
------------------------------------
- communicating with the executive
------------------------------------
-
-A service connection is created with the following API;
-
-SaErrorT
-saServiceConnect (
-	int *responseOut,
-	int *callbackOut,
-	enum service_types service);
-
-
-The responseOut parameter specifies the file descriptor where response messages
-will be delivered.  The callback out parameter describes the file descriptor
-where callback messages are delivered.
-
-The service specifies the service to use.
-
-Messages are sent and received from the executive with the following functions:
-
-SaAisErrorT saSendMsgRetry (
-	int s,
-	struct iovec *iov,
-	unsigned int iov_len);
-
-the s member is the socket to use retrieved with saServiceConnect
-The iov is the iovector used to send a message.
-the iov_len is the number of elements in iov.
-
-This sends an IO-vectorized message.
-
-SaErrorT
-saSendRetry (
-	int s,
-	const void *msg,
-	size_t len,
-	int flags);
-
-the s member is the socket to use retrieved with saServiceConnect
-the msg member is a pointer to the message to send to the service
-the len member is the length of the message to send
-the flags parameter is the flags to use with the sendmsg system call
-
-
-This sends a data blob to the exective.
-
-A message is received from the executive with the function:
-
-SaErrorT
-saRecvRetry (
-	int s,
-	void *msg,
-	size_t len,
-	int flags);
-
-the s member is the socket to use retrieved with saServiceConnect
-the msg member is a pointer to the message to receive to the service
-the len member is the length of the message to receive
-the flags parameter is the flags to use with the sendmsg system call
-
-A message may be send and a reply waited for with the following function:
-SaAisErrorT saSendMsgReceiveReply (
-        int s,
-        struct iovec *iov,
-        unsigned int iov_len,
-        void *responseMessage,
-        int responseLen)
-
-s is the socket to send and receive the response.
-iov is the iovector to send.
-iov_len is the number of elements in iov.
-responseMessage is the data block used to store the response.
-responesLen is the length of the data block that is expected to be received.
-
-Waiting for a file descriptor using poll systemcall is done with the api:
-
-SaErrorT
-saPollRetry (
-	struct pollfd *ufds,
-	unsigned int nfds,
-	int timeout);
-
-where the parameters are the standard poll parameters.
-
-Messages can be received out of order searching for a specific message id with:
-
-----------
- messages
-----------
-Please follow the style of the messages.  It makes debugging much easier
-if parallel style is used.
-
-An service should be added to service_types enumeration in ipc_gen or in the
-case of an external project, a number should be registered with the project.
-
-enum service_types {
-        EVS_SERVICE = 0,
-        CLM_SERVICE = 1,
-        AMF_SERVICE = 2,
-        CKPT_SERVICE = 3,
-        EVT_SERVICE = 4,
-        LCK_SERVICE = 5,
-        MSG_SERVICE = 6,
-        CFG_SERVICE = 7,
-        CPG_SERVICE = 8
-};
-
-These are the request CLM message identifiers:
-
-Each library should have an ipc_APINAME.h file in include.  It should define
-request types and response types.
-
-enum req_clm_types {
-	MESSAGE_REQ_CLM_TRACKSTART = 0,
-	MESSAGE_REQ_CLM_TRACKSTOP = 1,
-	MESSAGE_REQ_CLM_NODEGET = 2,
-	MESSAGE_REQ_CLM_NODEGETASYNC = 3
-};
-
-These are the response CLM message identifiers:
-
-enum res_clm_types {
-        MESSAGE_RES_CLM_TRACKCALLBACK = 0,
-        MESSAGE_RES_CLM_TRACKSTART = 1,
-        MESSAGE_RES_CLM_TRACKSTOP = 2,
-        MESSAGE_RES_CLM_NODEGET = 3,
-        MESSAGE_RES_CLM_NODEGETASYNC = 4,
-        MESSAGE_RES_CLM_NODEGETCALLBACK = 5
-};
-
-A request header should be placed at the front of every message send by
-the library.
-
-typedef struct {
-        int size __attribute__((aligned(8)));
-        int id __attribute__((aligned(8)));
-} mar_req_header_t __attribute__((aligned(8)));
-
-There is also a response message header which should start every response
-message:
-
-typedef struct {
-        int size; __attribute__((aligned(8)))
-        int id __attribute__((aligned(8)));
-        SaAisErrorT error __attribute__((aligned(8)));
-} mar_res_header_t __attribute__((aligned(8)));
-
-the error parameter is used to pass errors from the executive to the library,
-including SA_ERR_TRY_AGAIN for flow control, which is described later.
-
-This is described later:
-
-typedef struct {
-        mar_uint32_t nodeid __attribute__((aligned(8)));
-        void *conn __attribute__((aligned(8)));
-} mar_message_source_t __attribute__((aligned(8)));
-
-This is the MESSAGE_REQ_CLM_TRACKSTART message id above:
-
-struct req_clm_trackstart {
-	mar_req_header_t header;
-	SaUint8T trackFlags;
-	SaClmClusterNotificationT *notificationBufferAddress;
-	SaUint32T numberOfItems;
-};
-
-The saClmClusterTrackStart api should create this message and send it to the
-executive.
-
-responses should be of:
-
-struct res_clm_trackstart
-
-------------
- some notes
-------------
-* Avoid doing anything tricky in the library itself.  Let the executive
-  handler do all of the work of the system.  minimize what the API does.
-* Once an api is developed, it must be added to the makefile.  Just add
-  a line for the file to EXECOBJS build line.
-* protect I/O send/recv with a mutex.
-* always look at other libraries when there is a question about how to
-  do something.  It has likely been thought out in another library.
-
--------------------------------------------------------------------------------
- adding services
--------------------------------------------------------------------------------
-Services are defined by service handlers and messages described in
-include/ipc_SERVICE.h.  These two peices of information are used by the
-executive to dispatch the correct messages to the correct receipients.
-
--------------------------------
- the service handler structure
--------------------------------
-
-A service is added by defining a structure defined in exec/service.h.  The
-structure is a little daunting:
-
-struct libais_handler {
-	int (*libais_handler_fn) (void *conn, void *msg);
-	int response_size;
-	int response_id;
-	enum corosync_flow_control flow_control;
-};
-
-The response_size, response_id, and flow_control for a library handler are
-used for flow control.  A response message will be sent to the library of the
-size response_size, with the header id of response_id if the totem message
-queue is full.  Some library APIs may not need to block in this condition
-(because they don't have to use totem), so they should specify
-COROSYNC_FLOW_CONTROL_NOT_REQUIREDin the flow control field.
-
-The libais_handler_fn is a function to be called when the library handler is
-requested to be executed.
-
-struct corosync_exec_handler {
-	void (*exec_handler_fn) (void *msg, unsigned int nodeid);
-	void (*exec_endian_convert_fn) (void *msg);
-};
-
-The exec_handler_fn is a function to be called when the executive handler is
-requested to execute.
-
-The exec_endian_convert_fn is a function to be called to convert the endianess
-of the executive message.  Note messages are not stored in big or little endian
-format before transmit.  Instead they are transmitted in either big endian or
-little endian depending on the byte order of the transmitter and converted to
-the host machine order on receipt of the message.
-
-struct corosync_service_handler {
-	unsigned char *name;
-	unsigned short id;
-	unsigned int private_data_size;
-	int (*lib_init_fn) (void *conn);
-	int (*lib_exit_fn) (void *conn);
-	struct corosync_lib_handler *lib_service;
-	int lib_service_count;
-	struct corosync_exec_handler *exec_service;
-	int (*exec_init_fn) (struct objdb_iface_ver0 *);
-	int (*config_init_fn) (struct objdb_iface_ver0 *);
-	void (*exec_dump_fn) (void);
-	int exec_service_count;
-	void (*confchg_fn) (
-		enum totem_configuration_type configuration_type,
-		const unsigned int *member_list, size_t member_list_entries,
-		const unsigned int *left_list, size_t left_list_entries,
-		const unsigned int *joined_list, size_t joined_list_entries,
-		const struct memb_ring_id *ring_id);
-	void (*sync_init) (void);
-	int (*sync_process) (void);
-	void (*sync_activate) (void);
-	void (*sync_abort) (void);
-};
-
-name is the name of the service.
-
-id is the identifier of the service.
-
-private_data_size is the size of the private data used by the connection
-which the library and executive handlers can reference.
-
-lib_init_fn is the function executed when a library connection is made to
-the service handler.
-
-lib_exit_fn is the function executed when a library connection is exited
-either because the application closed the file descriptor, or the OS
-closed the file descriptor.
-
-lib_service is an array of corosync_lib_handler data structures which define
-the library service handler.
-
-lib_service_count is the number of elements in lib_service.
-
-exec_service is an array of corosync_exec_handler data structures which define
-the executive service handler.
-
-exec_init_fn is a function used to initialize the executive service.  This
-is only called once.
-
-config_init_fn is called to parse config files and populate the object
-database.
-
-exec_dump_fn is called when SIGUSR2 is sent to the executive to dump the
-current state of the service.
-
-exec_service_count is the number of entries in the exec_service array.
-
-confchg_fn is called every time a configuration change occurs.
-
-sync_init is called when the service should begin synchronization.
-
-sync_process is called to process synchronization messages.
-
-sync_activate is called to activate the current service synchronization.
-
-sync_abort is called to abort the current service synchronization.
-
---------------
- flow control
---------------
-The totem protocol includes flow control so that it doesn't send too many
-messages when the network is completely full.  But the library can
-still send messages to the executive much faster then the executive can send
-them over totem.  So the library relies on the group messaging flow control to
-control flow of messages sent from the library.  If the totem queues are full,
-no more messages may be sent, so the executive in ipc.c automatically detects
-this scenario and returns an SA_ERR_TRY_AGAIN error.
-
-When a library gets SA_ERR_TRY_AGAIN, the library may either retry, or return
-this error to the user if the error is allowed by the API definitions.  The
-The other information is critical to ensuring that the library reads the correct
-message and size of message.  Make sure the libais_handler matches the messages
-used in the handler function.
-
-------------------------------------------------
- dynamically linking the service handler plugin
-------------------------------------------------
-
-The service handler needs some special magic to dynamically be linked into
-corosync.
-
-/*
- * Dynamic loader definition
- */
-static struct corosync_service_handler *clm_get_service_handler_ver0 (void);
-
-static struct corosync_service_handler_iface_ver0 clm_service_handler_iface = {
-        .corosync_get_service_handler_ver0       = clm_get_service_handler_ver0
-};
-
-static struct lcr_iface corosync_clm_ver0[1] = {
-        {
-                .name                   = "corosync_clm",
-                .version                = 0,
-                .versions_replace       = 0,
-                .versions_replace_count = 0,
-                .dependencies           = 0,
-                .dependency_count       = 0,
-                .constructor            = NULL,
-                .destructor             = NULL,
-                .interfaces             = NULL
-        }
-};
-
-static struct lcr_comp clm_comp_ver0 = {
-        .iface_count                    = 1,
-        .ifaces                         = corosync_clm_ver0
-};
-
-static struct corosync_service_handler *clm_get_service_handler_ver0 (void)
-{
-        return (&clm_service_handler);
-}
-
-__attribute__ ((constructor)) static void clm_comp_register (void) {
-        lcr_interfaces_set (&corosync_clm_ver0[0], &clm_service_handler_iface);
-
-        lcr_component_register (&clm_comp_ver0);
-}
-
-Once this code is added (substitute clm for the service being implemented),
-the service will be loaded if its in the default services list.
-
-The default service list is specified in service.c:default_services.  If
-creating an external plugin, there are configuration parameters which may
-be used to add your plugin into the corosync scanning of plugins.
-
----------------------------------
- Connection specific information
----------------------------------
-Every connection may have specific connection information if private data
-is greater then zero for the service handler.  This is used to allow each
-library connection to maintain private state to that connection.  The private
-data for a connection can be retrieved with:
-struct service_pd service_pd = (struct service_pd *)corosync_conn_private_data_get (conn);
-
-where service is the name of the service implemented and conn is the connection
-information likely passed into the library handler or stored in a
-message_source structure for later use by an executive handler.
-
-------------------------------
- sending responses to the api
-------------------------------
-
-A message is sent to the library from the executive message handler using
-the function:
-
-extern int corosync_conn_send_response (void *conn_info, void *msg,
-	int mlen);
-
-conn_info is passed into the library message handler or stored in the
-executive message.  This member describes the connection to send the response.
-
-msg is the message to send
-mlen is the length of the message to send
-
-Keep in mind that struct res_message should be at the beginning of the response
-message so that it follows the style used in the rest of corosync.
-
---------------------------------------------
- deferring response to an executive message
---------------------------------------------
-
-The message source structure is used to store information about the source of a
-message so a later executive message can respond to a library request.  In
-a library handler, the source field should be set up with:
-
-message_source_set (&req_exec_ZZZZZZZ.source, conn);
-gmi_mcast (req_exec_ZZZZZZZ)
-
-In this case conn_info is passed into the library message handler
-
-Then the executive message handler determines if this processor is responsible
-for responding:
-
-if (message_source_is_local (conn)) {
-	corosync_conn_send_response ();
-
-}
-
----------------
- Using totempg
----------------
-To send a message to every processor and the local processor for self
-delivery according to virtual synchrony semantics use:
-
-The totempg interface supports multiple users at one time and if you need
-to use a full totempg interface (defined in totempg.h) please ask for
-assistance on the mailing list.  If you simply want to use multicast
-transmissions in corosync, do the following:
-
-       assert (totempg_groups_mcast_joined (corosync_group_handle, &req_exec_clm_iovec, 1, TOTEMPG_AGREED) == 0);
-
------------------
- library handler
------------------
-Every library handler has the prototype:
-
-static int message_handler_req_clm_init (void *conn, void *msg);
-
-The start of the handler function should look something like this:
-
-int message_handler_req_clm_trackstart (void *conn *conn,
-	void *msg)
-{
-        struct req_clm_trackstart *req_clm_trackstart =
-		(struct req_clm_trackstart *)message;
-
- { package up library handler message into executive message }
- { multicast message using totempg interface }
-}
-
-This assigns the void *message to a structure that can be used by the
-library handler.
-
-The conn field is used to indicate where the response should respond to.
-Use the tricks described in deferring a response to the executive handler to
-have the executive handler respond to the message.
-
-avoid doing anything tricky in a library handler.  Do all the work in the
-executive handler at first.  If later, it is possible to optimize, optimize
-away.
-
--------------------
- executive handler
--------------------
-Every executive handler has the prototype:
-
-static int message_handler_req_exec_clm_nodejoin (void *msg,
-	unsigned int nodeid);
-
-The start of the handler function should look something like this:
-
-static int message_handler_req_exec_clm_nodejoin (void *msg,
-	unsigned int nodeid);
-{
-        struct req_exec_clm_nodejoin *req_exec_clm_nodejoin = (struct req_exec_clm_nodejoin *)message;
-
- { do real work of executing request, this is done on every node }
-}
-
-The conn_info structure is not available.  If it is needed, it can be stored
-in the message sent by the library message handler in a source structure.
-
-The msg field contains the message sent by the library handler
-
-The nodeid is a unique node identifier of the node that originated the message.
-
---------------------
- the libais_init_fn
---------------------
-This should be used to initialize any state for the connection.
-
---------------------
- the libais_exit_fn
---------------------
-This function is called every time a service connection is disconnected by
-the executive.  Free memory, change structures, or whatever work needs to
-be done to clean up.
-
-If the exit_fn couldn't complete because it is waiting for some event, it may
-return -1, which will allow the executive to make some forward progress.  Then
-exit_fn will be called again.  Return 0 when the exit was completed.  This is
-most useful when toteom should be used to queue a message, but the queue is
-full.  In this case, waiting a few more seconds may open up the queue, so
-return -1, and then the executive will try again to call exit_fn.  Do NOT
-return -1 forever or the ais executive will spin.
-
-If -1 is returned, ENSURE that the state of the library hasn't changed so much that
-exit_fn cannot be called again.  If exit_fn returns -1, it WILL be called again
-so expect it in the code.
-
-----------------
- the confchg_fn
-----------------
-This function is called whenever a configuration change occurs.  Some
-services may not need this function, while others may.  This is a good way
-to sync up joining nodes with the current state of the information stored
-on a particular processor.
-
--------------------------------------------------------------------------------
-Final comments
--------------------------------------------------------------------------------
-GDB is your friend, especially the "where" command.  But it stops execution.
-This has a nasty side effect of killing the current configuration.  In this
-case GDB may become your enemy.
-
-printf is your friend when GDB is your enemy.
-
-If stuck, ask on the mailing list, send your patches.  Alot of time has been
-spent designing corosync, and even more time debugging it.  There are people
-that can help you debug problems, especially around things like message
-delivery.
-
-Submit patches early to get feedback, especially around things like parallel
-style.  Parallel style is very important to ensure maintainability by the
-corosync community.
-
-If this document is wrong or incomplete, complain so we can get it fixed
-for other people.
-
-Have fun!