corosync/man/corosync.conf.5

.\"/*
.\" * Copyright (c) 2005 MontaVista Software, Inc.
.\" * Copyright (c) 2006-2012 Red Hat, Inc.
.\" *
.\" * All rights reserved.
.\" *
.\" * Author: Steven Dake (sdake@redhat.com)
.\" *
.\" * This software licensed under BSD license, the text of which follows:
.\" *
.\" * Redistribution and use in source and binary forms, with or without
.\" * modification, are permitted provided that the following conditions are met:
.\" *
.\" * - Redistributions of source code must retain the above copyright notice,
.\" *   this list of conditions and the following disclaimer.
.\" * - Redistributions in binary form must reproduce the above copyright notice,
.\" *   this list of conditions and the following disclaimer in the documentation
.\" *   and/or other materials provided with the distribution.
.\" * - Neither the name of the MontaVista Software, Inc. nor the names of its
.\" *   contributors may be used to endorse or promote products derived from this
.\" *   software without specific prior written permission.
.\" *
.\" * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
.\" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
.\" * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
.\" * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
.\" * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
.\" * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
.\" * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
.\" * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
.\" * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
.\" * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
.\" * THE POSSIBILITY OF SUCH DAMAGE.
.\" */
.TH COROSYNC_CONF 5 2012-10-10 "corosync Man Page" "Corosync Cluster Engine Programmer's Manual"
.SH NAME
corosync.conf - corosync executive configuration file

.SH SYNOPSIS
/etc/corosync/corosync.conf

.SH DESCRIPTION
The corosync.conf instructs the corosync executive about various parameters
needed to control the corosync executive.  Empty lines and lines starting with
# character are ignored.  The configuration file consists of bracketed top level
directives.  The possible directive choices are:

.TP
totem { }
This top level directive contains configuration options for the totem protocol.
.TP
logging { }
This top level directive contains configuration options for logging.
.TP
quorum { }
This top level directive contains configuration options for quorum.
.TP
nodelist { }
This top level directive contains configuration options for nodes in cluster.
.TP
qb { }
This top level directive contains configuration options related to libqb.
.TP
resources { }
This top level directive contains configuration options for resources.

.PP
The 
.B interface sub-directive of totem is optional for UDP and knet transports.

For knet, multiple interface subsections define parameters for each knet link on the
system.

For UDPU an interface section is not needed and it is recommended that the nodelist
is used to define cluster nodes.

.TP
linknumber
This specifies the link number for the interface.  When using the knet
protocol, each interface should specify separate link numbers to uniquely
identify to the membership protocol which interface to use for which link. 
The linknumber must start at 0. For UDP the only supported linknumber is 0.

.TP
knet_link_priority
This specifies the priority for the link when knet is used in 'passive'
mode. (see link_mode below)

.TP
knet_ping_interval
This specifies the interval between knet link pings. 
(default 1000 ms)

.TP
knet_ping_timeout
If no ping is received within this time, the knet link is declared dead. 
(default 2000 ms)

.TP
knet_ping_precision
How many values of latency are used to calculate
the average link latency. (default 2048 samples)

.TP
knet_pong_count
How many valid ping/pongs before a link is marked UP. (default 5)
.TP

knet_transport
Which IP transport knet should use. valid values are "sctp" or "udp". (default: udp)

.TP
bindnetaddr (udp only)
This specifies the network address the corosync executive should bind
to when using udp.

bindnetaddr (udp only) 
should be an IP address configured on the system, or a network
address.

For example, if the local interface is 192.168.5.92 with netmask
255.255.255.0, you should set bindnetaddr to 192.168.5.92 or 192.168.5.0.
If the local interface is 192.168.5.92 with netmask 255.255.255.192,
set bindnetaddr to 192.168.5.92 or 192.168.5.64, and so forth.

This may also be an IPV6 address, in which case IPV6 networking will be used.
In this case, the exact address must be specified and there is no automatic
selection of the network interface within a specific subnet as with IPv4.

If IPv6 networking is used, the nodeid field in nodelist must be specified.

.TP
broadcast (udp only)
This is optional and can be set to yes.  If it is set to yes, the broadcast
address will be used for communication.  If this option is set, mcastaddr
should not be set.

.TP
mcastaddr (udp only)
This is the multicast address used by corosync executive.  The default
should work for most networks, but the network administrator should be queried
about a multicast address to use.  Avoid 224.x.x.x because this is a "config"
multicast address.

This may also be an IPV6 multicast address, in which case IPV6 networking
will be used.  If IPv6 networking is used, the nodeid field in nodelist must
be specified.

It's not necessary to use this option if cluster_name option is used. If both options
are used, mcastaddr has higher priority.

.TP
mcastport (udp only)
This specifies the UDP port number.  It is possible to use the same multicast
address on a network with the corosync services configured for different
UDP ports.
Please note corosync uses two UDP ports mcastport (for mcast receives) and 
mcastport - 1 (for mcast sends).
If you have multiple clusters on the same network using the same mcastaddr 
please configure the mcastports with a gap.

.TP
ttl (udp only)
This specifies the Time To Live (TTL). If you run your cluster on a routed
network then the default of "1" will be too small. This option provides
a way to increase this up to 255. The valid range is 0..255.

.PP
.PP
Within the
.B totem
directive, there are seven configuration options of which one is required,
five are optional, and one is required when IPV6 is configured in the interface
subdirective.  The required directive controls the version of the totem
configuration.  The optional option unless using IPV6 directive controls
identification of the processor.  The optional options control secrecy and
authentication, the network mode of operation and maximum network MTU
field.

.TP
version
This specifies the version of the configuration file.  Currently the only
valid version for this directive is 2.

.PP
clear_node_high_bit
This configuration option is optional and is only relevant when no nodeid is
specified.  Some corosync clients require a signed 32 bit nodeid that is greater
than zero however by default corosync uses all 32 bits of the IPv4 address space
when generating a nodeid.  Set this option to yes to force the high bit to be
zero and therefor ensure the nodeid is a positive signed 32 bit integer.

WARNING: The clusters behavior is undefined if this option is enabled on only
a subset of the cluster (for example during a rolling upgrade).

.TP
crypto_hash
This specifies which HMAC authentication should be used to authenticate all
messages. Valid values are none (no authentication), md5, sha1, sha256,
sha384 and sha512. Encrypted transmission is only supported for
the knet transport.

The default is sha1.

.TP
crypto_cipher
This specifies which cipher should be used to encrypt all messages.
Valid values are none (no encryption), aes256, aes192, aes128 and 3des.
Enabling crypto_cipher, requires also enabling of crypto_hash. Encrypted
transmission is only supported for the knet transport.

The default is aes256.

.TP
link_mode
This specifies the Kronosnet mode, which may be passive, active, or
rr (round-robin). 
.B passive:
the active link with the lowest priority will be used. If one or more 
links share the same priority the one with the lowest link ID will
be used.
.B active: 
All active links will be used simultaneously to send traffic.
link priority is ignored.
.B rr:
Round-Robin policy. Each packet will be sent to the next active link in 
order.

If only one interface directive is specified, passive is automatically chosen.

The maximum number of interface directives that is allowed with Kronosnet
is 8. For other transports it is 1.

When using multiple interfaces, make sure to use different multicast
address/port (port for same address must differ by at least two) pair
for each interface (this is checked by parser).

.TP
netmtu
This specifies the network maximum transmit unit.  To set this value beyond
1500, the regular frame MTU, requires ethernet devices that support large, or
also called jumbo, frames.  If any device in the network doesn't support large
frames, the protocol will not operate properly.  The hosts must also have their
mtu size set from 1500 to whatever frame size is specified here.

Please note while some NICs or switches claim large frame support, they support
9000 MTU as the maximum frame size including the IP header.  Setting the netmtu
and host MTUs to 9000 will cause totem to use the full 9000 bytes of the frame.
Then Linux will add a 18 byte header moving the full frame size to 9018.  As a
result some hardware will not operate properly with this size of data.  A netmtu
of 8982 seems to work for the few large frame devices that have been tested.
Some manufacturers claim large frame support when in fact they support frame
sizes of 4500 bytes.

When sending multicast traffic, if the network frequently reconfigures, chances are
that some device in the network doesn't support large frames.

Choose hardware carefully if intending to use large frame support.

The default is 1500.

.TP
transport
This directive controls the transport mechanism used.  
The default is knet.  The transport type can also be set to udpu or udp.
Only knet allows crypto or multiple interfaces per node.

.TP
cluster_name
This specifies the name of cluster and it's used for automatic generating
of multicast address.

.TP
config_version
This specifies version of config file. This is converted to unsigned 64-bit int.
By default it's 0. Option is used to prevent joining old nodes with not
up-to-date configuration. If value is not 0, and node is going for first time
(only for first time, join after split doesn't follow this rules)
from single-node membership to multiple nodes membership, other nodes
config_versions are collected. If current node config_version is not
equal to highest of collected versions, corosync is terminated.

.TP
ip_version
Specifies version of IP to use for communication. Value can be one of
ipv4 or ipv6. Default (if unspecified) is ipv4.


Within the
.B totem
directive, there are several configuration options which are used to control
the operation of the protocol.  It is generally not recommended to change any
of these values without proper guidance and sufficient testing.  Some networks
may require larger values if suffering from frequent reconfigurations.  Some
applications may require faster failure detection times which can be achieved
by reducing the token timeout.

.TP
token
This timeout is used directly or as a base for real token timeout calculation (explained in
.B token_coefficient
section). Token timeout specifies in milliseconds until a token loss is declared after not
receiving a token.  This is the time spent detecting a failure of a processor
in the current configuration.  Reforming a new configuration takes about 50
milliseconds in addition to this timeout.

For real token timeout used by totem it's possible to read cmap value of
.B runtime.config.token
key.

The default is 1000 milliseconds.

.TP
token_coefficient
This value is used only when
.B nodelist
section is specified and contains at least 3 nodes. If so, real token timeout
is then computed as token + (number_of_nodes - 2) * token_coefficient.
This allows cluster to scale without manually changing token timeout
every time new node is added. This value can be set to 0 resulting
in effective removal of this feature.

The default is 650 milliseconds.

.TP
token_retransmit
This timeout specifies in milliseconds after how long before receiving a token
the token is retransmitted.  This will be automatically calculated if token
is modified.  It is not recommended to alter this value without guidance from
the corosync community.

The default is 238 milliseconds.

.TP
hold
This timeout specifies in milliseconds how long the token should be held by
the representative when the protocol is under low utilization.   It is not
recommended to alter this value without guidance from the corosync community.

The default is 180 milliseconds.

.TP
token_retransmits_before_loss_const
This value identifies how many token retransmits should be attempted before
forming a new configuration.  If this value is set, retransmit and hold will
be automatically calculated from retransmits_before_loss and token.

The default is 4 retransmissions.

.TP
join
This timeout specifies in milliseconds how long to wait for join messages in
the membership protocol.

The default is 50 milliseconds.

.TP
send_join
This timeout specifies in milliseconds an upper range between 0 and send_join
to wait before sending a join message.  For configurations with less than
32 nodes, this parameter is not necessary.  For larger rings, this parameter
is necessary to ensure the NIC is not overflowed with join messages on
formation of a new ring.  A reasonable value for large rings (128 nodes) would
be 80msec.  Other timer values must also change if this value is changed.  Seek
advice from the corosync mailing list if trying to run larger configurations.

The default is 0 milliseconds.

.TP
consensus
This timeout specifies in milliseconds how long to wait for consensus to be
achieved before starting a new round of membership configuration.  The minimum
value for consensus must be 1.2 * token.  This value will be automatically
calculated at 1.2 * token if the user doesn't specify a consensus value.

For two node clusters, a consensus larger than the join timeout but less than
token is safe.  For three node or larger clusters, consensus should be larger
than token.  There is an increasing risk of odd membership changes, which still
guarantee virtual synchrony,  as node count grows if consensus is less than
token.

The default is 1200 milliseconds.

.TP
merge
This timeout specifies in milliseconds how long to wait before checking for
a partition when no multicast traffic is being sent.  If multicast traffic
is being sent, the merge detection happens automatically as a function of
the protocol.

The default is 200 milliseconds.

.TP
downcheck
This timeout specifies in milliseconds how long to wait before checking
that a network interface is back up after it has been downed.

The default is 1000 milliseconds.

.TP
fail_recv_const
This constant specifies how many rotations of the token without receiving any
of the messages when messages should be received may occur before a new
configuration is formed.

The default is 2500 failures to receive a message.

.TP
seqno_unchanged_const
This constant specifies how many rotations of the token without any multicast
traffic should occur before the hold timer is started.

The default is 30 rotations.

.TP
heartbeat_failures_allowed
[HeartBeating mechanism]
Configures the optional HeartBeating mechanism for faster failure detection. Keep in
mind that engaging this mechanism in lossy networks could cause faulty loss declaration
as the mechanism relies on the network for heartbeating.

So as a rule of thumb use this mechanism if you require improved failure in low to
medium utilized networks.

This constant specifies the number of heartbeat failures the system should tolerate
before declaring heartbeat failure e.g 3. Also if this value is not set or is 0 then the
heartbeat mechanism is not engaged in the system and token rotation is the method
of failure detection

The default is 0 (disabled).

.TP
max_network_delay
[HeartBeating mechanism]
This constant specifies in milliseconds the approximate delay that your network takes
to transport one packet from one machine to another. This value is to be set by system
engineers and please don't change if not sure as this effects the failure detection
mechanism using heartbeat.

The default is 50 milliseconds.

.TP
window_size
This constant specifies the maximum number of messages that may be sent on one
token rotation.  If all processors perform equally well, this value could be
large (300), which would introduce higher latency from origination to delivery
for very large rings.  To reduce latency in large rings(16+), the defaults are
a safe compromise.  If 1 or more slow processor(s) are present among fast
processors, window_size should be no larger than 256000 / netmtu to avoid
overflow of the kernel receive buffers.  The user is notified of this by
the display of a retransmit list in the notification logs.  There is no loss
of data, but performance is reduced when these errors occur.

The default is 50 messages.

.TP
max_messages
This constant specifies the maximum number of messages that may be sent by one
processor on receipt of the token.  The max_messages parameter is limited to
256000 / netmtu to prevent overflow of the kernel transmit buffers.

The default is 17 messages.

.TP
miss_count_const
This constant defines the maximum number of times on receipt of a token
a message is checked for retransmission before a retransmission occurs.  This
parameter is useful to modify for switches that delay multicast packets
compared to unicast packets.  The default setting works well for nearly all
modern switches.

The default is 5 messages.

.TP
knet_pmtud_interval
How often the knet PMTUd runs to look for network MTU changes.
Value in seconds, default: 60

.PP
Within the
.B logging
directive, there are several configuration options which are all optional.

.PP
The following 3 options are valid only for the top level logging directive:

.TP
timestamp
This specifies that a timestamp is placed on all log messages.

The default is off.

.TP
fileline
This specifies that file and line should be printed.

The default is off.

.TP
function_name
This specifies that the code function name should be printed.

The default is off.

.PP
The following options are valid both for top level logging directive
and they can be overridden in logger_subsys entries.

.TP
to_stderr
.TP
to_logfile
.TP
to_syslog
These specify the destination of logging output. Any combination of
these options may be specified. Valid options are
.B yes
and
.B no.

The default is syslog and stderr.

Please note, if you are using to_logfile and want to rotate the file, use logrotate(8)
with the option 
.B
copytruncate.
eg.
.ne 18
.RS
.nf
.ft CW
/var/log/corosync.log {
	missingok
	compress
	notifempty
	daily
	rotate 7
	copytruncate
}
.ft
.fi
.RE

.TP
logfile
If the
.B to_logfile
directive is set to
.B yes
, this option specifies the pathname of the log file.

No default.

.TP
logfile_priority
This specifies the logfile priority for this particular subsystem. Ignored if debug is on.
Possible values are: alert, crit, debug (same as debug = on), emerg, err, info, notice, warning.

The default is: info.

.TP
syslog_facility
This specifies the syslog facility type that will be used for any messages
sent to syslog. options are daemon, local0, local1, local2, local3, local4,
local5, local6 & local7.

The default is daemon.

.TP
syslog_priority
This specifies the syslog level for this particular subsystem. Ignored if debug is on.
Possible values are: alert, crit, debug (same as debug = on), emerg, err, info, notice, warning.

The default is: info.

.TP
debug
This specifies whether debug output is logged for this particular logger. Also can contain
value trace, what is highest level of debug information.

The default is off.

.PP
Within the
.B logging
directive, logger_subsys directives are optional.

.PP
Within the
.B logger_subsys
sub-directive, all of the above logging configuration options are valid and
can be used to override the default settings.
The subsys entry, described below, is mandatory to identify the subsystem.

.TP
subsys
This specifies the subsystem identity (name) for which logging is specified. This is the
name used by a service in the log_init() call. E.g. 'CPG'. This directive is
required.

.PP
Within the
.B quorum
directive it is possible to specify the quorum algorithm to use with the

.TP
provider
directive. At the time of writing only corosync_votequorum is supported.
See votequorum(5) for configuration options.

.PP
Within the
.B nodelist
directive it is possible to specify specific information about nodes in cluster. Directive
can contain only
.B node
sub-directive, which specifies every node that should be a member of the membership, and where
non-default options are needed. Every node must have at least ring0_addr field filled.

Every node that should be a member of the membership must be specified.

Possible options are:
.TP
ringX_addr
This specifies IP address of one of the nodes. X is link number. 

.TP
nodeid
This configuration option is required for each node. It is a 32 bit value 
specifying the node identifier delivered to the
cluster membership service. The node identifier value of zero is
reserved and should not be used.

.PP
Within the
.B qb
directive it is possible to specify options for libqb.

Possible option is:
.TP
ipc_type
This specifies type of IPC to use. Can be one of native (default), shm and socket.
Native means one of shm or socket, depending on what is supported by OS. On systems
with support for both, SHM is selected. SHM is generally faster, but need to allocate
ring buffer file in /dev/shm.

.PP
Within the
.B resources
directive it is possible to specify options for resources.

Possible option is:
.TP
watchdog_device
This configuration option controls which watchdog device corosync try to bind to.
The default value is /dev/watchdog. The value "no" disables the watchdog feature.

.SH "FILES"
.TP
/etc/corosync/corosync.conf
The corosync executive configuration file.

.SH "SEE ALSO"
.BR corosync_overview (8),
.BR votequorum (5),
.BR corosync-qdevice (8),
.BR logrotate (8)
.PP