votequorum: major rework to fix qdevice API and integration with core

qdevice is a very special node in the cluster and it adds a certain
amount of complexity and special cases across the code.

most of the qdevice data are shared across the cluster (name/votes)
but effectively each node has a different view of the qdevice
(registered/unregistered/voting/etc.)

with this change, we align the qdevice view across the node,
exchanging more data between nodes and we fix how qdevice behaves
and it is configured.

The only side effect is that the amount of data transmitted on wire
is slightly higher.

The qdevice API is still disabled by default. This means that
the amount of real changes in current code are a lot smaller
than it appears by this patch.

TODO: documentation/man pages needs to be updated once
      this change is in (and behavior finalized).

User visible changes:

- configuration (coroparse, exec/votequorum):
  the quorum device section is now standalone within the quorum.

  quorum {
    provider: corosync_votequorum
    device {
      model: (name)
      timeout: (millisec)
      votes:
    }
  }

  the keyword "model:" is mandatory to enable qdevice in configuration
  and should express the name of the script/daemon that will provide
  the qdevice. Looking into the future, an init script or systemd
  service will look for that name in /path/to/be/decided/name
  and start/stop qdevice.

  timeout: defines the maximum interval the qdevice implementation
  has available between poll (see votequorum_qdevice_poll.3) before
  the device is considered dead and votes discarded

  votes: is now a configuration parameter and not an API call.
  quorum devices don't care what they need to vote.
  votes is autocalculated when a nodelist is available and all
  nodes in the list vote 1. Otherwise this parameter is mandatory.

- configuration (exec/votequorum):
  startup and runtime configuration changes have been improved.
  errors at startup are considered fatal. errors at runtime
  have different exit paths.

  startup:

  * quorum.two_node and qdevice are incompatible.
  * quorum.expected_votes requires quorum.device.votes.
  * quorum.expected_votes - quorum.device.votes cannot be lower
    than 2.
  * qdevice and last_man_standing are mutually exclusive.
  * qdevice and auto_tie_breaker are mutually exclusive.

  runtime config changes:

  * quorum.two_node and qdevice are incompatible:
    if quorum device is alive, two_node is disabled.
    if quorum device is not alive and node count is 2, two_node is
       enabled, and quorum device cannot be registered

  * if either last_man_standing or auto_tie_breaker were enabled
    at startup, and at runtime quorum device is configured,
    quorum device registration will be blocked.

  * if quorum.expected_votes is configured but not quorum.device.votes,
    quorum device registration will be blocked.

  * if quorum.device.votes is not configured and we cannot
    automatically calculate it, quorum device registration will be blocked.

  * An error in configuring quorum.expected_votes and quorum.device.votes
    will block quorum device registration.

blocking quorum device registation, also means dropping the votes.

quorum.device.votes (either set or automatically calculated) is now
used to determine current expected_votes in the cluster.

- logging (exec/votequorum):

  all errors from configuration are treated as WARNING/CRITICAL.

  lots of extra DEBUG output is added (see internal changes too).

- corosync-quorumtool (tools/corosync-quorumtool):

  * added option to forcefully kick out a quorum device from the local
    node. This is for emergency recovery only and it is only
    available when qdevice API is built-in.

  * Improved status output, specifically add node state and qdevice
    information

[root@fedora-master-node2 coro]# corosync-quorumtool -s
Version:          1.99.4.12-9c7d-dirty
Quorum type:      corosync_votequorum
Nodes:            2
Ring ID:          132
Quorate:          Yes
Node votes:       1
Node state:       Member
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2
Flags:            Quorate Qdevice
Nodeid     Votes  Name
   1     1  fedora-master-node1.int.fabbione.net
   2     1  fedora-master-node2.int.fabbione.net
   0     1  QDEVICE (Voting)

  * allow to print status for any node in the cluster known to
    local node.

[root@fedora-master-node1 coro]# corosync-quorumtool -s
Version:          1.99.4.12-9c7d-dirty
Quorum type:      corosync_votequorum
Nodes:            2
Ring ID:          144
Quorate:          Yes
Node votes:       1
Node state:       Member
Expected votes:   3
Highest expected: 3
Total votes:      2
Quorum:           2
Flags:            Quorate
Nodeid     Votes  Name
   1     1  fedora-master-node1.int.fabbione.net
   2     1  fedora-master-node2.int.fabbione.net

[root@fedora-master-node1 coro]# corosync-quorumtool -s -n 2
Version:          1.99.4.12-9c7d-dirty
Quorum type:      corosync_votequorum
Nodes:            2
Ring ID:          144
Quorate:          Yes
Node votes:       1
Node state:       Member
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2
Flags:            Quorate Qdevice
Nodeid     Votes  Name
   1     1  fedora-master-node1.int.fabbione.net
   2     1  fedora-master-node2.int.fabbione.net
         0     1  QDEVICE (Voting)

Internal changes:

- change qdevice timer to not run all time, but only when necessary.
- change votequorum_nodeinfo on wire data to use flags instead of uint8_t
  and add QDEVICE status.
- allocate nodeid 0 to qdevice since it's the only real
  nodeid that be reserved.
- change send_nodeinfo to allow to send nodeinfo for any node
  so that we can share qdevice info across the cluster
  (and this might be useful in future if we need to sync
   internal cluster view).
- add votequorum api call to update qdevice name
- add runtime data if quorum device has been forcefully disabled
  by config error
- add qdevice votes to expected_votes calculation (this
  is probably the biggest difference vs cman)
- change votequorum_read_nodelist_configuration so that
  we can autocalculate votes for qdevice (we need the nodecount
  vs votes).
- add all checks for startup/runtime config (see above).
- do not make qdevice part of the membership_list received from
  totem. None of our users care about it and it is not a real node.
- change onwire message handlers to deal with "data for this node from any node"
  case and undersand nodeid 0 for qdevice info
- always allocate qdevice at startup. this simplifies code a lot.
- dispatch qdevice nodeinfo on membership changes.
- inform libvotequorum users when a qdevice is registered
- improve substantially qdevice api and add a simple
  barrier based on qdevice name.
- add qdevice API barrier at cluster level. This feature allow
  only one qdevice name to be active in the cluster at any time.
- qdevice getinfo can now report status for qdevice on any node.
- change slightly the way the qdevice API is built-in/out:
  only the libvotequorum calls are #ifdef'out now. Doing so in
  the core is too complex and would make the code unreadable
  with the risk of missing a bit or two effectively introducing
  an on-wire incompatibility if we will ever turn the API on.
- probably added some bugs on the way...

TODO: update qdevice_* API once the above is settled and test
      qdevice integration with other features.

Signed-off-by: Fabio M. Di Nitto <fdinitto@redhat.com>
Reviewed-by: Christine Caulfield <ccaulfie@redhat.com>
Reviewed-by: Steven Dake <sdake@redhat.com> (only second part)

exec/coroparse.c

@@ -87,6 +87,7 @@ enum main_cp_cb_data_state {
 	MAIN_CP_CB_DATA_STATE_LOGGING_DAEMON,
 	MAIN_CP_CB_DATA_STATE_MEMBER,
 	MAIN_CP_CB_DATA_STATE_QUORUM,
+	MAIN_CP_CB_DATA_STATE_QDEVICE,
 	MAIN_CP_CB_DATA_STATE_NODELIST,
 	MAIN_CP_CB_DATA_STATE_NODELIST_NODE,
 };
@@ -388,7 +389,6 @@ static int main_config_parser_cb(const char *path,
 		case MAIN_CP_CB_DATA_STATE_QUORUM:
 			if ((strcmp(path, "quorum.expected_votes") == 0) ||
 			    (strcmp(path, "quorum.votes") == 0) ||
-			    (strcmp(path, "quorum.quorumdev_poll") == 0) ||
 			    (strcmp(path, "quorum.last_man_standing_window") == 0) ||
 			    (strcmp(path, "quorum.leaving_timeout") == 0)) {
 				if (safe_atoi(value, &i) != 0) {
@@ -410,6 +410,15 @@ static int main_config_parser_cb(const char *path,
 				add_as_string = 0;
 			}
 			break;
+		case MAIN_CP_CB_DATA_STATE_QDEVICE:
+			if ((strcmp(path, "quorum.device.timeout") == 0) ||
+			    (strcmp(path, "quorum.device.votes") == 0)) {
+				if (safe_atoi(value, &i) != 0) {
+					goto atoi_error;
+				}
+				icmap_set_uint32(path, i);
+				add_as_string = 0;
+			}
 		case MAIN_CP_CB_DATA_STATE_TOTEM:
 			if ((strcmp(path, "totem.version") == 0) ||
 			    (strcmp(path, "totem.nodeid") == 0) ||
@@ -671,6 +680,9 @@ static int main_config_parser_cb(const char *path,
 		if (strcmp(path, "quorum") == 0) {
 			data->state = MAIN_CP_CB_DATA_STATE_QUORUM;
 		}
+		if (strcmp(path, "quorum.device") == 0) {
+			data->state = MAIN_CP_CB_DATA_STATE_QDEVICE;
+		}
 		if (strcmp(path, "nodelist") == 0) {
 			data->state = MAIN_CP_CB_DATA_STATE_NODELIST;
 			data->node_number = 0;
@@ -858,6 +870,9 @@ static int main_config_parser_cb(const char *path,
 		case MAIN_CP_CB_DATA_STATE_QUORUM:
 			data->state = MAIN_CP_CB_DATA_STATE_NORMAL;
 			break;
+		case MAIN_CP_CB_DATA_STATE_QDEVICE:
+			data->state = MAIN_CP_CB_DATA_STATE_QUORUM;
+			break;
 		case MAIN_CP_CB_DATA_STATE_NODELIST:
 			data->state = MAIN_CP_CB_DATA_STATE_NORMAL;
 			break;

include/corosync/ipc_votequorum.h

@@ -48,6 +48,7 @@ enum req_votequorum_types {
 	,
 	MESSAGE_REQ_VOTEQUORUM_QDEVICE_REGISTER,
 	MESSAGE_REQ_VOTEQUORUM_QDEVICE_UNREGISTER,
+	MESSAGE_REQ_VOTEQUORUM_QDEVICE_UPDATE,
 	MESSAGE_REQ_VOTEQUORUM_QDEVICE_POLL,
 	MESSAGE_REQ_VOTEQUORUM_QDEVICE_GETINFO
 #endif
@@ -62,23 +63,39 @@ enum res_votequorum_types {
 	MESSAGE_RES_VOTEQUORUM_EXPECTEDVOTES_NOTIFICATION
 };
 
-struct req_lib_votequorum_setvotes {
+struct req_lib_votequorum_qdevice_register {
 	struct qb_ipc_request_header header __attribute__((aligned(8)));
-	unsigned int votes;
-	int nodeid;
+	char name[VOTEQUORUM_MAX_QDEVICE_NAME_LEN];
 };
 
-struct req_lib_votequorum_qdevice_register {
+struct req_lib_votequorum_qdevice_unregister {
 	struct qb_ipc_request_header header __attribute__((aligned(8)));
-	unsigned int votes;
 	char name[VOTEQUORUM_MAX_QDEVICE_NAME_LEN];
 };
 
+struct req_lib_votequorum_qdevice_update {
+	struct qb_ipc_request_header header __attribute__((aligned(8)));
+	char oldname[VOTEQUORUM_MAX_QDEVICE_NAME_LEN];
+	char newname[VOTEQUORUM_MAX_QDEVICE_NAME_LEN];
+};
+
 struct req_lib_votequorum_qdevice_poll {
 	struct qb_ipc_request_header header __attribute__((aligned(8)));
+	char name[VOTEQUORUM_MAX_QDEVICE_NAME_LEN];
 	int state;
 };
 
+struct req_lib_votequorum_qdevice_getinfo {
+	struct qb_ipc_request_header header __attribute__((aligned(8)));
+	int nodeid;
+};
+
+struct req_lib_votequorum_setvotes {
+	struct qb_ipc_request_header header __attribute__((aligned(8)));
+	unsigned int votes;
+	int nodeid;
+};
+
 struct req_lib_votequorum_setexpected {
 	struct qb_ipc_request_header header __attribute__((aligned(8)));
 	unsigned int expected_votes;
@@ -109,10 +126,12 @@ struct res_lib_votequorum_status {
 #define VOTEQUORUM_INFO_LAST_MAN_STANDING       8
 #define VOTEQUORUM_INFO_AUTO_TIE_BREAKER       16
 #define VOTEQUORUM_INFO_LEAVE_REMOVE           32
+#define VOTEQUORUM_INFO_QDEVICE                64
 
 struct res_lib_votequorum_getinfo {
 	struct qb_ipc_response_header header __attribute__((aligned(8)));
-	int nodeid;
+	unsigned int nodeid;
+	unsigned int state;
 	unsigned int votes;
 	unsigned int expected_votes;
 	unsigned int highest_expected;

include/corosync/votequorum.h

@@ -42,19 +42,16 @@ extern "C" {
 
 typedef uint64_t votequorum_handle_t;
 
-#ifdef EXPERIMENTAL_QUORUM_DEVICE_API
-#define VOTEQUORUM_MAX_QDEVICE_NAME_LEN 255
-#endif
-
 #define VOTEQUORUM_INFO_FLAG_TWONODE            1
 #define VOTEQUORUM_INFO_FLAG_QUORATE            2
 #define VOTEQUORUM_INFO_WAIT_FOR_ALL            4
 #define VOTEQUORUM_INFO_LAST_MAN_STANDING       8
 #define VOTEQUORUM_INFO_AUTO_TIE_BREAKER       16
 #define VOTEQUORUM_INFO_LEAVE_REMOVE           32
+#define VOTEQUORUM_INFO_QDEVICE                64
 
-#define VOTEQUORUM_NODEID_US 0
-#define VOTEQUORUM_NODEID_QDEVICE -1
+#define VOTEQUORUM_NODEID_QDEVICE 0
+#define VOTEQUORUM_MAX_QDEVICE_NAME_LEN 255
 
 #define NODESTATE_MEMBER     1
 #define NODESTATE_DEAD       2
@@ -64,6 +61,7 @@ typedef uint64_t votequorum_handle_t;
 
 struct votequorum_info {
 	unsigned int node_id;
+	unsigned int state;
 	unsigned int node_votes;
 	unsigned int node_expected_votes;
 	unsigned int highest_expected;
@@ -187,20 +185,29 @@ cs_error_t votequorum_context_set (
  */
 cs_error_t votequorum_qdevice_register (
 	votequorum_handle_t handle,
-	const char *name,
-	unsigned int votes);
+	const char *name);
 
 /**
  * Unregister a quorum device
  */
 cs_error_t votequorum_qdevice_unregister (
-	votequorum_handle_t handle);
+	votequorum_handle_t handle,
+	const char *name);
+
+/**
+ * Update registered name of a quorum device
+ */
+cs_error_t votequorum_qdevice_update (
+	votequorum_handle_t handle,
+	const char *oldname,
+	const char *newname);
 
 /**
  * Poll a quorum device
  */
 cs_error_t votequorum_qdevice_poll (
 	votequorum_handle_t handle,
+	const char *name,
 	unsigned int state);
 
 /**
@@ -208,6 +215,7 @@ cs_error_t votequorum_qdevice_poll (
  */
 cs_error_t votequorum_qdevice_getinfo (
 	votequorum_handle_t handle,
+	unsigned int nodeid,
 	struct votequorum_qdevice_info *info);
 
 #endif

lib/votequorum.c

@@ -174,6 +174,7 @@ cs_error_t votequorum_getinfo (
 	error = res_lib_votequorum_getinfo.header.error;
 
 	info->node_id = res_lib_votequorum_getinfo.nodeid;
+	info->state = res_lib_votequorum_getinfo.state;
 	info->node_votes = res_lib_votequorum_getinfo.votes;
 	info->node_expected_votes = res_lib_votequorum_getinfo.expected_votes;
 	info->highest_expected = res_lib_votequorum_getinfo.highest_expected;
@@ -539,8 +540,7 @@ error_put:
 #ifdef EXPERIMENTAL_QUORUM_DEVICE_API
 cs_error_t votequorum_qdevice_register (
 	votequorum_handle_t handle,
-	const char *name,
-	unsigned int votes)
+	const char *name)
 {
 	cs_error_t error;
 	struct votequorum_inst *votequorum_inst;
@@ -548,8 +548,9 @@ cs_error_t votequorum_qdevice_register (
 	struct req_lib_votequorum_qdevice_register req_lib_votequorum_qdevice_register;
 	struct res_lib_votequorum_status res_lib_votequorum_status;
 
-	if (strlen(name) >= VOTEQUORUM_MAX_QDEVICE_NAME_LEN)
+	if (strlen(name) >= VOTEQUORUM_MAX_QDEVICE_NAME_LEN) {
 		return CS_ERR_INVALID_PARAM;
+	}
 
 	error = hdb_error_to_cs(hdb_handle_get (&votequorum_handle_t_db, handle, (void *)&votequorum_inst));
 	if (error != CS_OK) {
@@ -560,7 +561,6 @@ cs_error_t votequorum_qdevice_register (
 	req_lib_votequorum_qdevice_register.header.size = sizeof (struct req_lib_votequorum_qdevice_register);
 	req_lib_votequorum_qdevice_register.header.id = MESSAGE_REQ_VOTEQUORUM_QDEVICE_REGISTER;
 	strcpy(req_lib_votequorum_qdevice_register.name, name);
-	req_lib_votequorum_qdevice_register.votes = votes;
 
 	iov.iov_base = (char *)&req_lib_votequorum_qdevice_register;
 	iov.iov_len = sizeof (struct req_lib_votequorum_qdevice_register);
@@ -586,6 +586,7 @@ error_exit:
 
 cs_error_t votequorum_qdevice_poll (
 	votequorum_handle_t handle,
+	const char *name,
 	unsigned int state)
 {
 	cs_error_t error;
@@ -594,14 +595,18 @@ cs_error_t votequorum_qdevice_poll (
 	struct req_lib_votequorum_qdevice_poll req_lib_votequorum_qdevice_poll;
 	struct res_lib_votequorum_status res_lib_votequorum_status;
 
+	if (strlen(name) >= VOTEQUORUM_MAX_QDEVICE_NAME_LEN) {
+		return CS_ERR_INVALID_PARAM;
+	}
+
 	error = hdb_error_to_cs(hdb_handle_get (&votequorum_handle_t_db, handle, (void *)&votequorum_inst));
 	if (error != CS_OK) {
 		return (error);
 	}
 
-
 	req_lib_votequorum_qdevice_poll.header.size = sizeof (struct req_lib_votequorum_qdevice_poll);
 	req_lib_votequorum_qdevice_poll.header.id = MESSAGE_REQ_VOTEQUORUM_QDEVICE_POLL;
+	strcpy(req_lib_votequorum_qdevice_poll.name, name);
 	req_lib_votequorum_qdevice_poll.state = state;
 
 	iov.iov_base = (char *)&req_lib_votequorum_qdevice_poll;
@@ -627,24 +632,30 @@ error_exit:
 }
 
 cs_error_t votequorum_qdevice_unregister (
-	votequorum_handle_t handle)
+	votequorum_handle_t handle,
+	const char *name)
 {
 	cs_error_t error;
 	struct votequorum_inst *votequorum_inst;
 	struct iovec iov;
-	struct req_lib_votequorum_general req_lib_votequorum_general;
+	struct req_lib_votequorum_qdevice_unregister req_lib_votequorum_qdevice_unregister;
 	struct res_lib_votequorum_status res_lib_votequorum_status;
 
+	if (strlen(name) >= VOTEQUORUM_MAX_QDEVICE_NAME_LEN) {
+		return CS_ERR_INVALID_PARAM;
+	}
+
 	error = hdb_error_to_cs(hdb_handle_get (&votequorum_handle_t_db, handle, (void *)&votequorum_inst));
 	if (error != CS_OK) {
 		return (error);
 	}
 
-	req_lib_votequorum_general.header.size = sizeof (struct req_lib_votequorum_general);
-	req_lib_votequorum_general.header.id = MESSAGE_REQ_VOTEQUORUM_QDEVICE_UNREGISTER;
+	req_lib_votequorum_qdevice_unregister.header.size = sizeof (struct req_lib_votequorum_qdevice_unregister);
+	req_lib_votequorum_qdevice_unregister.header.id = MESSAGE_REQ_VOTEQUORUM_QDEVICE_UNREGISTER;
+	strcpy(req_lib_votequorum_qdevice_unregister.name, name);
 
-	iov.iov_base = (char *)&req_lib_votequorum_general;
-	iov.iov_len = sizeof (struct req_lib_votequorum_general);
+	iov.iov_base = (char *)&req_lib_votequorum_qdevice_unregister;
+	iov.iov_len = sizeof (struct req_lib_votequorum_qdevice_unregister);
 
         error = qb_to_cs_error(qb_ipcc_sendv_recv (
 		votequorum_inst->c,
@@ -667,12 +678,13 @@ error_exit:
 
 cs_error_t votequorum_qdevice_getinfo (
 	votequorum_handle_t handle,
+	unsigned int nodeid,
 	struct votequorum_qdevice_info *qinfo)
 {
 	cs_error_t error;
 	struct votequorum_inst *votequorum_inst;
 	struct iovec iov;
-	struct req_lib_votequorum_general req_lib_votequorum_general;
+	struct req_lib_votequorum_qdevice_getinfo req_lib_votequorum_qdevice_getinfo;
 	struct res_lib_votequorum_qdevice_getinfo res_lib_votequorum_qdevice_getinfo;
 
 	error = hdb_error_to_cs(hdb_handle_get (&votequorum_handle_t_db, handle, (void *)&votequorum_inst));
@@ -680,12 +692,12 @@ cs_error_t votequorum_qdevice_getinfo (
 		return (error);
 	}
 
+	req_lib_votequorum_qdevice_getinfo.header.size = sizeof (struct req_lib_votequorum_qdevice_getinfo);
+	req_lib_votequorum_qdevice_getinfo.header.id = MESSAGE_REQ_VOTEQUORUM_QDEVICE_GETINFO;
+	req_lib_votequorum_qdevice_getinfo.nodeid = nodeid;
 
-	req_lib_votequorum_general.header.size = sizeof (struct req_lib_votequorum_general);
-	req_lib_votequorum_general.header.id = MESSAGE_REQ_VOTEQUORUM_QDEVICE_GETINFO;
-
-	iov.iov_base = (char *)&req_lib_votequorum_general;
-	iov.iov_len = sizeof (struct req_lib_votequorum_general);
+	iov.iov_base = (char *)&req_lib_votequorum_qdevice_getinfo;
+	iov.iov_len = sizeof (struct req_lib_votequorum_qdevice_getinfo);
 
         error = qb_to_cs_error(qb_ipcc_sendv_recv (
 		votequorum_inst->c,

test/testvotequorum2.c

@@ -50,7 +50,7 @@ static void print_info(int ok_to_fail)
 	struct votequorum_qdevice_info qinfo;
 	int err;
 
-	if ( (err=votequorum_qdevice_getinfo(handle, &qinfo)) != CS_OK)
+	if ( (err=votequorum_qdevice_getinfo(handle, VOTEQUORUM_NODEID_QDEVICE, &qinfo)) != CS_OK)
 		fprintf(stderr, "votequorum_qdevice_getinfo error %d: %s\n", err, ok_to_fail?"OK":"FAILED");
 	else {
 		printf("qdevice votes  %d\n", qinfo.votes);
@@ -63,6 +63,7 @@ static void print_info(int ok_to_fail)
 
 int main(int argc, char *argv[])
 {
+	int ret = 0;
 #ifdef EXPERIMENTAL_QUORUM_DEVICE_API
 	int pollcount=0, polltime=1;
 	int err;
@@ -82,24 +83,34 @@ int main(int argc, char *argv[])
 	}
 
 	if (argc >= 2) {
-		if ( (err=votequorum_qdevice_register(handle, "QDEVICE", 4)) != CS_OK)
+		if ( (err=votequorum_qdevice_register(handle, "QDEVICE")) != CS_OK) {
 			fprintf(stderr, "qdevice_register FAILED: %d\n", err);
+			ret = -1;
+			goto out;
+		}
 
 		while (pollcount--) {
 			print_info(0);
-			if ((err=votequorum_qdevice_poll(handle, 1)) != CS_OK)
+			if ((err=votequorum_qdevice_poll(handle, "QDEVICE", 1)) != CS_OK) {
 				fprintf(stderr, "qdevice poll FAILED: %d\n", err);
+				ret = -1;
+				goto out;
+			}
 			print_info(0);
 			sleep(polltime);
 		}
-		if ((err= votequorum_qdevice_unregister(handle)) != CS_OK)
+		if ((err= votequorum_qdevice_unregister(handle, "QDEVICE")) != CS_OK) {
 			fprintf(stderr, "qdevice unregister FAILED: %d\n", err);
+			ret = -1;
+			goto out;
+		}
 	}
 	print_info(1);
 
+out:
 	votequorum_finalize(handle);
 #else
-	fprintf(stderr, "qdevice support is not built in corosync/votequorum\n");	
+	fprintf(stderr, "qdevice support is not built in corosync/votequorum\n");
 #endif
-	return 0;
+	return ret;
 }

tools/corosync-quorumtool.c

@@ -63,7 +63,8 @@ typedef enum {
 	CMD_SHOWSTATUS,
 	CMD_SETVOTES,
 	CMD_SETEXPECTED,
-	CMD_MONITOR
+	CMD_MONITOR,
+	CMD_UNREGISTER_QDEVICE
 } command_t;
 
 /*
@@ -108,6 +109,8 @@ static votequorum_callbacks_t v_callbacks = {
 	.votequorum_notify_fn = NULL,
 	.votequorum_expectedvotes_notify_fn = NULL
 };
+static uint32_t our_nodeid = 0;
+static uint8_t display_qdevice = 0;
 
 /*
  * cfg bits
@@ -132,6 +135,7 @@ static void show_usage(const char *name)
 	printf("  -e <expected>  change expected votes for the cluster *\n");
 	printf("  -H             show nodeids in hexadecimal rather than decimal\n");
 	printf("  -i             show node IP addresses instead of the resolved name\n");
+	printf("  -f             forcefully unregister a quorum device *DANGEROUS*\n");
 	printf("  -h             show this help text\n");
 	printf("\n");
 	printf("  * Starred items only work if votequorum is the quorum provider for corosync\n");
@@ -283,7 +287,7 @@ static void quorum_notification_fn(
 	}
 }
 
-static void display_nodes_data(nodeid_format_t nodeid_format, name_format_t name_format)
+static void display_nodes_data(uint32_t nodeid, nodeid_format_t nodeid_format, name_format_t name_format)
 {
 	int i;
 
@@ -305,13 +309,51 @@ static void display_nodes_data(nodeid_format_t nodeid_format, name_format_t name
 			printf("%s\n", node_name(g_view_list[i], name_format));
 		}
 	}
+
 	if (g_view_list_entries) {
 		free(g_view_list);
 		g_view_list = NULL;
 	}
+
+#if EXPERIMENTAL_QUORUM_DEVICE_API
+	if ((display_qdevice) && (v_handle)) {
+		int err;
+		struct votequorum_qdevice_info qinfo;
+
+		err = votequorum_qdevice_getinfo(v_handle, nodeid, &qinfo);
+		if (err != CS_OK) {
+			fprintf(stderr, "votequorum_qdevice_getinfo error: %d\n", err);
+		} else {
+			if (nodeid_format == NODEID_FORMAT_DECIMAL) {
+				printf("%10u   ", VOTEQUORUM_NODEID_QDEVICE);
+			} else {
+				printf("0x%08x   ", VOTEQUORUM_NODEID_QDEVICE);
+			}
+			printf("%3d  %s (%s)\n", qinfo.votes, qinfo.name, qinfo.state?"Voting":"Not voting");
+		}
+	}
+#endif
+}
+
+static const char *decode_state(int state)
+{
+	switch(state) {
+		case NODESTATE_MEMBER:
+			return "Member";
+			break;
+		case NODESTATE_DEAD:
+			return "Dead";
+			break;
+		case NODESTATE_LEAVING:
+			return "Leaving";
+			break;
+		default:
+			return "Unknown state (this is bad)";
+			break;
+	}
 }
 
-static int display_quorum_data(int is_quorate, int loop)
+static int display_quorum_data(int is_quorate, uint32_t nodeid, int loop)
 {
 	struct votequorum_info info;
 	int err;
@@ -335,8 +377,9 @@ static int display_quorum_data(int is_quorate, int loop)
 		return CS_OK;
 	}
 
-	if ((err=votequorum_getinfo(v_handle, 0, &info)) == CS_OK) {
+	if ((err=votequorum_getinfo(v_handle, nodeid, &info)) == CS_OK) {
 		printf("Node votes:       %d\n", info.node_votes);
+		printf("Node state:       %s\n", decode_state(info.state));
 		printf("Expected votes:   %d\n", info.node_expected_votes);
 		printf("Highest expected: %d\n", info.highest_expected);
 		printf("Total votes:      %d\n", info.total_votes);
@@ -348,6 +391,12 @@ static int display_quorum_data(int is_quorate, int loop)
 		if (info.flags & VOTEQUORUM_INFO_LAST_MAN_STANDING) printf("LastManStanding ");
 		if (info.flags & VOTEQUORUM_INFO_AUTO_TIE_BREAKER) printf("AutoTieBreaker ");
 		if (info.flags & VOTEQUORUM_INFO_LEAVE_REMOVE) printf("LeaveRemove ");
+		if (info.flags & VOTEQUORUM_INFO_QDEVICE) {
+			printf("Qdevice ");
+			display_qdevice = 1;
+		} else {
+			display_qdevice = 0;
+		}
 		printf("\n");
 	} else {
 		fprintf(stderr, "votequorum_getinfo FAILED: %d\n", err);
@@ -361,7 +410,7 @@ static int display_quorum_data(int is_quorate, int loop)
  *         0 if not quorate
  *        -1 on error
  */
-static int show_status(nodeid_format_t nodeid_format, name_format_t name_format)
+static int show_status(uint32_t nodeid, nodeid_format_t nodeid_format, name_format_t name_format)
 {
 	int is_quorate;
 	int err;
@@ -395,11 +444,11 @@ quorum_err:
 		return -1;
 	}
 
-	err = display_quorum_data(is_quorate, 0);
+	err = display_quorum_data(is_quorate, nodeid, 0);
 	if (err != CS_OK) {
 		return -1;
 	}
-	display_nodes_data(nodeid_format, name_format);
+	display_nodes_data(nodeid, nodeid_format, name_format);
 
 	return is_quorate;
 }
@@ -410,7 +459,7 @@ static int monitor_status(nodeid_format_t nodeid_format, name_format_t name_form
 
 	if (q_type == QUORUM_FREE) {
 		printf("\nQuorum is not configured - cannot monitor\n");
-		return show_status(nodeid_format, name_format);
+		return show_status(our_nodeid, nodeid_format, name_format);
 	}
 
 	err=quorum_trackstart(q_handle, CS_TRACK_CHANGES);
@@ -429,8 +478,8 @@ static int monitor_status(nodeid_format_t nodeid_format, name_format_t name_form
 		}
 		time(&t);
 		printf("date:             %s", ctime((const time_t *)&t));
-		err = display_quorum_data(g_quorate, loop);
-		display_nodes_data(nodeid_format, name_format);
+		err = display_quorum_data(g_quorate, our_nodeid, loop);
+		display_nodes_data(our_nodeid, nodeid_format, name_format);
 		printf("\n");
 		loop = 1;
 		if (err != CS_OK) {
@@ -463,13 +512,37 @@ static int show_nodes(nodeid_format_t nodeid_format, name_format_t name_format)
 		}
 	}
 
-	display_nodes_data(nodeid_format, name_format);
+	display_nodes_data(our_nodeid, nodeid_format, name_format);
 
 	result = EXIT_SUCCESS;
 err_exit:
 	return result;
 }
 
+static int unregister_qdevice(void)
+{
+#ifdef EXPERIMENTAL_QUORUM_DEVICE_API 
+	int err;
+	struct votequorum_qdevice_info qinfo;
+
+	err = votequorum_qdevice_getinfo(v_handle, our_nodeid, &qinfo);
+	if (err != CS_OK) {
+		fprintf(stderr, "votequorum_qdevice_getinfo FAILED: %d\n", err);
+		return -1;
+	}
+
+	err = votequorum_qdevice_unregister(v_handle, qinfo.name);
+	if (err != CS_OK) {
+		fprintf(stderr, "votequorum_qdevice_unregister FAILED: %d\n", err);
+		return -1;
+	}
+	return 0;
+#else
+	fprintf(stderr, "votequorum quorum device support is not built-in\n");
+	return -1;
+#endif
+}
+
 /*
  * return -1 on error
  *         0 if OK
@@ -509,6 +582,11 @@ static int init_all(void) {
 		goto out;
 	}
 
+	if (cmap_get_uint32(cmap_handle, "runtime.votequorum.this_node_id", &our_nodeid) != CS_OK) {
+		fprintf(stderr, "Unable to retrive this node nodeid\n");
+		goto out;
+	}
+
 	return 0;
 out:
 	return -1;
@@ -530,12 +608,13 @@ static void close_all(void) {
 }
 
 int main (int argc, char *argv[]) {
-	const char *options = "VHslme:v:hin:d:";
+	const char *options = "VHslmfe:v:hin:d:";
 	char *endptr;
 	int opt;
 	int votes = 0;
 	int ret = 0;
-	uint32_t nodeid = VOTEQUORUM_NODEID_US;
+	uint32_t nodeid = 0;
+	uint32_t nodeid_set = 0;
 	nodeid_format_t nodeid_format = NODEID_FORMAT_DECIMAL;
 	name_format_t address_format = ADDRESS_FORMAT_NAME;
 	command_t command_opt = CMD_UNKNOWN;
@@ -552,6 +631,14 @@ int main (int argc, char *argv[]) {
 
 	while ( (opt = getopt(argc, argv, options)) != -1 ) {
 		switch (opt) {
+		case 'f':
+			if (using_votequorum() > 0) {
+				command_opt = CMD_UNREGISTER_QDEVICE;
+			} else {
+				fprintf(stderr, "You cannot unregister quorum device, corosync is not using votequorum\n");
+				exit(2);
+			}
+			break;
 		case 's':
 			command_opt = CMD_SHOWSTATUS;
 			break;
@@ -582,15 +669,18 @@ int main (int argc, char *argv[]) {
 			break;
 		case 'n':
 			nodeid = strtol(optarg, &endptr, 0);
-			if ((nodeid == 0 && endptr == optarg) || nodeid <= 0) {
+			if ((nodeid == 0 && endptr == optarg) || nodeid < 0) {
 				fprintf(stderr, "The nodeid was not valid, try a positive number\n");
+				exit(2);
 			}
+			nodeid_set = 1;
 			break;
 		case 'v':
 			if (using_votequorum() > 0) {
 				votes = strtol(optarg, &endptr, 0);
 				if ((votes == 0 && endptr == optarg) || votes < 0) {
 					fprintf(stderr, "New votes value was not valid, try a positive number or zero\n");
+					exit(2);
 				} else {
 					command_opt = CMD_SETVOTES;
 				}
@@ -600,10 +690,12 @@ int main (int argc, char *argv[]) {
 				exit(2);
 			}
 			break;
+		case ':':
 		case 'h':
 		case '?':
 		default:
-		break;
+			command_opt = CMD_UNKNOWN;
+			break;
 		}
 	}
 
@@ -616,9 +708,15 @@ int main (int argc, char *argv[]) {
 		ret = show_nodes(nodeid_format, address_format);
 		break;
 	case CMD_SHOWSTATUS:
-		ret = show_status(nodeid_format, address_format);
+		if (!nodeid_set) {
+			nodeid = our_nodeid;
+		}
+		ret = show_status(nodeid, nodeid_format, address_format);
 		break;
 	case CMD_SETVOTES:
+		if (!nodeid_set) {
+			nodeid = our_nodeid;
+		}
 		ret = set_votes(nodeid, votes);
 		break;
 	case CMD_SETEXPECTED:
@@ -627,6 +725,9 @@ int main (int argc, char *argv[]) {
 	case CMD_MONITOR:
 		ret = monitor_status(nodeid_format, address_format);
 		break;
+	case CMD_UNREGISTER_QDEVICE:
+		ret = unregister_qdevice();
+		break;
 	}
 
 	close_all();