qdevice: Add support for heuristics

Heuristics are set of commands executed locally on startup, cluster
membership change, successful connect to corosync-qnetd and optionally
also at regular times. When all commands finish successfully
(their return error code is zero) on time, heuristics have passed,
otherwise they have failed. The heuristics result is sent to
corosync-qnetd and there it's used in calculations to determine which
partition should be quorate.

Right know, there are some problems (bugs):
- Regular heuristics is supported only by ffsplit. This is not a
problem for clusters with power fencing, but deployments where
non-quorate partition continues to operate may see this as a problem.
- Qdevice-tool status doesn't contain detailed information about
heuristics.
- Qdevice-tool doesn't have a possibility to trigger heuristics
re-execute.

Thanks Chrissie Caulfield for Englishify the man pages.

Signed-off-by: Jan Friesse <jfriesse@redhat.com>
Reviewed-by: Christine Caulfield <ccaulfie@redhat.com>

man/corosync-qdevice-tool.8

@@ -1,5 +1,5 @@
 .\"/*
-.\" * Copyright (C) 2016 Red Hat, Inc.
+.\" * Copyright (C) 2016-2017 Red Hat, Inc.
 .\" *
 .\" * All rights reserved.
 .\" *
@@ -31,7 +31,7 @@
 .\" * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
 .\" * THE POSSIBILITY OF SUCH DAMAGE.
 .\" */
-.TH COROSYNC-QDEVICE-TOOL 8 2016-06-24
+.TH COROSYNC-QDEVICE-TOOL 8 2017-10-17
 .SH NAME
 corosync-qdevice-tool \- corosync-qdevice control interface.
 .SH SYNOPSIS
@@ -76,6 +76,7 @@ HB interval:            10000ms
 Sync HB interval:       30000ms
 Configured node list:
     0   Node ID = 1
+Heuristics:             Enabled
 Ring ID:                1.a00000000021b48
 Membership node list:   1
 Quorate:                Yes
@@ -96,6 +97,7 @@ Algorithm:              Fifty-Fifty split
 Tie-breaker:            Node with lowest node ID
 Poll timer running:     Yes (cast vote)
 State:                  Connected
+Heuristics result:      Pass (regular: Pass, membership: Fail, connect: Fail)
 TLS active:             Yes (client certificate sent)
 Connected since:        2016-06-24T17:02:35
 Echo reply received:    2016-06-24T17:05:15
@@ -113,10 +115,13 @@ For model net, it's good to check the
 .I Poll timer running
 state. Internally, model net supports 3 states. Not voting (when
 .I Poll timer running
-is No it means that the
+is No, which means
+.B corosync-qdevice
+is waiting for
+.B corosync-qnetd
+to reply), voting (without cast vote, it means that the
 .B corosync-qnetd
-algorithm decides that the current node shouldn't get a vote),
-voting (without cast vote) and voting (with cast vote).
+algorithm decides that the current node shouldn't get a vote) and voting (with cast vote).
 .SH SEE ALSO
 .BR corosync-qnetd (8)
 .BR corosync-qdevice (8)

man/corosync-qdevice.8

@@ -1,5 +1,5 @@
 .\"/*
-.\" * Copyright (C) 2016 Red Hat, Inc.
+.\" * Copyright (C) 2016-2017 Red Hat, Inc.
 .\" *
 .\" * All rights reserved.
 .\" *
@@ -31,7 +31,7 @@
 .\" * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
 .\" * THE POSSIBILITY OF SUCH DAMAGE.
 .\" */
-.TH COROSYNC-QDEVICE 8 2016-06-29
+.TH COROSYNC-QDEVICE 8 2017-10-17
 .SH NAME
 corosync-qdevice \- QDevice daemon
 .SH SYNOPSIS
@@ -93,9 +93,59 @@ Default is 30000.
 .B votes
 The number of votes provided to the cluster by qdevice. Default is (number_of_nodes - 1) or generally
 sum(votes_per_node) - 1.
+
+.PP
+.B quorum.device.heuristics
+subkey holds the configuration of the heuristics. Heuristics are set of commands executed locally on
+startup, cluster membership change, successful connect to
+.B corosync-qnetd
+and optionally also at regular times. When all commands finish successfully
+(their return error code is zero) on time,
+heuristics have passed, otherwise they have failed. The heuristics result is sent to
+.B corosync-qnetd
+and there it's used in calculations to determine which partition should be quorate.
+.TP
+.B timeout
+Specifies maximum time in milliseconds how long
+.B corosync-qdevice
+waits till the heuristics commands finish. If some command doesn't finish before the timeout, it's
+killed and heuristics fail. This timeout is used for heuristics executed at regular times.
+Default value is half of the quorum.device.timeout, so 5000.
+.TP
+.B sync_timeout
+Similar to quorum.device.heuristics.timeout but used during membership changes. Default
+value is half of the quorum.device.sync_timeout, so 15000.
+.TP
+.B interval
+Specifies interval between two regular heuristics execution. Default value is
+3 * quorum.device.timeout, so 30000.
+.TP
+.B mode
+Can be on of
+.I on, sync or off
+and specifies mode of operation of heuristics. Default is
+.I off
+what means heuristics are disabled. When
+.I sync
+is set, heuristics are executed only during startup, membership change and when connection
+to
+.B corosync-qnetd
+is established. When heuristics should be running also on regular basis, this option
+should be set to
+.I on
+value.
+.TP
+.B exec_NAME
+defines executables.
+.I NAME
+can be arbitrary valid cmap key name string and it has no special meaning.
+The value of this variable must contain a command to execute. The alue is parsed (split)
+into arguments similarly as Bourne shell would do. Quoting is possible by
+using backslash and double quotes.
+
 .PP
 .B quorum.device.net
-holds the configuration for model 'net'.
+subkey holds the configuration for model 'net'.
 .TP
 .B tls
 Can be one of
@@ -206,9 +256,13 @@ Algorithms are used to change behavior of how
 provides votes to a given node/partition. Currently there are two algorithms supported.
 .TP
 .B ffsplit
-This one makes sense only for clusters with even number of nodes. It provides exactly one
-vote to the partition with the highest number of active nodes. If there are two exactly similar partitions,
-it provides its vote to the partition that has the most clients connected to the qnetd
+This one makes sense only for clusters with an even number of nodes. It provides exactly one
+vote to the partition with the highest number of active nodes. If there are two exactly
+similar partitions,
+it provides its vote to the partition with higher score. The score is computed
+as (number_of_connected_nodes +
+number_of_connected_nodes_with_passed_heuristics - number_of_connected_nodes_with_failed_heuristics)
+If the scores are equal, the vote is provided to partition with the most clients connected to the qnetd
 server. If this number is also equal, then the tie_breaker is used. It is able to transition
 its vote if the currently active partition becomes partitioned and a non-active partition
 still has at least 50% of the active nodes. Because of this, a vote is not provided
@@ -225,8 +279,12 @@ qnetd server then we return a vote.
 
 If more than one node can see the qnetd server but some nodes can't
 see each other then the cluster is divided up into 'partitions' based on
-their ring_id and this algorithm returns a vote to the largest active partition or, 
-if there is more than 1 equal partition, the partition that contains the tie_breaker 
+their ring_id and this algorithm returns a vote to the partition with highest
+heuristics score (computed the same way as for the
+.B ffsplit
+algorithm), or if there is more than 1 partition with equal scores,
+the largest active partition or,
+if there is more than 1 equal partition, the partition that contains the tie_breaker
 node (lowest, highest, etc). For LMS to work, the number
 of qdevice votes has to be set to default (so just delete
 .B quorum.device.votes
@@ -267,6 +325,36 @@ Maximum size of a message allowed to be sent to an IPC client. (65536)
 .B master_wins
 Force enable/disable master wins. (default is model)
 .TP
+.B heuristics_ipc_max_send_buffers
+Maximum number of heuristics worker send buffers. (128)
+.TP
+.B heuristics_ipc_max_send_receive_size
+Maximum size of a message allowed to be send to, or received from heuristics worker. (4096)
+.TP
+.B heuristics_min_timeout
+Minimum heuristics timeout accepted by client in ms. (1000)
+.TP
+.B heuristics_max_timeout
+Maximum heuristics timeout accepted by client in ms. (120000)
+.TP
+.B heuristics_min_interval
+Minimum heuristics interval accepted by client in ms. (1000)
+.TP
+.B heuristics_max_interval
+Maximum heuristics interval accepted by client in ms. (3600000)
+.TP
+.B heuristics_max_execs
+Maximum number of exec_ commands. (32)
+.TP
+.B heuristics_use_execvp
+Use execvp instead of execv for executing commands. (off)
+.TP
+.B heuristics_max_processes
+Maximum number of processes running at one time. (160)
+.B heuristics_kill_list_interval
+Interval between status is gathered and eventually signal is sent
+to processes which didn't finished on time in ms. (5000)
+.TP
 .B net_nss_db_dir
 NSS database directory. (/etc/corosync/qdevice/net/nssdb)
 .TP
@@ -308,6 +396,33 @@ Maximum connection timeout accepted by client in ms. (120000)
 .TP
 .B net_test_algorithm_enabled
 Enable test algorithm. (if built with --enable-debug on, otherwise off)
+
+.SH EXAMPLE
+Define qdevice with
+.I net
+model connecting to qnetd running on qnetd.example.org host, using ffsplit algorithm.
+Heuristics is set to
+.I sync
+mode and executes two commands.
+
+.nf
+quorum {
+  provider: corosync_votequorum
+  device {
+    votes: 1
+    model: net
+    net {
+      tls: on
+      host: qnetd.example.org
+      algorithm: ffsplit
+    }
+    heuristics {
+      mode: sync
+      exec_ping: /bin/ping -q -c 1 "www.example.org"
+      exec_test_txt_exists: /usr/bin/test -f /tmp/test.txt
+    }
+}
+.fi
 .SH SEE ALSO
 .BR corosync-qdevice-tool (8)
 .BR corosync-qdevice-net-certutil (8)

qdevices/Makefile.am

@@ -1,7 +1,7 @@
-# Copyright (c) 2012-2016 Red Hat, Inc.
+# Copyright (c) 2012-2017 Red Hat, Inc.
 #
-# Authors: Fabio M. Di Nitto (fdinitto@redhat.com)
-#          Jan Friesse (jfriesse@redhat.com)
+# Authors: Jan Friesse (jfriesse@redhat.com)
+#          Fabio M. Di Nitto (fdinitto@redhat.com)
 #
 # This software licensed under BSD license, the text of which follows:
 #
@@ -73,7 +73,7 @@ corosync_qnetd_SOURCES	= corosync-qnetd.c \
                           dynar-getopt-lex.h qnetd-advanced-settings.c qnetd-advanced-settings.h
 
 corosync_qnetd_tool_SOURCES = corosync-qnetd-tool.c unix-socket.c unix-socket.h dynar.c dynar.h \
-                              dynar-str.c dynar-str.h
+                              dynar-str.c dynar-str.h utils.c utils.h
 
 corosync_qnetd_CFLAGS		= $(nss_CFLAGS)
 corosync_qnetd_LDADD		= $(nss_LIBS)
@@ -124,10 +124,26 @@ corosync_qdevice_SOURCES = corosync-qdevice.c \
                            qdevice-net-poll-array-user-data.h \
                            qdevice-config.h qnet-config.h qdevice-net-disconnect-reason.h \
                            qdevice-model-type.h qdevice-advanced-settings.c \
-                           qdevice-advanced-settings.h dynar-getopt-lex.c dynar-getopt-lex.h
+                           qdevice-advanced-settings.h dynar-getopt-lex.c dynar-getopt-lex.h \
+                           qdevice-heuristics.h qdevice-heuristics.c \
+                           qdevice-heuristics-worker.h qdevice-heuristics-worker.c \
+                           qdevice-heuristics-io.h qdevice-heuristics-io.c \
+                           qdevice-heuristics-worker-instance.h \
+                           qdevice-heuristics-worker-log.h qdevice-heuristics-worker-log.c \
+                           qdevice-heuristics-log.h qdevice-heuristics-log.c \
+                           qdevice-heuristics-instance.h qdevice-heuristics-instance.c \
+                           qdevice-heuristics-mode.h qdevice-heuristics-mode.c \
+                           qdevice-heuristics-exec-list.c qdevice-heuristics-exec-list.h \
+                           qdevice-heuristics-cmd.c qdevice-heuristics-cmd.h \
+                           qdevice-heuristics-worker-cmd.c qdevice-heuristics-worker-cmd.h \
+                           qdevice-heuristics-cmd-str.h \
+                           qdevice-heuristics-exec-result.c qdevice-heuristics-exec-result.h \
+                           process-list.h process-list.c \
+                           qdevice-net-heuristics.c qdevice-net-heuristics.h \
+                           qdevice-heuristics-result-notifier.c qdevice-heuristics-result-notifier.h
 
 corosync_qdevice_tool_SOURCES = corosync-qdevice-tool.c unix-socket.c unix-socket.h dynar.c dynar.h \
-                                dynar-str.c dynar-str.h
+                                dynar-str.c dynar-str.h utils.c utils.h
 
 corosync_qdevice_CFLAGS	= $(nss_CFLAGS)
 corosync_qdevice_LDADD	= $(nss_LIBS) $(LIBQB_LIBS) $(top_builddir)/lib/libcmap.la \
@@ -140,9 +156,9 @@ corosync-qdevice-net-certutil: corosync-qdevice-net-certutil.sh
 	    $< > $@
 
 TESTS				= qnetd-cluster-list.test dynar.test dynar-simple-lex.test \
-                                  dynar-getopt-lex.test
+                                  dynar-getopt-lex.test process-list.test
 check_PROGRAMS			= qnetd-cluster-list.test dynar.test dynar-simple-lex.test \
-                                  dynar-getopt-lex.test
+                                  dynar-getopt-lex.test process-list.test
 
 qnetd_cluster_list_test_SOURCES	= qnetd-cluster-list.c test-qnetd-cluster-list.c \
                                   qnetd-cluster.c qnetd-cluster.h \
@@ -154,5 +170,7 @@ qnetd_cluster_list_test_LDADD	= $(nss_LIBS)
 dynar_test_SOURCES		= test-dynar.c dynar.c dynar-str.c
 dynar_simple_lex_test_SOURCES	= test-dynar-simple-lex.c dynar.c dynar-str.c dynar-simple-lex.c
 dynar_getopt_lex_test_SOURCES	= test-dynar-getopt-lex.c dynar.c dynar-str.c dynar-getopt-lex.c
+process_list_test_SOURCES	= test-process-list.c dynar.c dynar-str.c dynar-simple-lex.c \
+                                  process-list.c
 
 endif

qdevices/corosync-qdevice-tool.c

@@ -249,7 +249,8 @@ main(int argc, char * const argv[])
 		errx(QDEVICE_TOOL_EXIT_CODE_INTERNAL_ERROR, "Can't store command");
 	}
 
-	if (fprintf(sock, "%s", dynar_data(&send_str)) != strlen(dynar_data(&send_str)) ||
+	res = fprintf(sock, "%s", dynar_data(&send_str));
+	if (res < 0 || (size_t)res != strlen(dynar_data(&send_str)) ||
 	    fflush(sock) != 0) {
 		errx(QDEVICE_TOOL_EXIT_CODE_INTERNAL_ERROR, "Can't send command");
 	}

qdevices/corosync-qdevice.c

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -41,6 +41,7 @@
 #include "qdevice-advanced-settings.h"
 #include "qdevice-config.h"
 #include "qdevice-cmap.h"
+#include "qdevice-heuristics.h"
 #include "qdevice-ipc.h"
 #include "qdevice-log.h"
 #include "qdevice-model.h"
@@ -79,6 +80,18 @@ signal_handlers_register(void)
 	act.sa_flags = SA_RESTART;
 
 	sigaction(SIGTERM, &act, NULL);
+
+	act.sa_handler = SIG_DFL;
+	sigemptyset(&act.sa_mask);
+	act.sa_flags = SA_RESTART;
+
+	sigaction(SIGCHLD, &act, NULL);
+
+	act.sa_handler = SIG_IGN;
+	sigemptyset(&act.sa_mask);
+	act.sa_flags = SA_RESTART;
+
+	sigaction(SIGPIPE, &act, NULL);
 }
 
 static void
@@ -170,6 +183,9 @@ main(int argc, char * const argv[])
 
 	qdevice_instance_init(&instance, &advanced_settings);
 
+	qdevice_heuristics_init(&instance.heuristics_instance, &advanced_settings);
+	instance.heuristics_instance.qdevice_instance_ptr = &instance;
+
 	qdevice_cmap_init(&instance);
 	qdevice_log_init(&instance, force_debug);
 
@@ -233,6 +249,11 @@ main(int argc, char * const argv[])
 		return (1);
 	}
 
+	qdevice_log(LOG_DEBUG, "Waiting for initial heuristics exec result");
+	if (qdevice_heuristics_wait_for_initial_exec_result(&instance.heuristics_instance) != 0) {
+		return (1);
+	}
+
 	global_instance = &instance;
 	signal_handlers_register();
 
@@ -249,12 +270,21 @@ main(int argc, char * const argv[])
 	qdevice_log(LOG_DEBUG, "Destroying qdevice model");
 	qdevice_model_destroy(&instance);
 
+	qdevice_log(LOG_DEBUG, "Destroying qdevice ipc");
 	qdevice_ipc_destroy(&instance);
 
+	qdevice_log(LOG_DEBUG, "Destroying votequorum and cmap");
 	qdevice_votequorum_destroy(&instance);
 	qdevice_cmap_destroy(&instance);
+
+	qdevice_log(LOG_DEBUG, "Destroying heuristics");
+	qdevice_heuristics_destroy(&instance.heuristics_instance);
+
+	qdevice_log(LOG_DEBUG, "Closing log");
 	qdevice_log_close(&instance);
+
 	qdevice_instance_destroy(&instance);
+
 	qdevice_advanced_settings_destroy(&advanced_settings);
 
 	return (0);

qdevices/corosync-qnetd-tool.c

@@ -277,7 +277,8 @@ main(int argc, char * const argv[])
 		errx(QNETD_TOOL_EXIT_CODE_INTERNAL_ERROR, "Can't store command");
 	}
 
-	if (fprintf(sock, "%s", dynar_data(&send_str)) != strlen(dynar_data(&send_str)) ||
+	res = fprintf(sock, "%s", dynar_data(&send_str));
+	if (res < 0 || (size_t)res != strlen(dynar_data(&send_str)) ||
 	    fflush(sock) != 0) {
 		errx(QNETD_TOOL_EXIT_CODE_INTERNAL_ERROR, "Can't send command");
 	}

qdevices/dynar-str.c

@@ -86,7 +86,7 @@ dynar_str_vcatf(struct dynar *dest, const char *format, va_list ap)
 		return (-1);
 	}
 
-	if (to_write < sizeof(buf)) {
+	if ((size_t)to_write < sizeof(buf)) {
 		/*
 		 * Writing 1 byte string (snprintf writes also '\0') means string is empty
 		 */
@@ -105,7 +105,11 @@ dynar_str_vcatf(struct dynar *dest, const char *format, va_list ap)
 	written = vsnprintf(p, allocated, format, ap_copy);
 	va_end(ap_copy);
 
-	if (written >= allocated) {
+	if (written < 0) {
+		return (-1);
+	}
+
+	if ((size_t)written >= allocated) {
 		return (-1);
 	}
 

qdevices/dynar.c

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -54,6 +54,25 @@ dynar_set_max_size(struct dynar *array, size_t maximum_size)
 	array->maximum_size = maximum_size;
 }
 
+int
+dynar_set_size(struct dynar *array, size_t size)
+{
+
+	if (size > dynar_max_size(array)) {
+		dynar_set_max_size(array, size);
+	}
+
+	if (size > dynar_size(array)) {
+		if (dynar_prealloc(array, size - dynar_size(array)) == -1) {
+			return (-1);
+		}
+	}
+
+	array->size = size;
+
+	return (0);
+}
+
 void
 dynar_destroy(struct dynar *array)
 {

qdevices/dynar.h

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -72,6 +72,8 @@ extern int	 dynar_prealloc(struct dynar *array, size_t size);
 
 extern int	 dynar_prepend(struct dynar *array, const void *src, size_t size);
 
+extern int	 dynar_set_size(struct dynar *array, size_t size);
+
 #ifdef __cplusplus
 }
 #endif

qdevices/msg.c

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -44,7 +44,7 @@
 #define MSG_TYPE_LENGTH		2
 #define MSG_LENGTH_LENGTH	4
 
-#define MSG_STATIC_SUPPORTED_MESSAGES_SIZE	16
+#define MSG_STATIC_SUPPORTED_MESSAGES_SIZE	18
 
 enum msg_type msg_static_supported_messages[MSG_STATIC_SUPPORTED_MESSAGES_SIZE] = {
     MSG_TYPE_PREINIT,
@@ -63,6 +63,8 @@ enum msg_type msg_static_supported_messages[MSG_STATIC_SUPPORTED_MESSAGES_SIZE]
     MSG_TYPE_ASK_FOR_VOTE_REPLY,
     MSG_TYPE_VOTE_INFO,
     MSG_TYPE_VOTE_INFO_REPLY,
+    MSG_TYPE_HEURISTICS_CHANGE,
+    MSG_TYPE_HEURISTICS_CHANGE_REPLY,
 };
 
 size_t
@@ -521,6 +523,7 @@ msg_create_node_list(struct dynar *msg,
     int add_ring_id, const struct tlv_ring_id *ring_id,
     int add_config_version, uint64_t config_version,
     int add_quorate, enum tlv_quorate quorate,
+    int add_heuristics, enum tlv_heuristics heuristics,
     const struct node_list *nodes)
 {
 	struct node_list_entry *node_info;
@@ -565,6 +568,12 @@ msg_create_node_list(struct dynar *msg,
 		}
 	}
 
+	if (add_heuristics && heuristics != TLV_HEURISTICS_UNDEFINED) {
+		if (tlv_add_heuristics(msg, heuristics) == -1) {
+			goto small_buf_err;
+		}
+	}
+
 	msg_set_len(msg, dynar_size(msg) - (MSG_TYPE_LENGTH + MSG_LENGTH_LENGTH));
 
 	return (dynar_size(msg));
@@ -710,6 +719,66 @@ small_buf_err:
 	return (0);
 }
 
+size_t
+msg_create_heuristics_change(struct dynar *msg, uint32_t msg_seq_number,
+    enum tlv_heuristics heuristics)
+{
+
+	dynar_clean(msg);
+
+	msg_add_type(msg, MSG_TYPE_HEURISTICS_CHANGE);
+	msg_add_len(msg);
+
+	if (tlv_add_msg_seq_number(msg, msg_seq_number) == -1) {
+		goto small_buf_err;
+	}
+
+	if (tlv_add_heuristics(msg, heuristics) == -1) {
+		goto small_buf_err;
+	}
+
+	msg_set_len(msg, dynar_size(msg) - (MSG_TYPE_LENGTH + MSG_LENGTH_LENGTH));
+
+	return (dynar_size(msg));
+
+small_buf_err:
+	return (0);
+}
+
+size_t
+msg_create_heuristics_change_reply(struct dynar *msg, uint32_t msg_seq_number,
+    const struct tlv_ring_id *ring_id, enum tlv_heuristics heuristics, enum tlv_vote vote)
+{
+
+	dynar_clean(msg);
+
+	msg_add_type(msg, MSG_TYPE_HEURISTICS_CHANGE_REPLY);
+	msg_add_len(msg);
+
+	if (tlv_add_msg_seq_number(msg, msg_seq_number) == -1) {
+		goto small_buf_err;
+	}
+
+	if (tlv_add_vote(msg, vote) == -1) {
+		goto small_buf_err;
+	}
+
+	if (tlv_add_ring_id(msg, ring_id) == -1) {
+		goto small_buf_err;
+	}
+
+	if (tlv_add_heuristics(msg, heuristics) == -1) {
+		goto small_buf_err;
+	}
+
+	msg_set_len(msg, dynar_size(msg) - (MSG_TYPE_LENGTH + MSG_LENGTH_LENGTH));
+
+	return (dynar_size(msg));
+
+small_buf_err:
+	return (0);
+}
+
 int
 msg_is_valid_msg_type(const struct dynar *msg)
 {
@@ -968,6 +1037,12 @@ msg_decode(const struct dynar *msg, struct msg_decoded *decoded_msg)
 			decoded_msg->tie_breaker_set = 1;
 			memcpy(&decoded_msg->tie_breaker, &tie_breaker, sizeof(tie_breaker));
 			break;
+		case TLV_OPT_HEURISTICS:
+			if ((res = tlv_iter_decode_heuristics(&tlv_iter,
+			    &decoded_msg->heuristics)) != 0) {
+				return (res);
+			}
+			break;
 		/*
 		 * Default is not defined intentionally. Compiler shows warning when
 		 * new tlv option is added. Also protocol ignores unknown options so
@@ -1012,6 +1087,8 @@ msg_type_to_str(enum msg_type type)
 	case MSG_TYPE_ASK_FOR_VOTE_REPLY: return ("Ask for vote reply"); break;
 	case MSG_TYPE_VOTE_INFO: return ("Vote info"); break;
 	case MSG_TYPE_VOTE_INFO_REPLY: return ("Vote info reply"); break;
+	case MSG_TYPE_HEURISTICS_CHANGE: return ("Heuristics change"); break;
+	case MSG_TYPE_HEURISTICS_CHANGE_REPLY: return ("Heuristics change reply"); break;
 	}
 
 	return ("Unknown message type");

qdevices/msg.h

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -63,6 +63,8 @@ enum msg_type {
 	MSG_TYPE_ASK_FOR_VOTE_REPLY = 13,
 	MSG_TYPE_VOTE_INFO = 14,
 	MSG_TYPE_VOTE_INFO_REPLY = 15,
+	MSG_TYPE_HEURISTICS_CHANGE = 16,
+	MSG_TYPE_HEURISTICS_CHANGE_REPLY = 17,
 };
 
 struct msg_decoded {
@@ -112,6 +114,7 @@ struct msg_decoded {
 	enum tlv_quorate quorate;	/* Valid only if quorate_set != 0 */
 	int tie_breaker_set;
 	struct tlv_tie_breaker tie_breaker;
+	enum tlv_heuristics heuristics;	/* Always valid but can be TLV_HEURISTICS_UNDEFINED */
 };
 
 extern size_t		msg_create_preinit(struct dynar *msg, const char *cluster_name,
@@ -159,6 +162,7 @@ extern size_t		msg_create_node_list(struct dynar *msg,
     int add_ring_id, const struct tlv_ring_id *ring_id,
     int add_config_version, uint64_t config_version,
     int add_quorate, enum tlv_quorate quorate,
+    int add_heuristics, enum tlv_heuristics heuristics,
     const struct node_list *nodes);
 
 extern size_t		msg_create_node_list_reply(struct dynar *msg, uint32_t msg_seq_number,
@@ -175,6 +179,13 @@ extern size_t		msg_create_vote_info(struct dynar *msg, uint32_t msg_seq_number,
 
 extern size_t		msg_create_vote_info_reply(struct dynar *msg, uint32_t msg_seq_number);
 
+extern size_t		msg_create_heuristics_change(struct dynar *msg, uint32_t msg_seq_number,
+    enum tlv_heuristics heuristics);
+
+extern size_t		msg_create_heuristics_change_reply(struct dynar *msg,
+    uint32_t msg_seq_number, const struct tlv_ring_id *ring_id, enum tlv_heuristics heuristics,
+    enum tlv_vote vote);
+
 extern size_t		msg_get_header_length(void);
 
 extern uint32_t		msg_get_len(const struct dynar *msg);

qdevices/process-list.c

@@ -0,0 +1,621 @@
+/*
+ * Copyright (c) 2015-2016 Red Hat, Inc.
+ *
+ * All rights reserved.
+ *
+ * Author: Jan Friesse (jfriesse@redhat.com)
+ *
+ * This software licensed under BSD license, the text of which follows:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright notice,
+ *   this list of conditions and the following disclaimer.
+ * - Redistributions in binary form must reproduce the above copyright notice,
+ *   this list of conditions and the following disclaimer in the documentation
+ *   and/or other materials provided with the distribution.
+ * - Neither the name of the Red Hat, Inc. nor the names of its
+ *   contributors may be used to endorse or promote products derived from this
+ *   software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+#include <sys/wait.h>
+
+#include <string.h>
+#include <stdlib.h>
+#include <assert.h>
+#include <err.h>
+#include <errno.h>
+#include <poll.h>
+#include <unistd.h>
+
+#include "dynar.h"
+#include "dynar-str.h"
+#include "dynar-simple-lex.h"
+#include "process-list.h"
+
+static void		process_list_free_argv(size_t no_params, char **argv);
+
+static void		process_list_entry_free(struct process_list_entry *entry);
+
+static char 		**process_list_parse_command(const char *command, size_t *no_params);
+
+static int		process_list_entry_exec(const struct process_list *plist,
+    struct process_list_entry *entry);
+
+void
+process_list_init(struct process_list *plist, size_t max_list_entries, int use_execvp,
+    process_list_notify_fn_t notify_fn, void *notify_fn_user_data)
+{
+
+	memset(plist, 0, sizeof(*plist));
+
+	plist->max_list_entries = max_list_entries;
+	plist->allocated_list_entries = 0;
+	plist->use_execvp = use_execvp;
+	plist->notify_fn = notify_fn;
+	plist->notify_fn_user_data = notify_fn_user_data;
+
+	TAILQ_INIT(&plist->active_list);
+	TAILQ_INIT(&plist->to_kill_list);
+}
+
+static void
+process_list_free_argv(size_t no_params, char **argv)
+{
+	size_t zi;
+
+	for (zi = 0; zi < no_params; zi++) {
+		free(argv[zi]);
+	}
+	free(argv);
+}
+
+static void
+process_list_entry_free(struct process_list_entry *entry)
+{
+
+	process_list_free_argv(entry->exec_argc, entry->exec_argv);
+	free(entry->name);
+	free(entry);
+}
+
+static char **
+process_list_parse_command(const char *command, size_t *no_params)
+{
+	struct dynar command_dstr;
+	struct dynar_simple_lex lex;
+	struct dynar *token;
+	int finished;
+	char **res_argv;
+	size_t zi;
+
+	res_argv = NULL;
+
+	dynar_init(&command_dstr, strlen(command) + 1);
+	if (dynar_str_cpy(&command_dstr, command) != 0) {
+		return (NULL);
+	}
+
+	dynar_simple_lex_init(&lex, &command_dstr, DYNAR_SIMPLE_LEX_TYPE_QUOTE);
+	*no_params = 0;
+	finished = 0;
+
+	while (!finished) {
+		token = dynar_simple_lex_token_next(&lex);
+		if (token == NULL) {
+			goto exit_res;
+		}
+
+		if (strcmp(dynar_data(token), "") == 0) {
+			finished = 1;
+		} else {
+			(*no_params)++;
+		}
+	}
+
+	if (*no_params < 1) {
+		goto exit_res;
+	}
+
+	dynar_simple_lex_destroy(&lex);
+
+	res_argv = malloc(sizeof(char *) * (*no_params + 1));
+	if (res_argv == NULL) {
+		goto exit_res;
+	}
+	memset(res_argv, 0, sizeof(char *) * (*no_params + 1));
+
+	dynar_simple_lex_init(&lex, &command_dstr, DYNAR_SIMPLE_LEX_TYPE_QUOTE);
+
+	finished = 0;
+	zi = 0;
+	while (!finished) {
+		token = dynar_simple_lex_token_next(&lex);
+		if (token == NULL) {
+			process_list_free_argv(*no_params, res_argv);
+			res_argv = NULL;
+			goto exit_res;
+		}
+
+		if (strcmp(dynar_data(token), "") == 0) {
+			finished = 1;
+		} else {
+			res_argv[zi] = strdup(dynar_data(token));
+			if (res_argv[zi] == NULL) {
+				process_list_free_argv(*no_params, res_argv);
+				res_argv = NULL;
+			}
+			zi++;
+		}
+	}
+
+	if (zi != *no_params) {
+		/*
+		 * If this happens it means something is seriously broken (memory corrupted)
+		 */
+		process_list_free_argv(*no_params, res_argv);
+		res_argv = NULL;
+		goto exit_res;
+	}
+
+exit_res:
+	dynar_simple_lex_destroy(&lex);
+	dynar_destroy(&command_dstr);
+	return (res_argv);
+}
+
+struct process_list_entry *
+process_list_add(struct process_list *plist, const char *name, const char *command)
+{
+	struct process_list_entry *entry;
+
+	if (plist->allocated_list_entries + 1 > plist->max_list_entries) {
+		return (NULL);
+	}
+
+	/*
+	 * Alloc new entry
+	 */
+	entry = malloc(sizeof(*entry));
+	if (entry == NULL) {
+		return (NULL);
+	}
+
+	memset(entry, 0, sizeof(*entry));
+	entry->name = strdup(name);
+	if (entry->name == NULL) {
+		process_list_entry_free(entry);
+
+		return (NULL);
+	}
+
+	entry->state = PROCESS_LIST_ENTRY_STATE_INITIALIZED;
+	entry->exec_argv = process_list_parse_command(command, &entry->exec_argc);
+	if (entry->exec_argv == NULL) {
+		process_list_entry_free(entry);
+
+		return (NULL);
+	}
+
+	plist->allocated_list_entries++;
+	TAILQ_INSERT_TAIL(&plist->active_list, entry, entries);
+
+	return (entry);
+}
+
+void
+process_list_free(struct process_list *plist)
+{
+	struct process_list_entry *entry;
+	struct process_list_entry *entry_next;
+
+	entry = TAILQ_FIRST(&plist->active_list);
+
+	while (entry != NULL) {
+		entry_next = TAILQ_NEXT(entry, entries);
+
+		process_list_entry_free(entry);
+
+		entry = entry_next;
+	}
+
+	entry = TAILQ_FIRST(&plist->to_kill_list);
+
+	while (entry != NULL) {
+		entry_next = TAILQ_NEXT(entry, entries);
+
+		process_list_entry_free(entry);
+
+		entry = entry_next;
+	}
+
+	plist->allocated_list_entries = 0;
+
+	TAILQ_INIT(&plist->active_list);
+	TAILQ_INIT(&plist->to_kill_list);
+}
+
+static void
+process_list_entry_exec_helper_set_stdfd(void)
+{
+	int devnull;
+
+	devnull = open("/dev/null", O_RDWR);
+	if (devnull == -1) {
+		err(1, "Can't open /dev/null");
+	}
+
+	if (dup2(devnull, 0) < 0 || dup2(devnull, 1) < 0 || dup2(devnull, 2) < 0) {
+		close(devnull);
+		err(1, "Can't dup2 stdin/out/err to /dev/null");
+	}
+
+	close(devnull);
+}
+
+static int
+process_list_entry_exec(const struct process_list *plist, struct process_list_entry *entry)
+{
+	pid_t pid;
+
+	if (entry->state != PROCESS_LIST_ENTRY_STATE_INITIALIZED) {
+		return (-1);
+	}
+
+	pid = fork();
+	if (pid == -1) {
+		return (-1);
+	} else if (pid == 0) {
+		process_list_entry_exec_helper_set_stdfd();
+
+		if (!plist->use_execvp) {
+			execv(entry->exec_argv[0], entry->exec_argv);
+		} else {
+			execvp(entry->exec_argv[0], entry->exec_argv);
+		}
+
+		/*
+		 * Exec returned -> exec failed
+		 */
+		err(1, "Can't execute command %s (%s)", entry->name, entry->exec_argv[0]);
+	} else {
+		entry->pid = pid;
+		entry->state = PROCESS_LIST_ENTRY_STATE_RUNNING;
+
+		if (plist->notify_fn != NULL) {
+			plist->notify_fn(PROCESS_LIST_NOTIFY_REASON_EXECUTED, entry,
+			    plist->notify_fn_user_data);
+		}
+	}
+
+	return (0);
+}
+
+int
+process_list_exec_initialized(struct process_list *plist)
+{
+	struct process_list_entry *entry;
+
+	TAILQ_FOREACH(entry, &plist->active_list, entries) {
+		if (entry->state == PROCESS_LIST_ENTRY_STATE_INITIALIZED) {
+			if (process_list_entry_exec(plist, entry) != 0) {
+				return (-1);
+			}
+		}
+	}
+
+	return (0);
+}
+
+static int
+process_list_entry_waitpid(const struct process_list *plist, struct process_list_entry *entry)
+{
+	pid_t wpid_res;
+	int status;
+
+	if (entry->state == PROCESS_LIST_ENTRY_STATE_INITIALIZED ||
+	    entry->state == PROCESS_LIST_ENTRY_STATE_FINISHED) {
+		return (0);
+	}
+
+	wpid_res = waitpid(entry->pid, &status, WNOHANG);
+	if (wpid_res == -1) {
+		return (-1);
+	}
+
+	if (wpid_res == 0) {
+		/*
+		 * No change
+		 */
+		return (0);
+	}
+
+	entry->exit_status = status;
+
+	if (entry->state == PROCESS_LIST_ENTRY_STATE_RUNNING) {
+		if (plist->notify_fn != NULL) {
+			plist->notify_fn(PROCESS_LIST_NOTIFY_REASON_FINISHED, entry,
+			    plist->notify_fn_user_data);
+		}
+	}
+
+	entry->state = PROCESS_LIST_ENTRY_STATE_FINISHED;
+
+	return (0);
+}
+
+int
+process_list_waitpid(struct process_list *plist)
+{
+	struct process_list_entry *entry;
+	struct process_list_entry *entry_next;
+
+	TAILQ_FOREACH(entry, &plist->active_list, entries) {
+		if (process_list_entry_waitpid(plist, entry) != 0) {
+			return (-1);
+		}
+	}
+
+	entry = TAILQ_FIRST(&plist->to_kill_list);
+
+	while (entry != NULL) {
+		entry_next = TAILQ_NEXT(entry, entries);
+
+		if (process_list_entry_waitpid(plist, entry) != 0) {
+			return (-1);
+		}
+
+		if (entry->state == PROCESS_LIST_ENTRY_STATE_FINISHED) {
+			/*
+			 * Process finished -> remove it from list
+			 */
+			TAILQ_REMOVE(&plist->to_kill_list, entry, entries);
+			process_list_entry_free(entry);
+			plist->allocated_list_entries--;
+		}
+
+		entry = entry_next;
+	}
+
+	return (0);
+}
+
+size_t
+process_list_get_no_running(struct process_list *plist)
+{
+	struct process_list_entry *entry;
+	size_t res;
+
+	res = 0;
+
+	TAILQ_FOREACH(entry, &plist->active_list, entries) {
+		if (entry->state == PROCESS_LIST_ENTRY_STATE_RUNNING) {
+			res++;
+		}
+	}
+
+	return (res);
+}
+
+/*
+ * -1 = Not all processes finished
+ *  0 = All processes finished sucesfully
+ *  1 - All processes finished but some of them not sucesfully
+ */
+int
+process_list_get_summary_result(struct process_list *plist)
+{
+	struct process_list_entry *entry;
+	int res;
+
+	res = 0;
+
+	TAILQ_FOREACH(entry, &plist->active_list, entries) {
+		if (entry->state != PROCESS_LIST_ENTRY_STATE_FINISHED) {
+			return (-1);
+		}
+
+		if (!WIFEXITED(entry->exit_status) || WEXITSTATUS(entry->exit_status) != 0) {
+			res = 1;
+		}
+	}
+
+	return (res);
+}
+
+/*
+ *  0 = All processes finished sucesfully
+ *  1 = Some process finished and failed
+ * -1 = Not all processed finished and none of finished failed
+ */
+int
+process_list_get_summary_result_short(struct process_list *plist)
+{
+	struct process_list_entry *entry;
+	int res;
+
+	res = 0;
+
+	TAILQ_FOREACH(entry, &plist->active_list, entries) {
+		if (entry->state == PROCESS_LIST_ENTRY_STATE_FINISHED) {
+			if (!WIFEXITED(entry->exit_status) || WEXITSTATUS(entry->exit_status) != 0) {
+				return (1);
+			}
+		} else {
+			res = -1;
+		}
+	}
+
+	return (res);
+}
+
+static void
+process_list_move_entry_to_kill_list(struct process_list *plist, struct process_list_entry *entry)
+{
+
+	TAILQ_REMOVE(&plist->active_list, entry, entries);
+	TAILQ_INSERT_TAIL(&plist->to_kill_list, entry, entries);
+}
+
+void
+process_list_move_active_entries_to_kill_list(struct process_list *plist)
+{
+	struct process_list_entry *entry;
+	struct process_list_entry *entry_next;
+
+	entry = TAILQ_FIRST(&plist->active_list);
+
+	while (entry != NULL) {
+		entry_next = TAILQ_NEXT(entry, entries);
+
+		if (entry->state == PROCESS_LIST_ENTRY_STATE_INITIALIZED ||
+		    entry->state == PROCESS_LIST_ENTRY_STATE_FINISHED) {
+			TAILQ_REMOVE(&plist->active_list, entry, entries);
+			process_list_entry_free(entry);
+			plist->allocated_list_entries--;
+		} else {
+			process_list_move_entry_to_kill_list(plist, entry);
+		}
+
+		entry = entry_next;
+	}
+}
+
+static int
+process_list_process_kill_list_entry(struct process_list *plist, struct process_list_entry *entry)
+{
+	int sig_to_send;
+	enum process_list_entry_state new_state;
+	int res;
+
+	sig_to_send = 0;
+	new_state = PROCESS_LIST_ENTRY_STATE_INITIALIZED;
+
+	switch (entry->state) {
+	case PROCESS_LIST_ENTRY_STATE_INITIALIZED:
+		/*
+		 * This shouldn't happen. If it does, process_list_move_active_entries_to_kill_list
+		 * doesn't work as expected or there is some kind of memory corruption.
+		 */
+		assert(entry->state != PROCESS_LIST_ENTRY_STATE_INITIALIZED);
+		break;
+	case PROCESS_LIST_ENTRY_STATE_FINISHED:
+		/*
+		 * This shouldn't happen. If it does, process_list_waitpid
+		 * doesn't work as expected or there is some kind of memory corruption.
+		 */
+		assert(entry->state != PROCESS_LIST_ENTRY_STATE_FINISHED);
+		break;
+	case PROCESS_LIST_ENTRY_STATE_RUNNING:
+		sig_to_send = SIGTERM;
+		new_state = PROCESS_LIST_ENTRY_STATE_SIGTERM_SENT;
+		break;
+	case PROCESS_LIST_ENTRY_STATE_SIGTERM_SENT:
+		sig_to_send = SIGKILL;
+		new_state = PROCESS_LIST_ENTRY_STATE_SIGKILL_SENT;
+		break;
+	case PROCESS_LIST_ENTRY_STATE_SIGKILL_SENT:
+		sig_to_send = SIGKILL;
+		new_state = PROCESS_LIST_ENTRY_STATE_SIGKILL_SENT;
+		break;
+	}
+
+	res = 0;
+
+	if (kill(entry->pid, sig_to_send) == -1) {
+		if (errno == EPERM || errno == EINVAL) {
+			res = -1;
+		}
+	}
+
+	entry->state = new_state;
+
+	return (res);
+}
+
+int
+process_list_process_kill_list(struct process_list *plist)
+{
+	struct process_list_entry *entry;
+
+	if (process_list_waitpid(plist) != 0) {
+		return (-1);
+	}
+
+	TAILQ_FOREACH(entry, &plist->to_kill_list, entries) {
+		if (process_list_process_kill_list_entry(plist, entry) != 0) {
+			return (-1);
+		}
+	}
+
+	return (0);
+}
+
+size_t
+process_list_get_kill_list_items(struct process_list *plist)
+{
+	struct process_list_entry *entry;
+	size_t res;
+
+	res = 0;
+
+	TAILQ_FOREACH(entry, &plist->to_kill_list, entries) {
+		res++;
+	}
+
+	return (res);
+}
+
+int
+process_list_killall(struct process_list *plist, uint32_t timeout)
+{
+	uint32_t action_timeout;
+	int i;
+
+	process_list_move_active_entries_to_kill_list(plist);
+
+	action_timeout = timeout / 10;
+	if (action_timeout < 1) {
+		action_timeout = 1;
+	}
+
+	for (i = 0; i < 10; i++) {
+		/*
+		 * Make sure all process got signal (quick phase)
+		 */
+		if (process_list_process_kill_list(plist) != 0) {
+			return (-1);
+		}
+	}
+
+	for (i = 0; i < 10 && process_list_get_kill_list_items(plist) > 0; i++) {
+		if (process_list_process_kill_list(plist) != 0) {
+			return (-1);
+		}
+
+		poll(NULL, 0, action_timeout);
+	}
+
+	if (process_list_get_kill_list_items(plist) > 0) {
+		return (-1);
+	}
+
+	return (0);
+}

qdevices/process-list.h

@@ -0,0 +1,119 @@
+/*
+ * Copyright (c) 2015-2017 Red Hat, Inc.
+ *
+ * All rights reserved.
+ *
+ * Author: Jan Friesse (jfriesse@redhat.com)
+ *
+ * This software licensed under BSD license, the text of which follows:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright notice,
+ *   this list of conditions and the following disclaimer.
+ * - Redistributions in binary form must reproduce the above copyright notice,
+ *   this list of conditions and the following disclaimer in the documentation
+ *   and/or other materials provided with the distribution.
+ * - Neither the name of the Red Hat, Inc. nor the names of its
+ *   contributors may be used to endorse or promote products derived from this
+ *   software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _PROCESS_LIST_H_
+#define _PROCESS_LIST_H_
+
+#include <sys/queue.h>
+
+#include "dynar.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+enum process_list_entry_state {
+	PROCESS_LIST_ENTRY_STATE_INITIALIZED,
+	PROCESS_LIST_ENTRY_STATE_RUNNING,
+	PROCESS_LIST_ENTRY_STATE_FINISHED,
+	PROCESS_LIST_ENTRY_STATE_SIGTERM_SENT,
+	PROCESS_LIST_ENTRY_STATE_SIGKILL_SENT,
+};
+
+enum process_list_notify_reason {
+	PROCESS_LIST_NOTIFY_REASON_EXECUTED,
+	PROCESS_LIST_NOTIFY_REASON_FINISHED,
+};
+
+struct process_list_entry {
+	char *name;
+	enum process_list_entry_state state;
+	char **exec_argv;
+	size_t exec_argc;
+	pid_t pid;
+	int exit_status;
+
+	TAILQ_ENTRY(process_list_entry) entries;
+};
+
+typedef void (*process_list_notify_fn_t) (enum process_list_notify_reason reason,
+    const struct process_list_entry *entry, void *user_data);
+
+struct process_list {
+	int use_execvp;
+	size_t max_list_entries;
+	size_t allocated_list_entries;
+	process_list_notify_fn_t notify_fn;
+	void *notify_fn_user_data;
+
+	TAILQ_HEAD(, process_list_entry) active_list;
+	TAILQ_HEAD(, process_list_entry) to_kill_list;
+};
+
+
+extern void				 process_list_init(struct process_list *plist,
+    size_t max_list_entries, int use_execvp, process_list_notify_fn_t notify_fn,
+    void *notify_fn_user_data);
+
+extern struct process_list_entry	*process_list_add(struct process_list *plist,
+    const char *name, const char *command);
+
+extern void				 process_list_free(struct process_list *plist);
+
+extern int				 process_list_exec_initialized(struct process_list *plist);
+
+extern int				 process_list_waitpid(struct process_list *plist);
+
+extern size_t				 process_list_get_no_running(struct process_list *plist);
+
+extern int				 process_list_get_summary_result(struct process_list *plist);
+
+extern int				 process_list_get_summary_result_short(
+    struct process_list *plist);
+
+extern void				 process_list_move_active_entries_to_kill_list(
+    struct process_list *plist);
+
+extern int				 process_list_process_kill_list(struct process_list *plist);
+
+extern size_t				 process_list_get_kill_list_items(struct process_list *plist);
+
+extern int				 process_list_killall(struct process_list *plist,
+    uint32_t timeout);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _PROCESS_LIST_H_ */

qdevices/qdevice-advanced-settings.c

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -63,6 +63,21 @@ qdevice_advanced_settings_init(struct qdevice_advanced_settings *settings)
 	settings->ipc_max_clients = QDEVICE_DEFAULT_IPC_MAX_CLIENTS;
 	settings->ipc_max_receive_size = QDEVICE_DEFAULT_IPC_MAX_RECEIVE_SIZE;
 	settings->ipc_max_send_size = QDEVICE_DEFAULT_IPC_MAX_SEND_SIZE;
+
+	settings->heuristics_ipc_max_send_buffers = QDEVICE_DEFAULT_HEURISTICS_IPC_MAX_SEND_BUFFERS;
+	settings->heuristics_ipc_max_send_receive_size = QDEVICE_DEFAULT_HEURISTICS_IPC_MAX_SEND_RECEIVE_SIZE;
+
+	settings->heuristics_min_timeout = QDEVICE_DEFAULT_HEURISTICS_MIN_TIMEOUT;
+	settings->heuristics_max_timeout = QDEVICE_DEFAULT_HEURISTICS_MAX_TIMEOUT;
+	settings->heuristics_min_interval = QDEVICE_DEFAULT_HEURISTICS_MIN_INTERVAL;
+	settings->heuristics_max_interval = QDEVICE_DEFAULT_HEURISTICS_MAX_INTERVAL;
+
+	settings->heuristics_max_execs = QDEVICE_DEFAULT_HEURISTICS_MAX_EXECS;
+
+	settings->heuristics_use_execvp = QDEVICE_DEFAULT_HEURISTICS_USE_EXECVP;
+	settings->heuristics_max_processes = QDEVICE_DEFAULT_HEURISTICS_MAX_PROCESSES;
+	settings->heuristics_kill_list_interval = QDEVICE_DEFAULT_HEURISTICS_KILL_LIST_INTERVAL;
+
 	if ((settings->net_nss_db_dir = strdup(QDEVICE_NET_DEFAULT_NSS_DB_DIR)) == NULL) {
 		return (-1);
 	}
@@ -166,6 +181,75 @@ qdevice_advanced_settings_set(struct qdevice_advanced_settings *settings,
 		}
 
 		settings->ipc_max_send_size = (size_t)tmpll;
+	} else if (strcasecmp(option, "heuristics_ipc_max_send_buffers") == 0) {
+		tmpll = strtoll(value, &ep, 10);
+		if (tmpll < QDEVICE_MIN_HEURISTICS_IPC_MAX_SEND_BUFFERS || errno != 0 || *ep != '\0') {
+			return (-2);
+		}
+
+		settings->heuristics_ipc_max_send_buffers = (size_t)tmpll;
+	} else if (strcasecmp(option, "heuristics_ipc_max_send_receive_size") == 0) {
+		tmpll = strtoll(value, &ep, 10);
+		if (tmpll < QDEVICE_MIN_HEURISTICS_IPC_MAX_SEND_RECEIVE_SIZE || errno != 0 || *ep != '\0') {
+			return (-2);
+		}
+
+		settings->heuristics_ipc_max_send_receive_size = (size_t)tmpll;
+	} else if (strcasecmp(option, "heuristics_min_timeout") == 0) {
+		tmpll = strtoll(value, &ep, 10);
+		if (tmpll < QDEVICE_MIN_HEURISTICS_TIMEOUT || errno != 0 || *ep != '\0') {
+			return (-2);
+		}
+
+		settings->heuristics_min_timeout = (uint32_t)tmpll;
+	} else if (strcasecmp(option, "heuristics_max_timeout") == 0) {
+		tmpll = strtoll(value, &ep, 10);
+		if (tmpll < QDEVICE_MIN_HEURISTICS_TIMEOUT || errno != 0 || *ep != '\0') {
+			return (-2);
+		}
+
+		settings->heuristics_max_timeout = (uint32_t)tmpll;
+	} else if (strcasecmp(option, "heuristics_min_interval") == 0) {
+		tmpll = strtoll(value, &ep, 10);
+		if (tmpll < QDEVICE_MIN_HEURISTICS_INTERVAL || errno != 0 || *ep != '\0') {
+			return (-2);
+		}
+
+		settings->heuristics_min_interval = (uint32_t)tmpll;
+	} else if (strcasecmp(option, "heuristics_max_interval") == 0) {
+		tmpll = strtoll(value, &ep, 10);
+		if (tmpll < QDEVICE_MIN_HEURISTICS_INTERVAL || errno != 0 || *ep != '\0') {
+			return (-2);
+		}
+
+		settings->heuristics_max_interval = (uint32_t)tmpll;
+	} else if (strcasecmp(option, "heuristics_max_execs") == 0) {
+		tmpll = strtoll(value, &ep, 10);
+		if (tmpll < QDEVICE_MIN_HEURISTICS_MAX_EXECS || errno != 0 || *ep != '\0') {
+			return (-2);
+		}
+
+		settings->heuristics_max_execs = (size_t)tmpll;
+	} else if (strcasecmp(option, "heuristics_use_execvp") == 0) {
+		if ((tmpll = utils_parse_bool_str(value)) == -1) {
+			return (-2);
+		}
+
+		settings->heuristics_use_execvp = (uint8_t)tmpll;
+	} else if (strcasecmp(option, "heuristics_max_processes") == 0) {
+		tmpll = strtoll(value, &ep, 10);
+		if (tmpll < QDEVICE_MIN_HEURISTICS_MAX_PROCESSES || errno != 0 || *ep != '\0') {
+			return (-2);
+		}
+
+		settings->heuristics_max_processes = (size_t)tmpll;
+	} else if (strcasecmp(option, "heuristics_kill_list_interval") == 0) {
+		tmpll = strtoll(value, &ep, 10);
+		if (tmpll < QDEVICE_MIN_HEURISTICS_KILL_LIST_INTERVAL || errno != 0 || *ep != '\0') {
+			return (-2);
+		}
+
+		settings->heuristics_kill_list_interval = (uint32_t)tmpll;
 	} else if (strcasecmp(option, "net_nss_db_dir") == 0) {
 		free(settings->net_nss_db_dir);
 

qdevices/qdevice-advanced-settings.h

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -55,6 +55,16 @@ struct qdevice_advanced_settings {
 	size_t ipc_max_send_size;
 	size_t ipc_max_receive_size;
 	enum qdevice_advanced_settings_master_wins master_wins;
+	size_t heuristics_ipc_max_send_buffers;
+	size_t heuristics_ipc_max_send_receive_size;
+	uint32_t heuristics_min_timeout;
+	uint32_t heuristics_max_timeout;
+	uint32_t heuristics_min_interval;
+	uint32_t heuristics_max_interval;
+	size_t heuristics_max_execs;
+	int heuristics_use_execvp;
+	size_t heuristics_max_processes;
+	uint32_t heuristics_kill_list_interval;
 
 	/*
 	 * Related to model NET

qdevices/qdevice-cmap.c

@@ -300,6 +300,17 @@ qdevice_cmap_logging_event(struct qdevice_instance *instance)
 	qdevice_log_configure(instance);
 }
 
+static void
+qdevice_cmap_heuristics_event(struct qdevice_instance *instance)
+{
+
+	qdevice_log(LOG_DEBUG, "Heuristics configuration possibly changed");
+	if (qdevice_instance_configure_from_cmap_heuristics(instance) != 0) {
+		qdevice_log(LOG_DEBUG, "qdevice_instance_configure_from_cmap_heuristics returned error -> exit");
+		exit(2);
+	}
+}
+
 static void
 qdevice_cmap_reload_cb(cmap_handle_t cmap_handle, cmap_track_handle_t cmap_track_handle,
     int32_t event, const char *key_name,
@@ -309,15 +320,15 @@ qdevice_cmap_reload_cb(cmap_handle_t cmap_handle, cmap_track_handle_t cmap_track
 	cs_error_t cs_res;
 	uint8_t reload;
 	struct qdevice_instance *instance;
-	int node_list_event;
-	int logging_event;
 	const char *node_list_prefix_str;
 	const char *logging_prefix_str;
+	const char *heuristics_prefix_str;
+	struct qdevice_cmap_change_events events;
 
-	node_list_event = 0;
-	logging_event = 0;
+	memset(&events, 0, sizeof(events));
 	node_list_prefix_str = "nodelist.";
 	logging_prefix_str = "logging.";
+	heuristics_prefix_str = "quorum.device.heuristics.";
 
 	if (cmap_context_get(cmap_handle, (const void **)&instance) != CS_OK) {
 		qdevice_log(LOG_ERR, "Fatal error. Can't get cmap context");
@@ -338,8 +349,9 @@ qdevice_cmap_reload_cb(cmap_handle_t cmap_handle, cmap_track_handle_t cmap_track
 			return ;
 		} else {
 			instance->cmap_reload_in_progress = 0;
-			node_list_event = 1;
-			logging_event = 1;
+			events.node_list = 1;
+			events.logging = 1;
+			events.heuristics = 1;
 		}
 	}
 
@@ -353,20 +365,36 @@ qdevice_cmap_reload_cb(cmap_handle_t cmap_handle, cmap_track_handle_t cmap_track
 	}
 
 	if (strncmp(key_name, node_list_prefix_str, strlen(node_list_prefix_str)) == 0) {
-		node_list_event = 1;
+		events.node_list = 1;
 	}
 
 	if (strncmp(key_name, logging_prefix_str, strlen(logging_prefix_str)) == 0) {
-		logging_event = 1;
+		events.logging = 1;
 	}
 
-	if (logging_event) {
+	if (strncmp(key_name, heuristics_prefix_str, strlen(heuristics_prefix_str)) == 0) {
+		events.heuristics = 1;
+	}
+
+	if (events.logging) {
 		qdevice_cmap_logging_event(instance);
 	}
 
-	if (node_list_event) {
+	if (events.node_list) {
 		qdevice_cmap_node_list_event(instance);
 	}
+
+	if (events.heuristics) {
+		qdevice_cmap_heuristics_event(instance);
+	}
+
+	/*
+	 * Inform model about change
+	 */
+	if (qdevice_model_cmap_changed(instance, &events) != 0) {
+		qdevice_log(LOG_DEBUG, "qdevice_model_cmap_changed returned error -> exit");
+		exit(2);
+	}
 }
 
 int
@@ -403,6 +431,16 @@ qdevice_cmap_add_track(struct qdevice_instance *instance)
 		return (-1);
 	}
 
+	res = cmap_track_add(instance->cmap_handle, "quorum.device.heuristics.",
+	    CMAP_TRACK_ADD | CMAP_TRACK_DELETE | CMAP_TRACK_MODIFY | CMAP_TRACK_PREFIX,
+	    qdevice_cmap_reload_cb,
+	    NULL, &instance->cmap_heuristics_track_handle);
+
+	if (res != CS_OK) {
+		qdevice_log(LOG_ERR, "Can't initialize logging tracking");
+		return (-1);
+	}
+
 	return (0);
 }
 
@@ -426,6 +464,11 @@ qdevice_cmap_del_track(struct qdevice_instance *instance)
 		qdevice_log(LOG_WARNING, "Can't delete cmap logging tracking");
 	}
 
+	res = cmap_track_delete(instance->cmap_handle, instance->cmap_heuristics_track_handle);
+	if (res != CS_OK) {
+		qdevice_log(LOG_WARNING, "Can't delete cmap heuristics tracking");
+	}
+
 	return (0);
 }
 

qdevices/qdevice-cmap.h

@@ -44,6 +44,12 @@
 extern "C" {
 #endif
 
+struct qdevice_cmap_change_events {
+	unsigned int logging	: 1;
+	unsigned int node_list	: 1;
+	unsigned int heuristics	: 1;
+};
+
 extern int		qdevice_cmap_get_nodelist(cmap_handle_t cmap_handle,
     struct node_list *list);
 

qdevices/qdevice-config.h

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -40,6 +40,8 @@
 #include <qb/qbdefs.h>
 #include <qb/qblog.h>
 
+#include "qdevice-heuristics-mode.h"
+
 #ifdef __cplusplus
 extern "C" {
 #endif
@@ -77,6 +79,31 @@ extern "C" {
 #define QDEVICE_DEFAULT_IPC_MAX_SEND_SIZE	(64*1024)
 #define QDEVICE_MIN_IPC_RECEIVE_SEND_SIZE	1024
 
+#define QDEVICE_DEFAULT_HEURISTICS_IPC_MAX_SEND_BUFFERS		128
+#define QDEVICE_MIN_HEURISTICS_IPC_MAX_SEND_BUFFERS		10
+#define QDEVICE_DEFAULT_HEURISTICS_IPC_MAX_SEND_RECEIVE_SIZE	(4 * 1024)
+#define QDEVICE_MIN_HEURISTICS_IPC_MAX_SEND_RECEIVE_SIZE	1024
+
+#define QDEVICE_DEFAULT_HEURISTICS_MIN_TIMEOUT			(1 * 1000)
+#define QDEVICE_DEFAULT_HEURISTICS_MAX_TIMEOUT			(2 * 60 * 1000)
+#define QDEVICE_MIN_HEURISTICS_TIMEOUT				250
+#define QDEVICE_DEFAULT_HEURISTICS_MIN_INTERVAL			QDEVICE_DEFAULT_HEURISTICS_MIN_TIMEOUT
+#define QDEVICE_DEFAULT_HEURISTICS_MAX_INTERVAL			(60 * 60 * 1000)
+#define QDEVICE_MIN_HEURISTICS_INTERVAL				QDEVICE_MIN_HEURISTICS_TIMEOUT
+
+#define QDEVICE_DEFAULT_HEURISTICS_MODE				QDEVICE_HEURISTICS_MODE_DISABLED
+
+#define QDEVICE_DEFAULT_HEURISTICS_MAX_EXECS			32
+#define QDEVICE_MIN_HEURISTICS_MAX_EXECS			1
+
+#define QDEVICE_DEFAULT_HEURISTICS_USE_EXECVP			0
+
+#define QDEVICE_DEFAULT_HEURISTICS_MAX_PROCESSES		(QDEVICE_DEFAULT_HEURISTICS_MAX_EXECS * 5)
+#define QDEVICE_MIN_HEURISTICS_MAX_PROCESSES			1
+
+#define QDEVICE_DEFAULT_HEURISTICS_KILL_LIST_INTERVAL		(5 * 1000)
+#define QDEVICE_MIN_HEURISTICS_KILL_LIST_INTERVAL		QDEVICE_MIN_HEURISTICS_TIMEOUT
+
 #define QDEVICE_TOOL_PROGRAM_NAME		"corosync-qdevice-tool"
 
 #ifdef __cplusplus

qdevices/qdevice-heuristics-cmd-str.h

@@ -0,0 +1,56 @@
+/*
+ * Copyright (c) 2015-2017 Red Hat, Inc.
+ *
+ * All rights reserved.
+ *
+ * Author: Jan Friesse (jfriesse@redhat.com)
+ *
+ * This software licensed under BSD license, the text of which follows:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright notice,
+ *   this list of conditions and the following disclaimer.
+ * - Redistributions in binary form must reproduce the above copyright notice,
+ *   this list of conditions and the following disclaimer in the documentation
+ *   and/or other materials provided with the distribution.
+ * - Neither the name of the Red Hat, Inc. nor the names of its
+ *   contributors may be used to endorse or promote products derived from this
+ *   software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _QDEVICE_HEURISTICS_CMD_STR_H_
+#define _QDEVICE_HEURISTICS_CMD_STR_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#define QDEVICE_HEURISTICS_CMD_STR_EXEC_LIST_CLEAR		"exec-list-clear"
+#define QDEVICE_HEURISTICS_CMD_STR_EXEC_LIST_ADD		"exec-list-add"
+#define QDEVICE_HEURISTICS_CMD_STR_EXEC_LIST_ADD_SPACE		\
+    QDEVICE_HEURISTICS_CMD_STR_EXEC_LIST_ADD " "
+#define QDEVICE_HEURISTICS_CMD_STR_EXEC				"exec"
+#define QDEVICE_HEURISTICS_CMD_STR_EXEC_ADD_SPACE		QDEVICE_HEURISTICS_CMD_STR_EXEC " "
+#define QDEVICE_HEURISTICS_CMD_STR_EXEC_RESULT			"exec-result"
+#define QDEVICE_HEURISTICS_CMD_STR_EXEC_RESULT_ADD_SPACE	\
+    QDEVICE_HEURISTICS_CMD_STR_EXEC_RESULT " "
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _QDEVICE_HEURISTICS_CMD_STR_H_ */

qdevices/qdevice-heuristics-cmd.c

@@ -0,0 +1,353 @@
+/*
+ * Copyright (c) 2015-2017 Red Hat, Inc.
+ *
+ * All rights reserved.
+ *
+ * Author: Jan Friesse (jfriesse@redhat.com)
+ *
+ * This software licensed under BSD license, the text of which follows:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright notice,
+ *   this list of conditions and the following disclaimer.
+ * - Redistributions in binary form must reproduce the above copyright notice,
+ *   this list of conditions and the following disclaimer in the documentation
+ *   and/or other materials provided with the distribution.
+ * - Neither the name of the Red Hat, Inc. nor the names of its
+ *   contributors may be used to endorse or promote products derived from this
+ *   software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <stdlib.h>
+
+#include "dynar.h"
+#include "dynar-str.h"
+#include "qdevice-heuristics-exec-result.h"
+#include "qdevice-heuristics-cmd.h"
+#include "qdevice-heuristics-cmd-str.h"
+#include "qdevice-heuristics-io.h"
+#include "qdevice-log.h"
+
+static int
+qdevice_heuristics_cmd_process_exec_result(struct qdevice_heuristics_instance *instance,
+    struct dynar *data)
+{
+	uint32_t seq_number;
+	char *str;
+	enum qdevice_heuristics_exec_result exec_result;
+
+	str = dynar_data(data);
+
+	if (sscanf(str, QDEVICE_HEURISTICS_CMD_STR_EXEC_RESULT_ADD_SPACE "%"PRIu32" %u", &seq_number,
+	    &exec_result) != 2) {
+		qdevice_log(LOG_CRIT, "Can't parse exec result command (sscanf)");
+
+		return (-1);
+	}
+
+	qdevice_log(LOG_DEBUG,
+	    "Received heuristics exec result command with seq_no \"%"PRIu32"\" and result \"%s\"", seq_number,
+	    qdevice_heuristics_exec_result_to_str(exec_result));
+
+	if (!instance->waiting_for_result) {
+		qdevice_log(LOG_DEBUG, "Received exec result is not expected. Ignoring.");
+
+		return (0);
+	}
+
+	if (seq_number != instance->expected_reply_seq_number) {
+		qdevice_log(LOG_DEBUG, "Received heuristics exec result seq number %"PRIu32
+		    " is not expected one (expected %"PRIu32"). Ignoring.", seq_number,
+		    instance->expected_reply_seq_number);
+
+		return (0);
+	}
+
+	instance->waiting_for_result = 0;
+
+	if (qdevice_heuristics_result_notifier_notify(&instance->exec_result_notifier_list,
+	    (void *)instance, seq_number, exec_result) != 0) {
+		qdevice_log(LOG_DEBUG, "qdevice_heuristics_result_notifier_notify returned non-zero result");
+		return (-1);
+	}
+
+	return (0);
+}
+
+/*
+ * 1 - Line processed
+ * 0 - No line to process - everything processed
+ * -1 - Error
+ */
+static int
+qdevice_heuristics_cmd_process_one_line(struct qdevice_heuristics_instance *instance,
+    struct dynar *data)
+{
+	char *str;
+	size_t str_len;
+	size_t nl_pos;
+	size_t zi;
+
+	str = dynar_data(data);
+	str_len = dynar_size(data);
+
+	/*
+	 * Find valid line
+	 */
+	for (zi = 0; zi < str_len && str[zi] != '\r' && str[zi] != '\n'; zi++) ;
+
+	if (zi >= str_len) {
+		/*
+		 * Command is not yet fully readed
+		 */
+		return (0);
+	}
+
+	nl_pos = zi;
+
+	str[nl_pos] = '\0';
+
+	if (strncmp(str, QDEVICE_HEURISTICS_CMD_STR_EXEC_RESULT_ADD_SPACE,
+	    strlen(QDEVICE_HEURISTICS_CMD_STR_EXEC_RESULT_ADD_SPACE)) == 0) {
+		if (qdevice_heuristics_cmd_process_exec_result(instance, data) != 0) {
+			return (-1);
+		}
+	} else {
+		qdevice_log(LOG_CRIT,
+		    "Heuristics worker sent unknown command \"%s\"", str);
+
+		    return (-1);
+	}
+
+	/*
+	 * Find place where is begining of new "valid" line
+	 */
+	for (zi = nl_pos + 1; zi < str_len && (str[zi] == '\0' || str[zi] == '\n' || str[zi] == '\r'); zi++) ;
+
+	memmove(str, str + zi, str_len - zi);
+	if (dynar_set_size(data, str_len - zi) == -1) {
+		qdevice_log(LOG_CRIT,
+		    "qdevice_heuristics_cmd_process_one_line: Can't set dynar size");
+		return (-1);
+	}
+
+	return (1);
+}
+
+/*
+ * 0 - No error
+ * -1 - Error
+ */
+static int
+qdevice_heuristics_cmd_process(struct qdevice_heuristics_instance *instance)
+{
+	int res;
+
+	while ((res =
+	    qdevice_heuristics_cmd_process_one_line(instance, &instance->cmd_in_buffer)) == 1) ;
+
+	return (res);
+}
+
+/*
+ * 0 - No error
+ * 1 - Error
+ */
+int
+qdevice_heuristics_cmd_read_from_pipe(struct qdevice_heuristics_instance *instance)
+{
+	int res;
+	int ret;
+
+	res = qdevice_heuristics_io_read(instance->pipe_cmd_recv, &instance->cmd_in_buffer);
+
+	ret = 0;
+
+	switch (res) {
+	case 0:
+		/*
+		 * Partial read
+		 */
+		break;
+	case -1:
+		qdevice_log(LOG_ERR, "Lost connection with heuristics worker");
+		ret = -1;
+		break;
+	case -2:
+		qdevice_log(LOG_ERR, "Heuristics worker sent too long cmd.");
+		ret = -1;
+		break;
+	case -3:
+		qdevice_log(LOG_ERR, "Unhandled error when reading from heuristics worker cmd fd");
+		ret = -1;
+		break;
+	case 1:
+		/*
+		 * At least one cmd line received
+		 */
+		ret = qdevice_heuristics_cmd_process(instance);
+		break;
+	}
+
+	return (ret);
+}
+
+/*
+ * 0 - No error
+ * 1 - Error
+ */
+int
+qdevice_heuristics_cmd_write(struct qdevice_heuristics_instance *instance)
+{
+	struct send_buffer_list_entry *send_buffer;
+	int res;
+
+	send_buffer = send_buffer_list_get_active(&instance->cmd_out_buffer_list);
+	if (send_buffer == NULL) {
+		qdevice_log(LOG_CRIT, "send_buffer_list_get_active in qdevice_heuristics_cmd_write returned NULL");
+
+		return (-1);
+	}
+
+	res = qdevice_heuristics_io_write(instance->pipe_cmd_send, &send_buffer->buffer,
+	    &send_buffer->msg_already_sent_bytes);
+
+	if (res == 1) {
+		send_buffer_list_delete(&instance->cmd_out_buffer_list, send_buffer);
+	}
+
+	if (res == -1) {
+		qdevice_log(LOG_CRIT, "qdevice_heuristics_io_write returned -1 (write returned 0)");
+
+		return (-1);
+	}
+
+	if (res == -2) {
+		qdevice_log(LOG_CRIT, "Unhandled error in during sending message to heuristics "
+		    "worker (qdevice_heuristics_io_write returned -2)");
+
+		return (-1);
+	}
+
+	return (0);
+}
+
+static int
+qdevice_heuristics_cmd_remove_newlines(struct dynar *str)
+{
+	size_t len;
+	size_t zi;
+	char *buf;
+
+	len = dynar_size(str);
+	buf = dynar_data(str);
+
+	for (zi = 0; zi < len ; zi++) {
+		if (buf[zi] == '\n' || buf[zi] == '\r') {
+			buf[zi] = ' ';
+		}
+	}
+
+	return (0);
+}
+
+int
+qdevice_heuristics_cmd_write_exec_list(struct qdevice_heuristics_instance *instance,
+    const struct qdevice_heuristics_exec_list *new_exec_list)
+{
+	struct send_buffer_list_entry *send_buffer;
+	struct qdevice_heuristics_exec_list_entry *entry;
+
+	send_buffer = send_buffer_list_get_new(&instance->cmd_out_buffer_list);
+	if (send_buffer == NULL) {
+		qdevice_log(LOG_ERR, "Can't alloc send list for cmd change exec list");
+
+		return (-1);
+	}
+
+	if (dynar_str_cpy(&send_buffer->buffer, QDEVICE_HEURISTICS_CMD_STR_EXEC_LIST_CLEAR) == -1 ||
+	    dynar_str_cat(&send_buffer->buffer, "\n") == -1) {
+		qdevice_log(LOG_ERR, "Can't alloc list clear message");
+
+		send_buffer_list_discard_new(&instance->cmd_out_buffer_list, send_buffer);
+
+		return (-1);
+	}
+
+	send_buffer_list_put(&instance->cmd_out_buffer_list, send_buffer);
+
+	if (new_exec_list == NULL) {
+		return (0);
+	}
+
+	/*
+	 * new_exec_list is not NULL, send it
+	 */
+	TAILQ_FOREACH(entry, new_exec_list, entries) {
+		send_buffer = send_buffer_list_get_new(&instance->cmd_out_buffer_list);
+		if (send_buffer == NULL) {
+			qdevice_log(LOG_ERR, "Can't alloc send list for cmd change exec list");
+
+			return (-1);
+		}
+
+		if (dynar_str_cpy(&send_buffer->buffer,
+		    QDEVICE_HEURISTICS_CMD_STR_EXEC_LIST_ADD_SPACE) == -1 ||
+		    dynar_str_cat(&send_buffer->buffer, entry->name) == -1 ||
+		    dynar_str_cat(&send_buffer->buffer, " ") == -1 ||
+		    dynar_str_cat(&send_buffer->buffer, entry->command) == -1 ||
+		    qdevice_heuristics_cmd_remove_newlines(&send_buffer->buffer) == -1 ||
+		    dynar_str_cat(&send_buffer->buffer, "\n") == -1) {
+			qdevice_log(LOG_ERR, "Can't alloc list add message");
+
+			send_buffer_list_discard_new(&instance->cmd_out_buffer_list, send_buffer);
+
+			return (-1);
+		}
+
+		send_buffer_list_put(&instance->cmd_out_buffer_list, send_buffer);
+	}
+
+	return (0);
+}
+
+int
+qdevice_heuristics_cmd_write_exec(struct qdevice_heuristics_instance *instance,
+    uint32_t timeout, uint32_t seq_number)
+{
+	struct send_buffer_list_entry *send_buffer;
+
+	send_buffer = send_buffer_list_get_new(&instance->cmd_out_buffer_list);
+	if (send_buffer == NULL) {
+		qdevice_log(LOG_ERR, "Can't alloc send list for cmd change exec list");
+
+		return (-1);
+	}
+
+	if (dynar_str_cpy(&send_buffer->buffer, QDEVICE_HEURISTICS_CMD_STR_EXEC_ADD_SPACE) == -1 ||
+	    dynar_str_catf(&send_buffer->buffer, "%"PRIu32" %"PRIu32"\n", timeout, seq_number) == -1) {
+		qdevice_log(LOG_ERR, "Can't alloc exec message");
+
+		send_buffer_list_discard_new(&instance->cmd_out_buffer_list, send_buffer);
+
+		return (-1);
+	}
+
+	send_buffer_list_put(&instance->cmd_out_buffer_list, send_buffer);
+
+	return (0);
+}

qdevices/qdevice-heuristics-cmd.h

@@ -0,0 +1,60 @@
+/*
+ * Copyright (c) 2015-2017 Red Hat, Inc.
+ *
+ * All rights reserved.
+ *
+ * Author: Jan Friesse (jfriesse@redhat.com)
+ *
+ * This software licensed under BSD license, the text of which follows:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright notice,
+ *   this list of conditions and the following disclaimer.
+ * - Redistributions in binary form must reproduce the above copyright notice,
+ *   this list of conditions and the following disclaimer in the documentation
+ *   and/or other materials provided with the distribution.
+ * - Neither the name of the Red Hat, Inc. nor the names of its
+ *   contributors may be used to endorse or promote products derived from this
+ *   software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _QDEVICE_HEURISTICS_CMD_H_
+#define _QDEVICE_HEURISTICS_CMD_H_
+
+#include "qdevice-heuristics-instance.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+extern int		qdevice_heuristics_cmd_write(
+    struct qdevice_heuristics_instance *instance);
+
+extern int		qdevice_heuristics_cmd_write_exec_list(
+    struct qdevice_heuristics_instance *instance, const struct qdevice_heuristics_exec_list *new_exec_list);
+
+extern int		qdevice_heuristics_cmd_write_exec(struct qdevice_heuristics_instance *instance,
+    uint32_t timeout, uint32_t seq_number);
+
+extern int		qdevice_heuristics_cmd_read_from_pipe(
+    struct qdevice_heuristics_instance *instance);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _QDEVICE_HEURISTICS_CMD_H_ */

qdevices/qdevice-heuristics-exec-list.c

@@ -0,0 +1,209 @@
+/*
+ * Copyright (c) 2015-2017 Red Hat, Inc.
+ *
+ * All rights reserved.
+ *
+ * Author: Jan Friesse (jfriesse@redhat.com)
+ *
+ * This software licensed under BSD license, the text of which follows:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright notice,
+ *   this list of conditions and the following disclaimer.
+ * - Redistributions in binary form must reproduce the above copyright notice,
+ *   this list of conditions and the following disclaimer in the documentation
+ *   and/or other materials provided with the distribution.
+ * - Neither the name of the Red Hat, Inc. nor the names of its
+ *   contributors may be used to endorse or promote products derived from this
+ *   software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <inttypes.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include "qdevice-heuristics-exec-list.h"
+
+void
+qdevice_heuristics_exec_list_init(struct qdevice_heuristics_exec_list *list)
+{
+
+	TAILQ_INIT(list);
+}
+
+struct qdevice_heuristics_exec_list_entry *
+qdevice_heuristics_exec_list_add(struct qdevice_heuristics_exec_list *list,
+    char *name, char *command)
+{
+	struct qdevice_heuristics_exec_list_entry *entry;
+
+	entry = (struct qdevice_heuristics_exec_list_entry *)malloc(sizeof(*entry));
+	if (entry == NULL) {
+		return (NULL);
+	}
+
+	memset(entry, 0, sizeof(*entry));
+
+	entry->name = strdup(name);
+	if (entry->name == NULL) {
+		free(entry);
+
+		return (NULL);
+	}
+
+	entry->command = strdup(command);
+	if (entry->command == NULL) {
+		free(entry->name);
+		free(entry);
+
+		return (NULL);
+	}
+
+	TAILQ_INSERT_TAIL(list, entry, entries);
+
+	return (entry);
+}
+
+void
+qdevice_heuristics_exec_list_free(struct qdevice_heuristics_exec_list *list)
+{
+	struct qdevice_heuristics_exec_list_entry *entry;
+	struct qdevice_heuristics_exec_list_entry *entry_next;
+
+	entry = TAILQ_FIRST(list);
+
+	while (entry != NULL) {
+		entry_next = TAILQ_NEXT(entry, entries);
+
+		free(entry->name);
+		free(entry->command);
+		free(entry);
+
+		entry = entry_next;
+	}
+
+	TAILQ_INIT(list);
+}
+
+size_t
+qdevice_heuristics_exec_list_size(const struct qdevice_heuristics_exec_list *list)
+{
+	struct qdevice_heuristics_exec_list_entry *entry;
+	size_t res;
+
+	res = 0;
+
+	TAILQ_FOREACH(entry, list, entries) {
+		res++;
+	}
+
+	return (res);
+}
+
+int
+qdevice_heuristics_exec_list_clone(struct qdevice_heuristics_exec_list *dst_list,
+    const struct qdevice_heuristics_exec_list *src_list)
+{
+	struct qdevice_heuristics_exec_list_entry *entry;
+
+	qdevice_heuristics_exec_list_init(dst_list);
+
+	TAILQ_FOREACH(entry, src_list, entries) {
+		if (qdevice_heuristics_exec_list_add(dst_list, entry->name, entry->command) == NULL) {
+			qdevice_heuristics_exec_list_free(dst_list);
+
+			return (-1);
+		}
+	}
+
+	return (0);
+}
+
+void
+qdevice_heuristics_exec_list_del(struct qdevice_heuristics_exec_list *list,
+    struct qdevice_heuristics_exec_list_entry *entry)
+{
+
+	TAILQ_REMOVE(list, entry, entries);
+
+	free(entry->name);
+	free(entry->command);
+	free(entry);
+}
+
+int
+qdevice_heuristics_exec_list_is_empty(const struct qdevice_heuristics_exec_list *list)
+{
+
+	return (TAILQ_EMPTY(list));
+}
+
+struct qdevice_heuristics_exec_list_entry *
+qdevice_heuristics_exec_list_find_name(const struct qdevice_heuristics_exec_list *list,
+    const char *name)
+{
+	struct qdevice_heuristics_exec_list_entry *entry;
+
+	TAILQ_FOREACH(entry, list, entries) {
+		if (strcmp(entry->name, name) == 0) {
+			return (entry);
+		}
+	}
+
+	return (NULL);
+}
+
+int
+qdevice_heuristics_exec_list_eq(const struct qdevice_heuristics_exec_list *list1,
+    const struct qdevice_heuristics_exec_list *list2)
+{
+	struct qdevice_heuristics_exec_list_entry *entry1;
+	struct qdevice_heuristics_exec_list_entry *entry2;
+	struct qdevice_heuristics_exec_list tmp_list;
+	int res;
+
+	res = 1;
+
+	if (qdevice_heuristics_exec_list_clone(&tmp_list, list2) != 0) {
+		return (-1);
+	}
+
+	TAILQ_FOREACH(entry1, list1, entries) {
+		entry2 = qdevice_heuristics_exec_list_find_name(&tmp_list, entry1->name);
+		if (entry2 == NULL) {
+			res = 0;
+			goto return_res;
+		}
+
+		if (strcmp(entry1->command, entry2->command) != 0) {
+			res = 0;
+			goto return_res;
+		}
+
+		qdevice_heuristics_exec_list_del(&tmp_list, entry2);
+	}
+
+	if (!qdevice_heuristics_exec_list_is_empty(&tmp_list)) {
+		res = 0;
+		goto return_res;
+	}
+
+return_res:
+	qdevice_heuristics_exec_list_free(&tmp_list);
+
+	return (res);
+}

qdevices/qdevice-heuristics-exec-list.h

@@ -0,0 +1,88 @@
+/*
+ * Copyright (c) 2015-2017 Red Hat, Inc.
+ *
+ * All rights reserved.
+ *
+ * Author: Jan Friesse (jfriesse@redhat.com)
+ *
+ * This software licensed under BSD license, the text of which follows:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright notice,
+ *   this list of conditions and the following disclaimer.
+ * - Redistributions in binary form must reproduce the above copyright notice,
+ *   this list of conditions and the following disclaimer in the documentation
+ *   and/or other materials provided with the distribution.
+ * - Neither the name of the Red Hat, Inc. nor the names of its
+ *   contributors may be used to endorse or promote products derived from this
+ *   software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _QDEVICE_HEURISTICS_EXEC_LIST_H_
+#define _QDEVICE_HEURISTICS_EXEC_LIST_H_
+
+#include <sys/types.h>
+
+#include <sys/queue.h>
+#include <inttypes.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+struct qdevice_heuristics_exec_list_entry {
+	char *name;
+	char *command;
+	TAILQ_ENTRY(qdevice_heuristics_exec_list_entry) entries;
+};
+
+TAILQ_HEAD(qdevice_heuristics_exec_list, qdevice_heuristics_exec_list_entry);
+
+extern void						 qdevice_heuristics_exec_list_init(
+    struct qdevice_heuristics_exec_list *list);
+
+extern struct qdevice_heuristics_exec_list_entry	*qdevice_heuristics_exec_list_add(
+    struct qdevice_heuristics_exec_list *list, char *name, char *command);
+
+extern void				 		 qdevice_heuristics_exec_list_free(
+    struct qdevice_heuristics_exec_list *list);
+
+extern size_t						 qdevice_heuristics_exec_list_size(
+    const struct qdevice_heuristics_exec_list *list);
+
+extern int						 qdevice_heuristics_exec_list_clone(
+    struct qdevice_heuristics_exec_list *dst_list,
+    const struct qdevice_heuristics_exec_list *src_list);
+
+extern void						 qdevice_heuristics_exec_list_del(
+    struct qdevice_heuristics_exec_list *list, struct qdevice_heuristics_exec_list_entry *entry);
+
+extern int						 qdevice_heuristics_exec_list_is_empty(
+    const struct qdevice_heuristics_exec_list *list);
+
+extern struct qdevice_heuristics_exec_list_entry	*qdevice_heuristics_exec_list_find_name(
+    const struct qdevice_heuristics_exec_list *list, const char *name);
+
+extern int						 qdevice_heuristics_exec_list_eq(
+    const struct qdevice_heuristics_exec_list *list1,
+    const struct qdevice_heuristics_exec_list *list2);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _QDEVICE_HEURISTICS_EXEC_LIST_H_ */

qdevices/qdevice-heuristics-exec-result.c

@@ -0,0 +1,48 @@
+/*
+ * Copyright (c) 2015-2017 Red Hat, Inc.
+ *
+ * All rights reserved.
+ *
+ * Author: Jan Friesse (jfriesse@redhat.com)
+ *
+ * This software licensed under BSD license, the text of which follows:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright notice,
+ *   this list of conditions and the following disclaimer.
+ * - Redistributions in binary form must reproduce the above copyright notice,
+ *   this list of conditions and the following disclaimer in the documentation
+ *   and/or other materials provided with the distribution.
+ * - Neither the name of the Red Hat, Inc. nor the names of its
+ *   contributors may be used to endorse or promote products derived from this
+ *   software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include "qdevice-heuristics-exec-result.h"
+
+const char *
+qdevice_heuristics_exec_result_to_str(enum qdevice_heuristics_exec_result exec_result)
+{
+
+	switch (exec_result) {
+	case QDEVICE_HEURISTICS_EXEC_RESULT_FAIL: return("Fail"); break;
+	case QDEVICE_HEURISTICS_EXEC_RESULT_PASS: return("Pass"); break;
+	case QDEVICE_HEURISTICS_EXEC_RESULT_DISABLED: return("Disabled"); break;
+	}
+
+	return ("Unknown heuristics exec result value");
+}

qdevices/qdevice-heuristics-exec-result.h

@@ -0,0 +1,65 @@
+/*
+ * Copyright (c) 2015-2017 Red Hat, Inc.
+ *
+ * All rights reserved.
+ *
+ * Author: Jan Friesse (jfriesse@redhat.com)
+ *
+ * This software licensed under BSD license, the text of which follows:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright notice,
+ *   this list of conditions and the following disclaimer.
+ * - Redistributions in binary form must reproduce the above copyright notice,
+ *   this list of conditions and the following disclaimer in the documentation
+ *   and/or other materials provided with the distribution.
+ * - Neither the name of the Red Hat, Inc. nor the names of its
+ *   contributors may be used to endorse or promote products derived from this
+ *   software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _QDEVICE_HEURISTICS_EXEC_RESULT_H_
+#define _QDEVICE_HEURISTICS_EXEC_RESULT_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+enum qdevice_heuristics_exec_result {
+	/*
+	 * Heuristics worker received command to exec heuristics but list is empty. This
+	 * is happening when heuristics is disabled.
+	 */
+	QDEVICE_HEURISTICS_EXEC_RESULT_DISABLED = 0,
+	/*
+	 * All executed commands passed
+	 */
+	QDEVICE_HEURISTICS_EXEC_RESULT_PASS = 1,
+	/*
+	 * One (or more) commands failed or timed-out
+	 */
+	QDEVICE_HEURISTICS_EXEC_RESULT_FAIL = 2,
+};
+
+extern const char *	qdevice_heuristics_exec_result_to_str(
+    enum qdevice_heuristics_exec_result exec_result);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _QDEVICE_HEURISTICS_EXEC_RESULT_H_ */

qdevices/qdevice-heuristics-instance.c

@@ -0,0 +1,60 @@
+/*
+ * Copyright (c) 2015-2017 Red Hat, Inc.
+ *
+ * All rights reserved.
+ *
+ * Author: Jan Friesse (jfriesse@redhat.com)
+ *
+ * This software licensed under BSD license, the text of which follows:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright notice,
+ *   this list of conditions and the following disclaimer.
+ * - Redistributions in binary form must reproduce the above copyright notice,
+ *   this list of conditions and the following disclaimer in the documentation
+ *   and/or other materials provided with the distribution.
+ * - Neither the name of the Red Hat, Inc. nor the names of its
+ *   contributors may be used to endorse or promote products derived from this
+ *   software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <string.h>
+
+#include "qdevice-heuristics-instance.h"
+#include "qdevice-heuristics-exec-list.h"
+
+int
+qdevice_heuristics_instance_init(struct qdevice_heuristics_instance *instance)
+{
+
+	memset(instance, 0, sizeof(*instance));
+
+	qdevice_heuristics_exec_list_init(&instance->exec_list);
+	qdevice_heuristics_result_notifier_list_init(&instance->exec_result_notifier_list);
+
+	return (0);
+}
+
+int
+qdevice_heuristics_instance_destroy(struct qdevice_heuristics_instance *instance)
+{
+
+	qdevice_heuristics_result_notifier_list_free(&instance->exec_result_notifier_list);
+	qdevice_heuristics_exec_list_free(&instance->exec_list);
+
+	return (0);
+}

qdevices/qdevice-heuristics-instance.h

@@ -0,0 +1,82 @@
+/*
+ * Copyright (c) 2015-2017 Red Hat, Inc.
+ *
+ * All rights reserved.
+ *
+ * Author: Jan Friesse (jfriesse@redhat.com)
+ *
+ * This software licensed under BSD license, the text of which follows:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright notice,
+ *   this list of conditions and the following disclaimer.
+ * - Redistributions in binary form must reproduce the above copyright notice,
+ *   this list of conditions and the following disclaimer in the documentation
+ *   and/or other materials provided with the distribution.
+ * - Neither the name of the Red Hat, Inc. nor the names of its
+ *   contributors may be used to endorse or promote products derived from this
+ *   software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _QDEVICE_HEURISTICS_INSTANCE_H_
+#define _QDEVICE_HEURISTICS_INSTANCE_H_
+
+#include "dynar.h"
+#include "send-buffer-list.h"
+#include "qdevice-heuristics-mode.h"
+#include "qdevice-heuristics-exec-list.h"
+#include "qdevice-heuristics-exec-result.h"
+#include "qdevice-heuristics-result-notifier.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+struct qdevice_heuristics_instance {
+	int pipe_cmd_send;
+	int pipe_cmd_recv;
+	int pipe_log_recv;
+	pid_t worker_pid;
+	struct send_buffer_list cmd_out_buffer_list;
+	struct dynar log_in_buffer;
+	struct dynar cmd_in_buffer;
+
+	uint32_t timeout;
+	uint32_t sync_timeout;
+	uint32_t interval;
+
+	enum qdevice_heuristics_mode mode;
+
+	int waiting_for_result;
+	uint32_t expected_reply_seq_number;
+
+	struct qdevice_heuristics_exec_list exec_list;
+
+	struct qdevice_instance *qdevice_instance_ptr;
+
+	struct qdevice_heuristics_result_notifier_list exec_result_notifier_list;
+};
+
+extern int	qdevice_heuristics_instance_init(struct qdevice_heuristics_instance *instance);
+
+extern int	qdevice_heuristics_instance_destroy(struct qdevice_heuristics_instance *instance);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _QDEVICE_HEURISTICS_INSTANCE_H_ */

qdevices/qdevice-heuristics-io.c

@@ -0,0 +1,151 @@
+/*
+ * Copyright (c) 2015-2017 Red Hat, Inc.
+ *
+ * All rights reserved.
+ *
+ * Author: Jan Friesse (jfriesse@redhat.com)
+ *
+ * This software licensed under BSD license, the text of which follows:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright notice,
+ *   this list of conditions and the following disclaimer.
+ * - Redistributions in binary form must reproduce the above copyright notice,
+ *   this list of conditions and the following disclaimer in the documentation
+ *   and/or other materials provided with the distribution.
+ * - Neither the name of the Red Hat, Inc. nor the names of its
+ *   contributors may be used to endorse or promote products derived from this
+ *   software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <limits.h>
+#include <errno.h>
+#include <signal.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+
+#include "qdevice-heuristics-io.h"
+
+#define QDEVICE_HEURISTICS_IO_BUFFER_SIZE	256
+
+ssize_t
+qdevice_heuristics_io_blocking_write(int fd, const void *buf, size_t count)
+{
+	ssize_t bytes_written;
+	ssize_t tmp_bytes_written;
+
+	bytes_written = 0;
+
+	do {
+		tmp_bytes_written = write(fd, (const char *)buf + bytes_written,
+		    (count - bytes_written > SSIZE_MAX) ? SSIZE_MAX : count - bytes_written);
+		if (tmp_bytes_written == -1) {
+			if (errno != EAGAIN && errno != EINTR && errno != EWOULDBLOCK) {
+				return (-1);
+			}
+		} else {
+			bytes_written += tmp_bytes_written;
+		}
+	} while ((size_t)bytes_written != count);
+
+	return (bytes_written);
+}
+
+/*
+ *  1 Full line readed (at least one \n found)
+ *  0 Partial read (no error)
+ * -1 End of connection
+ * -2 Buffer too long
+ * -3 Unhandled error
+ */
+int
+qdevice_heuristics_io_read(int fd, struct dynar *dest)
+{
+	char buf[QDEVICE_HEURISTICS_IO_BUFFER_SIZE];
+	ssize_t readed;
+	int res;
+	size_t zi;
+
+	res = 0;
+	readed = read(fd, buf, sizeof(buf));
+	if (readed > 0) {
+		if (dynar_cat(dest, buf, readed) == -1) {
+			res = -2;
+			goto exit_err;
+		}
+
+		for (zi = 0; zi < (size_t)readed; zi++) {
+			if (buf[zi] == '\n') {
+				res = 1;
+			}
+		}
+	}
+
+	if (readed == 0) {
+		res = -1;
+	}
+
+	if (readed < 0 && errno != EAGAIN && errno != EWOULDBLOCK && errno != EINTR) {
+		res = -3;
+	}
+
+exit_err:
+	return (res);
+}
+
+/*
+ * 1 All data succesfully sent
+ *  0 Partial send (no error)
+ * -1 send returned 0,
+ * -2 Unhandled error
+ */
+int
+qdevice_heuristics_io_write(int fd, const struct dynar *msg, size_t *already_sent_bytes)
+{
+	ssize_t sent;
+	size_t to_send;
+	int res;
+
+	res = 0;
+
+	to_send = dynar_size(msg) - *already_sent_bytes;
+	if (to_send > QDEVICE_HEURISTICS_IO_BUFFER_SIZE) {
+		to_send = QDEVICE_HEURISTICS_IO_BUFFER_SIZE;
+	}
+
+	sent = write(fd, dynar_data(msg) + *already_sent_bytes,
+	    to_send);
+
+	if (sent > 0) {
+		*already_sent_bytes += sent;
+
+		if (*already_sent_bytes == dynar_size(msg)) {
+			return (1);
+		}
+	}
+
+	if (sent == 0) {
+		res = -1;
+	}
+
+	if (sent < 0 && errno != EAGAIN && errno != EWOULDBLOCK && errno != EINTR) {
+		res = -2;
+	}
+
+	return (res);
+}

qdevices/qdevice-heuristics-io.h

@@ -0,0 +1,56 @@
+/*
+ * Copyright (c) 2015-2017 Red Hat, Inc.
+ *
+ * All rights reserved.
+ *
+ * Author: Jan Friesse (jfriesse@redhat.com)
+ *
+ * This software licensed under BSD license, the text of which follows:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright notice,
+ *   this list of conditions and the following disclaimer.
+ * - Redistributions in binary form must reproduce the above copyright notice,
+ *   this list of conditions and the following disclaimer in the documentation
+ *   and/or other materials provided with the distribution.
+ * - Neither the name of the Red Hat, Inc. nor the names of its
+ *   contributors may be used to endorse or promote products derived from this
+ *   software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _QDEVICE_HEURISTICS_IO_H_
+#define _QDEVICE_HEURISTICS_IO_H_
+
+#include "dynar.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+extern ssize_t          qdevice_heuristics_io_blocking_write(int fd, const void *buf,
+    size_t count);
+
+extern int		qdevice_heuristics_io_read(int fd, struct dynar *dest);
+
+extern int		qdevice_heuristics_io_write(int fd, const struct dynar *msg,
+    size_t *already_sent_bytes);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _QDEVICE_HEURISTICS_IO_H_ */

qdevices/qdevice-heuristics-log.c

@@ -0,0 +1,173 @@
+/*
+ * Copyright (c) 2015-2017 Red Hat, Inc.
+ *
+ * All rights reserved.
+ *
+ * Author: Jan Friesse (jfriesse@redhat.com)
+ *
+ * This software licensed under BSD license, the text of which follows:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright notice,
+ *   this list of conditions and the following disclaimer.
+ * - Redistributions in binary form must reproduce the above copyright notice,
+ *   this list of conditions and the following disclaimer in the documentation
+ *   and/or other materials provided with the distribution.
+ * - Neither the name of the Red Hat, Inc. nor the names of its
+ *   contributors may be used to endorse or promote products derived from this
+ *   software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include "qdevice-heuristics-io.h"
+#include "qdevice-heuristics-log.h"
+#include "qdevice-log.h"
+
+/*
+ * 1 - Line logged
+ * 0 - No line to log - everything processed
+ * -1 - Error
+ */
+static int
+qdevice_heuristics_log_process_one_line(struct dynar *data)
+{
+	char *str;
+	char *log_str_start;
+	size_t str_len;
+	size_t nl_pos;
+	size_t zi;
+	int status;
+	unsigned int log_priority;
+
+	str = dynar_data(data);
+	str_len = dynar_size(data);
+	log_str_start = str;
+
+	status = 0;
+	log_priority = 0;
+	/*
+	 * Find start of log message and end of line
+	 */
+	for (zi = 0; zi < str_len && status != -1; zi++) {
+		switch (status) {
+		case 0:
+			if (str[zi] >= '0' && str[zi] <= '9') {
+				log_priority = log_priority * 10 + (str[zi] - '0');
+			} else if (str[zi] == ' ') {
+				status = 1;
+			} else {
+				qdevice_log(LOG_ERR, "Parsing of heuristics log line failed. "
+				    "Unexpected char '%c'", str[zi]);
+				return (-1);
+			}
+			break;
+		case 1:
+			if (str[zi] != ' ') {
+				status = 2;
+				log_str_start = str + zi;
+			}
+			break;
+		case 2:
+			if (str[zi] == '\n' || str[zi] == '\r') {
+				str[zi] = '\0';
+				nl_pos = zi;
+				status = -1;
+			}
+			break;
+		}
+	}
+
+	if (status != -1) {
+		return (0);
+	}
+
+	/*
+	 * Do actual logging
+	 */
+	qb_log_from_external_source(__func__, __FILE__, "worker: %s", log_priority, __LINE__, 0, log_str_start);
+
+	/*
+	 * Find place where is begining of new "valid" line
+	 */
+	for (zi = nl_pos + 1; zi < str_len && (str[zi] == '\0' || str[zi] == '\n' || str[zi] == '\r'); zi++) ;
+
+	memmove(str, str + zi, str_len - zi);
+	if (dynar_set_size(data, str_len - zi) == -1) {
+		qdevice_log(LOG_ERR, "qdevice_heuristics_log_process_one_line: Can't set dynar size");
+		return (-1);
+	}
+
+	return (1);
+
+}
+
+
+/*
+ * 0 - No error
+ * -1 - Error
+ */
+static int
+qdevice_heuristics_log_process(struct qdevice_heuristics_instance *instance)
+{
+	int res;
+
+	while ((res = qdevice_heuristics_log_process_one_line(&instance->log_in_buffer)) == 1) ;
+
+	return (res);
+}
+
+/*
+ * 0 - No error
+ * 1 - Error
+ */
+int
+qdevice_heuristics_log_read_from_pipe(struct qdevice_heuristics_instance *instance)
+{
+	int res;
+	int ret;
+
+	res = qdevice_heuristics_io_read(instance->pipe_log_recv, &instance->log_in_buffer);
+
+	ret = 0;
+
+	switch (res) {
+	case 0:
+		/*
+		 * Partial read
+		 */
+		break;
+	case -1:
+		qdevice_log(LOG_ERR, "Lost connection with heuristics worker");
+		ret = -1;
+		break;
+	case -2:
+		qdevice_log(LOG_ERR, "Heuristics worker sent too long log. Ignoring line");
+		dynar_clean(&instance->log_in_buffer);
+		break;
+	case -3:
+		qdevice_log(LOG_ERR, "Unhandled error when reading from heuristics worker log fd");
+		ret = -1;
+		break;
+	case 1:
+		/*
+		 * At least one log line received
+		 */
+		ret = qdevice_heuristics_log_process(instance);
+		break;
+	}
+
+	return (ret);
+}

qdevices/qdevice-heuristics-log.h

@@ -0,0 +1,51 @@
+/*
+ * Copyright (c) 2015-2017 Red Hat, Inc.
+ *
+ * All rights reserved.
+ *
+ * Author: Jan Friesse (jfriesse@redhat.com)
+ *
+ * This software licensed under BSD license, the text of which follows:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright notice,
+ *   this list of conditions and the following disclaimer.
+ * - Redistributions in binary form must reproduce the above copyright notice,
+ *   this list of conditions and the following disclaimer in the documentation
+ *   and/or other materials provided with the distribution.
+ * - Neither the name of the Red Hat, Inc. nor the names of its
+ *   contributors may be used to endorse or promote products derived from this
+ *   software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _QDEVICE_HEURISTICS_LOG_H_
+#define _QDEVICE_HEURISTICS_LOG_H_
+
+#include "qdevice-heuristics-instance.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+extern int		qdevice_heuristics_log_read_from_pipe(
+    struct qdevice_heuristics_instance *instance);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _QDEVICE_HEURISTICS_LOG_H_ */

qdevices/qdevice-heuristics-mode.c

@@ -0,0 +1,51 @@
+/*
+ * Copyright (c) 2017 Red Hat, Inc.
+ *
+ * All rights reserved.
+ *
+ * Author: Jan Friesse (jfriesse@redhat.com)
+ *
+ * This software licensed under BSD license, the text of which follows:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright notice,
+ *   this list of conditions and the following disclaimer.
+ * - Redistributions in binary form must reproduce the above copyright notice,
+ *   this list of conditions and the following disclaimer in the documentation
+ *   and/or other materials provided with the distribution.
+ * - Neither the name of the Red Hat, Inc. nor the names of its
+ *   contributors may be used to endorse or promote products derived from this
+ *   software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <inttypes.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include "qdevice-heuristics-mode.h"
+
+const char*
+qdevice_heuristics_mode_to_str(enum qdevice_heuristics_mode mode)
+{
+	switch (mode) {
+	case QDEVICE_HEURISTICS_MODE_DISABLED: return ("Disabled"); break;
+	case QDEVICE_HEURISTICS_MODE_ENABLED: return ("Enabled"); break;
+	case QDEVICE_HEURISTICS_MODE_SYNC: return ("Enabled only on sync"); break;
+	}
+
+	return ("Undefined");
+}

qdevices/qdevice-heuristics-mode.h

@@ -0,0 +1,55 @@
+/*
+ * Copyright (c) 2015-2017 Red Hat, Inc.
+ *
+ * All rights reserved.
+ *
+ * Author: Jan Friesse (jfriesse@redhat.com)
+ *
+ * This software licensed under BSD license, the text of which follows:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright notice,
+ *   this list of conditions and the following disclaimer.
+ * - Redistributions in binary form must reproduce the above copyright notice,
+ *   this list of conditions and the following disclaimer in the documentation
+ *   and/or other materials provided with the distribution.
+ * - Neither the name of the Red Hat, Inc. nor the names of its
+ *   contributors may be used to endorse or promote products derived from this
+ *   software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _QDEVICE_HEURISTICS_MODE_H_
+#define _QDEVICE_HEURISTICS_MODE_H_
+
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+enum qdevice_heuristics_mode {
+	QDEVICE_HEURISTICS_MODE_DISABLED = 0,
+	QDEVICE_HEURISTICS_MODE_ENABLED = 1,
+	QDEVICE_HEURISTICS_MODE_SYNC = 2,
+};
+
+extern const char	 *qdevice_heuristics_mode_to_str(enum qdevice_heuristics_mode mode);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _QDEVICE_HEURISTICS_MODE_H_ */

qdevices/qdevice-heuristics-result-notifier.c

@@ -0,0 +1,134 @@
+/*
+ * Copyright (c) 2015-2017 Red Hat, Inc.
+ *
+ * All rights reserved.
+ *
+ * Author: Jan Friesse (jfriesse@redhat.com)
+ *
+ * This software licensed under BSD license, the text of which follows:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright notice,
+ *   this list of conditions and the following disclaimer.
+ * - Redistributions in binary form must reproduce the above copyright notice,
+ *   this list of conditions and the following disclaimer in the documentation
+ *   and/or other materials provided with the distribution.
+ * - Neither the name of the Red Hat, Inc. nor the names of its
+ *   contributors may be used to endorse or promote products derived from this
+ *   software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <stdlib.h>
+#include <string.h>
+
+#include "qdevice-heuristics-result-notifier.h"
+
+void
+qdevice_heuristics_result_notifier_list_init(struct qdevice_heuristics_result_notifier_list *notifier_list)
+{
+
+        TAILQ_INIT(notifier_list);
+}
+
+struct qdevice_heuristics_result_notifier_item *
+qdevice_heuristics_result_notifier_list_get(struct qdevice_heuristics_result_notifier_list *notifier_list,
+    qdevice_heuristics_result_notifier_callback callback)
+{
+	struct qdevice_heuristics_result_notifier_item *item;
+
+	TAILQ_FOREACH(item, notifier_list, entries) {
+		if (item->callback == callback) {
+			return (item);
+		}
+	}
+
+	return (NULL);
+}
+
+struct qdevice_heuristics_result_notifier_item *
+qdevice_heuristics_result_notifier_list_add(struct qdevice_heuristics_result_notifier_list *notifier_list,
+    qdevice_heuristics_result_notifier_callback callback)
+{
+	struct qdevice_heuristics_result_notifier_item *item;
+
+	item = qdevice_heuristics_result_notifier_list_get(notifier_list, callback);
+	if (item != NULL) {
+		return (item);
+	}
+
+	item = (struct qdevice_heuristics_result_notifier_item *)malloc(sizeof(*item));
+	if (item == NULL) {
+		return (NULL);
+	}
+	memset(item, 0, sizeof(*item));
+	item->callback = callback;
+	item->active = 0;
+
+	TAILQ_INSERT_TAIL(notifier_list, item, entries);
+
+	return (item);
+}
+
+int
+qdevice_heuristics_result_notifier_list_set_active(struct qdevice_heuristics_result_notifier_list *notifier_list,
+    qdevice_heuristics_result_notifier_callback callback, int active)
+{
+	struct qdevice_heuristics_result_notifier_item *item;
+
+	item = qdevice_heuristics_result_notifier_list_get(notifier_list, callback);
+	if (item == NULL) {
+		return (-1);
+	}
+
+	item->active = active;
+
+	return (0);
+}
+
+void
+qdevice_heuristics_result_notifier_list_free(struct qdevice_heuristics_result_notifier_list *notifier_list)
+{
+	struct qdevice_heuristics_result_notifier_item *item;
+	struct qdevice_heuristics_result_notifier_item *item_next;
+
+	item = TAILQ_FIRST(notifier_list);
+	while (item != NULL) {
+		item_next = TAILQ_NEXT(item, entries);
+
+		free(item);
+		item = item_next;
+	}
+}
+
+int
+qdevice_heuristics_result_notifier_notify(struct qdevice_heuristics_result_notifier_list *notifier_list,
+    void *heuristics_instance, uint32_t seq_number, enum qdevice_heuristics_exec_result exec_result)
+{
+	struct qdevice_heuristics_result_notifier_item *item;
+
+	TAILQ_FOREACH(item, notifier_list, entries) {
+		if (!item->active) {
+			continue ;
+		}
+
+		if (item->callback(heuristics_instance, seq_number, exec_result) != 0) {
+			return (-1);
+		}
+	}
+
+	return (0);
+}

qdevices/qdevice-heuristics-result-notifier.h

@@ -0,0 +1,87 @@
+/*
+ * Copyright (c) 2015-2017 Red Hat, Inc.
+ *
+ * All rights reserved.
+ *
+ * Author: Jan Friesse (jfriesse@redhat.com)
+ *
+ * This software licensed under BSD license, the text of which follows:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright notice,
+ *   this list of conditions and the following disclaimer.
+ * - Redistributions in binary form must reproduce the above copyright notice,
+ *   this list of conditions and the following disclaimer in the documentation
+ *   and/or other materials provided with the distribution.
+ * - Neither the name of the Red Hat, Inc. nor the names of its
+ *   contributors may be used to endorse or promote products derived from this
+ *   software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _QDEVICE_HEURISTICS_RESULT_NOTIFIER_H_
+#define _QDEVICE_HEURISTICS_RESULT_NOTIFIER_H_
+
+#include <sys/queue.h>
+#include <sys/types.h>
+#include <stdint.h>
+#include <stdlib.h>
+
+#include "qdevice-heuristics-exec-result.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+typedef int (*qdevice_heuristics_result_notifier_callback)(void *heuristics_instance, uint32_t seq_number,
+    enum qdevice_heuristics_exec_result exec_result);
+
+struct qdevice_heuristics_result_notifier_item {
+	qdevice_heuristics_result_notifier_callback callback;
+	int active;
+	TAILQ_ENTRY(qdevice_heuristics_result_notifier_item) entries;
+};
+
+TAILQ_HEAD(qdevice_heuristics_result_notifier_list, qdevice_heuristics_result_notifier_item);
+
+extern void						 qdevice_heuristics_result_notifier_list_init(
+    struct qdevice_heuristics_result_notifier_list *notifier_list);
+
+extern struct qdevice_heuristics_result_notifier_item	*qdevice_heuristics_result_notifier_list_add(
+    struct qdevice_heuristics_result_notifier_list *notifier_list,
+    qdevice_heuristics_result_notifier_callback callback);
+
+extern struct qdevice_heuristics_result_notifier_item	*qdevice_heuristics_result_notifier_list_get(
+    struct qdevice_heuristics_result_notifier_list *notifier_list,
+    qdevice_heuristics_result_notifier_callback callback);
+
+extern int						 qdevice_heuristics_result_notifier_list_set_active(
+    struct qdevice_heuristics_result_notifier_list *notifier_list,
+    qdevice_heuristics_result_notifier_callback callback, int active);
+
+extern void						 qdevice_heuristics_result_notifier_list_free(
+    struct qdevice_heuristics_result_notifier_list *notifier_list);
+
+extern int						 qdevice_heuristics_result_notifier_notify(
+    struct qdevice_heuristics_result_notifier_list *notifier_list,
+    void *heuristics_instance, uint32_t seq_number,
+    enum qdevice_heuristics_exec_result exec_result);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _QDEVICE_HEURISTICS_RESULT_NOTIFIER_H_ */

qdevices/qdevice-heuristics-worker-cmd.c

@@ -0,0 +1,384 @@
+/*
+ * Copyright (c) 2015-2017 Red Hat, Inc.
+ *
+ * All rights reserved.
+ *
+ * Author: Jan Friesse (jfriesse@redhat.com)
+ *
+ * This software licensed under BSD license, the text of which follows:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright notice,
+ *   this list of conditions and the following disclaimer.
+ * - Redistributions in binary form must reproduce the above copyright notice,
+ *   this list of conditions and the following disclaimer in the documentation
+ *   and/or other materials provided with the distribution.
+ * - Neither the name of the Red Hat, Inc. nor the names of its
+ *   contributors may be used to endorse or promote products derived from this
+ *   software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <stdlib.h>
+#include <string.h>
+#include <stdio.h>
+
+#include "dynar.h"
+#include "dynar-str.h"
+#include "qdevice-heuristics-io.h"
+#include "qdevice-heuristics-worker.h"
+#include "qdevice-heuristics-worker-cmd.h"
+#include "qdevice-heuristics-cmd-str.h"
+#include "qdevice-heuristics-worker-log.h"
+
+static int
+qdevice_heuristics_worker_cmd_process_exec_list_add(struct qdevice_heuristics_worker_instance *instance,
+    struct dynar *data)
+{
+	size_t zi;
+	char *exec_name;
+	char *exec_command;
+	char *str;
+
+	str = dynar_data(data);
+
+	/*
+	 * Skip to first space
+	 */
+	for (zi = 0; str[zi] != ' ' && str[zi] != '\0'; zi++) ;
+
+	if (str[zi] == '\0') {
+		qdevice_heuristics_worker_log_printf(instance, LOG_CRIT,
+		    "qdevice_heuristics_worker_cmd_process_exec_list_add: Can't find first space");
+		return (-1);
+	}
+
+	/*
+	 * Skip to the end of spaces
+	 */
+	for (; str[zi] == ' '; zi++) ;
+
+	if (str[zi] == '\0') {
+		qdevice_heuristics_worker_log_printf(instance, LOG_CRIT,
+		    "qdevice_heuristics_worker_cmd_process_exec_list_add: Can't find start of exec name");
+		return (-1);
+	}
+
+	/*
+	 * Found exec name
+	 */
+	exec_name = str + zi;
+
+	/*
+	 * Skip to the next spaces
+	 */
+	for (; str[zi] != ' ' && str[zi] != '\0'; zi++) ;
+
+	if (str[zi] == '\0') {
+		qdevice_heuristics_worker_log_printf(instance, LOG_CRIT,
+		    "qdevice_heuristics_worker_cmd_process_exec_list_add: Can't find second space");
+		return (-1);
+	}
+
+	/*
+	 * Put trailing \0 into exec_name
+	 */
+	str[zi] = '\0';
+	zi++;
+
+	/*
+	 * Skip to the end of next spaces
+	 */
+	for (; str[zi] == ' '; zi++) ;
+
+	if (str[zi] == '\0') {
+		qdevice_heuristics_worker_log_printf(instance, LOG_CRIT,
+		    "qdevice_heuristics_worker_cmd_process_exec_list_add: Can't find start of exec command");
+		return (-1);
+	}
+
+	/*
+	 * Found exec_command
+	 */
+	exec_command = str + zi;
+
+	qdevice_heuristics_worker_log_printf(instance, LOG_DEBUG,
+	    "qdevice_heuristics_worker_cmd_process_one_line: Received exec-list-add command "
+	    "with name \"%s\" and command \"%s\"", exec_name, exec_command);
+
+	if (qdevice_heuristics_exec_list_add(&instance->exec_list, exec_name, exec_command) == NULL) {
+		qdevice_heuristics_worker_log_printf(instance, LOG_CRIT,
+		    "qdevice_heuristics_worker_cmd_process_exec_list_add: Can't alloc exec list entry");
+		return (-1);
+	}
+
+	return (0);
+}
+
+static int
+qdevice_heuristics_worker_cmd_process_exec(struct qdevice_heuristics_worker_instance *instance,
+    struct dynar *data)
+{
+	uint32_t timeout;
+	uint32_t seq_number;
+	char *str;
+	struct qdevice_heuristics_exec_list_entry *exec_list_entry;
+	struct process_list_entry *plist_entry;
+
+	str = dynar_data(data);
+
+	if (sscanf(str, QDEVICE_HEURISTICS_CMD_STR_EXEC_ADD_SPACE "%"PRIu32" %"PRIu32, &timeout,
+	    &seq_number) != 2) {
+		qdevice_heuristics_worker_log_printf(instance, LOG_CRIT,
+		    "qdevice_heuristics_worker_cmd_process_exec: Can't parse command (sscanf)");
+		return (-1);
+	}
+
+	qdevice_heuristics_worker_log_printf(instance, LOG_DEBUG,
+	    "qdevice_heuristics_worker_cmd_process_exec: Received exec command "
+	    "with seq_no \"%"PRIu32"\" and timeout \"%"PRIu32"\"", seq_number, timeout);
+
+	if (instance->exec_timeout_timer != NULL) {
+		process_list_move_active_entries_to_kill_list(&instance->main_process_list);
+
+		timer_list_delete(&instance->main_timer_list, instance->exec_timeout_timer);
+		instance->exec_timeout_timer = NULL;
+	}
+
+	instance->last_exec_seq_number = seq_number;
+
+	if (qdevice_heuristics_exec_list_is_empty(&instance->exec_list)) {
+		if (qdevice_heuristics_worker_cmd_write_exec_result(instance,
+		    instance->last_exec_seq_number,
+		    QDEVICE_HEURISTICS_EXEC_RESULT_DISABLED) != 0) {
+			return (-1);
+		}
+	} else {
+		/*
+		 * Initialize process list (from exec list)
+		 */
+		TAILQ_FOREACH(exec_list_entry, &instance->exec_list, entries) {
+			plist_entry = process_list_add(&instance->main_process_list,
+			    exec_list_entry->name, exec_list_entry->command);
+
+			if (plist_entry == NULL) {
+				qdevice_heuristics_worker_log_printf(instance, LOG_ERR,
+				    "qdevice_heuristics_worker_cmd_process_exec: Can't allocate "
+				    "process list entry");
+
+				process_list_move_active_entries_to_kill_list(
+				    &instance->main_process_list);
+
+				if (qdevice_heuristics_worker_cmd_write_exec_result(instance,
+				    instance->last_exec_seq_number,
+				    QDEVICE_HEURISTICS_EXEC_RESULT_FAIL) != 0) {
+					return (-1);
+				}
+
+				return (0);
+			}
+		}
+
+		if (process_list_exec_initialized(&instance->main_process_list) != 0) {
+			qdevice_heuristics_worker_log_printf(instance, LOG_ERR,
+			    "qdevice_heuristics_worker_cmd_process_exec: Can't execute "
+			    "process list");
+
+			process_list_move_active_entries_to_kill_list(&instance->main_process_list);
+
+			if (qdevice_heuristics_worker_cmd_write_exec_result(instance,
+			    instance->last_exec_seq_number,
+			    QDEVICE_HEURISTICS_EXEC_RESULT_FAIL) != 0) {
+				return (-1);
+			}
+
+			return (0);
+		}
+
+		instance->exec_timeout_timer = timer_list_add(&instance->main_timer_list,
+		    timeout, qdevice_heuristics_worker_exec_timeout_timer_callback,
+		    (void *)instance, NULL);
+		if (instance->exec_timeout_timer == NULL) {
+			qdevice_heuristics_worker_log_printf(instance, LOG_ERR,
+			    "qdevice_heuristics_worker_cmd_process_exec: Can't add exec timeout "
+			    "timer to timer list");
+
+			process_list_move_active_entries_to_kill_list(&instance->main_process_list);
+
+			if (qdevice_heuristics_worker_cmd_write_exec_result(instance,
+			    instance->last_exec_seq_number,
+			    QDEVICE_HEURISTICS_EXEC_RESULT_FAIL) != 0) {
+				return (-1);
+			}
+
+			return (0);
+		}
+	}
+
+	return (0);
+}
+
+/*
+ * 1 - Line processed
+ * 0 - No line to process - everything processed
+ * -1 - Error
+ */
+static int
+qdevice_heuristics_worker_cmd_process_one_line(struct qdevice_heuristics_worker_instance *instance,
+    struct dynar *data)
+{
+	char *str;
+	size_t str_len;
+	size_t nl_pos;
+	size_t zi;
+
+	str = dynar_data(data);
+	str_len = dynar_size(data);
+
+	/*
+	 * Find valid line
+	 */
+	for (zi = 0; zi < str_len && str[zi] != '\r' && str[zi] != '\n'; zi++) ;
+
+	if (zi >= str_len) {
+		/*
+		 * Command is not yet fully readed
+		 */
+		return (0);
+	}
+
+	nl_pos = zi;
+
+	str[nl_pos] = '\0';
+
+	if (strcmp(str, QDEVICE_HEURISTICS_CMD_STR_EXEC_LIST_CLEAR) == 0) {
+		qdevice_heuristics_worker_log_printf(instance, LOG_DEBUG,
+		    "qdevice_heuristics_worker_cmd_process_one_line: Received exec-list-clear command");
+
+		qdevice_heuristics_exec_list_free(&instance->exec_list);
+	} else if (strncmp(str, QDEVICE_HEURISTICS_CMD_STR_EXEC_LIST_ADD_SPACE,
+	    strlen(QDEVICE_HEURISTICS_CMD_STR_EXEC_LIST_ADD)) == 0) {
+		if (qdevice_heuristics_worker_cmd_process_exec_list_add(instance, data) != 0) {
+			return (-1);
+		}
+	} else if (strncmp(str, QDEVICE_HEURISTICS_CMD_STR_EXEC_ADD_SPACE,
+	    strlen(QDEVICE_HEURISTICS_CMD_STR_EXEC_ADD_SPACE)) == 0) {
+		if (qdevice_heuristics_worker_cmd_process_exec(instance, data) != 0) {
+			return (-1);
+		}
+	} else {
+		qdevice_heuristics_worker_log_printf(instance, LOG_CRIT,
+		    "qdevice_heuristics_worker_cmd_process_one_line: Unknown command \"%s\" "
+		    "received from main qdevice process", str);
+
+		    return (-1);
+	}
+
+	/*
+	 * Find place where is begining of new "valid" line
+	 */
+	for (zi = nl_pos + 1; zi < str_len && (str[zi] == '\0' || str[zi] == '\n' || str[zi] == '\r'); zi++) ;
+
+	memmove(str, str + zi, str_len - zi);
+	if (dynar_set_size(data, str_len - zi) == -1) {
+		qdevice_heuristics_worker_log_printf(instance, LOG_CRIT,
+		    "qdevice_heuristics_worker_cmd_process_one_line: Can't set dynar size");
+		return (-1);
+	}
+
+	return (1);
+}
+
+/*
+ * 0 - No error
+ * -1 - Error
+ */
+static int
+qdevice_heuristics_worker_cmd_process(struct qdevice_heuristics_worker_instance *instance)
+{
+	int res;
+
+	while ((res =
+	    qdevice_heuristics_worker_cmd_process_one_line(instance, &instance->cmd_in_buffer)) == 1) ;
+
+	return (res);
+}
+
+/*
+ * 0 - No error
+ * 1 - Error
+ */
+int
+qdevice_heuristics_worker_cmd_read_from_pipe(struct qdevice_heuristics_worker_instance *instance)
+{
+	int res;
+	int ret;
+
+	res = qdevice_heuristics_io_read(QDEVICE_HEURISTICS_WORKER_CMD_IN_FD, &instance->cmd_in_buffer);
+
+	ret = 0;
+
+	switch (res) {
+	case 0:
+		/*
+		 * Partial read
+		 */
+		break;
+	case -1:
+		qdevice_heuristics_worker_log_printf(instance, LOG_ERR,
+		    "Lost connection with main qdevice process");
+		ret = -1;
+		break;
+	case -2:
+		qdevice_heuristics_worker_log_printf(instance, LOG_ERR,
+		    "Heuristics sent too long command line");
+		ret = -1;
+		break;
+	case -3:
+		qdevice_heuristics_worker_log_printf(instance, LOG_ERR,
+		    "Unhandled error when reading from heuristics command in fd");
+		ret = -1;
+		break;
+	case 1:
+		/*
+		 * At least one log line received
+		 */
+		ret = qdevice_heuristics_worker_cmd_process(instance);
+		break;
+	}
+
+	return (ret);
+}
+
+int
+qdevice_heuristics_worker_cmd_write_exec_result(struct qdevice_heuristics_worker_instance *instance,
+    uint32_t seq_number, enum qdevice_heuristics_exec_result exec_result)
+{
+	if (dynar_str_cpy(&instance->cmd_out_buffer,
+	    QDEVICE_HEURISTICS_CMD_STR_EXEC_RESULT_ADD_SPACE) != -1 &&
+	    dynar_str_catf(&instance->cmd_out_buffer, "%"PRIu32" %u\n", seq_number,
+	    (int)exec_result) != -1) {
+		(void)qdevice_heuristics_io_blocking_write(QDEVICE_HEURISTICS_WORKER_CMD_OUT_FD,
+		    dynar_data(&instance->cmd_out_buffer), dynar_size(&instance->cmd_out_buffer));
+	} else {
+		qdevice_heuristics_worker_log_printf(instance, LOG_CRIT,
+		    "Can't alloc memory for exec result");
+
+		return (-1);
+	}
+
+	return (0);
+}

qdevices/qdevice-heuristics-worker-cmd.h

@@ -0,0 +1,56 @@
+/*
+ * Copyright (c) 2015-2017 Red Hat, Inc.
+ *
+ * All rights reserved.
+ *
+ * Author: Jan Friesse (jfriesse@redhat.com)
+ *
+ * This software licensed under BSD license, the text of which follows:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright notice,
+ *   this list of conditions and the following disclaimer.
+ * - Redistributions in binary form must reproduce the above copyright notice,
+ *   this list of conditions and the following disclaimer in the documentation
+ *   and/or other materials provided with the distribution.
+ * - Neither the name of the Red Hat, Inc. nor the names of its
+ *   contributors may be used to endorse or promote products derived from this
+ *   software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _QDEVICE_HEURISTICS_WORKER_CMD_H_
+#define _QDEVICE_HEURISTICS_WORKER_CMD_H_
+
+#include "qdevice-heuristics-worker-instance.h"
+#include "qdevice-heuristics-exec-result.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+extern int		qdevice_heuristics_worker_cmd_read_from_pipe(
+    struct qdevice_heuristics_worker_instance *instance);
+
+extern int		qdevice_heuristics_worker_cmd_write_exec_result(
+    struct qdevice_heuristics_worker_instance *instance, uint32_t seq_number,
+    enum qdevice_heuristics_exec_result exec_result);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _QDEVICE_HEURISTICS_WORKER_CMD_H_ */

qdevices/qdevice-heuristics-worker-instance.h

@@ -0,0 +1,69 @@
+/*
+ * Copyright (c) 2015-2017 Red Hat, Inc.
+ *
+ * All rights reserved.
+ *
+ * Author: Jan Friesse (jfriesse@redhat.com)
+ *
+ * This software licensed under BSD license, the text of which follows:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright notice,
+ *   this list of conditions and the following disclaimer.
+ * - Redistributions in binary form must reproduce the above copyright notice,
+ *   this list of conditions and the following disclaimer in the documentation
+ *   and/or other materials provided with the distribution.
+ * - Neither the name of the Red Hat, Inc. nor the names of its
+ *   contributors may be used to endorse or promote products derived from this
+ *   software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _QDEVICE_HEURISTICS_WORKER_INSTANCE_H_
+#define _QDEVICE_HEURISTICS_WORKER_INSTANCE_H_
+
+#include "dynar.h"
+
+#include "qdevice-heuristics-exec-list.h"
+#include "process-list.h"
+#include "timer-list.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+struct qdevice_heuristics_worker_instance {
+	struct dynar cmd_in_buffer;
+	struct dynar cmd_out_buffer;
+	struct dynar log_out_buffer;
+
+	struct qdevice_heuristics_exec_list exec_list;
+	struct process_list main_process_list;
+	struct timer_list main_timer_list;
+
+	struct timer_list_entry *kill_list_timer;
+	struct timer_list_entry *exec_timeout_timer;
+
+	uint32_t last_exec_seq_number;
+
+	int schedule_exit;
+};
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _QDEVICE_HEURISTICS_WORKER_INSTANCE_H_ */

qdevices/qdevice-heuristics-worker-log.c

@@ -0,0 +1,96 @@
+/*
+ * Copyright (c) 2015-2017 Red Hat, Inc.
+ *
+ * All rights reserved.
+ *
+ * Author: Jan Friesse (jfriesse@redhat.com)
+ *
+ * This software licensed under BSD license, the text of which follows:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright notice,
+ *   this list of conditions and the following disclaimer.
+ * - Redistributions in binary form must reproduce the above copyright notice,
+ *   this list of conditions and the following disclaimer in the documentation
+ *   and/or other materials provided with the distribution.
+ * - Neither the name of the Red Hat, Inc. nor the names of its
+ *   contributors may be used to endorse or promote products derived from this
+ *   software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <stdarg.h>
+#include <stdlib.h>
+#include <syslog.h>
+
+#include "dynar.h"
+#include "dynar-str.h"
+#include "qdevice-heuristics-io.h"
+#include "qdevice-heuristics-worker.h"
+#include "qdevice-heuristics-worker-log.h"
+
+static int
+qdevice_heuristics_worker_log_remove_newlines(struct dynar *str)
+{
+	size_t len;
+	size_t zi;
+	char *buf;
+
+	len = dynar_size(str);
+	buf = dynar_data(str);
+
+	for (zi = 0; zi < len ; zi++) {
+		if (buf[zi] == '\n' || buf[zi] == '\r') {
+			buf[zi] = ' ';
+		}
+	}
+
+	return (0);
+}
+
+void
+qdevice_heuristics_worker_log_printf(struct qdevice_heuristics_worker_instance *instance,
+    int priority, const char *format, ...)
+{
+	va_list ap;
+	va_list ap_copy;
+
+	va_start(ap, format);
+
+	if (dynar_str_cpy(&instance->log_out_buffer, "") != -1 &&
+	    dynar_str_catf(&instance->log_out_buffer, "%u ", priority) != -1 &&
+	    dynar_str_vcatf(&instance->log_out_buffer, format, ap) != -1 &&
+	    qdevice_heuristics_worker_log_remove_newlines(&instance->log_out_buffer) != -1 &&
+	    dynar_str_cat(&instance->log_out_buffer, "\n") != -1) {
+		/*
+		 * It was possible to log everything
+		 */
+		(void)qdevice_heuristics_io_blocking_write(QDEVICE_HEURISTICS_WORKER_LOG_OUT_FD,
+		    dynar_data(&instance->log_out_buffer), dynar_size(&instance->log_out_buffer));
+	} else {
+		/*
+		 * As a fallback try to log to syslog
+		 */
+		va_copy(ap_copy, ap);
+		openlog("qdevice_heuristics_worker", LOG_PID, LOG_DAEMON);
+		syslog(LOG_ERR, "Log entry sent to syslog instead of parent process");
+		vsyslog(priority, format, ap_copy);
+		closelog();
+		va_end(ap_copy);
+	}
+
+	va_end(ap);
+}

qdevices/qdevice-heuristics-worker-log.h

@@ -0,0 +1,54 @@
+/*
+ * Copyright (c) 2015-2017 Red Hat, Inc.
+ *
+ * All rights reserved.
+ *
+ * Author: Jan Friesse (jfriesse@redhat.com)
+ *
+ * This software licensed under BSD license, the text of which follows:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright notice,
+ *   this list of conditions and the following disclaimer.
+ * - Redistributions in binary form must reproduce the above copyright notice,
+ *   this list of conditions and the following disclaimer in the documentation
+ *   and/or other materials provided with the distribution.
+ * - Neither the name of the Red Hat, Inc. nor the names of its
+ *   contributors may be used to endorse or promote products derived from this
+ *   software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _QDEVICE_HEURISTICS_WORKER_LOG_H_
+#define _QDEVICE_HEURISTICS_WORKER_LOG_H_
+
+#include <syslog.h>
+
+#include "qdevice-heuristics-worker-instance.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+extern void		qdevice_heuristics_worker_log_printf(
+    struct qdevice_heuristics_worker_instance *instance,
+    int priority, const char *format, ...) __attribute__((__format__(__printf__, 3, 0)));
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _QDEVICE_HEURISTICS_WORKER_LOG_H_ */

qdevices/qdevice-heuristics-worker.c

@@ -0,0 +1,353 @@
+/*
+ * Copyright (c) 2015-2017 Red Hat, Inc.
+ *
+ * All rights reserved.
+ *
+ * Author: Jan Friesse (jfriesse@redhat.com)
+ *
+ * This software licensed under BSD license, the text of which follows:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright notice,
+ *   this list of conditions and the following disclaimer.
+ * - Redistributions in binary form must reproduce the above copyright notice,
+ *   this list of conditions and the following disclaimer in the documentation
+ *   and/or other materials provided with the distribution.
+ * - Neither the name of the Red Hat, Inc. nor the names of its
+ *   contributors may be used to endorse or promote products derived from this
+ *   software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <limits.h>
+#include <errno.h>
+#include <poll.h>
+#include <signal.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+
+#include "dynar-str.h"
+#include "qdevice-config.h"
+#include "qdevice-heuristics-io.h"
+#include "qdevice-heuristics-worker.h"
+#include "qdevice-heuristics-worker-instance.h"
+#include "qdevice-heuristics-worker-log.h"
+#include "qdevice-heuristics-worker-cmd.h"
+
+/*
+ * Declarations
+ */
+static int		qdevice_heuristics_worker_kill_list_timer_callback(void *data1,
+    void *data2);
+
+static void		qdevice_heuristics_worker_process_list_notify(
+    enum process_list_notify_reason reason, const struct process_list_entry *entry,
+    void *user_data);
+
+static void		qdevice_heuristics_worker_signal_handlers_register(void);
+
+
+/*
+ * Definitions
+ */
+static void
+qdevice_heuristics_worker_process_list_notify(enum process_list_notify_reason reason,
+    const struct process_list_entry *entry, void *user_data)
+{
+	struct qdevice_heuristics_worker_instance *instance;
+
+	instance = (struct qdevice_heuristics_worker_instance *)user_data;
+
+	switch (reason) {
+	case PROCESS_LIST_NOTIFY_REASON_EXECUTED:
+		qdevice_heuristics_worker_log_printf(instance, LOG_DEBUG,
+		    "process %s executed", entry->name);
+		break;
+	case PROCESS_LIST_NOTIFY_REASON_FINISHED:
+		if (!WIFEXITED(entry->exit_status) || WEXITSTATUS(entry->exit_status) != 0) {
+			if (WIFEXITED(entry->exit_status)) {
+				qdevice_heuristics_worker_log_printf(instance, LOG_WARNING,
+				    "process %s finished with status %d", entry->name,
+				    WEXITSTATUS(entry->exit_status));
+			} else if (WIFSIGNALED(entry->exit_status)) {
+				qdevice_heuristics_worker_log_printf(instance, LOG_WARNING,
+				    "process %s killed by signal %d", entry->name,
+				    WTERMSIG(entry->exit_status));
+			} else {
+				qdevice_heuristics_worker_log_printf(instance, LOG_WARNING,
+				    "process %s finished with non zero status", entry->name);
+			}
+		} else {
+			qdevice_heuristics_worker_log_printf(instance, LOG_DEBUG,
+			    "process %s sucesfully finished", entry->name);
+		}
+		break;
+	}
+}
+
+static void
+qdevice_heuristics_worker_signal_handlers_register(void)
+{
+	struct sigaction act;
+
+	act.sa_handler = SIG_DFL;
+	sigemptyset(&act.sa_mask);
+	act.sa_flags = SA_RESTART;
+
+	sigaction(SIGCHLD, &act, NULL);
+
+	act.sa_handler = SIG_IGN;
+	sigemptyset(&act.sa_mask);
+	act.sa_flags = SA_RESTART;
+
+	sigaction(SIGPIPE, &act, NULL);
+
+	act.sa_handler = SIG_IGN;
+	sigemptyset(&act.sa_mask);
+	act.sa_flags = SA_RESTART;
+
+	sigaction(SIGINT, &act, NULL);
+}
+
+static int
+qdevice_heuristics_worker_kill_list_timer_callback(void *data1, void *data2)
+{
+	struct qdevice_heuristics_worker_instance *instance;
+	size_t kill_list_size;
+
+	instance = (struct qdevice_heuristics_worker_instance *)data1;
+
+	if (process_list_process_kill_list(&instance->main_process_list) != 0) {
+		qdevice_heuristics_worker_log_printf(instance, LOG_CRIT,
+		    "qdevice_heuristics_worker_kill_list_timer_callback: process kill list failed. "
+		    "Shutting down worker");
+
+		instance->schedule_exit = 1;
+		return (0);
+	}
+
+	kill_list_size = process_list_get_kill_list_items(&instance->main_process_list);
+
+	if (kill_list_size > 0) {
+		qdevice_heuristics_worker_log_printf(instance, LOG_DEBUG,
+		    "Still waiting for %zu processes exit", kill_list_size);
+	}
+
+	/*
+	 * Schedule this timer again
+	 */
+	return (-1);
+}
+
+int
+qdevice_heuristics_worker_exec_timeout_timer_callback(void *data1, void *data2)
+{
+	struct qdevice_heuristics_worker_instance *instance;
+
+	instance = (struct qdevice_heuristics_worker_instance *)data1;
+
+	qdevice_heuristics_worker_log_printf(instance, LOG_WARNING,
+	    "Not all heuristics execs finished on time");
+
+	process_list_move_active_entries_to_kill_list(&instance->main_process_list);
+
+	instance->exec_timeout_timer = NULL;
+
+	if (qdevice_heuristics_worker_cmd_write_exec_result(instance, instance->last_exec_seq_number,
+	    QDEVICE_HEURISTICS_EXEC_RESULT_FAIL) != 0) {
+		instance->schedule_exit = 1;
+
+		return (0);
+	}
+
+	return (0);
+}
+
+static int
+qdevice_heuristics_worker_poll(struct qdevice_heuristics_worker_instance *instance)
+{
+	int poll_res;
+	struct pollfd poll_input_fd;
+	uint32_t timeout;
+	int plist_summary;
+
+	/*
+	 * Poll command input
+	 */
+	poll_input_fd.fd = QDEVICE_HEURISTICS_WORKER_CMD_IN_FD;
+	poll_input_fd.events = POLLIN;
+	poll_input_fd.revents = 0;
+
+	timeout = timer_list_time_to_expire_ms(&instance->main_timer_list);
+	if (timeout > QDEVICE_MIN_HEURISTICS_TIMEOUT) {
+		timeout = QDEVICE_MIN_HEURISTICS_TIMEOUT;
+	}
+
+	if ((poll_res = poll(&poll_input_fd, 1, timeout)) >= 0) {
+		if (poll_input_fd.revents & POLLIN) {
+			/*
+			 * POLLIN
+			 */
+			if (qdevice_heuristics_worker_cmd_read_from_pipe(instance) != 0) {
+				return (-1);
+			}
+		}
+
+		if (poll_input_fd.revents & POLLOUT) {
+			/*
+			 * Pollout shouldn't happen (critical error)
+			 */
+			qdevice_heuristics_worker_log_printf(instance, LOG_CRIT,
+			    "qdevice_heuristics_worker_poll: POLLOUT set. Shutting down worker");
+
+			return (-1);
+		}
+
+		if (poll_input_fd.revents & (POLLERR|POLLHUP|POLLNVAL) &&
+		    !(poll_input_fd.revents & (POLLIN|POLLOUT))) {
+			/*
+			 * Qdevice closed pipe
+			 */
+
+			return (-1);
+		}
+	}
+
+	if (process_list_waitpid(&instance->main_process_list) != 0) {
+		qdevice_heuristics_worker_log_printf(instance, LOG_CRIT,
+		    "qdevice_heuristics_worker_poll: Waitpid failed. Shutting down worker");
+
+		return (-1);
+	}
+
+	if (instance->exec_timeout_timer != NULL) {
+		plist_summary = process_list_get_summary_result_short(&instance->main_process_list);
+
+		switch (plist_summary) {
+		case -1:
+			/*
+			 * Processes not finished -> continue
+			 */
+			break;
+		case 0:
+			/*
+			 * All processes finished sucesfully
+			 */
+			if (qdevice_heuristics_worker_cmd_write_exec_result(instance,
+			    instance->last_exec_seq_number, QDEVICE_HEURISTICS_EXEC_RESULT_PASS) != 0) {
+				return (-1);
+			}
+
+			process_list_move_active_entries_to_kill_list(&instance->main_process_list);
+
+			timer_list_delete(&instance->main_timer_list, instance->exec_timeout_timer);
+			instance->exec_timeout_timer = NULL;
+
+			break;
+		case 1:
+			/*
+			 * Some processes failed
+			 */
+			if (qdevice_heuristics_worker_cmd_write_exec_result(instance,
+			    instance->last_exec_seq_number, QDEVICE_HEURISTICS_EXEC_RESULT_FAIL) != 0) {
+				return (-1);
+			}
+
+			process_list_move_active_entries_to_kill_list(&instance->main_process_list);
+
+			timer_list_delete(&instance->main_timer_list, instance->exec_timeout_timer);
+			instance->exec_timeout_timer = NULL;
+			break;
+		default:
+			qdevice_heuristics_worker_log_printf(instance, LOG_CRIT,
+			    "qdevice_heuristics_worker_poll: Unhandled "
+			    "process_list_get_summary_result. Shutting down worker");
+
+			return (-1);
+			break;
+		}
+	}
+
+	timer_list_expire(&instance->main_timer_list);
+
+	if (instance->schedule_exit) {
+		return (-1);
+	}
+
+	return (0);
+}
+
+void
+qdevice_heuristics_worker_start(size_t ipc_max_send_receive_size, int use_execvp,
+    size_t max_processes, uint32_t kill_list_interval)
+{
+	struct qdevice_heuristics_worker_instance instance;
+
+	memset(&instance, 0, sizeof(instance));
+
+	instance.schedule_exit = 0;
+
+	dynar_init(&instance.cmd_in_buffer, ipc_max_send_receive_size);
+	dynar_init(&instance.cmd_out_buffer, ipc_max_send_receive_size);
+	dynar_init(&instance.log_out_buffer, ipc_max_send_receive_size);
+
+	process_list_init(&instance.main_process_list, max_processes, use_execvp,
+	    qdevice_heuristics_worker_process_list_notify, (void *)&instance);
+
+	timer_list_init(&instance.main_timer_list);
+	instance.kill_list_timer = timer_list_add(&instance.main_timer_list,
+	    kill_list_interval, qdevice_heuristics_worker_kill_list_timer_callback,
+	    (void *)&instance, NULL);
+
+	if (instance.kill_list_timer == NULL) {
+		qdevice_heuristics_worker_log_printf(&instance, LOG_CRIT,
+		    "Can't create kill list timer");
+		return ;
+	}
+
+	instance.exec_timeout_timer = NULL;
+
+	qdevice_heuristics_exec_list_init(&instance.exec_list);
+
+	qdevice_heuristics_worker_signal_handlers_register();
+
+	qdevice_heuristics_worker_log_printf(&instance, LOG_DEBUG, "Heuristic worker initialized");
+
+	while (qdevice_heuristics_worker_poll(&instance) == 0) {
+	}
+
+	qdevice_heuristics_worker_log_printf(&instance, LOG_DEBUG, "Heuristic worker shutdown "
+	    "requested");
+
+	qdevice_heuristics_exec_list_free(&instance.exec_list);
+
+	timer_list_free(&instance.main_timer_list);
+
+	qdevice_heuristics_worker_log_printf(&instance, LOG_DEBUG,
+	    "Waiting for all processes to exit");
+
+	if (process_list_killall(&instance.main_process_list, kill_list_interval) != 0) {
+		qdevice_heuristics_worker_log_printf(&instance, LOG_WARNING,
+		    "Not all process exited");
+	}
+
+	process_list_free(&instance.main_process_list);
+
+	dynar_destroy(&instance.cmd_in_buffer);
+	dynar_destroy(&instance.cmd_out_buffer);
+	dynar_destroy(&instance.log_out_buffer);
+}

qdevices/qdevice-heuristics-worker.h

@@ -0,0 +1,55 @@
+/*
+ * Copyright (c) 2015-2017 Red Hat, Inc.
+ *
+ * All rights reserved.
+ *
+ * Author: Jan Friesse (jfriesse@redhat.com)
+ *
+ * This software licensed under BSD license, the text of which follows:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright notice,
+ *   this list of conditions and the following disclaimer.
+ * - Redistributions in binary form must reproduce the above copyright notice,
+ *   this list of conditions and the following disclaimer in the documentation
+ *   and/or other materials provided with the distribution.
+ * - Neither the name of the Red Hat, Inc. nor the names of its
+ *   contributors may be used to endorse or promote products derived from this
+ *   software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _QDEVICE_HEURISTICS_WORKER_H_
+#define _QDEVICE_HEURISTICS_WORKER_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#define QDEVICE_HEURISTICS_WORKER_CMD_IN_FD		0
+#define QDEVICE_HEURISTICS_WORKER_CMD_OUT_FD		1
+#define QDEVICE_HEURISTICS_WORKER_LOG_OUT_FD		2
+
+void		qdevice_heuristics_worker_start(size_t ipc_max_send_receive_size,
+    int use_execvp, size_t max_processes, uint32_t kill_list_interval);
+
+int		qdevice_heuristics_worker_exec_timeout_timer_callback(void *data1, void *data2);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _QDEVICE_HEURISTICS_WORKER_H_ */

qdevices/qdevice-heuristics.c

@@ -0,0 +1,372 @@
+/*
+ * Copyright (c) 2015-2017 Red Hat, Inc.
+ *
+ * All rights reserved.
+ *
+ * Author: Jan Friesse (jfriesse@redhat.com)
+ *
+ * This software licensed under BSD license, the text of which follows:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright notice,
+ *   this list of conditions and the following disclaimer.
+ * - Redistributions in binary form must reproduce the above copyright notice,
+ *   this list of conditions and the following disclaimer in the documentation
+ *   and/or other materials provided with the distribution.
+ * - Neither the name of the Red Hat, Inc. nor the names of its
+ *   contributors may be used to endorse or promote products derived from this
+ *   software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <sys/types.h>
+#include <sys/wait.h>
+
+#include <err.h>
+#include <poll.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+
+#include "qdevice-log.h"
+#include "qdevice-heuristics.h"
+#include "qdevice-heuristics-cmd.h"
+#include "qdevice-heuristics-worker.h"
+#include "qdevice-heuristics-io.h"
+#include "qdevice-votequorum.h"
+#include "utils.h"
+
+#define QDEVICE_HEURISTICS_WAIT_FOR_INITIAL_EXEC_RESULT_MAX_PFDS	5
+
+void
+qdevice_heuristics_init(struct qdevice_heuristics_instance *instance,
+    struct qdevice_advanced_settings *advanced_settings)
+{
+	int pipe_cmd_in[2], pipe_cmd_out[2], pipe_log_out[2];
+	pid_t pid;
+
+	if (pipe(pipe_cmd_in) != 0) {
+		err(1, "Can't create command input pipe");
+	}
+
+	if (pipe(pipe_cmd_out) != 0) {
+		err(1, "Can't create command output pipe");
+	}
+
+	if (pipe(pipe_log_out) != 0) {
+		err(1, "Can't create logging output pipe");
+	}
+
+	pid = fork();
+	if (pid == -1) {
+		err(1, "Can't create child process");
+	} else if (pid == 0) {
+		/*
+		 * Child
+		 */
+		(void)setsid();
+		if (dup2(pipe_cmd_in[0], 0) == -1) {
+			err(1, "Can't dup2 command input pipe");
+		}
+		close(pipe_cmd_in[1]);
+		close(pipe_cmd_in[0]);
+		if (utils_fd_set_non_blocking(0) == -1) {
+			err(1, "Can't set non blocking flag on command input pipe");
+		}
+
+		if (dup2(pipe_cmd_out[1], 1) == -1) {
+			err(1, "Can't dup2 command output pipe");
+		}
+		close(pipe_cmd_out[0]);
+		close(pipe_cmd_out[1]);
+
+		if (dup2(pipe_log_out[1], 2) == -1) {
+			err(1, "Can't dup2 logging output pipe");
+		}
+		close(pipe_log_out[0]);
+		close(pipe_log_out[1]);
+
+		qdevice_heuristics_worker_start(advanced_settings->heuristics_ipc_max_send_receive_size,
+		    advanced_settings->heuristics_use_execvp, advanced_settings->heuristics_max_processes,
+		    advanced_settings->heuristics_kill_list_interval);
+
+		qdevice_advanced_settings_destroy(advanced_settings);
+
+		exit(0);
+	} else {
+		close(pipe_cmd_in[0]);
+		close(pipe_cmd_out[1]);
+		close(pipe_log_out[1]);
+
+		qdevice_heuristics_instance_init(instance);
+
+		instance->pipe_cmd_send = pipe_cmd_in[1];
+		if (utils_fd_set_non_blocking(instance->pipe_cmd_send) == -1) {
+			err(1, "Can't set non blocking flag on command input pipe");
+		}
+		instance->pipe_cmd_recv = pipe_cmd_out[0];
+		if (utils_fd_set_non_blocking(instance->pipe_cmd_recv) == -1) {
+			err(1, "Can't set non blocking flag on command output pipe");
+		}
+		instance->pipe_log_recv = pipe_log_out[0];
+		if (utils_fd_set_non_blocking(instance->pipe_cmd_recv) == -1) {
+			err(1, "Can't set non blocking flag on logging output pipe");
+		}
+		instance->worker_pid = pid;
+
+		send_buffer_list_init(&instance->cmd_out_buffer_list,
+		    advanced_settings->heuristics_ipc_max_send_buffers,
+		    advanced_settings->heuristics_ipc_max_send_receive_size);
+		dynar_init(&instance->log_in_buffer,
+		    advanced_settings->heuristics_ipc_max_send_receive_size);
+		dynar_init(&instance->cmd_in_buffer,
+		    advanced_settings->heuristics_ipc_max_send_receive_size);
+	}
+}
+
+void
+qdevice_heuristics_destroy(struct qdevice_heuristics_instance *instance)
+{
+	int status;
+
+	/*
+	 * Close of pipe_cmd_send result is correct and almost instant exit of worker
+	 */
+	close(instance->pipe_cmd_send);
+
+	qdevice_log(LOG_DEBUG, "Waiting for heuristics worker to finish");
+	if (waitpid(instance->worker_pid, &status, 0) == -1) {
+		qdevice_log_err(LOG_ERR, "Heuristics worker waitpid failed");
+	} else {
+		/*
+		 * Log what left in worker log buffer. Errors can be ignored
+		 */
+		(void)qdevice_heuristics_log_read_from_pipe(instance);
+	}
+
+	close(instance->pipe_cmd_recv);
+	close(instance->pipe_log_recv);
+
+	dynar_destroy(&instance->log_in_buffer);
+	dynar_destroy(&instance->cmd_in_buffer);
+	send_buffer_list_free(&instance->cmd_out_buffer_list);
+
+	qdevice_heuristics_instance_destroy(instance);
+}
+
+int
+qdevice_heuristics_exec(struct qdevice_heuristics_instance *instance, int sync_in_progress)
+{
+	uint32_t timeout;
+
+	instance->expected_reply_seq_number++;
+	instance->waiting_for_result = 1;
+
+	if (sync_in_progress) {
+		timeout = instance->sync_timeout;
+	} else {
+		timeout = instance->timeout;
+	}
+
+	return (qdevice_heuristics_cmd_write_exec(instance, timeout,
+	    instance->expected_reply_seq_number));
+}
+
+int
+qdevice_heuristics_waiting_for_result(const struct qdevice_heuristics_instance *instance)
+{
+
+	return (instance->waiting_for_result);
+}
+
+int
+qdevice_heuristics_change_exec_list(struct qdevice_heuristics_instance *instance,
+    const struct qdevice_heuristics_exec_list *new_exec_list, int sync_in_progress)
+{
+
+	if (qdevice_heuristics_cmd_write_exec_list(instance, new_exec_list) != 0) {
+		return (-1);
+	}
+
+	qdevice_heuristics_exec_list_free(&instance->exec_list);
+
+	if (new_exec_list != NULL) {
+		if (qdevice_heuristics_exec_list_clone(&instance->exec_list, new_exec_list) != 0) {
+			qdevice_log(LOG_ERR, "Can't clone exec list");
+
+			return (-1);
+		}
+	}
+
+	if (qdevice_heuristics_waiting_for_result(instance)) {
+		if (qdevice_heuristics_exec(instance, sync_in_progress) != 0) {
+			qdevice_log(LOG_ERR, "Can't execute heuristics");
+
+			return (-1);
+		}
+	}
+
+	return (0);
+}
+
+
+int
+qdevice_heuristics_wait_for_initial_exec_result(struct qdevice_heuristics_instance *instance)
+{
+	struct pollfd pfds[QDEVICE_HEURISTICS_WAIT_FOR_INITIAL_EXEC_RESULT_MAX_PFDS];
+	int no_pfds;
+	int poll_res;
+	int timeout;
+	int i;
+	int case_processed;
+	int res;
+
+	while (!instance->qdevice_instance_ptr->vq_node_list_initial_heuristics_finished) {
+		no_pfds = 0;
+
+		assert(no_pfds < QDEVICE_HEURISTICS_WAIT_FOR_INITIAL_EXEC_RESULT_MAX_PFDS);
+		pfds[no_pfds].fd = instance->pipe_log_recv;
+		pfds[no_pfds].events = POLLIN;
+		pfds[no_pfds].revents = 0;
+		no_pfds++;
+
+		assert(no_pfds < QDEVICE_HEURISTICS_WAIT_FOR_INITIAL_EXEC_RESULT_MAX_PFDS);
+		pfds[no_pfds].fd = instance->pipe_cmd_recv;
+		pfds[no_pfds].events = POLLIN;
+		pfds[no_pfds].revents = 0;
+		no_pfds++;
+
+		assert(no_pfds < QDEVICE_HEURISTICS_WAIT_FOR_INITIAL_EXEC_RESULT_MAX_PFDS);
+		pfds[no_pfds].fd = instance->qdevice_instance_ptr->votequorum_poll_fd;
+		pfds[no_pfds].events = POLLIN;
+		pfds[no_pfds].revents = 0;
+		no_pfds++;
+
+		if (!send_buffer_list_empty(&instance->cmd_out_buffer_list)) {
+			assert(no_pfds < QDEVICE_HEURISTICS_WAIT_FOR_INITIAL_EXEC_RESULT_MAX_PFDS);
+			pfds[no_pfds].fd = instance->pipe_cmd_send;
+			pfds[no_pfds].events = POLLOUT;
+			pfds[no_pfds].revents = 0;
+			no_pfds++;
+		}
+
+		/*
+		 * We know this is never larger than QDEVICE_DEFAULT_HEURISTICS_MAX_TIMEOUT * 2
+		 */
+		timeout = (int)instance->sync_timeout * 2;
+
+		poll_res = poll(pfds, no_pfds, timeout);
+		if (poll_res > 0) {
+			for (i = 0; i < no_pfds; i++) {
+				if (pfds[i].revents & POLLIN) {
+					case_processed = 0;
+					switch (i) {
+					case 0:
+						case_processed = 1;
+
+						res = qdevice_heuristics_log_read_from_pipe(instance);
+						if (res == -1) {
+							return (-1);
+						}
+						break;
+					case 1:
+						case_processed = 1;
+						res = qdevice_heuristics_cmd_read_from_pipe(instance);
+						if (res == -1) {
+							return (-1);
+						}
+						break;
+					case 2:
+						case_processed = 1;
+						res = qdevice_votequorum_dispatch(instance->qdevice_instance_ptr);
+						if (res == -1) {
+							return (-1);
+						}
+					case 3:
+						/*
+						 * Read on heuristics cmd send fs shouldn't happen
+						 */
+						 break;
+					}
+
+					if (!case_processed) {
+						qdevice_log(LOG_CRIT, "Unhandled read on poll descriptor %u", i);
+						exit(1);
+					}
+				}
+
+				if (pfds[i].revents & POLLOUT) {
+					case_processed = 0;
+					switch (i) {
+					case 0:
+					case 1:
+					case 2:
+						/*
+						 * Write on heuristics log, cmd recv or vq shouldn't happen
+						 */
+						break;
+					case 3:
+						case_processed = 1;
+						res = qdevice_heuristics_cmd_write(instance);
+						if (res == -1) {
+							return (-1);
+						}
+						break;
+					}
+
+					if (!case_processed) {
+						qdevice_log(LOG_CRIT, "Unhandled write on poll descriptor %u", i);
+						exit(1);
+					}
+				}
+
+				if ((pfds[i].revents & (POLLERR|POLLHUP|POLLNVAL)) &&
+				    !(pfds[i].revents & (POLLIN|POLLOUT))) {
+					switch (i) {
+					case 0:
+					case 1:
+					case 3:
+						/*
+						 *  Closed pipe doesn't mean return of POLLIN. To display
+						 * better log message, we call read log as if POLLIN would
+						 * be set.
+						 */
+						res = qdevice_heuristics_log_read_from_pipe(instance);
+						if (res == -1) {
+							return (-1);
+						}
+
+						qdevice_log(LOG_ERR, "POLLERR (%u) on heuristics pipe. Exiting");
+						return (-1);
+						break;
+					case 2:
+						qdevice_log(LOG_ERR, "POLLERR (%u) on corosync socket. Exiting");
+						return (-1);
+						break;
+					}
+				}
+			}
+		} else if (poll_res == 0) {
+			qdevice_log(LOG_ERR, "Timeout waiting for initial heuristics exec result");
+			return (-1);
+		} else {
+			qdevice_log_err(LOG_ERR, "Initial heuristics exec result poll failed");
+			return (-1);
+		}
+	}
+
+	return (0);
+}

qdevices/qdevice-heuristics.h

@@ -0,0 +1,70 @@
+/*
+ * Copyright (c) 2015-2017 Red Hat, Inc.
+ *
+ * All rights reserved.
+ *
+ * Author: Jan Friesse (jfriesse@redhat.com)
+ *
+ * This software licensed under BSD license, the text of which follows:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright notice,
+ *   this list of conditions and the following disclaimer.
+ * - Redistributions in binary form must reproduce the above copyright notice,
+ *   this list of conditions and the following disclaimer in the documentation
+ *   and/or other materials provided with the distribution.
+ * - Neither the name of the Red Hat, Inc. nor the names of its
+ *   contributors may be used to endorse or promote products derived from this
+ *   software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _QDEVICE_HEURISTICS_H_
+#define _QDEVICE_HEURISTICS_H_
+
+#include "dynar.h"
+#include "qdevice-advanced-settings.h"
+#include "send-buffer-list.h"
+#include "qdevice-heuristics-instance.h"
+#include "qdevice-heuristics-log.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+extern void		qdevice_heuristics_init(struct qdevice_heuristics_instance *instance,
+    struct qdevice_advanced_settings *advanced_settings);
+
+extern void		qdevice_heuristics_destroy(struct qdevice_heuristics_instance *instance);
+
+extern int		qdevice_heuristics_exec(struct qdevice_heuristics_instance *instance,
+    int sync_in_progress);
+
+extern int		qdevice_heuristics_waiting_for_result(
+    const struct qdevice_heuristics_instance *instance);
+
+extern int		qdevice_heuristics_change_exec_list(
+    struct qdevice_heuristics_instance *instance,
+    const struct qdevice_heuristics_exec_list *new_exec_list, int sync_in_progress);
+
+extern int		qdevice_heuristics_wait_for_initial_exec_result(
+    struct qdevice_heuristics_instance *instance);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _QDEVICE_HEURISTICS_H_ */

qdevices/qdevice-instance.c

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -32,9 +32,12 @@
  * THE POSSIBILITY OF SUCH DAMAGE.
  */
 
+#include "qdevice-config.h"
 #include "qdevice-instance.h"
+#include "qdevice-heuristics-exec-list.h"
 #include "qdevice-log.h"
 #include "qdevice-model.h"
+#include "utils.h"
 
 int
 qdevice_instance_init(struct qdevice_instance *instance,
@@ -60,6 +63,197 @@ qdevice_instance_destroy(struct qdevice_instance *instance)
 	return (0);
 }
 
+int
+qdevice_instance_configure_from_cmap_heuristics(struct qdevice_instance *instance)
+{
+	char *str;
+	long int li;
+	char *ep;
+	int i;
+	int res;
+	cs_error_t cs_err;
+	cmap_iter_handle_t iter_handle;
+	char key_name[CMAP_KEYNAME_MAXLEN + 1];
+	size_t value_len;
+	cmap_value_types_t type;
+	struct qdevice_heuristics_exec_list tmp_exec_list;
+	struct qdevice_heuristics_exec_list *exec_list;
+	char *command;
+	char exec_name[CMAP_KEYNAME_MAXLEN + 1];
+	char tmp_key[CMAP_KEYNAME_MAXLEN + 1];
+	size_t no_execs;
+	int send_exec_list;
+
+	instance->heuristics_instance.timeout = instance->heartbeat_interval / 2;
+	if (cmap_get_string(instance->cmap_handle,
+	    "quorum.device.heuristics.timeout", &str) == CS_OK) {
+		li = strtol(str, &ep, 10);
+		if (li < instance->advanced_settings->heuristics_min_timeout ||
+		    li > instance->advanced_settings->heuristics_max_timeout || *ep != '\0') {
+			qdevice_log(LOG_ERR, "heuristics.timeout must be valid number in "
+			    "range <%"PRIu32",%"PRIu32">",
+			    instance->advanced_settings->heuristics_min_timeout,
+			    instance->advanced_settings->heuristics_max_timeout);
+
+			free(str);
+			return (-1);
+		} else {
+			instance->heuristics_instance.timeout = li;
+		}
+
+		free(str);
+	}
+
+	instance->heuristics_instance.sync_timeout = instance->sync_heartbeat_interval / 2;
+	if (cmap_get_string(instance->cmap_handle,
+	    "quorum.device.heuristics.sync_timeout", &str) == CS_OK) {
+		li = strtol(str, &ep, 10);
+		if (li < instance->advanced_settings->heuristics_min_timeout ||
+		    li > instance->advanced_settings->heuristics_max_timeout || *ep != '\0') {
+			qdevice_log(LOG_ERR, "heuristics.sync_timeout must be valid number in "
+			    "range <%"PRIu32",%"PRIu32">",
+			    instance->advanced_settings->heuristics_min_timeout,
+			    instance->advanced_settings->heuristics_max_timeout);
+
+			free(str);
+			return (-1);
+		} else {
+			instance->heuristics_instance.sync_timeout = li;
+		}
+
+		free(str);
+	}
+
+	instance->heuristics_instance.interval = instance->heartbeat_interval * 3;
+	if (cmap_get_string(instance->cmap_handle,
+	    "quorum.device.heuristics.interval", &str) == CS_OK) {
+		li = strtol(str, &ep, 10);
+		if (li < instance->advanced_settings->heuristics_min_interval ||
+		    li > instance->advanced_settings->heuristics_max_interval || *ep != '\0') {
+			qdevice_log(LOG_ERR, "heuristics.interval must be valid number in "
+			    "range <%"PRIu32",%"PRIu32">",
+			    instance->advanced_settings->heuristics_min_interval,
+			    instance->advanced_settings->heuristics_max_interval);
+
+			free(str);
+			return (-1);
+		} else {
+			instance->heuristics_instance.interval = li;
+		}
+
+		free(str);
+	}
+
+	instance->heuristics_instance.mode = QDEVICE_DEFAULT_HEURISTICS_MODE;
+
+	if (cmap_get_string(instance->cmap_handle, "quorum.device.heuristics.mode", &str) == CS_OK) {
+		if ((i = utils_parse_bool_str(str)) == -1) {
+			if (strcasecmp(str, "sync") != 0) {
+				qdevice_log(LOG_ERR, "quorum.device.heuristics.mode value is not valid.");
+
+				free(str);
+				return (-1);
+			} else {
+				instance->heuristics_instance.mode = QDEVICE_HEURISTICS_MODE_SYNC;
+			}
+		} else {
+			if (i == 1) {
+				instance->heuristics_instance.mode = QDEVICE_HEURISTICS_MODE_ENABLED;
+			} else {
+				instance->heuristics_instance.mode = QDEVICE_HEURISTICS_MODE_DISABLED;
+			}
+		}
+
+		free(str);
+	}
+
+	send_exec_list = 0;
+	exec_list = NULL;
+	qdevice_heuristics_exec_list_init(&tmp_exec_list);
+
+	if (instance->heuristics_instance.mode == QDEVICE_HEURISTICS_MODE_DISABLED) {
+		exec_list = NULL;
+		send_exec_list = 1;
+	} else if (instance->heuristics_instance.mode == QDEVICE_HEURISTICS_MODE_ENABLED ||
+	    instance->heuristics_instance.mode == QDEVICE_HEURISTICS_MODE_SYNC) {
+		/*
+		 * Walk thru list of commands to exec
+		 */
+		cs_err = cmap_iter_init(instance->cmap_handle, "quorum.device.heuristics.exec_", &iter_handle);
+		if (cs_err != CS_OK) {
+			qdevice_log(LOG_ERR, "Can't iterate quorum.device.heuristics.exec_ keys. "
+			    "Error %s", cs_strerror(cs_err));
+
+			return (-1);
+		}
+
+		while ((cs_err = cmap_iter_next(instance->cmap_handle, iter_handle, key_name,
+		    &value_len, &type)) == CS_OK) {
+			if (type != CMAP_VALUETYPE_STRING) {
+				qdevice_log(LOG_WARNING, "%s key is not of string type. Ignoring");
+				continue ;
+			}
+
+			res = sscanf(key_name, "quorum.device.heuristics.exec_%[^.]%s", exec_name, tmp_key);
+			if (res != 1) {
+				qdevice_log(LOG_WARNING, "%s key is not correct heuristics exec name. Ignoring");
+				continue ;
+			}
+
+			cs_err = cmap_get_string(instance->cmap_handle, key_name, &command);
+			if (cs_err != CS_OK) {
+				qdevice_log(LOG_WARNING, "Can't get value of %s key. Ignoring");
+				continue ;
+			}
+
+			if (qdevice_heuristics_exec_list_add(&tmp_exec_list, exec_name, command) == NULL) {
+				qdevice_log(LOG_WARNING, "Can't store value of %s key into list. Ignoring");
+			}
+
+			free(command);
+		}
+
+		no_execs = qdevice_heuristics_exec_list_size(&tmp_exec_list);
+
+		if (no_execs == 0) {
+			qdevice_log(LOG_INFO, "No valid heuristics execs defined. Disabling heuristics.");
+			instance->heuristics_instance.mode = QDEVICE_HEURISTICS_MODE_DISABLED;
+			exec_list = NULL;
+			send_exec_list = 1;
+		} else if (no_execs > instance->advanced_settings->heuristics_max_execs) {
+			qdevice_log(LOG_ERR, "Too much (%zu) heuristics execs defined (max is %zu)."
+			    " Disabling heuristics.", no_execs,
+			    instance->advanced_settings->heuristics_max_execs);
+			instance->heuristics_instance.mode = QDEVICE_HEURISTICS_MODE_DISABLED;
+			exec_list = NULL;
+			send_exec_list = 1;
+		} else if (qdevice_heuristics_exec_list_eq(&tmp_exec_list,
+		    &instance->heuristics_instance.exec_list) == 1) {
+			qdevice_log(LOG_DEBUG, "Heuristics list is unchanged");
+			send_exec_list = 0;
+		} else {
+			qdevice_log(LOG_DEBUG, "Heuristics list changed");
+			exec_list = &tmp_exec_list;
+			send_exec_list = 1;
+		}
+
+	} else {
+		qdevice_log(LOG_CRIT, "Undefined heuristics mode");
+		exit(1);
+	}
+
+	if (send_exec_list) {
+		if (qdevice_heuristics_change_exec_list(&instance->heuristics_instance,
+		    exec_list, instance->sync_in_progress) != 0) {
+			return (-1);
+		}
+	}
+
+	qdevice_heuristics_exec_list_free(&tmp_exec_list);
+
+	return (0);
+}
+
 int
 qdevice_instance_configure_from_cmap(struct qdevice_instance *instance)
 {
@@ -95,5 +289,9 @@ qdevice_instance_configure_from_cmap(struct qdevice_instance *instance)
 		instance->sync_heartbeat_interval = VOTEQUORUM_QDEVICE_DEFAULT_SYNC_TIMEOUT;
 	}
 
+	if (qdevice_instance_configure_from_cmap_heuristics(instance) != 0) {
+		return (-1);
+	}
+
 	return (0);
 }

qdevices/qdevice-instance.h

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -44,6 +44,7 @@
 #include <votequorum.h>
 
 #include "qdevice-advanced-settings.h"
+#include "qdevice-heuristics.h"
 #include "qdevice-model-type.h"
 #include "node-list.h"
 #include "unix-socket-ipc.h"
@@ -59,6 +60,7 @@ struct qdevice_instance {
 	cmap_track_handle_t cmap_reload_track_handle;
 	cmap_track_handle_t cmap_nodelist_track_handle;
 	cmap_track_handle_t cmap_logging_track_handle;
+	cmap_track_handle_t cmap_heuristics_track_handle;
 
 	votequorum_handle_t votequorum_handle;
 	int votequorum_poll_fd;
@@ -84,13 +86,21 @@ struct qdevice_instance {
 	votequorum_node_t *vq_quorum_node_list;
 
 	/*
-	 * Copy of votequorum_nodelist_notify_fn callback paramters.
-	 * Set after model callback is called.
+	 * Copy of current votequorum_nodelist_notify_fn callback parameters.
+	 * Set after model callback qdevice_votequorum_node_list_notify_callback is called.
 	 */
-	uint8_t vq_node_list_ring_id_set;
+	uint8_t vq_node_list_initial_ring_id_set;
 	votequorum_ring_id_t vq_node_list_ring_id;
 	uint32_t vq_node_list_entries;
 	uint32_t *vq_node_list;
+	uint8_t vq_node_list_initial_heuristics_finished;
+	enum qdevice_heuristics_exec_result vq_node_list_heuristics_result;
+
+	/*
+	 * Copy of current votequorum_nodelist_notify_fn callback ring id
+	 * It's set before any callback is called and used for qdevice_votequorum_poll
+	 */
+	votequorum_ring_id_t vq_poll_ring_id;
 
 	/*
 	 * Copy of votequorum_expectedvotes_notify_fn callback parameters.
@@ -106,6 +116,8 @@ struct qdevice_instance {
 	const struct qdevice_advanced_settings *advanced_settings;
 
 	int sync_in_progress;
+
+	struct qdevice_heuristics_instance heuristics_instance;
 };
 
 extern int	qdevice_instance_init(struct qdevice_instance *instance,
@@ -115,6 +127,8 @@ extern int	qdevice_instance_destroy(struct qdevice_instance *instance);
 
 extern int	qdevice_instance_configure_from_cmap(struct qdevice_instance *instance);
 
+extern int	qdevice_instance_configure_from_cmap_heuristics(struct qdevice_instance *instance);
+
 #ifdef __cplusplus
 }
 #endif

qdevices/qdevice-ipc-cmd.c

@@ -243,6 +243,19 @@ qdevice_ipc_cmd_status_add_last_poll(struct qdevice_instance *instance, struct d
 	return (1);
 }
 
+static int
+qdevice_ipc_cmd_status_add_heuristics(struct qdevice_instance *instance, struct dynar *outbuf,
+    int verbose)
+{
+
+	if (!verbose) {
+		return (1);
+	}
+
+	return (dynar_str_catf(outbuf, "Heuristics:\t\t%s\n",
+	    qdevice_heuristics_mode_to_str(instance->heuristics_instance.mode)) != 0);
+}
+
 int
 qdevice_ipc_cmd_status(struct qdevice_instance *instance, struct dynar *outbuf, int verbose)
 {
@@ -252,6 +265,7 @@ qdevice_ipc_cmd_status(struct qdevice_instance *instance, struct dynar *outbuf,
 	    qdevice_ipc_cmd_status_add_nodeid(instance, outbuf, verbose) &&
 	    qdevice_ipc_cmd_status_add_intervals(instance, outbuf, verbose) &&
 	    qdevice_ipc_cmd_status_add_config_node_list(instance, outbuf, verbose) &&
+	    qdevice_ipc_cmd_status_add_heuristics(instance, outbuf, verbose) &&
 	    qdevice_ipc_cmd_status_add_membership_node_list(instance, outbuf, verbose) &&
 	    qdevice_ipc_cmd_status_add_quorum_node_list(instance, outbuf, verbose) &&
 	    qdevice_ipc_cmd_status_add_expected_votes(instance, outbuf, verbose) &&

qdevices/qdevice-model-net.c

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -41,6 +41,7 @@
 #include "qdevice-net-instance.h"
 #include "qdevice-net-ipc-cmd.h"
 #include "qdevice-net-algorithm.h"
+#include "qdevice-net-heuristics.h"
 #include "qdevice-net-poll.h"
 #include "qdevice-net-send.h"
 #include "qdevice-net-votequorum.h"
@@ -82,6 +83,11 @@ qdevice_model_net_init(struct qdevice_instance *instance)
 		return (-1);
 	}
 
+	if (qdevice_net_heuristics_init(net_instance) != 0) {
+		qdevice_log(LOG_ERR, "Can't initialize net heuristics");
+		return (-1);
+	}
+
 	return (0);
 }
 
@@ -208,6 +214,11 @@ qdevice_model_net_run(struct qdevice_instance *instance)
 
 		try_connect = qdevice_net_disconnect_reason_try_reconnect(net_instance->disconnect_reason);
 
+		/*
+		 * Unpause cast vote timer, because if it is paused we cannot remove tracking
+		 */
+		qdevice_net_cast_vote_timer_set_paused(net_instance, 0);
+
 		vote = TLV_VOTE_NO_CHANGE;
 
 		if (qdevice_net_algorithm_disconnected(net_instance,
@@ -381,7 +392,7 @@ qdevice_model_net_votequorum_quorum_notify(struct qdevice_instance *instance,
 
 	if (qdevice_net_algorithm_votequorum_quorum_notify(net_instance, quorate,
 	    node_list_entries, node_list, &send_node_list, &vote) != 0) {
-		qdevice_log(LOG_DEBUG, "Algorithm returned error. Disconnecting.");
+		qdevice_log(LOG_ERR, "Algorithm returned error. Disconnecting.");
 
 		net_instance->disconnect_reason = QDEVICE_NET_DISCONNECT_REASON_ALGO_VOTEQUORUM_QUORUM_NOTIFY_ERR;
 		net_instance->schedule_disconnect = 1;
@@ -419,17 +430,20 @@ qdevice_model_net_votequorum_quorum_notify(struct qdevice_instance *instance,
 }
 
 int
-qdevice_model_net_votequorum_node_list_notify(struct qdevice_instance *instance,
-    votequorum_ring_id_t votequorum_ring_id, uint32_t node_list_entries, uint32_t node_list[])
+qdevice_model_net_votequorum_node_list_heuristics_notify(struct qdevice_instance *instance,
+    votequorum_ring_id_t votequorum_ring_id, uint32_t node_list_entries, uint32_t node_list[],
+    enum qdevice_heuristics_exec_result heuristics_exec_result)
 {
 	struct qdevice_net_instance *net_instance;
 	struct tlv_ring_id tlv_rid;
 	enum tlv_vote vote;
+	enum tlv_heuristics heuristics;
 	int send_node_list;
 
 	net_instance = instance->model_data;
 
 	qdevice_net_votequorum_ring_id_to_tlv(&tlv_rid, &votequorum_ring_id);
+	heuristics = qdevice_net_heuristics_exec_result_to_tlv(heuristics_exec_result);
 
 	if (net_instance->state != QDEVICE_NET_INSTANCE_STATE_WAITING_VOTEQUORUM_CMAP_EVENTS) {
 		/*
@@ -447,22 +461,24 @@ qdevice_model_net_votequorum_node_list_notify(struct qdevice_instance *instance,
 		vote = TLV_VOTE_WAIT_FOR_REPLY;
 	}
 
-	if (qdevice_net_algorithm_votequorum_node_list_notify(net_instance, &tlv_rid,
-	    node_list_entries, node_list, &send_node_list, &vote) != 0) {
-		qdevice_log(LOG_DEBUG, "Algorithm returned error. Disconnecting.");
+	if (qdevice_net_algorithm_votequorum_node_list_heuristics_notify(net_instance, &tlv_rid,
+	    node_list_entries, node_list, &send_node_list, &vote, &heuristics) != 0) {
+		qdevice_log(LOG_ERR, "Algorithm returned error. Disconnecting.");
 
-		net_instance->disconnect_reason = QDEVICE_NET_DISCONNECT_REASON_ALGO_VOTEQUORUM_NODE_LIST_NOTIFY_ERR;
+		net_instance->disconnect_reason =
+		    QDEVICE_NET_DISCONNECT_REASON_ALGO_VOTEQUORUM_NODE_LIST_HEURISTICS_NOTIFY_ERR;
 		net_instance->schedule_disconnect = 1;
 
 		return (0);
 	} else {
-		qdevice_log(LOG_DEBUG, "Algorithm decided to %s list and result vote is %s",
-		    (send_node_list ? "send" : "not send"), tlv_vote_to_str(vote));
+		qdevice_log(LOG_DEBUG, "Algorithm decided to %s list, result vote is %s and heuristics is %s",
+		    (send_node_list ? "send" : "not send"), tlv_vote_to_str(vote),
+		    tlv_heuristics_to_str(heuristics));
 	}
 
 	if (send_node_list) {
 		if (qdevice_net_send_membership_node_list(net_instance, &tlv_rid,
-		    node_list_entries, node_list) != 0) {
+		    node_list_entries, node_list, heuristics) != 0) {
 			/*
 			 * Fatal error -> schedule disconnect
 			 */
@@ -473,6 +489,78 @@ qdevice_model_net_votequorum_node_list_notify(struct qdevice_instance *instance,
 		}
 	}
 
+	/*
+	 * Unpause cast vote timer
+	 */
+	qdevice_net_cast_vote_timer_set_paused(net_instance, 0);
+
+	if (qdevice_net_cast_vote_timer_update(net_instance, vote) != 0) {
+		qdevice_log(LOG_CRIT, "qdevice_model_net_votequorum_node_list_notify fatal error "
+		    "Can't update cast vote timer");
+		net_instance->disconnect_reason = QDEVICE_NET_DISCONNECT_REASON_CANT_SCHEDULE_VOTING_TIMER;
+		net_instance->schedule_disconnect = 1;
+
+		return (0);
+	}
+
+	net_instance->latest_vq_heuristics_result = heuristics;
+	net_instance->latest_heuristics_result = heuristics;
+
+	if (qdevice_net_heuristics_schedule_timer(net_instance) != 0) {
+		return (0);
+	}
+
+	return (0);
+}
+
+int
+qdevice_model_net_votequorum_node_list_notify(struct qdevice_instance *instance,
+    votequorum_ring_id_t votequorum_ring_id, uint32_t node_list_entries, uint32_t node_list[])
+{
+	struct qdevice_net_instance *net_instance;
+	struct tlv_ring_id tlv_rid;
+	enum tlv_vote vote;
+	int pause_cast_vote_timer;
+
+	net_instance = instance->model_data;
+
+	/*
+	 * Stop regular heuristics till qdevice_model_net_votequorum_node_list_heuristics_notify
+	 * is called
+	 */
+	if (qdevice_net_heuristics_stop_timer(net_instance) != 0) {
+		return (0);
+	}
+
+	pause_cast_vote_timer = 1;
+	vote = TLV_VOTE_NO_CHANGE;
+
+	if (net_instance->state != QDEVICE_NET_INSTANCE_STATE_WAITING_VOTEQUORUM_CMAP_EVENTS &&
+	    net_instance->cast_vote_timer_vote == TLV_VOTE_ACK) {
+		/*
+		 * Nodelist changed and vote timer still votes ACK. It's needed to start voting
+		 * NACK.
+		 */
+		if (net_instance->cast_vote_timer_vote == TLV_VOTE_ACK) {
+			vote = TLV_VOTE_NACK;
+		}
+	}
+
+	qdevice_net_votequorum_ring_id_to_tlv(&tlv_rid, &votequorum_ring_id);
+
+	if (qdevice_net_algorithm_votequorum_node_list_notify(net_instance, &tlv_rid,
+	    node_list_entries, node_list, &pause_cast_vote_timer, &vote) != 0) {
+		qdevice_log(LOG_ERR, "Algorithm returned error. Disconnecting.");
+
+		net_instance->disconnect_reason = QDEVICE_NET_DISCONNECT_REASON_ALGO_VOTEQUORUM_NODE_LIST_NOTIFY_ERR;
+		net_instance->schedule_disconnect = 1;
+
+		return (0);
+	} else {
+		qdevice_log(LOG_DEBUG, "Algorithm decided to %s cast vote timer and result vote is %s ",
+		    (pause_cast_vote_timer ? "pause" : "not pause"), tlv_vote_to_str(vote));
+	}
+
 	if (qdevice_net_cast_vote_timer_update(net_instance, vote) != 0) {
 		qdevice_log(LOG_CRIT, "qdevice_model_net_votequorum_node_list_notify fatal error "
 		    "Can't update cast vote timer");
@@ -482,6 +570,8 @@ qdevice_model_net_votequorum_node_list_notify(struct qdevice_instance *instance,
 		return (0);
 	}
 
+	qdevice_net_cast_vote_timer_set_paused(net_instance, pause_cast_vote_timer);
+
 	return (0);
 }
 
@@ -524,6 +614,41 @@ qdevice_model_net_votequorum_expected_votes_notify(struct qdevice_instance *inst
 	return (0);
 }
 
+int
+qdevice_model_net_cmap_changed(struct qdevice_instance *instance,
+    const struct qdevice_cmap_change_events *events)
+{
+	struct qdevice_net_instance *net_instance;
+	enum qdevice_heuristics_mode active_heuristics_mode;
+	int heuristics_enabled;
+
+	net_instance = instance->model_data;
+
+	if (events->heuristics) {
+		active_heuristics_mode = instance->heuristics_instance.mode;
+		heuristics_enabled = (active_heuristics_mode == QDEVICE_HEURISTICS_MODE_ENABLED ||
+		    active_heuristics_mode == QDEVICE_HEURISTICS_MODE_SYNC);
+
+		if (net_instance->state == QDEVICE_NET_INSTANCE_STATE_WAITING_VOTEQUORUM_CMAP_EVENTS &&
+		    !net_instance->server_supports_heuristics && heuristics_enabled) {
+			qdevice_log(LOG_ERR, "Heuristics are enabled but not supported by the server");
+
+			net_instance->disconnect_reason =
+			    QDEVICE_NET_DISCONNECT_REASON_SERVER_DOESNT_SUPPORT_REQUIRED_OPT;
+
+			net_instance->schedule_disconnect = 1;
+
+			return (0);
+		}
+
+		if (qdevice_net_heuristics_schedule_timer(net_instance) != 0) {
+			return (0);
+		}
+	}
+
+	return (0);
+}
+
 int
 qdevice_model_net_ipc_cmd_status(struct qdevice_instance *instance,
     struct dynar *outbuf, int verbose)
@@ -548,7 +673,9 @@ static struct qdevice_model qdevice_model_net = {
 	.config_node_list_changed		= qdevice_model_net_config_node_list_changed,
 	.votequorum_quorum_notify		= qdevice_model_net_votequorum_quorum_notify,
 	.votequorum_node_list_notify		= qdevice_model_net_votequorum_node_list_notify,
+	.votequorum_node_list_heuristics_notify	= qdevice_model_net_votequorum_node_list_heuristics_notify,
 	.votequorum_expected_votes_notify	= qdevice_model_net_votequorum_expected_votes_notify,
+	.cmap_changed				= qdevice_model_net_cmap_changed,
 	.ipc_cmd_status				= qdevice_model_net_ipc_cmd_status,
 };
 

qdevices/qdevice-model-net.h

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -59,9 +59,17 @@ extern int	qdevice_model_net_votequorum_quorum_notify(struct qdevice_instance *i
 extern int	qdevice_model_net_votequorum_node_list_notify(struct qdevice_instance *instance,
     votequorum_ring_id_t votequorum_ring_id, uint32_t node_list_entries, uint32_t node_list[]);
 
+extern int	qdevice_model_net_votequorum_node_list_heuristics_notify(
+    struct qdevice_instance *instance,
+    votequorum_ring_id_t votequorum_ring_id, uint32_t node_list_entries, uint32_t node_list[],
+    enum qdevice_heuristics_exec_result heuristics_exec_result);
+
 extern int 	qdevice_model_net_votequorum_expected_votes_notify(struct qdevice_instance *instance,
     uint32_t expected_votes);
 
+extern int      qdevice_model_net_cmap_changed(struct qdevice_instance *instance,
+    const struct qdevice_cmap_change_events *events);
+
 extern int	qdevice_model_net_ipc_cmd_status(struct qdevice_instance *instance,
     struct dynar *outbuf, int verbose);
 

qdevices/qdevice-model.c

@@ -135,6 +135,23 @@ qdevice_model_votequorum_node_list_notify(struct qdevice_instance *instance,
 	    votequorum_node_list_notify(instance, votequorum_ring_id, node_list_entries, node_list));
 }
 
+int
+qdevice_model_votequorum_node_list_heuristics_notify(struct qdevice_instance *instance,
+    votequorum_ring_id_t votequorum_ring_id, uint32_t node_list_entries, uint32_t node_list[],
+    enum qdevice_heuristics_exec_result heuristics_exec_result)
+{
+
+	if (instance->model_type >= QDEVICE_MODEL_TYPE_ARRAY_SIZE ||
+	    qdevice_model_array[instance->model_type] == NULL) {
+		qdevice_log(LOG_CRIT, "qdevice_model_votequorum_node_list_heuristics_notify unhandled model");
+		exit(1);
+	}
+
+	return (qdevice_model_array[instance->model_type]->
+	    votequorum_node_list_heuristics_notify(instance, votequorum_ring_id, node_list_entries,
+		node_list, heuristics_exec_result));
+}
+
 int
 qdevice_model_votequorum_expected_votes_notify(struct qdevice_instance *instance,
     uint32_t expected_votes)
@@ -164,6 +181,21 @@ qdevice_model_ipc_cmd_status(struct qdevice_instance *instance, struct dynar *ou
 	    ipc_cmd_status(instance, outbuf, verbose));
 }
 
+int
+qdevice_model_cmap_changed(struct qdevice_instance *instance,
+    const struct qdevice_cmap_change_events *events)
+{
+
+	if (instance->model_type >= QDEVICE_MODEL_TYPE_ARRAY_SIZE ||
+	    qdevice_model_array[instance->model_type] == NULL) {
+		qdevice_log(LOG_CRIT, "qdevice_model_cmap_chaged unhandled model");
+		exit(1);
+	}
+
+	return (qdevice_model_array[instance->model_type]->
+	    cmap_changed(instance, events));
+}
+
 int
 qdevice_model_register(enum qdevice_model_type model_type,
     struct qdevice_model *model)

qdevices/qdevice-model.h

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -35,8 +35,10 @@
 #ifndef _QDEVICE_MODEL_H_
 #define _QDEVICE_MODEL_H_
 
+#include "qdevice-cmap.h"
 #include "qdevice-instance.h"
 #include "qdevice-model-type.h"
+#include "qdevice-heuristics-exec-result.h"
 
 #ifdef __cplusplus
 extern "C" {
@@ -59,12 +61,19 @@ extern int	qdevice_model_votequorum_quorum_notify(struct qdevice_instance *insta
 extern int	qdevice_model_votequorum_node_list_notify(struct qdevice_instance *instance,
     votequorum_ring_id_t votequorum_ring_id, uint32_t node_list_entries, uint32_t node_list[]);
 
+extern int	qdevice_model_votequorum_node_list_heuristics_notify(struct qdevice_instance *instance,
+    votequorum_ring_id_t votequorum_ring_id, uint32_t node_list_entries, uint32_t node_list[],
+    enum qdevice_heuristics_exec_result heuristics_exec_result);
+
 extern int 	qdevice_model_votequorum_expected_votes_notify(struct qdevice_instance *instance,
     uint32_t expected_votes);
 
 extern int	qdevice_model_ipc_cmd_status(struct qdevice_instance *instance,
     struct dynar *outbuf, int verbose);
 
+extern int	qdevice_model_cmap_changed(struct qdevice_instance *instance,
+    const struct qdevice_cmap_change_events *events);
+
 struct qdevice_model {
 	const char *name;
 	int (*init)(struct qdevice_instance *instance);
@@ -78,9 +87,14 @@ struct qdevice_model {
 	int (*votequorum_node_list_notify)(struct qdevice_instance *instance,
 	    votequorum_ring_id_t votequorum_ring_id,uint32_t node_list_entries,
 	    uint32_t node_list[]);
+	int (*votequorum_node_list_heuristics_notify)(struct qdevice_instance *instance,
+	    votequorum_ring_id_t votequorum_ring_id,uint32_t node_list_entries,
+	    uint32_t node_list[], enum qdevice_heuristics_exec_result heuristics_exec_result);
 	int (*votequorum_expected_votes_notify)(struct qdevice_instance *instance,
 	    uint32_t expected_votes);
 	int (*ipc_cmd_status)(struct qdevice_instance *instance, struct dynar *outbuf, int verbose);
+	int (*cmap_changed)(struct qdevice_instance *instance,
+	    const struct qdevice_cmap_change_events *events);
 };
 
 extern int		 qdevice_model_register(

qdevices/qdevice-net-algo-2nodelms.c

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -45,13 +45,12 @@ int
 qdevice_net_algo_2nodelms_init(struct qdevice_net_instance *instance)
 {
 
-
 	return (0);
 }
 
 int
-qdevice_net_algo_2nodelms_connected(struct qdevice_net_instance *instance, int *send_config_node_list,
-    int *send_membership_node_list, int *send_quorum_node_list, enum tlv_vote *vote)
+qdevice_net_algo_2nodelms_connected(struct qdevice_net_instance *instance, enum tlv_heuristics *heuristics,
+    int *send_config_node_list, int *send_membership_node_list, int *send_quorum_node_list, enum tlv_vote *vote)
 {
 
 	return (0);
@@ -69,7 +68,16 @@ qdevice_net_algo_2nodelms_config_node_list_changed(struct qdevice_net_instance *
 int
 qdevice_net_algo_2nodelms_votequorum_node_list_notify(struct qdevice_net_instance *instance,
     const struct tlv_ring_id *ring_id, uint32_t node_list_entries, uint32_t node_list[],
-    int *send_node_list, enum tlv_vote *vote)
+    int *pause_cast_vote_timer, enum tlv_vote *vote)
+{
+
+	return (0);
+}
+
+int
+qdevice_net_algo_2nodelms_votequorum_node_list_heuristics_notify(struct qdevice_net_instance *instance,
+    const struct tlv_ring_id *ring_id, uint32_t node_list_entries, uint32_t node_list[],
+    int *send_node_list, enum tlv_vote *vote, enum tlv_heuristics *heuristics)
 {
 
 	return (0);
@@ -168,6 +176,27 @@ qdevice_net_algo_2nodelms_echo_reply_not_received(struct qdevice_net_instance *i
 	return (-1);
 }
 
+int
+qdevice_net_algo_2nodelms_heuristics_change(struct qdevice_net_instance *instance,
+    enum tlv_heuristics *heuristics, int *send_msg, enum tlv_vote *vote)
+{
+
+	return (0);
+}
+
+int
+qdevice_net_algo_2nodelms_heuristics_change_reply_received(struct qdevice_net_instance *instance,
+    uint32_t seq_number, const struct tlv_ring_id *ring_id, int ring_id_is_valid,
+    enum tlv_heuristics heuristics, enum tlv_vote *vote)
+{
+
+	if (!ring_id_is_valid) {
+		*vote = TLV_VOTE_NO_CHANGE;
+	}
+
+	return (0);
+}
+
 int
 qdevice_net_algo_2nodelms_disconnected(struct qdevice_net_instance *instance,
     enum qdevice_net_disconnect_reason disconnect_reason, int *try_reconnect, enum tlv_vote *vote)
@@ -187,6 +216,7 @@ static struct qdevice_net_algorithm qdevice_net_algo_2nodelms = {
 	.connected				= qdevice_net_algo_2nodelms_connected,
 	.config_node_list_changed		= qdevice_net_algo_2nodelms_config_node_list_changed,
 	.votequorum_node_list_notify		= qdevice_net_algo_2nodelms_votequorum_node_list_notify,
+	.votequorum_node_list_heuristics_notify	= qdevice_net_algo_2nodelms_votequorum_node_list_heuristics_notify,
 	.votequorum_quorum_notify		= qdevice_net_algo_2nodelms_votequorum_quorum_notify,
 	.votequorum_expected_votes_notify	= qdevice_net_algo_2nodelms_votequorum_expected_votes_notify,
 	.config_node_list_reply_received	= qdevice_net_algo_2nodelms_config_node_list_reply_received,
@@ -196,6 +226,8 @@ static struct qdevice_net_algorithm qdevice_net_algo_2nodelms = {
 	.vote_info_received			= qdevice_net_algo_2nodelms_vote_info_received,
 	.echo_reply_received			= qdevice_net_algo_2nodelms_echo_reply_received,
 	.echo_reply_not_received		= qdevice_net_algo_2nodelms_echo_reply_not_received,
+	.heuristics_change			= qdevice_net_algo_2nodelms_heuristics_change,
+	.heuristics_change_reply_received	= qdevice_net_algo_2nodelms_heuristics_change_reply_received,
 	.disconnected				= qdevice_net_algo_2nodelms_disconnected,
 	.destroy				= qdevice_net_algo_2nodelms_destroy,
 };

qdevices/qdevice-net-algo-2nodelms.h

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -44,8 +44,8 @@ extern "C" {
 extern int	qdevice_net_algo_2nodelms_init(struct qdevice_net_instance *instance);
 
 extern int	qdevice_net_algo_2nodelms_connected(struct qdevice_net_instance *instance,
-    int *send_config_node_list, int *send_membership_node_list, int *send_quorum_node_list,
-    enum tlv_vote *vote);
+    enum tlv_heuristics *heuristics, int *send_config_node_list, int *send_membership_node_list,
+    int *send_quorum_node_list, enum tlv_vote *vote);
 
 extern int	qdevice_net_algo_2nodelms_config_node_list_changed(
     struct qdevice_net_instance *instance, const struct node_list *nlist,
@@ -53,7 +53,13 @@ extern int	qdevice_net_algo_2nodelms_config_node_list_changed(
 
 extern int	qdevice_net_algo_2nodelms_votequorum_node_list_notify(
     struct qdevice_net_instance *instance, const struct tlv_ring_id *ring_id,
-    uint32_t node_list_entries, uint32_t node_list[], int *send_node_list, enum tlv_vote *vote);
+    uint32_t node_list_entries, uint32_t node_list[], int *pause_cast_vote_timer,
+    enum tlv_vote *vote);
+
+extern int	qdevice_net_algo_2nodelms_votequorum_node_list_heuristics_notify(
+    struct qdevice_net_instance *instance, const struct tlv_ring_id *ring_id,
+    uint32_t node_list_entries, uint32_t node_list[], int *send_node_list, enum tlv_vote *vote,
+    enum tlv_heuristics *heuristics);
 
 extern int	qdevice_net_algo_2nodelms_votequorum_quorum_notify(
     struct qdevice_net_instance *instance, uint32_t quorate, uint32_t node_list_entries,
@@ -88,6 +94,13 @@ extern int	qdevice_net_algo_2nodelms_echo_reply_received(
 extern int	qdevice_net_algo_2nodelms_echo_reply_not_received(
     struct qdevice_net_instance *instance);
 
+extern int	qdevice_net_algo_2nodelms_heuristics_change(struct qdevice_net_instance *instance,
+    enum tlv_heuristics *heuristics, int *send_msg, enum tlv_vote *vote);
+
+extern int	qdevice_net_algo_2nodelms_heuristics_change_reply_received(struct qdevice_net_instance *instance,
+    uint32_t seq_number, const struct tlv_ring_id *ring_id, int ring_id_is_valid, enum tlv_heuristics heuristics,
+    enum tlv_vote *vote);
+
 extern int	qdevice_net_algo_2nodelms_disconnected(struct qdevice_net_instance *instance,
     enum qdevice_net_disconnect_reason disconnect_reason, int *try_reconnect, enum tlv_vote *vote);
 

qdevices/qdevice-net-algo-ffsplit.c

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -120,8 +120,8 @@ qdevice_net_algo_ffsplit_init(struct qdevice_net_instance *instance)
 }
 
 int
-qdevice_net_algo_ffsplit_connected(struct qdevice_net_instance *instance, int *send_config_node_list,
-    int *send_membership_node_list, int *send_quorum_node_list, enum tlv_vote *vote)
+qdevice_net_algo_ffsplit_connected(struct qdevice_net_instance *instance, enum tlv_heuristics *heuristics,
+    int *send_config_node_list, int *send_membership_node_list, int *send_quorum_node_list, enum tlv_vote *vote)
 {
 
 	return (0);
@@ -143,7 +143,16 @@ qdevice_net_algo_ffsplit_config_node_list_changed(struct qdevice_net_instance *i
 int
 qdevice_net_algo_ffsplit_votequorum_node_list_notify(struct qdevice_net_instance *instance,
     const struct tlv_ring_id *ring_id, uint32_t node_list_entries, uint32_t node_list[],
-    int *send_node_list, enum tlv_vote *vote)
+    int *pause_cast_vote_timer, enum tlv_vote *vote)
+{
+
+	return (0);
+}
+
+int
+qdevice_net_algo_ffsplit_votequorum_node_list_heuristics_notify(struct qdevice_net_instance *instance,
+    const struct tlv_ring_id *ring_id, uint32_t node_list_entries, uint32_t node_list[],
+    int *send_node_list, enum tlv_vote *vote, enum tlv_heuristics *heuristics)
 {
 
 	return (0);
@@ -246,6 +255,27 @@ qdevice_net_algo_ffsplit_echo_reply_not_received(struct qdevice_net_instance *in
 	return (-1);
 }
 
+int
+qdevice_net_algo_ffsplit_heuristics_change(struct qdevice_net_instance *instance,
+    enum tlv_heuristics *heuristics, int *send_msg, enum tlv_vote *vote)
+{
+
+	return (0);
+}
+
+int
+qdevice_net_algo_ffsplit_heuristics_change_reply_received(struct qdevice_net_instance *instance,
+    uint32_t seq_number, const struct tlv_ring_id *ring_id, int ring_id_is_valid,
+    enum tlv_heuristics heuristics, enum tlv_vote *vote)
+{
+
+	if (!ring_id_is_valid) {
+		*vote = TLV_VOTE_NO_CHANGE;
+	}
+
+	return (0);
+}
+
 int
 qdevice_net_algo_ffsplit_disconnected(struct qdevice_net_instance *instance,
     enum qdevice_net_disconnect_reason disconnect_reason, int *try_reconnect, enum tlv_vote *vote)
@@ -272,6 +302,7 @@ static struct qdevice_net_algorithm qdevice_net_algo_ffsplit = {
 	.connected				= qdevice_net_algo_ffsplit_connected,
 	.config_node_list_changed		= qdevice_net_algo_ffsplit_config_node_list_changed,
 	.votequorum_node_list_notify		= qdevice_net_algo_ffsplit_votequorum_node_list_notify,
+	.votequorum_node_list_heuristics_notify	= qdevice_net_algo_ffsplit_votequorum_node_list_heuristics_notify,
 	.votequorum_quorum_notify		= qdevice_net_algo_ffsplit_votequorum_quorum_notify,
 	.votequorum_expected_votes_notify	= qdevice_net_algo_ffsplit_votequorum_expected_votes_notify,
 	.config_node_list_reply_received	= qdevice_net_algo_ffsplit_config_node_list_reply_received,
@@ -281,6 +312,8 @@ static struct qdevice_net_algorithm qdevice_net_algo_ffsplit = {
 	.vote_info_received			= qdevice_net_algo_ffsplit_vote_info_received,
 	.echo_reply_received			= qdevice_net_algo_ffsplit_echo_reply_received,
 	.echo_reply_not_received		= qdevice_net_algo_ffsplit_echo_reply_not_received,
+	.heuristics_change			= qdevice_net_algo_ffsplit_heuristics_change,
+	.heuristics_change_reply_received	= qdevice_net_algo_ffsplit_heuristics_change_reply_received,
 	.disconnected				= qdevice_net_algo_ffsplit_disconnected,
 	.destroy				= qdevice_net_algo_ffsplit_destroy,
 };

qdevices/qdevice-net-algo-ffsplit.h

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -44,8 +44,8 @@ extern "C" {
 extern int	qdevice_net_algo_ffsplit_init(struct qdevice_net_instance *instance);
 
 extern int	qdevice_net_algo_ffsplit_connected(struct qdevice_net_instance *instance,
-    int *send_config_node_list, int *send_membership_node_list, int *send_quorum_node_list,
-    enum tlv_vote *vote);
+    enum tlv_heuristics *heuristics, int *send_config_node_list, int *send_membership_node_list,
+    int *send_quorum_node_list, enum tlv_vote *vote);
 
 extern int	qdevice_net_algo_ffsplit_config_node_list_changed(
     struct qdevice_net_instance *instance, const struct node_list *nlist,
@@ -53,7 +53,13 @@ extern int	qdevice_net_algo_ffsplit_config_node_list_changed(
 
 extern int	qdevice_net_algo_ffsplit_votequorum_node_list_notify(
     struct qdevice_net_instance *instance, const struct tlv_ring_id *ring_id,
-    uint32_t node_list_entries, uint32_t node_list[], int *send_node_list, enum tlv_vote *vote);
+    uint32_t node_list_entries, uint32_t node_list[], int *pause_cast_vote_timer,
+    enum tlv_vote *vote);
+
+extern int	qdevice_net_algo_ffsplit_votequorum_node_list_heuristics_notify(
+    struct qdevice_net_instance *instance, const struct tlv_ring_id *ring_id,
+    uint32_t node_list_entries, uint32_t node_list[], int *send_node_list, enum tlv_vote *vote,
+    enum tlv_heuristics *heuristics);
 
 extern int	qdevice_net_algo_ffsplit_votequorum_quorum_notify(
     struct qdevice_net_instance *instance, uint32_t quorate, uint32_t node_list_entries,
@@ -88,6 +94,13 @@ extern int	qdevice_net_algo_ffsplit_echo_reply_received(
 extern int	qdevice_net_algo_ffsplit_echo_reply_not_received(
     struct qdevice_net_instance *instance);
 
+extern int	qdevice_net_algo_ffsplit_heuristics_change(struct qdevice_net_instance *instance,
+    enum tlv_heuristics *heuristics, int *send_msg, enum tlv_vote *vote);
+
+extern int	qdevice_net_algo_ffsplit_heuristics_change_reply_received(struct qdevice_net_instance *instance,
+    uint32_t seq_number, const struct tlv_ring_id *ring_id, int ring_id_is_valid, enum tlv_heuristics heuristics,
+    enum tlv_vote *vote);
+
 extern int	qdevice_net_algo_ffsplit_disconnected(struct qdevice_net_instance *instance,
     enum qdevice_net_disconnect_reason disconnect_reason, int *try_reconnect, enum tlv_vote *vote);
 

qdevices/qdevice-net-algo-lms.c

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -74,9 +74,10 @@ qdevice_net_algo_lms_init(struct qdevice_net_instance *instance)
 
 int
 qdevice_net_algo_lms_connected(struct qdevice_net_instance *instance,
-    int *send_config_node_list, int *send_membership_node_list, int *send_quorum_node_list,
-    enum tlv_vote *vote)
+    enum tlv_heuristics *heuristics, int *send_config_node_list, int *send_membership_node_list,
+    int *send_quorum_node_list, enum tlv_vote *vote)
 {
+
 	return (0);
 }
 
@@ -92,8 +93,18 @@ qdevice_net_algo_lms_config_node_list_changed(struct qdevice_net_instance *insta
 int
 qdevice_net_algo_lms_votequorum_node_list_notify(struct qdevice_net_instance *instance,
     const struct tlv_ring_id *ring_id, uint32_t node_list_entries, uint32_t node_list[],
-    int *send_node_list, enum tlv_vote *vote)
+    int *pause_cast_vote_timer, enum tlv_vote *vote)
+{
+
+	return (0);
+}
+
+int
+qdevice_net_algo_lms_votequorum_node_list_heuristics_notify(struct qdevice_net_instance *instance,
+    const struct tlv_ring_id *ring_id, uint32_t node_list_entries, uint32_t node_list[],
+    int *send_node_list, enum tlv_vote *vote, enum tlv_heuristics *heuristics)
 {
+
 	return (0);
 }
 
@@ -209,6 +220,27 @@ qdevice_net_algo_lms_echo_reply_not_received(struct qdevice_net_instance *instan
 	return (-1);
 }
 
+int
+qdevice_net_algo_lms_heuristics_change(struct qdevice_net_instance *instance,
+    enum tlv_heuristics *heuristics, int *send_msg, enum tlv_vote *vote)
+{
+
+	return (0);
+}
+
+int
+qdevice_net_algo_lms_heuristics_change_reply_received(struct qdevice_net_instance *instance,
+    uint32_t seq_number, const struct tlv_ring_id *ring_id, int ring_id_is_valid,
+    enum tlv_heuristics heuristics, enum tlv_vote *vote)
+{
+
+	if (!ring_id_is_valid) {
+		*vote = TLV_VOTE_NO_CHANGE;
+	}
+
+	return (0);
+}
+
 int
 qdevice_net_algo_lms_disconnected(struct qdevice_net_instance *instance,
     enum qdevice_net_disconnect_reason disconnect_reason, int *try_reconnect, enum tlv_vote *vote)
@@ -236,6 +268,7 @@ static struct qdevice_net_algorithm qdevice_net_algo_lms = {
 	.connected				= qdevice_net_algo_lms_connected,
 	.config_node_list_changed		= qdevice_net_algo_lms_config_node_list_changed,
 	.votequorum_node_list_notify		= qdevice_net_algo_lms_votequorum_node_list_notify,
+	.votequorum_node_list_heuristics_notify	= qdevice_net_algo_lms_votequorum_node_list_heuristics_notify,
 	.votequorum_quorum_notify		= qdevice_net_algo_lms_votequorum_quorum_notify,
 	.votequorum_expected_votes_notify	= qdevice_net_algo_lms_votequorum_expected_votes_notify,
 	.config_node_list_reply_received	= qdevice_net_algo_lms_config_node_list_reply_received,
@@ -245,6 +278,8 @@ static struct qdevice_net_algorithm qdevice_net_algo_lms = {
 	.vote_info_received			= qdevice_net_algo_lms_vote_info_received,
 	.echo_reply_received			= qdevice_net_algo_lms_echo_reply_received,
 	.echo_reply_not_received		= qdevice_net_algo_lms_echo_reply_not_received,
+	.heuristics_change			= qdevice_net_algo_lms_heuristics_change,
+	.heuristics_change_reply_received	= qdevice_net_algo_lms_heuristics_change_reply_received,
 	.disconnected				= qdevice_net_algo_lms_disconnected,
 	.destroy				= qdevice_net_algo_lms_destroy,
 };

qdevices/qdevice-net-algo-lms.h

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -44,8 +44,8 @@ extern "C" {
 extern int	qdevice_net_algo_lms_init(struct qdevice_net_instance *instance);
 
 extern int	qdevice_net_algo_lms_connected(struct qdevice_net_instance *instance,
-    int *send_config_node_list, int *send_membership_node_list, int *send_quorum_node_list,
-    enum tlv_vote *vote);
+    enum tlv_heuristics *heuristics, int *send_config_node_list, int *send_membership_node_list,
+    int *send_quorum_node_list, enum tlv_vote *vote);
 
 extern int	qdevice_net_algo_lms_config_node_list_changed(
     struct qdevice_net_instance *instance, const struct node_list *nlist,
@@ -53,7 +53,13 @@ extern int	qdevice_net_algo_lms_config_node_list_changed(
 
 extern int	qdevice_net_algo_lms_votequorum_node_list_notify(
     struct qdevice_net_instance *instance, const struct tlv_ring_id *ring_id,
-    uint32_t node_list_entries, uint32_t node_list[], int *send_node_list, enum tlv_vote *vote);
+    uint32_t node_list_entries, uint32_t node_list[], int *pause_cast_vote_timer,
+    enum tlv_vote *vote);
+
+extern int	qdevice_net_algo_lms_votequorum_node_list_heuristics_notify(
+    struct qdevice_net_instance *instance, const struct tlv_ring_id *ring_id,
+    uint32_t node_list_entries, uint32_t node_list[], int *send_node_list, enum tlv_vote *vote,
+    enum tlv_heuristics *heuristics);
 
 extern int	qdevice_net_algo_lms_votequorum_quorum_notify(
     struct qdevice_net_instance *instance, uint32_t quorate, uint32_t node_list_entries,
@@ -88,6 +94,13 @@ extern int	qdevice_net_algo_lms_echo_reply_received(
 extern int	qdevice_net_algo_lms_echo_reply_not_received(
     struct qdevice_net_instance *instance);
 
+extern int	qdevice_net_algo_lms_heuristics_change(struct qdevice_net_instance *instance,
+    enum tlv_heuristics *heuristics, int *send_msg, enum tlv_vote *vote);
+
+extern int	qdevice_net_algo_lms_heuristics_change_reply_received(struct qdevice_net_instance *instance,
+    uint32_t seq_number, const struct tlv_ring_id *ring_id, int ring_id_is_valid, enum tlv_heuristics heuristics,
+    enum tlv_vote *vote);
+
 extern int	qdevice_net_algo_lms_disconnected(struct qdevice_net_instance *instance,
     enum qdevice_net_disconnect_reason disconnect_reason, int *try_reconnect, enum tlv_vote *vote);
 

qdevices/qdevice-net-algo-test.c

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -62,12 +62,13 @@ qdevice_net_algo_test_init(struct qdevice_net_instance *instance)
  * send_config_node_list, send_membership_node_list and send_quorum_node_list can be set to
  * nonzero (default) to make qdevice-net send given lists to qnetd
  * vote (default TLV_VOTE_WAIT_FOR_REPLY) can be set to update voting timer
+ * heuristics is result of pre connect heuristics and can be updated
  *
  * Callback should return 0 on success or -1 on failure (-> disconnect client).
  */
 int
-qdevice_net_algo_test_connected(struct qdevice_net_instance *instance, int *send_config_node_list,
-    int *send_membership_node_list, int *send_quorum_node_list, enum tlv_vote *vote)
+qdevice_net_algo_test_connected(struct qdevice_net_instance *instance, enum tlv_heuristics *heuristics,
+    int *send_config_node_list, int *send_membership_node_list, int *send_quorum_node_list, enum tlv_vote *vote)
 {
 
 	qdevice_log(LOG_INFO, "algo-test: Connected");
@@ -101,7 +102,35 @@ qdevice_net_algo_test_config_node_list_changed(struct qdevice_net_instance *inst
 }
 
 /*
- * Called after votequorum node list notify is dispatched.
+ * Called after votequorum node list notify is dispatched but heuristics result is not
+ * yet known (heuristics were just executed). Full list together with result of heuristics
+ * are received in qdevice_net_algo_test_votequorum_node_list_heuristics_notify.
+ * This applies also if heuristics are disabled.
+ *
+ * pause_cast_vote_timer is preset to 1. If after callback this value is still != 0,
+ * cast vote timer is paused until heuristics finishes.
+ *
+ * vote is by default set to TLV_VOTE_NO_CHANGE with exception of situation when
+ *   net_instance->state != QDEVICE_NET_INSTANCE_STATE_WAITING_VOTEQUORUM_CMAP_EVENTS and
+ *   cast_vote_timer_vote = TLV_VOTE_ACK
+ * then vote is set to TLV_VOTE_NACK. This is to allow qnetd disconnection and still vote
+ * ACK as long as no change happens.
+ *
+ * Callback should return 0 on success or -1 on failure (-> disconnect client).
+ */
+int
+qdevice_net_algo_test_votequorum_node_list_notify(struct qdevice_net_instance *instance,
+    const struct tlv_ring_id *ring_id, uint32_t node_list_entries, uint32_t node_list[],
+    int *pause_cast_vote_timer, enum tlv_vote *vote)
+{
+
+	qdevice_log(LOG_INFO, "algo-test: Votequorum list notify");
+
+	return (0);
+}
+
+/*
+ * Called after votequorum node list notify is dispatched and heuristics result is known.
  *
  * Callback should return 0 on success or -1 on failure (-> disconnect client).
  *
@@ -113,14 +142,15 @@ qdevice_net_algo_test_config_node_list_changed(struct qdevice_net_instance *inst
  *     vote = TLV_VOTE_NACK.
  * Otherwise send_node_list = 1 and vote = TLV_VOTE_WAIT_FOR_REPLY
  * If send_node_list is set to non zero, node list is sent to qnetd
+ * heuristics results can be used (and overwrite)
  */
 int
-qdevice_net_algo_test_votequorum_node_list_notify(struct qdevice_net_instance *instance,
+qdevice_net_algo_test_votequorum_node_list_heuristics_notify(struct qdevice_net_instance *instance,
     const struct tlv_ring_id *ring_id, uint32_t node_list_entries, uint32_t node_list[],
-    int *send_node_list, enum tlv_vote *vote)
+    int *send_node_list, enum tlv_vote *vote, enum tlv_heuristics *heuristics)
 {
 
-	qdevice_log(LOG_INFO, "algo-test: Votequorum list notify");
+	qdevice_log(LOG_INFO, "algo-test: Votequorum heuristics list notify");
 
 	return (0);
 }
@@ -172,7 +202,8 @@ qdevice_net_algo_test_votequorum_expected_votes_notify(struct qdevice_net_instan
 /*
  * Called when config node list reply is received. Vote is set to value returned by server (and can
  * be overwriten by algorithm). ring_id is returned by server. ring_id_is_valid is set to 1 only if
- * received ring id matches last sent ring id. It usually makes sense to not update timer.
+ * received ring id matches last sent ring id. If it is 0, then it usually makes sense to not
+ * update timer.
  *
  * Callback should return 0 on success or -1 on failure (-> disconnect client).
  */
@@ -321,6 +352,54 @@ qdevice_net_algo_test_echo_reply_not_received(struct qdevice_net_instance *insta
 	return (-1);
 }
 
+/*
+ * Called when regular heuristics finished and it's result differs from previous heuristics.
+ *
+ * heuristics contains result of heuristics
+ * send_msg distinquish if message should be sent to qnetd. Default value is 0 if qnetd is not
+ *   connected and 1 otherwise
+ * vote can be set to change cast_vote_timer voting value (default is TLV_VOTE_NO_CHANGE)
+ *
+ * It's not possible to set send_msg to 1 and heuristics to TLV_HEURISTICS_UNDEFINED
+ *
+ * Callback should return 0 on success, -1 on failure (-> force exit)
+ */
+int
+qdevice_net_algo_test_heuristics_change(struct qdevice_net_instance *instance,
+    enum tlv_heuristics *heuristics, int *send_msg, enum tlv_vote *vote)
+{
+
+	qdevice_log(LOG_INFO, "algo-test: Heuristics change");
+
+	return (0);
+}
+
+/*
+ * Called when reply for heuristics change was received.
+ * Vote is set to value returned by server (and can be overwriten by algorithm).
+ *
+ * ring_id and ring_id_is_valid have same meaning as for
+ * qdevice_net_algo_test_config_node_list_reply_received
+ *
+ * heuristics is unmodified value sent to server (and set by qdevice_net_algo_test_heuristics_change)
+ *
+ * Callback should return 0 on success or -1 on failure (-> disconnect client).
+ */
+int
+qdevice_net_algo_test_heuristics_change_reply_received(struct qdevice_net_instance *instance,
+    uint32_t seq_number, const struct tlv_ring_id *ring_id, int ring_id_is_valid,
+    enum tlv_heuristics heuristics, enum tlv_vote *vote)
+{
+
+	qdevice_log(LOG_INFO, "algo-test: heuristics change reply received");
+
+	if (!ring_id_is_valid) {
+		*vote = TLV_VOTE_NO_CHANGE;
+	}
+
+	return (0);
+}
+
 /*
  * Called when client disconnects from server.
  *
@@ -355,6 +434,7 @@ static struct qdevice_net_algorithm qdevice_net_algo_test = {
 	.connected				= qdevice_net_algo_test_connected,
 	.config_node_list_changed		= qdevice_net_algo_test_config_node_list_changed,
 	.votequorum_node_list_notify		= qdevice_net_algo_test_votequorum_node_list_notify,
+	.votequorum_node_list_heuristics_notify	= qdevice_net_algo_test_votequorum_node_list_heuristics_notify,
 	.votequorum_quorum_notify		= qdevice_net_algo_test_votequorum_quorum_notify,
 	.votequorum_expected_votes_notify	= qdevice_net_algo_test_votequorum_expected_votes_notify,
 	.config_node_list_reply_received	= qdevice_net_algo_test_config_node_list_reply_received,
@@ -364,6 +444,8 @@ static struct qdevice_net_algorithm qdevice_net_algo_test = {
 	.vote_info_received			= qdevice_net_algo_test_vote_info_received,
 	.echo_reply_received			= qdevice_net_algo_test_echo_reply_received,
 	.echo_reply_not_received		= qdevice_net_algo_test_echo_reply_not_received,
+	.heuristics_change			= qdevice_net_algo_test_heuristics_change,
+	.heuristics_change_reply_received	= qdevice_net_algo_test_heuristics_change_reply_received,
 	.disconnected				= qdevice_net_algo_test_disconnected,
 	.destroy				= qdevice_net_algo_test_destroy,
 };

qdevices/qdevice-net-algo-test.h

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -44,8 +44,8 @@ extern "C" {
 extern int	qdevice_net_algo_test_init(struct qdevice_net_instance *instance);
 
 extern int	qdevice_net_algo_test_connected(struct qdevice_net_instance *instance,
-    int *send_config_node_list, int *send_membership_node_list, int *send_quorum_node_list,
-    enum tlv_vote *vote);
+    enum tlv_heuristics *heuristics, int *send_config_node_list, int *send_membership_node_list,
+    int *send_quorum_node_list, enum tlv_vote *vote);
 
 extern int	qdevice_net_algo_test_config_node_list_changed(
     struct qdevice_net_instance *instance, const struct node_list *nlist,
@@ -53,7 +53,13 @@ extern int	qdevice_net_algo_test_config_node_list_changed(
 
 extern int	qdevice_net_algo_test_votequorum_node_list_notify(
     struct qdevice_net_instance *instance, const struct tlv_ring_id *ring_id,
-    uint32_t node_list_entries, uint32_t node_list[], int *send_node_list, enum tlv_vote *vote);
+    uint32_t node_list_entries, uint32_t node_list[], int *pause_cast_vote_timer,
+    enum tlv_vote *vote);
+
+extern int	qdevice_net_algo_test_votequorum_node_list_heuristics_notify(
+    struct qdevice_net_instance *instance, const struct tlv_ring_id *ring_id,
+    uint32_t node_list_entries, uint32_t node_list[], int *send_node_list, enum tlv_vote *vote,
+    enum tlv_heuristics *heuristics);
 
 extern int	qdevice_net_algo_test_votequorum_quorum_notify(
     struct qdevice_net_instance *instance, uint32_t quorate, uint32_t node_list_entries,
@@ -88,6 +94,13 @@ extern int	qdevice_net_algo_test_echo_reply_received(
 extern int	qdevice_net_algo_test_echo_reply_not_received(
     struct qdevice_net_instance *instance);
 
+extern int	qdevice_net_algo_test_heuristics_change(struct qdevice_net_instance *instance,
+    enum tlv_heuristics *heuristics, int *send_msg, enum tlv_vote *vote);
+
+extern int	qdevice_net_algo_test_heuristics_change_reply_received(struct qdevice_net_instance *instance,
+    uint32_t seq_number, const struct tlv_ring_id *ring_id, int ring_id_is_valid, enum tlv_heuristics heuristics,
+    enum tlv_vote *vote);
+
 extern int	qdevice_net_algo_test_disconnected(struct qdevice_net_instance *instance,
     enum qdevice_net_disconnect_reason disconnect_reason, int *try_reconnect, enum tlv_vote *vote);
 

qdevices/qdevice-net-algorithm.c

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -59,8 +59,8 @@ qdevice_net_algorithm_init(struct qdevice_net_instance *instance)
 }
 
 int
-qdevice_net_algorithm_connected(struct qdevice_net_instance *instance, int *send_config_node_list,
-    int *send_membership_node_list, int *send_quorum_node_list, enum tlv_vote *vote)
+qdevice_net_algorithm_connected(struct qdevice_net_instance *instance, enum tlv_heuristics *heuristics,
+    int *send_config_node_list, int *send_membership_node_list, int *send_quorum_node_list, enum tlv_vote *vote)
 {
 
 	if (instance->decision_algorithm >= QDEVICE_NET_STATIC_SUPPORTED_DECISION_ALGORITHMS_SIZE ||
@@ -70,7 +70,7 @@ qdevice_net_algorithm_connected(struct qdevice_net_instance *instance, int *send
 	}
 
 	return (qdevice_net_algorithm_array[instance->decision_algorithm]->connected(instance,
-	    send_config_node_list, send_membership_node_list, send_quorum_node_list, vote));
+	    heuristics, send_config_node_list, send_membership_node_list, send_quorum_node_list, vote));
 }
 
 int
@@ -93,7 +93,7 @@ qdevice_net_algorithm_config_node_list_changed(struct qdevice_net_instance *inst
 int
 qdevice_net_algorithm_votequorum_node_list_notify(struct qdevice_net_instance *instance,
     const struct tlv_ring_id *ring_id, uint32_t node_list_entries, uint32_t node_list[],
-    int *send_node_list, enum tlv_vote *vote)
+    int *pause_cast_vote_timer, enum tlv_vote *vote)
 {
 
 	if (instance->decision_algorithm >= QDEVICE_NET_STATIC_SUPPORTED_DECISION_ALGORITHMS_SIZE ||
@@ -104,7 +104,25 @@ qdevice_net_algorithm_votequorum_node_list_notify(struct qdevice_net_instance *i
 	}
 
 	return (qdevice_net_algorithm_array[instance->decision_algorithm]->votequorum_node_list_notify(
-	    instance, ring_id, node_list_entries, node_list, send_node_list, vote));
+	    instance, ring_id, node_list_entries, node_list, pause_cast_vote_timer, vote));
+}
+
+int
+qdevice_net_algorithm_votequorum_node_list_heuristics_notify(struct qdevice_net_instance *instance,
+    const struct tlv_ring_id *ring_id, uint32_t node_list_entries, uint32_t node_list[],
+    int *send_node_list, enum tlv_vote *vote, enum tlv_heuristics *heuristics)
+{
+
+	if (instance->decision_algorithm >= QDEVICE_NET_STATIC_SUPPORTED_DECISION_ALGORITHMS_SIZE ||
+	    qdevice_net_algorithm_array[instance->decision_algorithm] == NULL) {
+		qdevice_log(LOG_CRIT, "qdevice_net_algorithm_votequorum_node_list_heuristics_notify "
+		    "unhandled decision algorithm");
+		exit(1);
+	}
+
+	return (qdevice_net_algorithm_array[instance->decision_algorithm]->
+	    votequorum_node_list_heuristics_notify(
+	    instance, ring_id, node_list_entries, node_list, send_node_list, vote, heuristics));
 }
 
 int
@@ -259,6 +277,40 @@ qdevice_net_algorithm_echo_reply_not_received(struct qdevice_net_instance *insta
 	    echo_reply_not_received(instance));
 }
 
+int
+qdevice_net_algorithm_heuristics_change(struct qdevice_net_instance *instance,
+    enum tlv_heuristics *heuristics, int *send_msg, enum tlv_vote *vote)
+{
+
+	if (instance->decision_algorithm >= QDEVICE_NET_STATIC_SUPPORTED_DECISION_ALGORITHMS_SIZE ||
+	    qdevice_net_algorithm_array[instance->decision_algorithm] == NULL) {
+		qdevice_log(LOG_CRIT, "qdevice_net_algorithm_heuristics_change "
+		    "unhandled decision algorithm");
+		exit(1);
+	}
+
+	return (qdevice_net_algorithm_array[instance->decision_algorithm]->
+	    heuristics_change(instance, heuristics, send_msg, vote));
+}
+
+int
+qdevice_net_algorithm_heuristics_change_reply_received(struct qdevice_net_instance *instance,
+    uint32_t seq_number, const struct tlv_ring_id *ring_id, int ring_id_is_valid,
+    enum tlv_heuristics heuristics, enum tlv_vote *vote)
+{
+
+	if (instance->decision_algorithm >= QDEVICE_NET_STATIC_SUPPORTED_DECISION_ALGORITHMS_SIZE ||
+	    qdevice_net_algorithm_array[instance->decision_algorithm] == NULL) {
+		qdevice_log(LOG_CRIT, "qdevice_net_algorithm_heuristics_change_reply_received "
+		    "unhandled decision algorithm");
+		exit(1);
+	}
+
+	return (qdevice_net_algorithm_array[instance->decision_algorithm]->
+	    heuristics_change_reply_received(instance, seq_number, ring_id, ring_id_is_valid,
+	    heuristics, vote));
+}
+
 int
 qdevice_net_algorithm_disconnected(struct qdevice_net_instance *instance,
     enum qdevice_net_disconnect_reason disconnect_reason, int *try_reconnect, enum tlv_vote *vote)

qdevices/qdevice-net-algorithm.h

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -48,8 +48,8 @@ extern "C" {
 extern int	qdevice_net_algorithm_init(struct qdevice_net_instance *instance);
 
 extern int	qdevice_net_algorithm_connected(struct qdevice_net_instance *instance,
-    int *send_config_node_list, int *send_membership_node_list, int *send_quorum_node_list,
-    enum tlv_vote *vote);
+    enum tlv_heuristics *heuristics, int *send_config_node_list, int *send_membership_node_list,
+    int *send_quorum_node_list, enum tlv_vote *vote);
 
 extern int	qdevice_net_algorithm_config_node_list_changed(struct qdevice_net_instance *instance,
     const struct node_list *nlist, int config_version_set, uint64_t config_version,
@@ -57,7 +57,12 @@ extern int	qdevice_net_algorithm_config_node_list_changed(struct qdevice_net_ins
 
 extern int	qdevice_net_algorithm_votequorum_node_list_notify(struct qdevice_net_instance *instance,
     const struct tlv_ring_id *ring_id, uint32_t node_list_entries, uint32_t node_list[],
-    int *send_node_list, enum tlv_vote *vote);
+    int *pause_cast_vote_timer, enum tlv_vote *vote);
+
+extern int	qdevice_net_algorithm_votequorum_node_list_heuristics_notify(
+    struct qdevice_net_instance *instance,
+    const struct tlv_ring_id *ring_id, uint32_t node_list_entries, uint32_t node_list[],
+    int *send_node_list, enum tlv_vote *vote, enum tlv_heuristics *heuristics);
 
 extern int	qdevice_net_algorithm_votequorum_quorum_notify(struct qdevice_net_instance *instance,
     uint32_t quorate, uint32_t node_list_entries, votequorum_node_t node_list[], int *send_node_list,
@@ -88,6 +93,13 @@ extern int	qdevice_net_algorithm_echo_reply_received(struct qdevice_net_instance
 
 extern int	qdevice_net_algorithm_echo_reply_not_received(struct qdevice_net_instance *instance);
 
+extern int	qdevice_net_algorithm_heuristics_change(struct qdevice_net_instance *instance,
+    enum tlv_heuristics *heuristics, int *send_msg, enum tlv_vote *vote);
+
+extern int	qdevice_net_algorithm_heuristics_change_reply_received(struct qdevice_net_instance *instance,
+    uint32_t seq_number, const struct tlv_ring_id *ring_id, int ring_id_is_valid, enum tlv_heuristics heuristics,
+    enum tlv_vote *vote);
+
 extern int	qdevice_net_algorithm_disconnected(struct qdevice_net_instance *instance,
     enum qdevice_net_disconnect_reason disconnect_reason, int *try_reconnect, enum tlv_vote *vote);
 
@@ -96,14 +108,17 @@ extern void	qdevice_net_algorithm_destroy(struct qdevice_net_instance *instance)
 struct qdevice_net_algorithm {
 	int (*init)(struct qdevice_net_instance *instance);
 	int (*connected)(struct qdevice_net_instance *instance,
-	    int *send_config_node_list, int *send_membership_node_list, int *send_quorum_node_list,
-	    enum tlv_vote *vote);
+	    enum tlv_heuristics *heuristics, int *send_config_node_list, int *send_membership_node_list,
+	    int *send_quorum_node_list, enum tlv_vote *vote);
 	int (*config_node_list_changed)(struct qdevice_net_instance *instance,
 	    const struct node_list *nlist, int config_version_set, uint64_t config_version,
 	    int *send_node_list, enum tlv_vote *vote);
 	int (*votequorum_node_list_notify)(struct qdevice_net_instance *instance,
 	    const struct tlv_ring_id *ring_id, uint32_t node_list_entries, uint32_t node_list[],
-	    int *send_node_list, enum tlv_vote *vote);
+	    int *pause_cast_vote_timer, enum tlv_vote *vote);
+	int (*votequorum_node_list_heuristics_notify)(struct qdevice_net_instance *instance,
+	    const struct tlv_ring_id *ring_id, uint32_t node_list_entries, uint32_t node_list[],
+	    int *send_node_list, enum tlv_vote *vote, enum tlv_heuristics *heuristics);
 	int (*votequorum_quorum_notify)(struct qdevice_net_instance *instance, uint32_t quorate,
 	    uint32_t node_list_entries, votequorum_node_t node_list[], int *send_node_list,
 	    enum tlv_vote *vote);
@@ -127,6 +142,11 @@ struct qdevice_net_algorithm {
 	int (*echo_reply_received)(struct qdevice_net_instance *instance,
 	    uint32_t seq_number, int is_expected_seq_number);
 	int (*echo_reply_not_received)(struct qdevice_net_instance *instance);
+	int (*heuristics_change)(struct qdevice_net_instance *instance,
+	    enum tlv_heuristics *heuristics, int *send_msg, enum tlv_vote *vote);
+	int (*heuristics_change_reply_received)(struct qdevice_net_instance *instance,
+	    uint32_t seq_number, const struct tlv_ring_id *ring_id, int ring_id_is_valid,
+	    enum tlv_heuristics heuristics, enum tlv_vote *vote);
 	int (*disconnected)(struct qdevice_net_instance *instance,
 	    enum qdevice_net_disconnect_reason disconnect_reason, int *try_reconnect,
 	    enum tlv_vote *vote);

qdevices/qdevice-net-cast-vote-timer.c

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -46,6 +46,10 @@ qdevice_net_cast_vote_timer_callback(void *data1, void *data2)
 
 	instance = (struct qdevice_net_instance *)data1;
 
+	if (instance->cast_vote_timer_paused) {
+		return (-1);
+	}
+
 	case_processed = 0;
 
 	switch (instance->cast_vote_timer_vote) {
@@ -167,3 +171,18 @@ qdevice_net_cast_vote_timer_update(struct qdevice_net_instance *instance, enum t
 
 	return (0);
 }
+
+void
+qdevice_net_cast_vote_timer_set_paused(struct qdevice_net_instance *instance, int paused)
+{
+
+	if (paused != instance->cast_vote_timer_paused) {
+		instance->cast_vote_timer_paused = paused;
+
+		if (instance->cast_vote_timer_paused) {
+			qdevice_log(LOG_DEBUG, "Cast vote timer is now paused.");
+		} else {
+			qdevice_log(LOG_DEBUG, "Cast vote timer is no longer paused.");
+		}
+	}
+}

qdevices/qdevice-net-cast-vote-timer.h

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -43,7 +43,10 @@ extern "C" {
 #endif
 
 extern int		qdevice_net_cast_vote_timer_update(
-	struct qdevice_net_instance *instance, enum tlv_vote vote);
+    struct qdevice_net_instance *instance, enum tlv_vote vote);
+
+extern void		qdevice_net_cast_vote_timer_set_paused(
+    struct qdevice_net_instance *instance, int paused);
 
 #ifdef __cplusplus
 }

qdevices/qdevice-net-disconnect-reason.h

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -61,6 +61,12 @@ enum qdevice_net_disconnect_reason {
 	QDEVICE_NET_DISCONNECT_REASON_CANT_SCHEDULE_HB_TIMER,
 	/* Impossible to create or update votequorum poll timer */
 	QDEVICE_NET_DISCONNECT_REASON_CANT_SCHEDULE_VOTING_TIMER,
+	/* Impossible to create or update regular heuristics timer */
+	QDEVICE_NET_DISCONNECT_REASON_CANT_SCHEDULE_HEURISTICS_TIMER,
+	/* Impossible to exec heuristics */
+	QDEVICE_NET_DISCONNECT_REASON_CANT_START_HEURISTICS,
+	/* Impossible to activate/deactive heuristics result notifier */
+	QDEVICE_NET_DISCONNECT_REASON_CANT_ACTIVATE_HEURISTICS_RESULT_NOTIFIER,
 	/* Impossible to register votequorum callback */
 	QDEVICE_NET_DISCONNECT_REASON_CANT_REGISTER_VOTEQUORUM_CALLBACK,
 	/* Impossible to register cmap callback */
@@ -79,6 +85,8 @@ enum qdevice_net_disconnect_reason {
 
 	/* Server doesn't support client selected decision algorithm */
 	QDEVICE_NET_DISCONNECT_REASON_SERVER_DOESNT_SUPPORT_REQUIRED_ALGORITHM,
+	/* Server doesn't support client required option */
+	QDEVICE_NET_DISCONNECT_REASON_SERVER_DOESNT_SUPPORT_REQUIRED_OPT,
 
 	/* Can't decode message sent by server */
 	QDEVICE_NET_DISCONNECT_REASON_MSG_DECODE_ERROR,
@@ -102,12 +110,18 @@ enum qdevice_net_disconnect_reason {
 	QDEVICE_NET_DISCONNECT_REASON_ALGO_CONFIG_NODE_LIST_CHANGED_ERR,
 	QDEVICE_NET_DISCONNECT_REASON_ALGO_VOTEQUORUM_QUORUM_NOTIFY_ERR,
 	QDEVICE_NET_DISCONNECT_REASON_ALGO_VOTEQUORUM_NODE_LIST_NOTIFY_ERR,
+	QDEVICE_NET_DISCONNECT_REASON_ALGO_VOTEQUORUM_NODE_LIST_HEURISTICS_NOTIFY_ERR,
 	QDEVICE_NET_DISCONNECT_REASON_ALGO_VOTEQUORUM_EXPECTED_VOTES_NOTIFY_ERR,
 	QDEVICE_NET_DISCONNECT_REASON_ALGO_NODE_LIST_REPLY_ERR,
 	QDEVICE_NET_DISCONNECT_REASON_ALGO_ASK_FOR_VOTE_REPLY_ERR,
 	QDEVICE_NET_DISCONNECT_REASON_ALGO_VOTE_INFO_ERR,
 	QDEVICE_NET_DISCONNECT_REASON_ALGO_ECHO_REPLY_RECEIVED_ERR,
 	QDEVICE_NET_DISCONNECT_REASON_ALGO_ECHO_REPLY_NOT_RECEIVED_ERR,
+	QDEVICE_NET_DISCONNECT_REASON_ALGO_HEURISTICS_CHANGE_ERR,
+	QDEVICE_NET_DISCONNECT_REASON_ALGO_HEURISTICS_CHANGE_REPLY_ERR,
+
+	QDEVICE_NET_DISCONNECT_REASON_HEURISTICS_WORKER_CLOSED,
+	QDEVICE_NET_DISCONNECT_REASON_HEURISTICS_CANT_SEND_RECEIVE_MSG,
 };
 
 #define qdevice_net_disconnect_reason_try_reconnect(reason) (						\
@@ -127,7 +141,9 @@ enum qdevice_net_disconnect_reason {
     reason == QDEVICE_NET_DISCONNECT_REASON_LOCAL_SOCKET_CLOSED ||			\
     reason == QDEVICE_NET_DISCONNECT_REASON_CANT_SCHEDULE_VOTING_TIMER ||		\
     reason == QDEVICE_NET_DISCONNECT_REASON_CANT_REGISTER_VOTEQUORUM_CALLBACK ||	\
-    reason == QDEVICE_NET_DISCONNECT_REASON_CANT_REGISTER_CMAP_CALLBACK)
+    reason == QDEVICE_NET_DISCONNECT_REASON_CANT_REGISTER_CMAP_CALLBACK ||		\
+    reason == QDEVICE_NET_DISCONNECT_REASON_HEURISTICS_WORKER_CLOSED ||			\
+    reason == QDEVICE_NET_DISCONNECT_REASON_HEURISTICS_CANT_SEND_RECEIVE_MSG)
 
 #ifdef __cplusplus
 }

qdevices/qdevice-net-heuristics.c

@@ -0,0 +1,462 @@
+/*
+ * Copyright (c) 2017 Red Hat, Inc.
+ *
+ * All rights reserved.
+ *
+ * Author: Jan Friesse (jfriesse@redhat.com)
+ *
+ * This software licensed under BSD license, the text of which follows:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright notice,
+ *   this list of conditions and the following disclaimer.
+ * - Redistributions in binary form must reproduce the above copyright notice,
+ *   this list of conditions and the following disclaimer in the documentation
+ *   and/or other materials provided with the distribution.
+ * - Neither the name of the Red Hat, Inc. nor the names of its
+ *   contributors may be used to endorse or promote products derived from this
+ *   software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include "qdevice-log.h"
+#include "qdevice-net-algorithm.h"
+#include "qdevice-net-cast-vote-timer.h"
+#include "qdevice-net-heuristics.h"
+#include "qdevice-net-send.h"
+#include "qdevice-net-votequorum.h"
+
+enum tlv_heuristics
+qdevice_net_heuristics_exec_result_to_tlv(enum qdevice_heuristics_exec_result exec_result)
+{
+	enum tlv_heuristics res;
+
+	switch (exec_result) {
+	case QDEVICE_HEURISTICS_EXEC_RESULT_DISABLED: res = TLV_HEURISTICS_UNDEFINED; break;
+	case QDEVICE_HEURISTICS_EXEC_RESULT_PASS: res = TLV_HEURISTICS_PASS; break;
+	case QDEVICE_HEURISTICS_EXEC_RESULT_FAIL: res = TLV_HEURISTICS_FAIL; break;
+	default:
+		qdevice_log(LOG_ERR, "qdevice_net_heuristics_exec_result_to_tlv: Unhandled "
+		    "heuristics exec result %s",
+		    qdevice_heuristics_exec_result_to_str(exec_result));
+		exit(1);
+		break;
+	}
+
+	return (res);
+}
+
+static int
+qdevice_net_regular_heuristics_exec_result_callback(void *heuristics_instance_ptr,
+    uint32_t seq_number, enum qdevice_heuristics_exec_result exec_result)
+{
+	struct qdevice_heuristics_instance *heuristics_instance;
+	struct qdevice_instance *instance;
+	struct qdevice_net_instance *net_instance;
+	int send_msg;
+	enum tlv_vote vote;
+	enum tlv_heuristics heuristics;
+
+	heuristics_instance = (struct qdevice_heuristics_instance *)heuristics_instance_ptr;
+	instance = heuristics_instance->qdevice_instance_ptr;
+	net_instance = instance->model_data;
+
+	if (qdevice_heuristics_result_notifier_list_set_active(&heuristics_instance->exec_result_notifier_list,
+	    qdevice_net_regular_heuristics_exec_result_callback, 0) != 0) {
+		qdevice_log(LOG_ERR, "Can't deactivate net regular heuristics exec callback notifier");
+
+		net_instance->disconnect_reason = QDEVICE_NET_DISCONNECT_REASON_CANT_ACTIVATE_HEURISTICS_RESULT_NOTIFIER;
+		net_instance->schedule_disconnect = 1;
+
+		return (0);
+	}
+
+	heuristics = qdevice_net_heuristics_exec_result_to_tlv(exec_result);
+
+	if (exec_result == QDEVICE_HEURISTICS_EXEC_RESULT_DISABLED) {
+		/*
+		 * Can happen when user disables heuristics during runtime
+		 */
+		return (0);
+	}
+
+	if (net_instance->latest_heuristics_result != heuristics) {
+		qdevice_log(LOG_ERR, "Heuristics result changed from %s to %s",
+		    tlv_heuristics_to_str(net_instance->latest_heuristics_result),
+		    tlv_heuristics_to_str(heuristics));
+
+		if (net_instance->state != QDEVICE_NET_INSTANCE_STATE_WAITING_VOTEQUORUM_CMAP_EVENTS) {
+			/*
+			 * Not connected to qnetd
+			 */
+			send_msg = 0;
+		} else {
+			send_msg = 1;
+		}
+
+		vote = TLV_VOTE_NO_CHANGE;
+
+		if (qdevice_net_algorithm_heuristics_change(net_instance, &heuristics, &send_msg,
+		    &vote) == -1) {
+			qdevice_log(LOG_ERR, "Algorithm returned error. Disconnecting.");
+
+			net_instance->disconnect_reason =
+			    QDEVICE_NET_DISCONNECT_REASON_ALGO_HEURISTICS_CHANGE_ERR;
+			net_instance->schedule_disconnect = 1;
+
+			return (0);
+		} else {
+			qdevice_log(LOG_DEBUG, "Algorithm decided to %s message with heuristics result "
+			    "%s and result vote is %s", (send_msg ? "send" : "not send"),
+			    tlv_heuristics_to_str(heuristics), tlv_vote_to_str(vote));
+		}
+
+		if (send_msg) {
+			if (heuristics == TLV_HEURISTICS_UNDEFINED) {
+				qdevice_log(LOG_ERR, "Inconsistent algorithm result. "
+				    "It's not possible to send message with undefined heuristics. "
+				    "Disconnecting.");
+
+				net_instance->disconnect_reason =
+				    QDEVICE_NET_DISCONNECT_REASON_ALGO_HEURISTICS_CHANGE_ERR;
+				net_instance->schedule_disconnect = 1;
+
+				return (0);
+			}
+
+			if (!net_instance->server_supports_heuristics) {
+				qdevice_log(LOG_ERR, "Server doesn't support heuristics. "
+				    "Disconnecting.");
+
+				net_instance->disconnect_reason =
+				    QDEVICE_NET_DISCONNECT_REASON_SERVER_DOESNT_SUPPORT_REQUIRED_OPT;
+				net_instance->schedule_disconnect = 1;
+
+				return (0);
+			}
+
+			if (qdevice_net_send_heuristics_change(net_instance, heuristics) != 0) {
+				net_instance->disconnect_reason = QDEVICE_NET_DISCONNECT_REASON_CANT_ALLOCATE_MSG_BUFFER;
+				net_instance->schedule_disconnect = 1;
+
+				return (0);
+			}
+		}
+
+		if (qdevice_net_cast_vote_timer_update(net_instance, vote) != 0) {
+			qdevice_log(LOG_CRIT, "qdevice_net_heuristics_exec_result_callback "
+			    "Can't update cast vote timer");
+
+			net_instance->disconnect_reason = QDEVICE_NET_DISCONNECT_REASON_CANT_SCHEDULE_VOTING_TIMER;
+			net_instance->schedule_disconnect = 1;
+
+			return (0);
+		}
+	}
+
+	net_instance->latest_regular_heuristics_result = heuristics;
+	net_instance->latest_heuristics_result = heuristics;
+
+	if (qdevice_net_heuristics_schedule_timer(net_instance) != 0) {
+		return (0);
+	}
+
+	return (0);
+}
+
+static int
+qdevice_net_connect_heuristics_exec_result_callback(void *heuristics_instance_ptr,
+    uint32_t seq_number, enum qdevice_heuristics_exec_result exec_result)
+{
+	struct qdevice_heuristics_instance *heuristics_instance;
+	struct qdevice_instance *instance;
+	struct qdevice_net_instance *net_instance;
+	enum tlv_vote vote;
+	enum tlv_heuristics heuristics;
+	int send_config_node_list;
+	int send_membership_node_list;
+	int send_quorum_node_list;
+	struct tlv_ring_id tlv_rid;
+	enum tlv_quorate quorate;
+
+	heuristics_instance = (struct qdevice_heuristics_instance *)heuristics_instance_ptr;
+	instance = heuristics_instance->qdevice_instance_ptr;
+	net_instance = instance->model_data;
+
+
+	if (qdevice_heuristics_result_notifier_list_set_active(&heuristics_instance->exec_result_notifier_list,
+	    qdevice_net_connect_heuristics_exec_result_callback, 0) != 0) {
+		qdevice_log(LOG_ERR, "Can't deactivate net connect heuristics exec callback notifier");
+
+		net_instance->disconnect_reason = QDEVICE_NET_DISCONNECT_REASON_CANT_ACTIVATE_HEURISTICS_RESULT_NOTIFIER;
+		net_instance->schedule_disconnect = 1;
+
+		return (0);
+	}
+
+	heuristics = qdevice_net_heuristics_exec_result_to_tlv(exec_result);
+
+	send_config_node_list = 1;
+	send_membership_node_list = 1;
+	send_quorum_node_list = 1;
+	vote = TLV_VOTE_WAIT_FOR_REPLY;
+
+	if (qdevice_net_algorithm_connected(net_instance, &heuristics, &send_config_node_list,
+	    &send_membership_node_list, &send_quorum_node_list, &vote) != 0) {
+		qdevice_log(LOG_DEBUG, "Algorithm returned error. Disconnecting.");
+		net_instance->disconnect_reason = QDEVICE_NET_DISCONNECT_REASON_ALGO_CONNECTED_ERR;
+		return (0);
+	} else {
+		qdevice_log(LOG_DEBUG, "Algorithm decided to %s config node list, %s membership "
+		    "node list, %s quorum node list, heuristics is %s and result vote is %s",
+		    (send_config_node_list ? "send" : "not send"),
+		    (send_membership_node_list ? "send" : "not send"),
+		    (send_quorum_node_list ? "send" : "not send"),
+		    tlv_heuristics_to_str(heuristics),
+		    tlv_vote_to_str(vote));
+	}
+
+	/*
+	 * Now we can finally really send node list, votequorum node list and update timer
+	 */
+	if (send_config_node_list) {
+		if (qdevice_net_send_config_node_list(net_instance,
+		    &instance->config_node_list,
+		    instance->config_node_list_version_set,
+		    instance->config_node_list_version, 1) != 0) {
+			net_instance->disconnect_reason = QDEVICE_NET_DISCONNECT_REASON_CANT_ALLOCATE_MSG_BUFFER;
+			return (0);
+		}
+	}
+
+	if (send_membership_node_list) {
+		qdevice_net_votequorum_ring_id_to_tlv(&tlv_rid,
+		    &instance->vq_node_list_ring_id);
+
+		if (qdevice_net_send_membership_node_list(net_instance, &tlv_rid,
+		    instance->vq_node_list_entries,
+		    instance->vq_node_list,
+		    heuristics) != 0) {
+			net_instance->disconnect_reason = QDEVICE_NET_DISCONNECT_REASON_CANT_ALLOCATE_MSG_BUFFER;
+			return (0);
+		}
+	}
+
+	if (send_quorum_node_list) {
+		quorate = (instance->vq_quorum_quorate ?
+		    TLV_QUORATE_QUORATE : TLV_QUORATE_INQUORATE);
+
+		if (qdevice_net_send_quorum_node_list(net_instance,
+		    quorate,
+		    instance->vq_quorum_node_list_entries,
+		    instance->vq_quorum_node_list) != 0) {
+			net_instance->disconnect_reason = QDEVICE_NET_DISCONNECT_REASON_CANT_ALLOCATE_MSG_BUFFER;
+			return (0);
+		}
+	}
+
+	if (qdevice_net_cast_vote_timer_update(net_instance, vote) != 0) {
+		qdevice_log(LOG_CRIT, "qdevice_net_msg_received_set_option_reply fatal error. "
+		    " Can't update cast vote timer vote");
+		net_instance->disconnect_reason = QDEVICE_NET_DISCONNECT_REASON_CANT_SCHEDULE_VOTING_TIMER;
+	}
+
+	net_instance->state = QDEVICE_NET_INSTANCE_STATE_WAITING_VOTEQUORUM_CMAP_EVENTS;
+	net_instance->connected_since_time = time(NULL);
+
+	net_instance->latest_connect_heuristics_result = heuristics;
+	net_instance->latest_heuristics_result = heuristics;
+
+	return (0);
+}
+
+static int
+qdevice_net_heuristics_timer_callback(void *data1, void *data2)
+{
+	struct qdevice_net_instance *net_instance;
+	struct qdevice_heuristics_instance *heuristics_instance;
+
+	net_instance = (struct qdevice_net_instance *)data1;
+	heuristics_instance = &net_instance->qdevice_instance_ptr->heuristics_instance;
+
+	if (qdevice_heuristics_waiting_for_result(heuristics_instance)) {
+		qdevice_log(LOG_DEBUG, "Not executing regular heuristics because other heuristics is already running.");
+
+		return (1);
+	}
+
+	net_instance->regular_heuristics_timer = NULL;
+
+	qdevice_log(LOG_DEBUG, "Executing regular heuristics.");
+
+	if (qdevice_heuristics_result_notifier_list_set_active(&heuristics_instance->exec_result_notifier_list,
+	    qdevice_net_regular_heuristics_exec_result_callback, 1) != 0) {
+		qdevice_log(LOG_ERR, "Can't activate net regular heuristics exec callback notifier");
+
+		net_instance->disconnect_reason = QDEVICE_NET_DISCONNECT_REASON_CANT_ACTIVATE_HEURISTICS_RESULT_NOTIFIER;
+		net_instance->schedule_disconnect = 1;
+
+		return (0);
+	}
+
+	if (qdevice_heuristics_exec(heuristics_instance,
+	    net_instance->qdevice_instance_ptr->sync_in_progress) != 0) {
+		qdevice_log(LOG_ERR, "Can't execute regular heuristics.");
+
+		net_instance->schedule_disconnect = 1;
+		net_instance->disconnect_reason = QDEVICE_NET_DISCONNECT_REASON_CANT_START_HEURISTICS;
+
+		return (0);
+	}
+
+	/*
+	 * Do not schedule this callback again. It's going to be scheduled in the
+	 * qdevice_net_heuristics_exec_result_callback
+	 */
+	return (0);
+}
+
+int
+qdevice_net_heuristics_stop_timer(struct qdevice_net_instance *net_instance)
+{
+	struct qdevice_instance *instance;
+	struct qdevice_heuristics_instance *heuristics_instance;
+
+	instance = net_instance->qdevice_instance_ptr;
+	heuristics_instance = &instance->heuristics_instance;
+
+	if (net_instance->regular_heuristics_timer != NULL) {
+		qdevice_log(LOG_DEBUG, "Regular heuristics timer stopped");
+
+		timer_list_delete(&net_instance->main_timer_list, net_instance->regular_heuristics_timer);
+		net_instance->regular_heuristics_timer = NULL;
+
+		if (qdevice_heuristics_result_notifier_list_set_active(&heuristics_instance->exec_result_notifier_list,
+		    qdevice_net_regular_heuristics_exec_result_callback, 0) != 0) {
+			qdevice_log(LOG_ERR, "Can't deactivate net regular heuristics exec callback notifier");
+
+			net_instance->disconnect_reason =
+			    QDEVICE_NET_DISCONNECT_REASON_CANT_ACTIVATE_HEURISTICS_RESULT_NOTIFIER;
+			net_instance->schedule_disconnect = 1;
+			return (-1);
+		}
+	}
+
+	return (0);
+}
+
+int
+qdevice_net_heuristics_schedule_timer(struct qdevice_net_instance *net_instance)
+{
+	uint32_t interval;
+	struct qdevice_instance *instance;
+	struct qdevice_heuristics_instance *heuristics_instance;
+
+	instance = net_instance->qdevice_instance_ptr;
+	heuristics_instance = &instance->heuristics_instance;
+
+        if (heuristics_instance->mode != QDEVICE_HEURISTICS_MODE_ENABLED) {
+		qdevice_log(LOG_DEBUG, "Not scheduling heuristics timer because mode is not enabled");
+
+		if (qdevice_net_heuristics_stop_timer(net_instance) != 0) {
+			return (-1);
+		}
+
+		return (0);
+        }
+
+	if (net_instance->regular_heuristics_timer != NULL) {
+		qdevice_log(LOG_DEBUG, "Not scheduling heuristics timer because it is already scheduled");
+
+		return (0);
+	}
+
+	interval = heuristics_instance->interval;
+
+	qdevice_log(LOG_DEBUG, "Scheduling next regular heuristics in %"PRIu32"ms", interval);
+
+	net_instance->regular_heuristics_timer = timer_list_add(&net_instance->main_timer_list,
+		interval,
+		qdevice_net_heuristics_timer_callback,
+	        (void *)net_instance, NULL);
+
+	if (net_instance->regular_heuristics_timer == NULL) {
+		qdevice_log(LOG_ERR, "Can't schedule regular heuristics.");
+
+		net_instance->disconnect_reason = QDEVICE_NET_DISCONNECT_REASON_CANT_SCHEDULE_HEURISTICS_TIMER;
+		net_instance->schedule_disconnect = 1;
+		return (-1);
+	}
+
+	return (0);
+}
+
+int
+qdevice_net_heuristics_init(struct qdevice_net_instance *net_instance)
+{
+
+	if (qdevice_heuristics_result_notifier_list_add(
+	    &net_instance->qdevice_instance_ptr->heuristics_instance.exec_result_notifier_list,
+	    qdevice_net_regular_heuristics_exec_result_callback) == NULL) {
+		qdevice_log(LOG_ERR, "Can't add net regular heuristics exec callback into notifier");
+
+		return (-1);
+	}
+
+	if (qdevice_heuristics_result_notifier_list_add(
+	    &net_instance->qdevice_instance_ptr->heuristics_instance.exec_result_notifier_list,
+	    qdevice_net_connect_heuristics_exec_result_callback) == NULL) {
+		qdevice_log(LOG_ERR, "Can't add net connect heuristics exec callback into notifier");
+
+		return (-1);
+	}
+
+	return (0);
+}
+
+int
+qdevice_net_heuristics_exec_after_connect(struct qdevice_net_instance *net_instance)
+{
+	struct qdevice_instance *instance;
+	struct qdevice_heuristics_instance *heuristics_instance;
+
+	instance = net_instance->qdevice_instance_ptr;
+	heuristics_instance = &instance->heuristics_instance;
+
+	qdevice_log(LOG_DEBUG, "Executing after-connect heuristics.");
+
+	if (qdevice_heuristics_result_notifier_list_set_active(&heuristics_instance->exec_result_notifier_list,
+	    qdevice_net_connect_heuristics_exec_result_callback, 1) != 0) {
+		qdevice_log(LOG_ERR, "Can't activate net connect heuristics exec callback notifier");
+
+		net_instance->disconnect_reason = QDEVICE_NET_DISCONNECT_REASON_CANT_ACTIVATE_HEURISTICS_RESULT_NOTIFIER;
+		net_instance->schedule_disconnect = 1;
+
+		return (-1);
+	}
+
+	if (qdevice_heuristics_exec(heuristics_instance,
+	    instance->sync_in_progress) != 0) {
+		qdevice_log(LOG_ERR, "Can't execute connect heuristics.");
+
+		net_instance->schedule_disconnect = 1;
+		net_instance->disconnect_reason = QDEVICE_NET_DISCONNECT_REASON_CANT_START_HEURISTICS;
+
+		return (-1);
+	}
+
+	return (0);
+}

qdevices/qdevice-net-heuristics.h

@@ -0,0 +1,61 @@
+/*
+ * Copyright (c) 2017 Red Hat, Inc.
+ *
+ * All rights reserved.
+ *
+ * Author: Jan Friesse (jfriesse@redhat.com)
+ *
+ * This software licensed under BSD license, the text of which follows:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright notice,
+ *   this list of conditions and the following disclaimer.
+ * - Redistributions in binary form must reproduce the above copyright notice,
+ *   this list of conditions and the following disclaimer in the documentation
+ *   and/or other materials provided with the distribution.
+ * - Neither the name of the Red Hat, Inc. nor the names of its
+ *   contributors may be used to endorse or promote products derived from this
+ *   software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _QDEVICE_NET_HEURISTICS_H_
+#define _QDEVICE_NET_HEURISTICS_H_
+
+#include "qdevice-heuristics-exec-result.h"
+#include "qdevice-net-instance.h"
+#include "tlv.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+extern enum tlv_heuristics	qdevice_net_heuristics_exec_result_to_tlv(
+    enum qdevice_heuristics_exec_result exec_result);
+
+extern int			qdevice_net_heuristics_stop_timer(struct qdevice_net_instance *net_instance);
+
+extern int			qdevice_net_heuristics_schedule_timer(struct qdevice_net_instance *net_instance);
+
+extern int			qdevice_net_heuristics_init(struct qdevice_net_instance *net_instance);
+
+extern int			qdevice_net_heuristics_exec_after_connect(struct qdevice_net_instance *net_instance);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _QDEVICE_NET_HEURISTICS_H_ */

qdevices/qdevice-net-instance.c

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -53,7 +53,9 @@ qdevice_net_instance_init(struct qdevice_net_instance *instance,
     const char *host_addr, uint16_t host_port, const char *cluster_name,
     const struct tlv_tie_breaker *tie_breaker, uint32_t connect_timeout,
     int force_ip_version, int cmap_fd, int votequorum_fd, int local_socket_fd,
-    const struct qdevice_advanced_settings *advanced_settings)
+    const struct qdevice_advanced_settings *advanced_settings,
+    int heuristics_pipe_cmd_send_fd, int heuristics_pipe_cmd_recv_fd,
+    int heuristics_pipe_log_recv_fd)
 {
 
 	memset(instance, 0, sizeof(*instance));
@@ -103,6 +105,24 @@ qdevice_net_instance_init(struct qdevice_net_instance *instance,
 		return (-1);
 	}
 
+	if ((instance->heuristics_pipe_cmd_send_poll_fd =
+	    PR_CreateSocketPollFd(heuristics_pipe_cmd_send_fd)) == NULL) {
+		qdevice_log_nss(LOG_CRIT, "Can't create NSPR heuristics pipe command send poll fd");
+		return (-1);
+	}
+
+	if ((instance->heuristics_pipe_cmd_recv_poll_fd =
+	    PR_CreateSocketPollFd(heuristics_pipe_cmd_recv_fd)) == NULL) {
+		qdevice_log_nss(LOG_CRIT, "Can't create NSPR heuristics pipe command recv poll fd");
+		return (-1);
+	}
+
+	if ((instance->heuristics_pipe_log_recv_poll_fd =
+	    PR_CreateSocketPollFd(heuristics_pipe_log_recv_fd)) == NULL) {
+		qdevice_log_nss(LOG_CRIT, "Can't create NSPR heuristics pipe log recv poll fd");
+		return (-1);
+	}
+
 	return (0);
 }
 
@@ -169,6 +189,21 @@ qdevice_net_instance_destroy(struct qdevice_net_instance *instance)
 		qdevice_log_nss(LOG_WARNING, "Unable to close local socket poll fd");
 	}
 
+	if (PR_DestroySocketPollFd(instance->heuristics_pipe_cmd_send_poll_fd) != PR_SUCCESS) {
+		qdevice_log_nss(LOG_WARNING, "Unable to close heuristics pipe command send poll fd");
+		return (-1);
+	}
+
+	if (PR_DestroySocketPollFd(instance->heuristics_pipe_cmd_recv_poll_fd) != PR_SUCCESS) {
+		qdevice_log_nss(LOG_WARNING, "Unable to close heuristics pipe command recv poll fd");
+		return (-1);
+	}
+
+	if (PR_DestroySocketPollFd(instance->heuristics_pipe_log_recv_poll_fd) != PR_SUCCESS) {
+		qdevice_log_nss(LOG_WARNING, "Unable to close heuristics pipe log recv poll fd");
+		return (-1);
+	}
+
 	return (0);
 }
 
@@ -378,7 +413,10 @@ qdevice_net_instance_init_from_cmap(struct qdevice_instance *instance)
 	    host_addr, host_port, cluster_name, &tie_breaker, connect_timeout,
 	    force_ip_version,
 	    instance->cmap_poll_fd, instance->votequorum_poll_fd,
-	    instance->local_ipc.socket, instance->advanced_settings) == -1) {
+	    instance->local_ipc.socket, instance->advanced_settings,
+	    instance->heuristics_instance.pipe_cmd_send,
+	    instance->heuristics_instance.pipe_cmd_recv,
+	    instance->heuristics_instance.pipe_log_recv) == -1) {
 		qdevice_log(LOG_ERR, "Can't initialize qdevice-net instance");
 		goto error_free_instance;
 	}

qdevices/qdevice-net-instance.h

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -84,6 +84,7 @@ struct qdevice_net_instance {
 	uint32_t connect_timeout;
 	struct timer_list_entry *cast_vote_timer;
 	enum tlv_vote cast_vote_timer_vote;
+	int cast_vote_timer_paused;
 	const char *host_addr;
 	uint16_t host_port;
 	const char *cluster_name;
@@ -106,6 +107,15 @@ struct qdevice_net_instance {
 	time_t last_echo_reply_received_time;
 	time_t connected_since_time;
 	const struct qdevice_advanced_settings *advanced_settings;
+	PRFileDesc *heuristics_pipe_cmd_send_poll_fd;
+	PRFileDesc *heuristics_pipe_cmd_recv_poll_fd;
+	PRFileDesc *heuristics_pipe_log_recv_poll_fd;
+	struct timer_list_entry *regular_heuristics_timer;
+	int server_supports_heuristics;
+	enum tlv_heuristics latest_regular_heuristics_result;
+	enum tlv_heuristics latest_connect_heuristics_result;
+	enum tlv_heuristics latest_vq_heuristics_result;
+	enum tlv_heuristics latest_heuristics_result;
 };
 
 extern int		qdevice_net_instance_init(struct qdevice_net_instance *instance,
@@ -115,7 +125,9 @@ extern int		qdevice_net_instance_init(struct qdevice_net_instance *instance,
     const char *host_addr, uint16_t host_port, const char *cluster_name,
     const struct tlv_tie_breaker *tie_breaker, uint32_t connect_timeout, int force_ip_version,
     int cmap_fd, int votequorum_fd, int local_socket_fd,
-    const struct qdevice_advanced_settings *advanced_settings);
+    const struct qdevice_advanced_settings *advanced_settings,
+    int heuristics_pipe_cmd_send_fd, int heuristics_pipe_cmd_recv_fd,
+    int heuristics_pipe_log_recv_fd);
 
 extern void		qdevice_net_instance_clean(struct qdevice_net_instance *instance);
 

qdevices/qdevice-net-ipc-cmd.c

@@ -169,6 +169,38 @@ qdevice_net_ipc_cmd_status_add_state(struct qdevice_net_instance *instance,
 	return (dynar_str_catf(outbuf, "State:\t\t\t%s\n", state) != -1);
 }
 
+static int
+qdevice_net_ipc_cmd_status_add_heuristics(struct qdevice_net_instance *instance,
+    struct dynar *outbuf, int verbose)
+{
+	enum qdevice_heuristics_mode active_heuristics_mode;
+	int heuristics_enabled;
+
+	active_heuristics_mode = instance->qdevice_instance_ptr->heuristics_instance.mode;
+	heuristics_enabled = (active_heuristics_mode == QDEVICE_HEURISTICS_MODE_ENABLED ||
+	    active_heuristics_mode == QDEVICE_HEURISTICS_MODE_SYNC);
+
+	if (!heuristics_enabled) {
+		return (1);
+	}
+
+	if (dynar_str_catf(outbuf, "Heuristics result:\t%s",
+	    tlv_heuristics_to_str(instance->latest_heuristics_result)) == -1) {
+		return (0);
+	}
+
+	if (verbose) {
+		if (dynar_str_catf(outbuf, " (regular: %s, membership: %s, connect: %s)",
+		    tlv_heuristics_to_str(instance->latest_regular_heuristics_result),
+		    tlv_heuristics_to_str(instance->latest_vq_heuristics_result),
+		    tlv_heuristics_to_str(instance->latest_connect_heuristics_result)) == -1) {
+			return (0);
+		}
+	}
+
+	return (dynar_str_catf(outbuf, "\n") != -1);
+}
+
 static int
 qdevice_net_ipc_cmd_status_add_tls_state(struct qdevice_net_instance *instance,
     struct dynar *outbuf, int verbose)
@@ -231,6 +263,7 @@ qdevice_net_ipc_cmd_status(struct qdevice_net_instance *instance, struct dynar *
 	    qdevice_net_ipc_cmd_status_add_tie_breaker(instance, outbuf, verbose) &&
 	    qdevice_net_ipc_cmd_status_add_poll_timer_status(instance, outbuf, verbose) &&
 	    qdevice_net_ipc_cmd_status_add_state(instance, outbuf, verbose) &&
+	    qdevice_net_ipc_cmd_status_add_heuristics(instance, outbuf, verbose) &&
 	    qdevice_net_ipc_cmd_status_add_tls_state(instance, outbuf, verbose) &&
 	    qdevice_net_ipc_cmd_status_add_times(instance, outbuf, verbose)) {
 		return (1);

qdevices/qdevice-net-msg-received.c

@@ -35,6 +35,7 @@
 #include "qdevice-log.h"
 #include "qdevice-net-algorithm.h"
 #include "qdevice-net-cast-vote-timer.h"
+#include "qdevice-net-heuristics.h"
 #include "qdevice-net-msg-received.h"
 #include "qdevice-net-send.h"
 #include "qdevice-net-votequorum.h"
@@ -241,12 +242,7 @@ qdevice_net_msg_received_init_reply(struct qdevice_net_instance *instance,
 {
 	size_t zi;
 	int res;
-	int send_config_node_list;
-	int send_membership_node_list;
-	int send_quorum_node_list;
-	enum tlv_vote vote;
-	struct tlv_ring_id tlv_rid;
-	enum tlv_quorate quorate;
+	enum qdevice_heuristics_mode active_heuristics_mode;
 
 	qdevice_log(LOG_DEBUG, "Received init reply msg");
 
@@ -373,6 +369,32 @@ qdevice_net_msg_received_init_reply(struct qdevice_net_instance *instance,
 		return (-1);
 	}
 
+	/*
+	 * Check if server supports heuristics
+	 */
+	res = 0;
+	for (zi = 0; zi < msg->no_supported_options; zi++) {
+		if (msg->supported_options[zi] == TLV_OPT_HEURISTICS) {
+			res = 1;
+		}
+	}
+
+	instance->server_supports_heuristics = res;
+
+	if (!res) {
+		active_heuristics_mode = instance->qdevice_instance_ptr->heuristics_instance.mode;
+
+		if (active_heuristics_mode == QDEVICE_HEURISTICS_MODE_ENABLED ||
+		    active_heuristics_mode == QDEVICE_HEURISTICS_MODE_SYNC) {
+			qdevice_log(LOG_ERR, "Heuristics are enabled but not supported by server");
+
+			instance->disconnect_reason =
+			    QDEVICE_NET_DISCONNECT_REASON_SERVER_DOESNT_SUPPORT_REQUIRED_OPT;
+
+			return (-1);
+		}
+	}
+
 	/*
 	 * Finally fully connected so it's possible to remove connection timer
 	 */
@@ -384,74 +406,18 @@ qdevice_net_msg_received_init_reply(struct qdevice_net_instance *instance,
 	/*
 	 * Server accepted heartbeat interval -> schedule regular sending of echo request
 	 */
-	qdevice_net_echo_request_timer_schedule(instance);
-
-	send_config_node_list = 1;
-	send_membership_node_list = 1;
-	send_quorum_node_list = 1;
-	vote = TLV_VOTE_WAIT_FOR_REPLY;
-
-	if (qdevice_net_algorithm_connected(instance, &send_config_node_list, &send_membership_node_list,
-	    &send_quorum_node_list, &vote) != 0) {
-		qdevice_log(LOG_DEBUG, "Algorithm returned error. Disconnecting.");
-		instance->disconnect_reason = QDEVICE_NET_DISCONNECT_REASON_ALGO_CONNECTED_ERR;
+	if (qdevice_net_echo_request_timer_schedule(instance) != 0) {
 		return (-1);
-	} else {
-		qdevice_log(LOG_DEBUG, "Algorithm decided to %s config node list, %s membership "
-		    "node list, %s quorum node list and result vote is %s",
-		    (send_config_node_list ? "send" : "not send"),
-		    (send_membership_node_list ? "send" : "not send"),
-		    (send_quorum_node_list ? "send" : "not send"),
-		    tlv_vote_to_str(vote));
 	}
 
 	/*
-	 * Now we can finally really send node list, votequorum node list and update timer
+	 * Run heuristics (even when it is disabled, undefined result is ok, rest of sending
+	 * is handled by qdevice_net_connect_heuristics_exec_result_callback
 	 */
-	if (send_config_node_list) {
-		if (qdevice_net_send_config_node_list(instance,
-		    &instance->qdevice_instance_ptr->config_node_list,
-		    instance->qdevice_instance_ptr->config_node_list_version_set,
-		    instance->qdevice_instance_ptr->config_node_list_version, 1) != 0) {
-			instance->disconnect_reason = QDEVICE_NET_DISCONNECT_REASON_CANT_ALLOCATE_MSG_BUFFER;
-			return (-1);
-		}
-	}
-
-	if (send_membership_node_list) {
-		qdevice_net_votequorum_ring_id_to_tlv(&tlv_rid,
-		    &instance->qdevice_instance_ptr->vq_node_list_ring_id);
-
-		if (qdevice_net_send_membership_node_list(instance, &tlv_rid,
-		    instance->qdevice_instance_ptr->vq_node_list_entries,
-		    instance->qdevice_instance_ptr->vq_node_list) != 0) {
-			instance->disconnect_reason = QDEVICE_NET_DISCONNECT_REASON_CANT_ALLOCATE_MSG_BUFFER;
-			return (-1);
-		}
-	}
-
-	if (send_quorum_node_list) {
-		quorate = (instance->qdevice_instance_ptr->vq_quorum_quorate ?
-		    TLV_QUORATE_QUORATE : TLV_QUORATE_INQUORATE);
-
-		if (qdevice_net_send_quorum_node_list(instance,
-		    quorate,
-		    instance->qdevice_instance_ptr->vq_quorum_node_list_entries,
-		    instance->qdevice_instance_ptr->vq_quorum_node_list) != 0) {
-			instance->disconnect_reason = QDEVICE_NET_DISCONNECT_REASON_CANT_ALLOCATE_MSG_BUFFER;
-			return (-1);
-		}
-	}
-
-	if (qdevice_net_cast_vote_timer_update(instance, vote) != 0) {
-		qdevice_log(LOG_CRIT, "qdevice_net_msg_received_set_option_reply fatal error. "
-		    " Can't update cast vote timer vote");
-		instance->disconnect_reason = QDEVICE_NET_DISCONNECT_REASON_CANT_SCHEDULE_VOTING_TIMER;
+	if (qdevice_net_heuristics_exec_after_connect(instance) != 0) {
+		return (-1);
 	}
 
-	instance->state = QDEVICE_NET_INSTANCE_STATE_WAITING_VOTEQUORUM_CMAP_EVENTS;
-	instance->connected_since_time = time(NULL);
-
 	return (0);
 }
 
@@ -511,7 +477,9 @@ qdevice_net_msg_received_set_option_reply(struct qdevice_net_instance *instance,
 		return (-1);
 	}
 
-	qdevice_net_echo_request_timer_schedule(instance);
+	if (qdevice_net_echo_request_timer_schedule(instance) != 0) {
+		return (-1);
+	}
 
 	return (0);
 }
@@ -705,7 +673,7 @@ qdevice_net_msg_received_ask_for_vote_reply(struct qdevice_net_instance *instanc
 	}
 
 	if (!msg->vote_set || !msg->seq_number_set || !msg->ring_id_set) {
-		qdevice_log(LOG_ERR, "Received node list reply message without "
+		qdevice_log(LOG_ERR, "Received ask for vote reply message without "
 		    "required options. Disconnecting from server");
 
 		instance->disconnect_reason = QDEVICE_NET_DISCONNECT_REASON_REQUIRED_OPTION_MISSING;
@@ -832,6 +800,72 @@ qdevice_net_msg_received_vote_info_reply(struct qdevice_net_instance *instance,
 	return (qdevice_net_msg_received_unexpected_msg(instance, msg, "vote info reply"));
 }
 
+static int
+qdevice_net_msg_received_heuristics_change(struct qdevice_net_instance *instance,
+    const struct msg_decoded *msg)
+{
+
+	return (qdevice_net_msg_received_unexpected_msg(instance, msg, "heuristics change"));
+}
+
+static int
+qdevice_net_msg_received_heuristics_change_reply(struct qdevice_net_instance *instance,
+    const struct msg_decoded *msg)
+{
+	enum tlv_vote result_vote;
+	int ring_id_is_valid;
+
+	if (instance->state != QDEVICE_NET_INSTANCE_STATE_WAITING_VOTEQUORUM_CMAP_EVENTS) {
+		qdevice_log(LOG_ERR, "Received unexpected heuristics change reply message. "
+		    "Disconnecting from server");
+
+		instance->disconnect_reason = QDEVICE_NET_DISCONNECT_REASON_UNEXPECTED_MSG;
+		return (-1);
+	}
+
+	if (!msg->vote_set || !msg->seq_number_set || !msg->ring_id_set ||
+	    msg->heuristics == TLV_HEURISTICS_UNDEFINED) {
+		qdevice_log(LOG_ERR, "Received heuristics change reply message without "
+		    "required options. Disconnecting from server");
+
+		instance->disconnect_reason = QDEVICE_NET_DISCONNECT_REASON_REQUIRED_OPTION_MISSING;
+		return (-1);
+	}
+
+	qdevice_log(LOG_DEBUG, "Received heuristics change reply");
+	qdevice_log(LOG_DEBUG, "  seq = "UTILS_PRI_MSG_SEQ, msg->seq_number);
+	qdevice_log(LOG_DEBUG, "  vote = %s", tlv_vote_to_str(msg->vote));
+	qdevice_log(LOG_DEBUG, "  ring id = ("UTILS_PRI_RING_ID")",
+		    msg->ring_id.node_id, msg->ring_id.seq);
+	qdevice_log(LOG_DEBUG, "  heuristics = %s", tlv_heuristics_to_str(msg->heuristics));
+
+	result_vote = msg->vote;
+
+	if (!tlv_ring_id_eq(&msg->ring_id, &instance->last_sent_ring_id)) {
+		ring_id_is_valid = 0;
+		qdevice_log(LOG_DEBUG, "Received heuristics change reply with old ring id.");
+	} else {
+		ring_id_is_valid = 1;
+	}
+
+	if (qdevice_net_algorithm_heuristics_change_reply_received(instance, msg->seq_number,
+	    &msg->ring_id, ring_id_is_valid, msg->heuristics, &result_vote) != 0) {
+		qdevice_log(LOG_DEBUG, "Algorithm returned error. Disconnecting.");
+
+		instance->disconnect_reason = QDEVICE_NET_DISCONNECT_REASON_ALGO_HEURISTICS_CHANGE_REPLY_ERR;
+		return (-1);
+	} else {
+		qdevice_log(LOG_DEBUG, "Algorithm result vote is %s", tlv_vote_to_str(result_vote));
+	}
+
+	if (qdevice_net_cast_vote_timer_update(instance, result_vote) != 0) {
+		instance->disconnect_reason = QDEVICE_NET_DISCONNECT_REASON_CANT_SCHEDULE_VOTING_TIMER;
+		return (-1);
+	}
+
+	return (0);
+}
+
 int
 qdevice_net_msg_received(struct qdevice_net_instance *instance)
 {
@@ -923,6 +957,13 @@ qdevice_net_msg_received(struct qdevice_net_instance *instance)
 		msg_processed = 1;
 		ret_val = qdevice_net_msg_received_vote_info_reply(instance, &msg);
 		break;
+	case MSG_TYPE_HEURISTICS_CHANGE:
+		msg_processed = 1;
+		ret_val = qdevice_net_msg_received_heuristics_change(instance, &msg);
+		break;
+	case MSG_TYPE_HEURISTICS_CHANGE_REPLY:
+		msg_processed = 1;
+		ret_val = qdevice_net_msg_received_heuristics_change_reply(instance, &msg);
 	/*
 	 * Default is not defined intentionally. Compiler shows warning when msg type is added
 	 */

qdevices/qdevice-net-poll-array-user-data.h

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -47,6 +47,9 @@ enum qdevice_net_poll_array_user_data_type {
 	QDEVICE_NET_POLL_ARRAY_USER_DATA_TYPE_IPC_SOCKET,
 	QDEVICE_NET_POLL_ARRAY_USER_DATA_TYPE_SOCKET,
 	QDEVICE_NET_POLL_ARRAY_USER_DATA_TYPE_IPC_CLIENT,
+	QDEVICE_NET_POLL_ARRAY_USER_DATA_TYPE_HEURISTICS_CMD_SEND,
+	QDEVICE_NET_POLL_ARRAY_USER_DATA_TYPE_HEURISTICS_CMD_RECV,
+	QDEVICE_NET_POLL_ARRAY_USER_DATA_TYPE_HEURISTICS_LOG_RECV,
 };
 
 struct qdevice_net_poll_array_user_data {

qdevices/qdevice-net-poll.c

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -40,6 +40,8 @@
 #include "qdevice-votequorum.h"
 #include "qdevice-ipc.h"
 #include "qdevice-net-poll-array-user-data.h"
+#include "qdevice-heuristics.h"
+#include "qdevice-heuristics-cmd.h"
 
 /*
  * Needed for creating nspr handle from unix fd
@@ -144,6 +146,43 @@ qdevice_net_poll_err_socket(struct qdevice_net_instance *instance, const PRPollD
 	}
 }
 
+static void
+qdevice_net_poll_read_heuristics_log(struct qdevice_net_instance *instance)
+{
+	int res;
+
+	res = qdevice_heuristics_log_read_from_pipe(&instance->qdevice_instance_ptr->heuristics_instance);
+	if (res == -1) {
+		instance->schedule_disconnect = 1;
+		instance->disconnect_reason = QDEVICE_NET_DISCONNECT_REASON_HEURISTICS_WORKER_CLOSED;
+	}
+}
+
+static void
+qdevice_net_poll_read_heuristics_cmd(struct qdevice_net_instance *instance)
+{
+	int res;
+
+	res = qdevice_heuristics_cmd_read_from_pipe(&instance->qdevice_instance_ptr->heuristics_instance);
+	if (res == -1) {
+		instance->schedule_disconnect = 1;
+		instance->disconnect_reason = QDEVICE_NET_DISCONNECT_REASON_HEURISTICS_WORKER_CLOSED;
+	}
+}
+
+static void
+qdevice_net_poll_write_heuristics_cmd(struct qdevice_net_instance *instance)
+{
+	int res;
+
+	res = qdevice_heuristics_cmd_write(&instance->qdevice_instance_ptr->heuristics_instance);
+	if (res == -1) {
+		instance->schedule_disconnect = 1;
+		instance->disconnect_reason =
+		    QDEVICE_NET_DISCONNECT_REASON_HEURISTICS_CANT_SEND_RECEIVE_MSG;
+	}
+}
+
 static void
 qdevice_net_poll_read_ipc_socket(struct qdevice_net_instance *instance)
 {
@@ -214,6 +253,37 @@ qdevice_net_pr_poll_array_create(struct qdevice_net_instance *instance)
 	poll_desc->in_flags = PR_POLL_READ;
 	user_data->type = QDEVICE_NET_POLL_ARRAY_USER_DATA_TYPE_IPC_SOCKET;
 
+	if (pr_poll_array_add(poll_array, &poll_desc, (void **)&user_data) < 0) {
+		return (NULL);
+	}
+
+	poll_desc->fd = instance->heuristics_pipe_log_recv_poll_fd;
+	poll_desc->in_flags = PR_POLL_READ;
+	user_data->type = QDEVICE_NET_POLL_ARRAY_USER_DATA_TYPE_HEURISTICS_LOG_RECV;
+
+	if (pr_poll_array_add(poll_array, &poll_desc, (void **)&user_data) < 0) {
+		return (NULL);
+	}
+
+	poll_desc->fd = instance->heuristics_pipe_cmd_recv_poll_fd;
+	poll_desc->in_flags = PR_POLL_READ;
+	user_data->type = QDEVICE_NET_POLL_ARRAY_USER_DATA_TYPE_HEURISTICS_CMD_RECV;
+
+	if (pr_poll_array_add(poll_array, &poll_desc, (void **)&user_data) < 0) {
+		return (NULL);
+	}
+
+	if (!send_buffer_list_empty(
+	    &instance->qdevice_instance_ptr->heuristics_instance.cmd_out_buffer_list)) {
+		poll_desc->fd = instance->heuristics_pipe_cmd_send_poll_fd;
+		poll_desc->in_flags = PR_POLL_WRITE;
+		user_data->type = QDEVICE_NET_POLL_ARRAY_USER_DATA_TYPE_HEURISTICS_CMD_SEND;
+
+		if (pr_poll_array_add(poll_array, &poll_desc, (void **)&user_data) < 0) {
+			return (NULL);
+		}
+	}
+
 	if (instance->state != QDEVICE_NET_INSTANCE_STATE_WAITING_CONNECT ||
 	    !instance->non_blocking_client.destroyed) {
 		if (pr_poll_array_add(poll_array, &poll_desc, (void **)&user_data) < 0) {
@@ -313,6 +383,19 @@ qdevice_net_poll(struct qdevice_net_instance *instance)
 					case_processed = 1;
 					qdevice_ipc_io_read(instance->qdevice_instance_ptr, ipc_client);
 					break;
+				case QDEVICE_NET_POLL_ARRAY_USER_DATA_TYPE_HEURISTICS_CMD_SEND:
+					/*
+					 * Read on heuristics cmd send fd shouldn't happen
+					 */
+					break;
+				case QDEVICE_NET_POLL_ARRAY_USER_DATA_TYPE_HEURISTICS_CMD_RECV:
+					case_processed = 1;
+					qdevice_net_poll_read_heuristics_cmd(instance);
+					break;
+				case QDEVICE_NET_POLL_ARRAY_USER_DATA_TYPE_HEURISTICS_LOG_RECV:
+					case_processed = 1;
+					qdevice_net_poll_read_heuristics_log(instance);
+					break;
 				/*
 				 * Default is not defined intentionally. Compiler shows warning when
 				 * new poll_array_user_data_type is added
@@ -337,11 +420,18 @@ qdevice_net_poll(struct qdevice_net_instance *instance)
 					case_processed = 1;
 					qdevice_ipc_io_write(instance->qdevice_instance_ptr, ipc_client);
 					break;
+				case QDEVICE_NET_POLL_ARRAY_USER_DATA_TYPE_HEURISTICS_CMD_SEND:
+					case_processed = 1;
+					qdevice_net_poll_write_heuristics_cmd(instance);
+					break;
 				case QDEVICE_NET_POLL_ARRAY_USER_DATA_TYPE_VOTEQUORUM:
 				case QDEVICE_NET_POLL_ARRAY_USER_DATA_TYPE_CMAP:
 				case QDEVICE_NET_POLL_ARRAY_USER_DATA_TYPE_IPC_SOCKET:
+				case QDEVICE_NET_POLL_ARRAY_USER_DATA_TYPE_HEURISTICS_CMD_RECV:
+				case QDEVICE_NET_POLL_ARRAY_USER_DATA_TYPE_HEURISTICS_LOG_RECV:
 					/*
-					 * Write on votequorum, cmap and ipc socket shouldn't happen.
+					 * Write on votequorum, cmap, ipc socket and
+					 * heuristics log shouldn't happen.
 					 */
 					break;
 				/*
@@ -395,6 +485,25 @@ qdevice_net_poll(struct qdevice_net_instance *instance)
 					instance->disconnect_reason =
 						QDEVICE_NET_DISCONNECT_REASON_COROSYNC_CONNECTION_CLOSED;
 					break;
+				case QDEVICE_NET_POLL_ARRAY_USER_DATA_TYPE_HEURISTICS_LOG_RECV:
+				case QDEVICE_NET_POLL_ARRAY_USER_DATA_TYPE_HEURISTICS_CMD_RECV:
+				case QDEVICE_NET_POLL_ARRAY_USER_DATA_TYPE_HEURISTICS_CMD_SEND:
+					case_processed = 1;
+
+					/*
+					 * Closed pipe doesn't mean return of PR_POLL_READ. To display
+					 * better log message, we call read log as if POLL_READ would
+					 * be set.
+					 */
+					qdevice_net_poll_read_heuristics_log(instance);
+
+					qdevice_log(LOG_DEBUG, "POLL_ERR (%u) on heuristics pipe. "
+					    "Disconnecting.",  pfds[i].out_flags);
+
+					instance->schedule_disconnect = 1;
+					instance->disconnect_reason =
+						QDEVICE_NET_DISCONNECT_REASON_HEURISTICS_WORKER_CLOSED;
+					break;
 				/*
 				 * Default is not defined intentionally. Compiler shows warning when
 				 * new poll_array_user_data_type is added

qdevices/qdevice-net-send.c

@@ -54,9 +54,11 @@ qdevice_net_send_echo_request(struct qdevice_net_instance *instance)
 	instance->echo_request_expected_msg_seq_num++;
 
 	if (msg_create_echo_request(&send_buffer->buffer, 1,
-	    instance->echo_request_expected_msg_seq_num) == -1) {
+	    instance->echo_request_expected_msg_seq_num) == 0) {
 		qdevice_log(LOG_ERR, "Can't allocate send buffer for echo request msg");
 
+		send_buffer_list_discard_new(&instance->send_buffer_list, send_buffer);
+
 		return (-1);
 	}
 
@@ -188,7 +190,8 @@ qdevice_net_send_config_node_list(struct qdevice_net_instance *instance,
 
 	if (msg_create_node_list(&send_buffer->buffer, instance->last_msg_seq_num,
 	    (initial ? TLV_NODE_LIST_TYPE_INITIAL_CONFIG : TLV_NODE_LIST_TYPE_CHANGED_CONFIG),
-	    0, NULL, config_version_set, config_version, 0, TLV_QUORATE_INQUORATE, nlist) == 0) {
+	    0, NULL, config_version_set, config_version, 0, TLV_QUORATE_INQUORATE,
+	    0, TLV_HEURISTICS_UNDEFINED, nlist) == 0) {
 		qdevice_log(LOG_ERR, "Can't allocate send buffer for config list msg");
 
 		send_buffer_list_discard_new(&instance->send_buffer_list, send_buffer);
@@ -200,10 +203,42 @@ qdevice_net_send_config_node_list(struct qdevice_net_instance *instance,
 	return (0);
 }
 
+int
+qdevice_net_send_heuristics_change(struct qdevice_net_instance *instance,
+    enum tlv_heuristics heuristics)
+{
+	struct send_buffer_list_entry *send_buffer;
+
+	send_buffer = send_buffer_list_get_new(&instance->send_buffer_list);
+	if (send_buffer == NULL) {
+		qdevice_log(LOG_ERR, "Can't allocate send list buffer for heuristics change msg");
+
+		return (-1);
+	}
+
+	instance->last_msg_seq_num++;
+
+	qdevice_log(LOG_DEBUG, "Sending heuristics change seq = "UTILS_PRI_MSG_SEQ
+	    ", heuristics = %s",
+	    instance->last_msg_seq_num, tlv_heuristics_to_str(heuristics));
+
+	if (msg_create_heuristics_change(&send_buffer->buffer, instance->last_msg_seq_num,
+	    heuristics) == 0) {
+		qdevice_log(LOG_ERR, "Can't allocate send buffer for heuristics change msg");
+
+		send_buffer_list_discard_new(&instance->send_buffer_list, send_buffer);
+		return (-1);
+	}
+
+	send_buffer_list_put(&instance->send_buffer_list, send_buffer);
+
+	return (0);
+}
+
 int
 qdevice_net_send_membership_node_list(struct qdevice_net_instance *instance,
     const struct tlv_ring_id *ring_id,
-    uint32_t node_list_entries, uint32_t node_list[])
+    uint32_t node_list_entries, uint32_t node_list[], enum tlv_heuristics heuristics)
 {
 	struct node_list nlist;
 	struct send_buffer_list_entry *send_buffer;
@@ -234,13 +269,13 @@ qdevice_net_send_membership_node_list(struct qdevice_net_instance *instance,
 	instance->last_msg_seq_num++;
 
 	qdevice_log(LOG_DEBUG, "Sending membership node list seq = "UTILS_PRI_MSG_SEQ", "
-	    "ringid = ("UTILS_PRI_RING_ID").", instance->last_msg_seq_num,
-	    ring_id->node_id, ring_id->seq);
+	    "ringid = ("UTILS_PRI_RING_ID"), heuristics = %s.", instance->last_msg_seq_num,
+	    ring_id->node_id, ring_id->seq, tlv_heuristics_to_str(heuristics));
 	qdevice_log_debug_dump_node_list(&nlist);
 
 	if (msg_create_node_list(&send_buffer->buffer, instance->last_msg_seq_num,
 	    TLV_NODE_LIST_TYPE_MEMBERSHIP,
-	    1, ring_id, 0, 0, 0, 0, &nlist) == 0) {
+	    1, ring_id, 0, 0, 0, 0, 1, heuristics, &nlist) == 0) {
 		qdevice_log(LOG_ERR, "Can't allocate send buffer for membership list msg");
 
 		node_list_free(&nlist);
@@ -302,7 +337,7 @@ qdevice_net_send_quorum_node_list(struct qdevice_net_instance *instance,
 
 	if (msg_create_node_list(&send_buffer->buffer, instance->last_msg_seq_num,
 	    TLV_NODE_LIST_TYPE_QUORUM,
-	    0, NULL, 0, 0, 1, quorate, &nlist) == 0) {
+	    0, NULL, 0, 0, 1, quorate, 0, TLV_HEURISTICS_UNDEFINED, &nlist) == 0) {
 		qdevice_log(LOG_ERR, "Can't allocate send buffer for quorum list msg");
 
 		node_list_free(&nlist);

qdevices/qdevice-net-send.h

@@ -55,9 +55,12 @@ extern int		qdevice_net_send_config_node_list(struct qdevice_net_instance *insta
     const struct node_list *nlist, int config_version_set, uint64_t config_version,
     int initial);
 
+extern int		qdevice_net_send_heuristics_change(struct qdevice_net_instance *instance,
+    enum tlv_heuristics heuristics);
+
 extern int		qdevice_net_send_membership_node_list(
     struct qdevice_net_instance *instance, const struct tlv_ring_id *ring_id,
-    uint32_t node_list_entries, uint32_t node_list[]);
+    uint32_t node_list_entries, uint32_t node_list[], enum tlv_heuristics heuristics);
 
 extern int		qdevice_net_send_quorum_node_list(
     struct qdevice_net_instance *instance, enum tlv_quorate quorate,

qdevices/qdevice-net-votequorum.c

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -32,8 +32,8 @@
  * THE POSSIBILITY OF SUCH DAMAGE.
  */
 
-#include "qdevice-net-votequorum.h"
 #include "qdevice-log.h"
+#include "qdevice-net-votequorum.h"
 
 enum tlv_node_state
 qdevice_net_votequorum_node_state_to_tlv(uint32_t votequorum_node_state)

qdevices/qdevice-votequorum.c

@@ -82,6 +82,40 @@ qdevice_votequorum_quorum_notify_callback(votequorum_handle_t votequorum_handle,
 	memcpy(instance->vq_quorum_node_list, node_list, sizeof(*node_list) * node_list_entries);
 }
 
+static int
+qdevice_votequorum_heuristics_exec_result_callback(
+    void *heuristics_instance_ptr,
+    uint32_t seq_number, enum qdevice_heuristics_exec_result exec_result)
+{
+	struct qdevice_heuristics_instance *heuristics_instance;
+	struct qdevice_instance *instance;
+
+	heuristics_instance = (struct qdevice_heuristics_instance *)heuristics_instance_ptr;
+	instance = heuristics_instance->qdevice_instance_ptr;
+
+	if (qdevice_heuristics_result_notifier_list_set_active(
+	    &instance->heuristics_instance.exec_result_notifier_list,
+	    qdevice_votequorum_heuristics_exec_result_callback, 0) != 0) {
+		qdevice_log(LOG_CRIT, "Can't deactivate votequrorum heuristics exec callback notifier");
+		exit(2);
+	}
+
+	qdevice_log(LOG_DEBUG, "Votequorum heuristics exec result callback:");
+	qdevice_log(LOG_DEBUG, "  seq_number = %"PRIu32", exec_result = %s",
+	    seq_number, qdevice_heuristics_exec_result_to_str(exec_result));
+
+	if (qdevice_model_votequorum_node_list_heuristics_notify(instance, instance->vq_node_list_ring_id,
+	    instance->vq_node_list_entries, instance->vq_node_list, exec_result) != 0) {
+		qdevice_log(LOG_DEBUG, "qdevice_votequorum_node_list_heuristics_notify_callback returned error -> exit");
+		exit(2);
+	}
+
+	instance->vq_node_list_initial_heuristics_finished = 1;
+	instance->vq_node_list_heuristics_result = exec_result;
+
+	return (0);
+}
+
 static void
 qdevice_votequorum_node_list_notify_callback(votequorum_handle_t votequorum_handle,
     uint64_t context, votequorum_ring_id_t votequorum_ring_id,
@@ -96,6 +130,7 @@ qdevice_votequorum_node_list_notify_callback(votequorum_handle_t votequorum_hand
 	}
 
 	instance->sync_in_progress = 1;
+	memcpy(&instance->vq_poll_ring_id, &votequorum_ring_id, sizeof(votequorum_ring_id));
 
 	qdevice_log(LOG_DEBUG, "Votequorum nodelist notify callback:");
 	qdevice_log(LOG_DEBUG, "  Ring_id = ("UTILS_PRI_RING_ID")",
@@ -113,7 +148,19 @@ qdevice_votequorum_node_list_notify_callback(votequorum_handle_t votequorum_hand
 		exit(2);
 	}
 
-	instance->vq_node_list_ring_id_set = 1;
+	if (qdevice_heuristics_result_notifier_list_set_active(
+	    &instance->heuristics_instance.exec_result_notifier_list,
+	    qdevice_votequorum_heuristics_exec_result_callback, 1) != 0) {
+		qdevice_log(LOG_CRIT, "Can't activate votequrorum heuristics exec callback notifier");
+		exit(2);
+	}
+
+	if (qdevice_heuristics_exec(&instance->heuristics_instance, instance->sync_in_progress) != 0) {
+		qdevice_log(LOG_CRIT, "Can't start heuristics -> exit");
+		exit(2);
+	}
+
+	instance->vq_node_list_initial_ring_id_set = 1;
 	memcpy(&instance->vq_node_list_ring_id, &votequorum_ring_id, sizeof(votequorum_ring_id));
 	instance->vq_node_list_entries = node_list_entries;
 	free(instance->vq_node_list);
@@ -208,6 +255,12 @@ qdevice_votequorum_init(struct qdevice_instance *instance)
 		    cs_strerror(res));
 		exit(1);
 	}
+
+	if (qdevice_heuristics_result_notifier_list_add(&instance->heuristics_instance.exec_result_notifier_list,
+	    qdevice_votequorum_heuristics_exec_result_callback) == NULL) {
+		qdevice_log(LOG_CRIT, "Can't add votequrorum heuristics exec callback into notifier");
+		exit(1);
+	}
 }
 
 void
@@ -246,11 +299,11 @@ qdevice_votequorum_wait_for_ring_id(struct qdevice_instance *instance)
 
 	while (qdevice_votequorum_dispatch(instance) != -1 &&
 	    no_retries++ < instance->advanced_settings->max_cs_try_again &&
-	    !instance->vq_node_list_ring_id_set) {
+	    !instance->vq_node_list_initial_ring_id_set) {
 		(void)poll(NULL, 0, 1000);
 	}
 
-	if (!instance->vq_node_list_ring_id_set) {
+	if (!instance->vq_node_list_initial_ring_id_set) {
 		qdevice_log(LOG_CRIT, "Can't get initial votequorum membership information.");
 		return (-1);
 	}
@@ -284,7 +337,7 @@ qdevice_votequorum_poll(struct qdevice_instance *instance, int cast_vote)
 
 	res = votequorum_qdevice_poll(instance->votequorum_handle,
 	    instance->advanced_settings->votequorum_device_name, cast_vote,
-	    instance->vq_node_list_ring_id);
+	    instance->vq_poll_ring_id);
 
 	if (res != CS_OK && res != CS_ERR_TRY_AGAIN) {
 		if (res == CS_ERR_MESSAGE_ERROR) {

qdevices/qnetd-algo-2nodelms.c

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -134,14 +134,16 @@ qnetd_algo_2nodelms_config_node_list_received(struct qnetd_client *client,
 enum tlv_reply_error_code
 qnetd_algo_2nodelms_membership_node_list_received(struct qnetd_client *client,
     uint32_t msg_seq_num, const struct tlv_ring_id *ring_id,
-    const struct node_list *nodes, enum tlv_vote *result_vote)
+    const struct node_list *nodes, enum tlv_heuristics heuristics,
+    enum tlv_vote *result_vote)
 {
 	struct node_list_entry *node_info;
  	struct qnetd_client *other_client;
 	struct qnetd_algo_2nodelms_info *info = client->algorithm_data;
 	int node_count = 0;
-	int low_node_id = INT_MAX;
-	int high_node_id = 0;
+	uint32_t low_node_id = UINT32_MAX;
+	uint32_t high_node_id = 0;
+	enum tlv_heuristics other_node_heuristics;
 
 	/* If we're a newcomer and there is another active partition, then we must NACK
 	 * to avoid quorum moving to us from already active nodes.
@@ -175,6 +177,7 @@ qnetd_algo_2nodelms_membership_node_list_received(struct qnetd_client *client,
 
 	/* Now look for other clients connected from this cluster that can't see us any more */
 	node_count = 0;
+	other_node_heuristics = TLV_HEURISTICS_UNDEFINED;
 	TAILQ_FOREACH(other_client, &client->cluster->client_list, cluster_entries) {
 		node_count++;
 
@@ -185,6 +188,9 @@ qnetd_algo_2nodelms_membership_node_list_received(struct qnetd_client *client,
 		if (other_client->node_id > high_node_id) {
 			high_node_id = other_client->node_id;
 		}
+		if (other_client != client) {
+			other_node_heuristics = other_client->last_heuristics;
+		}
 	}
 	qnetd_log(LOG_DEBUG, "algo-2nodelms: cluster %s %d nodes running independently", client->cluster_name, node_count);
 
@@ -195,7 +201,21 @@ qnetd_algo_2nodelms_membership_node_list_received(struct qnetd_client *client,
 		return (TLV_REPLY_ERROR_CODE_NO_ERROR);
 	}
 
-	/* Both nodes are alive. Only give a vote to the nominated tie-breaker node */
+	/*
+	 * Both nodes are alive.
+	 * Check their heuristics.
+	 */
+	if (tlv_heuristics_cmp(heuristics, other_node_heuristics) > 0) {
+		*result_vote = info->last_result = TLV_VOTE_ACK;
+
+		return (TLV_REPLY_ERROR_CODE_NO_ERROR);
+	} else if (tlv_heuristics_cmp(heuristics, other_node_heuristics) < 0) {
+		*result_vote = info->last_result = TLV_VOTE_NACK;
+
+		return (TLV_REPLY_ERROR_CODE_NO_ERROR);
+	}
+
+	/* Heuristics are equal -> Only give a vote to the nominated tie-breaker node */
 	switch (client->tie_breaker.mode) {
 
 	case TLV_TIE_BREAKER_MODE_LOWEST:
@@ -296,6 +316,18 @@ qnetd_algo_2nodelms_vote_info_reply_received(struct qnetd_client *client, uint32
 	return (TLV_REPLY_ERROR_CODE_NO_ERROR);
 }
 
+enum tlv_reply_error_code
+qnetd_algo_2nodelms_heuristics_change_received(struct qnetd_client *client, uint32_t msg_seq_num,
+    enum tlv_heuristics heuristics, enum tlv_vote *result_vote)
+{
+
+	qnetd_log(LOG_INFO, "algo-2nodelms: heuristics change is not supported.");
+
+	*result_vote = TLV_VOTE_NO_CHANGE;
+
+	return (TLV_REPLY_ERROR_CODE_NO_ERROR);
+}
+
 enum tlv_reply_error_code
 qnetd_algo_2nodelms_timer_callback(struct qnetd_client *client, int *reschedule_timer,
     int *send_vote, enum tlv_vote *result_vote)
@@ -312,6 +344,7 @@ static struct qnetd_algorithm qnetd_algo_2nodelms = {
 	.client_disconnect		= qnetd_algo_2nodelms_client_disconnect,
 	.ask_for_vote_received		= qnetd_algo_2nodelms_ask_for_vote_received,
 	.vote_info_reply_received	= qnetd_algo_2nodelms_vote_info_reply_received,
+	.heuristics_change_received	= qnetd_algo_2nodelms_heuristics_change_received,
 	.timer_callback			= qnetd_algo_2nodelms_timer_callback,
 };
 

qdevices/qnetd-algo-2nodelms.h

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -51,7 +51,7 @@ extern enum tlv_reply_error_code	qnetd_algo_2nodelms_config_node_list_received(
 extern enum tlv_reply_error_code	qnetd_algo_2nodelms_membership_node_list_received(
     struct qnetd_client *client, uint32_t msg_seq_num,
     const struct tlv_ring_id *ring_id,
-    const struct node_list *nodes, enum tlv_vote *result_vote);
+    const struct node_list *nodes, enum tlv_heuristics heuristics, enum tlv_vote *result_vote);
 
 extern enum tlv_reply_error_code	qnetd_algo_2nodelms_quorum_node_list_received(
     struct qnetd_client *client, uint32_t msg_seq_num,
@@ -66,6 +66,10 @@ extern enum tlv_reply_error_code	qnetd_algo_2nodelms_ask_for_vote_received(
 extern enum tlv_reply_error_code	qnetd_algo_2nodelms_vote_info_reply_received(
     struct qnetd_client *client, uint32_t msg_seq_num);
 
+extern enum tlv_reply_error_code	qnetd_algo_2nodelms_heuristics_change_received(
+    struct qnetd_client *client, uint32_t msg_seq_num,
+    enum tlv_heuristics heuristics, enum tlv_vote *result_vote);
+
 extern enum tlv_reply_error_code	qnetd_algo_2nodelms_timer_callback(
     struct qnetd_client *client, int *reschedule_timer,
     int *send_vote, enum tlv_vote *result_vote);

qdevices/qnetd-algo-ffsplit.c

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -288,29 +288,42 @@ qnetd_algo_ffsplit_is_membership_stable(const struct qnetd_client *client, int c
 	return (1);
 }
 
-static size_t
-qnetd_algo_ffsplit_no_active_clients_in_partition(const struct qnetd_client *client,
-    const struct node_list *membership_node_list)
+static void
+qnetd_algo_ffsplit_get_active_clients_in_partition_stats(const struct qnetd_client *client,
+    const struct node_list *client_membership_node_list, enum tlv_heuristics client_heuristics,
+    size_t *no_clients, size_t *no_heuristics_pass, size_t *no_heuristics_fail)
 {
 	const struct node_list_entry *iter_node;
 	const struct qnetd_client *iter_client;
-	size_t res;
+	enum tlv_heuristics iter_heuristics;
 
-	res = 0;
+	*no_clients = 0;
+	*no_heuristics_pass = 0;
+	*no_heuristics_fail = 0;
 
-	if (client == NULL || membership_node_list == NULL) {
-		return (0);
+	if (client == NULL || client_membership_node_list == NULL) {
+		return ;
 	}
 
-	TAILQ_FOREACH(iter_node, membership_node_list, entries) {
+	TAILQ_FOREACH(iter_node, client_membership_node_list, entries) {
 		iter_client = qnetd_cluster_find_client_by_node_id(client->cluster,
 		    iter_node->node_id);
 		if (iter_client != NULL) {
-			res++;
+			(*no_clients)++;
+
+			if (iter_client == client) {
+				iter_heuristics = client_heuristics;
+			} else {
+				iter_heuristics = iter_client->last_heuristics;
+			}
+
+			if (iter_heuristics == TLV_HEURISTICS_PASS) {
+				(*no_heuristics_pass)++;
+			} else if (iter_heuristics == TLV_HEURISTICS_FAIL) {
+				(*no_heuristics_fail)++;
+			}
 		}
 	}
-
-	return (res);
 }
 
 /*
@@ -320,10 +333,16 @@ qnetd_algo_ffsplit_no_active_clients_in_partition(const struct qnetd_client *cli
 static int
 qnetd_algo_ffsplit_partition_cmp(const struct qnetd_client *client1,
     const struct node_list *config_node_list1, const struct node_list *membership_node_list1,
+    enum tlv_heuristics heuristics_1,
     const struct qnetd_client *client2,
-    const struct node_list *config_node_list2, const struct node_list *membership_node_list2)
+    const struct node_list *config_node_list2, const struct node_list *membership_node_list2,
+    enum tlv_heuristics heuristics_2)
 {
 	size_t part1_active_clients, part2_active_clients;
+	size_t part1_no_heuristics_pass, part2_no_heuristics_pass;
+	size_t part1_no_heuristics_fail, part2_no_heuristics_fail;
+	size_t part1_score, part2_score;
+
 	int res;
 
 	res = -1;
@@ -349,12 +368,37 @@ qnetd_algo_ffsplit_partition_cmp(const struct qnetd_client *client1,
 		 */
 
 		/*
-		 * Check how many active clients are in partitions
+		 * Check how many active clients are in partitions and heuristics results
+		 */
+		qnetd_algo_ffsplit_get_active_clients_in_partition_stats(client1,
+		    membership_node_list1, heuristics_1, &part1_active_clients,
+		    &part1_no_heuristics_pass, &part1_no_heuristics_fail);
+		qnetd_algo_ffsplit_get_active_clients_in_partition_stats(client2,
+		    membership_node_list2, heuristics_2, &part2_active_clients,
+		    &part2_no_heuristics_pass, &part2_no_heuristics_fail);
+
+		/*
+		 * Partition can contain clients with one of 4 states:
+		 * 1. Not-connected to qnetd (D)
+		 * 2. Disabled heuristics (U)
+		 * 3. Enabled heuristics with pass result (P)
+		 * 4. Enabled heuristics with fail result (F)
+		 *
+		 * The question is, what partition should get vote is kind of hard with
+		 * so much states. Following simple "score" seems to be good enough, but may
+		 * be suboptimal in some cases. As and example let's say there are
+		 * 2 partitions with 4 nodes each. Partition 1 looks like PDDD and partition 2 looks
+		 * like FUUU. Partition 1 score is 1 + (1 - 0), partition 2 score is 4 + (0 - 1).
+		 * Partition 2 wins eventho there is one processor with failed heuristics.
 		 */
-		part1_active_clients = qnetd_algo_ffsplit_no_active_clients_in_partition(
-		    client1, membership_node_list1);
-		part2_active_clients = qnetd_algo_ffsplit_no_active_clients_in_partition(
-		    client2, membership_node_list2);
+		part1_score = part1_active_clients + (part1_no_heuristics_pass - part1_no_heuristics_fail);
+		part2_score = part2_active_clients + (part2_no_heuristics_pass - part2_no_heuristics_fail);
+
+		if (part1_score > part2_score) {
+			res = 1; goto exit_res;
+		} else if (part1_score < part2_score) {
+			res = 0; goto exit_res;
+		}
 
 		if (part1_active_clients > part2_active_clients) {
 			res = 1; goto exit_res;
@@ -390,15 +434,18 @@ exit_res:
  */
 static const struct node_list *
 qnetd_algo_ffsplit_select_partition(const struct qnetd_client *client, int client_leaving,
-    const struct node_list *config_node_list, const struct node_list *membership_node_list)
+    const struct node_list *config_node_list, const struct node_list *membership_node_list,
+    enum tlv_heuristics client_heuristics)
 {
 	const struct qnetd_client *iter_client;
 	const struct qnetd_client *best_client;
 	const struct node_list *best_config_node_list, *best_membership_node_list;
 	const struct node_list *iter_config_node_list, *iter_membership_node_list;
+	enum tlv_heuristics iter_heuristics, best_heuristics;
 
 	best_client = NULL;
 	best_config_node_list = best_membership_node_list = NULL;
+	best_heuristics = TLV_HEURISTICS_UNDEFINED;
 
 	/*
 	 * Get highest score
@@ -411,17 +458,20 @@ qnetd_algo_ffsplit_select_partition(const struct qnetd_client *client, int clien
 
 			iter_config_node_list = config_node_list;
 			iter_membership_node_list = membership_node_list;
+			iter_heuristics = client_heuristics;
 		} else {
 			iter_config_node_list = &iter_client->configuration_node_list;
 			iter_membership_node_list = &iter_client->last_membership_node_list;
+			iter_heuristics = iter_client->last_heuristics;
 		}
 
 		if (qnetd_algo_ffsplit_partition_cmp(iter_client, iter_config_node_list,
-		    iter_membership_node_list, best_client, best_config_node_list,
-		    best_membership_node_list) > 0) {
+		    iter_membership_node_list, iter_heuristics, best_client, best_config_node_list,
+		    best_membership_node_list, best_heuristics) > 0) {
 			best_client = iter_client;
 			best_config_node_list = iter_config_node_list;
 			best_membership_node_list = iter_membership_node_list;
+			best_heuristics = iter_heuristics;
 		}
 	}
 
@@ -548,7 +598,7 @@ qnetd_algo_ffsplit_no_clients_in_sending_state(struct qnetd_client *client, int
 static enum tlv_vote
 qnetd_algo_ffsplit_do(struct qnetd_client *client, int client_leaving,
     const struct tlv_ring_id *ring_id, const struct node_list *config_node_list,
-    const struct node_list *membership_node_list)
+    const struct node_list *membership_node_list, enum tlv_heuristics client_heuristics)
 {
 	struct qnetd_algo_ffsplit_cluster_data *cluster_data;
 	const struct node_list *quorate_partition_node_list;
@@ -570,7 +620,7 @@ qnetd_algo_ffsplit_do(struct qnetd_client *client, int client_leaving,
 	qnetd_log(LOG_DEBUG, "ffsplit: Membership for cluster %s is now stable", client->cluster_name);
 
 	quorate_partition_node_list = qnetd_algo_ffsplit_select_partition(client, client_leaving,
-	    config_node_list, membership_node_list);
+	    config_node_list, membership_node_list, client_heuristics);
 	cluster_data->quorate_partition_node_list = quorate_partition_node_list;
 
 	if (quorate_partition_node_list == NULL) {
@@ -636,7 +686,7 @@ qnetd_algo_ffsplit_config_node_list_received(struct qnetd_client *client,
 		*result_vote = TLV_VOTE_ASK_LATER;
 	} else {
 		*result_vote = qnetd_algo_ffsplit_do(client, 0, &client->last_ring_id,
-		    nodes, &client->last_membership_node_list);
+		    nodes, &client->last_membership_node_list, client->last_heuristics);
 	}
 
 	return (TLV_REPLY_ERROR_CODE_NO_ERROR);
@@ -659,7 +709,7 @@ qnetd_algo_ffsplit_config_node_list_received(struct qnetd_client *client,
 enum tlv_reply_error_code
 qnetd_algo_ffsplit_membership_node_list_received(struct qnetd_client *client,
     uint32_t msg_seq_num, const struct tlv_ring_id *ring_id,
-    const struct node_list *nodes, enum tlv_vote *result_vote)
+    const struct node_list *nodes, enum tlv_heuristics heuristics, enum tlv_vote *result_vote)
 {
 
 	if (node_list_size(nodes) == 0) {
@@ -689,7 +739,7 @@ qnetd_algo_ffsplit_membership_node_list_received(struct qnetd_client *client,
 		*result_vote = TLV_VOTE_ASK_LATER;
 	} else {
 		*result_vote = qnetd_algo_ffsplit_do(client, 0, ring_id,
-		    &client->configuration_node_list, nodes);
+		    &client->configuration_node_list, nodes, heuristics);
 	}
 
 	return (TLV_REPLY_ERROR_CODE_NO_ERROR);
@@ -714,7 +764,8 @@ qnetd_algo_ffsplit_client_disconnect(struct qnetd_client *client, int server_goi
 {
 
 	(void)qnetd_algo_ffsplit_do(client, 1, &client->last_ring_id,
-	    &client->configuration_node_list, &client->last_membership_node_list);
+	    &client->configuration_node_list, &client->last_membership_node_list,
+	    client->last_heuristics);
 
 	free(client->algorithm_data);
 
@@ -787,6 +838,26 @@ qnetd_algo_ffsplit_vote_info_reply_received(struct qnetd_client *client, uint32_
 	return (TLV_REPLY_ERROR_CODE_NO_ERROR);
 }
 
+enum tlv_reply_error_code
+qnetd_algo_ffsplit_heuristics_change_received(struct qnetd_client *client, uint32_t msg_seq_num,
+    enum tlv_heuristics heuristics, enum tlv_vote *result_vote)
+{
+
+	if (node_list_size(&client->configuration_node_list) == 0 ||
+	    node_list_size(&client->last_membership_node_list) == 0) {
+		/*
+		 * Config or membership node list not received -> it's going to be sent later
+		 */
+		*result_vote = TLV_VOTE_ASK_LATER;
+	} else {
+		*result_vote = qnetd_algo_ffsplit_do(client, 0, &client->last_ring_id,
+		    &client->configuration_node_list, &client->last_membership_node_list,
+		    heuristics);
+	}
+
+	return (TLV_REPLY_ERROR_CODE_NO_ERROR);
+}
+
 enum tlv_reply_error_code
 qnetd_algo_ffsplit_timer_callback(struct qnetd_client *client, int *reschedule_timer,
     int *send_vote, enum tlv_vote *result_vote)
@@ -803,6 +874,7 @@ static struct qnetd_algorithm qnetd_algo_ffsplit = {
 	.client_disconnect		= qnetd_algo_ffsplit_client_disconnect,
 	.ask_for_vote_received		= qnetd_algo_ffsplit_ask_for_vote_received,
 	.vote_info_reply_received	= qnetd_algo_ffsplit_vote_info_reply_received,
+	.heuristics_change_received	= qnetd_algo_ffsplit_heuristics_change_received,
 	.timer_callback			= qnetd_algo_ffsplit_timer_callback,
 };
 

qdevices/qnetd-algo-ffsplit.h

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -51,7 +51,7 @@ extern enum tlv_reply_error_code	qnetd_algo_ffsplit_config_node_list_received(
 extern enum tlv_reply_error_code	qnetd_algo_ffsplit_membership_node_list_received(
     struct qnetd_client *client, uint32_t msg_seq_num,
     const struct tlv_ring_id *ring_id,
-    const struct node_list *nodes, enum tlv_vote *result_vote);
+    const struct node_list *nodes, enum tlv_heuristics heuristics, enum tlv_vote *result_vote);
 
 extern enum tlv_reply_error_code	qnetd_algo_ffsplit_quorum_node_list_received(
     struct qnetd_client *client, uint32_t msg_seq_num,
@@ -66,6 +66,10 @@ extern enum tlv_reply_error_code	qnetd_algo_ffsplit_ask_for_vote_received(
 extern enum tlv_reply_error_code	qnetd_algo_ffsplit_vote_info_reply_received(
     struct qnetd_client *client, uint32_t msg_seq_num);
 
+extern enum tlv_reply_error_code	qnetd_algo_ffsplit_heuristics_change_received(
+    struct qnetd_client *client, uint32_t msg_seq_num,
+    enum tlv_heuristics heuristics, enum tlv_vote *result_vote);
+
 extern enum tlv_reply_error_code	qnetd_algo_ffsplit_timer_callback(
     struct qnetd_client *client, int *reschedule_timer,
     int *send_vote, enum tlv_vote *result_vote);

qdevices/qnetd-algo-lms.c

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -71,6 +71,7 @@ static enum tlv_reply_error_code do_lms_algorithm(struct qnetd_client *client, c
 	struct qnetd_algo_lms_info *info = client->algorithm_data;
 	struct qnetd_algo_partition *cur_partition;
 	struct qnetd_algo_partition *largest_partition;
+	struct qnetd_algo_partition *best_score_partition;
 	const struct tlv_ring_id *ring_id = cur_ring_id;
 	int num_partitions;
 	int joint_leader;
@@ -131,7 +132,47 @@ static enum tlv_reply_error_code do_lms_algorithm(struct qnetd_client *client, c
 		}
 	}
 
-	/* Find the largest partition */
+	/*
+	 * Find the partition with highest score
+	 */
+	best_score_partition = NULL;
+	TAILQ_FOREACH(cur_partition, &info->partition_list, entries) {
+		if (!best_score_partition ||
+		    best_score_partition->score < cur_partition->score) {
+			best_score_partition = cur_partition;
+		}
+	}
+	qnetd_log(LOG_DEBUG, "algo-lms: best score partition is (" UTILS_PRI_RING_ID ") with score %d",
+		  best_score_partition->ring_id.node_id, best_score_partition->ring_id.seq, best_score_partition->score);
+
+	/* Now check if it's really the highest score, and not just the joint-highest */
+	joint_leader = 0;
+	TAILQ_FOREACH(cur_partition, &info->partition_list, entries) {
+		if (best_score_partition != cur_partition &&
+		    best_score_partition->score == cur_partition->score) {
+			joint_leader = 1;
+		}
+	}
+
+	if (!joint_leader) {
+		/* Partition with highest score is unique, allow us to run if we're in that partition. */
+		if (tlv_ring_id_eq(&best_score_partition->ring_id, ring_id)) {
+			qnetd_log(LOG_DEBUG, "algo-lms: We are in the best score partition. ACK");
+			*result_vote = info->last_result = TLV_VOTE_ACK;
+		}
+		else {
+			qnetd_log(LOG_DEBUG, "algo-lms: We are NOT in the best score partition. NACK");
+			*result_vote = info->last_result = TLV_VOTE_NACK;
+		}
+
+		qnetd_algo_free_partitions(&info->partition_list);
+
+		return (TLV_REPLY_ERROR_CODE_NO_ERROR);
+	}
+
+	/*
+	 * There are multiple partitions with same score. Find the largest partition
+	 */
 	largest_partition = NULL;
 	TAILQ_FOREACH(cur_partition, &info->partition_list, entries) {
 		if (!largest_partition ||
@@ -164,7 +205,7 @@ static enum tlv_reply_error_code do_lms_algorithm(struct qnetd_client *client, c
 		}
 	}
 	else {
-		int tb_node_id;
+		uint32_t tb_node_id;
 		struct tlv_ring_id tb_node_ring_id = {0LL, 0};
 
 		/* Look for the tie-breaker node */
@@ -283,7 +324,7 @@ qnetd_algo_lms_config_node_list_received(struct qnetd_client *client,
 enum tlv_reply_error_code
 qnetd_algo_lms_membership_node_list_received(struct qnetd_client *client,
     uint32_t msg_seq_num, const struct tlv_ring_id *ring_id,
-    const struct node_list *nodes, enum tlv_vote *result_vote)
+    const struct node_list *nodes, enum tlv_heuristics heuristics, enum tlv_vote *result_vote)
 {
 	qnetd_log(LOG_DEBUG, " ");
 	qnetd_log(LOG_DEBUG, "algo-lms: membership list from node %d partition (" UTILS_PRI_RING_ID ")", client->node_id, ring_id->node_id, ring_id->seq);
@@ -346,6 +387,18 @@ qnetd_algo_lms_vote_info_reply_received(struct qnetd_client *client, uint32_t ms
 	return (TLV_REPLY_ERROR_CODE_NO_ERROR);
 }
 
+enum tlv_reply_error_code
+qnetd_algo_lms_heuristics_change_received(struct qnetd_client *client, uint32_t msg_seq_num,
+    enum tlv_heuristics heuristics, enum tlv_vote *result_vote)
+{
+
+	qnetd_log(LOG_INFO, "algo-lms: heuristics change is not supported.");
+
+	*result_vote = TLV_VOTE_NO_CHANGE;
+
+	return (TLV_REPLY_ERROR_CODE_NO_ERROR);
+}
+
 enum tlv_reply_error_code
 qnetd_algo_lms_timer_callback(struct qnetd_client *client, int *reschedule_timer,
     int *send_vote, enum tlv_vote *result_vote)
@@ -384,6 +437,7 @@ static struct qnetd_algorithm qnetd_algo_lms = {
 	.client_disconnect		= qnetd_algo_lms_client_disconnect,
 	.ask_for_vote_received		= qnetd_algo_lms_ask_for_vote_received,
 	.vote_info_reply_received	= qnetd_algo_lms_vote_info_reply_received,
+	.heuristics_change_received	= qnetd_algo_lms_heuristics_change_received,
 	.timer_callback			= qnetd_algo_lms_timer_callback,
 };
 

qdevices/qnetd-algo-lms.h

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -51,7 +51,7 @@ extern enum tlv_reply_error_code	qnetd_algo_lms_config_node_list_received(
 extern enum tlv_reply_error_code	qnetd_algo_lms_membership_node_list_received(
     struct qnetd_client *client, uint32_t msg_seq_num,
     const struct tlv_ring_id *ring_id,
-    const struct node_list *nodes, enum tlv_vote *result_vote);
+    const struct node_list *nodes, enum tlv_heuristics heuristics, enum tlv_vote *result_vote);
 
 extern enum tlv_reply_error_code	qnetd_algo_lms_quorum_node_list_received(
     struct qnetd_client *client, uint32_t msg_seq_num,
@@ -66,6 +66,10 @@ extern enum tlv_reply_error_code	qnetd_algo_lms_ask_for_vote_received(
 extern enum tlv_reply_error_code	qnetd_algo_lms_vote_info_reply_received(
     struct qnetd_client *client, uint32_t msg_seq_num);
 
+extern enum tlv_reply_error_code	qnetd_algo_lms_heuristics_change_received(
+    struct qnetd_client *client, uint32_t msg_seq_num,
+    enum tlv_heuristics heuristics, enum tlv_vote *result_vote);
+
 extern enum tlv_reply_error_code	qnetd_algo_lms_timer_callback(
     struct qnetd_client *client, int *reschedule_timer,
     int *send_vote, enum tlv_vote *result_vote);

qdevices/qnetd-algo-test.c

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -132,6 +132,8 @@ qnetd_algo_test_config_node_list_received(struct qnetd_client *client,
  * All client fields are already set. Nodes is actual node list.
  * msg_seq_num is 32-bit number set by client.
  * ring_id is copied from client votequorum callback.
+ * heuristics is result of client heuristics (or TLV_HEURISTICS_UNDEFINED if heuristics
+ *  are disabled or not supported by client)
  *
  * Function has to return result_vote. This can be one of ack/nack, ask_later (client
  * should ask later for a vote) or wait_for_reply (client should wait for reply).
@@ -143,7 +145,7 @@ qnetd_algo_test_config_node_list_received(struct qnetd_client *client,
 enum tlv_reply_error_code
 qnetd_algo_test_membership_node_list_received(struct qnetd_client *client,
     uint32_t msg_seq_num, const struct tlv_ring_id *ring_id,
-    const struct node_list *nodes, enum tlv_vote *result_vote)
+    const struct node_list *nodes, enum tlv_heuristics heuristics, enum tlv_vote *result_vote)
 {
 
 	qnetd_log(LOG_INFO, "algo-test: membership_node_list_received");
@@ -225,6 +227,24 @@ qnetd_algo_test_vote_info_reply_received(struct qnetd_client *client, uint32_t m
 	return (TLV_REPLY_ERROR_CODE_NO_ERROR);
 }
 
+/*
+ * Called after client sent heuristics change message.
+ * heuristics is result of client regular heuristics (cannot be TLV_HEURISTICS_UNDEFINED)
+ * Variables client->last_regular_heuristics and client->last_heuristics are updated after
+ * the call.
+ */
+enum tlv_reply_error_code
+qnetd_algo_test_heuristics_change_received(struct qnetd_client *client, uint32_t msg_seq_num,
+    enum tlv_heuristics heuristics, enum tlv_vote *result_vote)
+{
+
+	qnetd_log(LOG_INFO, "algo-test: heuristics_change_received");
+
+	*result_vote = TLV_VOTE_NO_CHANGE;
+
+	return (TLV_REPLY_ERROR_CODE_NO_ERROR);
+}
+
 /*
  * Called as a result of qnetd_client_algo_timer_schedule function call after timeout expires.
  *
@@ -253,6 +273,7 @@ static struct qnetd_algorithm qnetd_algo_test = {
 	.client_disconnect		= qnetd_algo_test_client_disconnect,
 	.ask_for_vote_received		= qnetd_algo_test_ask_for_vote_received,
 	.vote_info_reply_received	= qnetd_algo_test_vote_info_reply_received,
+	.heuristics_change_received	= qnetd_algo_test_heuristics_change_received,
 	.timer_callback			= qnetd_algo_test_timer_callback,
 };
 

qdevices/qnetd-algo-test.h

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -51,7 +51,8 @@ extern enum tlv_reply_error_code	qnetd_algo_test_config_node_list_received(
 extern enum tlv_reply_error_code	qnetd_algo_test_membership_node_list_received(
     struct qnetd_client *client, uint32_t msg_seq_num,
     const struct tlv_ring_id *ring_id,
-    const struct node_list *nodes, enum tlv_vote *result_vote);
+    const struct node_list *nodes, enum tlv_heuristics heuristics,
+    enum tlv_vote *result_vote);
 
 extern enum tlv_reply_error_code	qnetd_algo_test_quorum_node_list_received(
     struct qnetd_client *client, uint32_t msg_seq_num,
@@ -66,6 +67,10 @@ extern enum tlv_reply_error_code	qnetd_algo_test_ask_for_vote_received(
 extern enum tlv_reply_error_code	qnetd_algo_test_vote_info_reply_received(
     struct qnetd_client *client, uint32_t msg_seq_num);
 
+extern enum tlv_reply_error_code	qnetd_algo_test_heuristics_change_received(
+    struct qnetd_client *client, uint32_t msg_seq_num,
+    enum tlv_heuristics heuristics, enum tlv_vote *result_vote);
+
 extern enum tlv_reply_error_code	qnetd_algo_test_timer_callback(
     struct qnetd_client *client, int *reschedule_timer,
     int *send_vote, enum tlv_vote *result_vote);

qdevices/qnetd-algo-utils.c

@@ -125,12 +125,23 @@ qnetd_algo_create_partitions(struct qnetd_client *client, partitions_list_t *par
 				return (-1);
 			}
 			partition->num_nodes = 0;
+			partition->score = 0;
 			memcpy(&partition->ring_id, &other_client->last_ring_id, sizeof(*ring_id));
 			num_partitions++;
 			TAILQ_INSERT_TAIL(partitions_list, partition, entries);
 		}
 		partition->num_nodes++;
 
+		/*
+		 * Score is computer similar way as in the ffsplit algorithm
+		 */
+		partition->score++;
+		if (other_client->last_heuristics == TLV_HEURISTICS_PASS) {
+			partition->score++;
+		} else if (other_client->last_heuristics == TLV_HEURISTICS_FAIL) {
+			partition->score--;
+		}
+
 	}
 
 	return (num_partitions);

qdevices/qnetd-algo-utils.h

@@ -42,6 +42,7 @@ extern "C" {
 struct qnetd_algo_partition {
 	struct tlv_ring_id ring_id;
 	int num_nodes;
+	int score;
 	TAILQ_ENTRY(qnetd_algo_partition) entries;
 };
 

qdevices/qnetd-algorithm.c

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -77,7 +77,7 @@ qnetd_algorithm_config_node_list_received(struct qnetd_client *client,
 enum tlv_reply_error_code
 qnetd_algorithm_membership_node_list_received(struct qnetd_client *client,
     uint32_t msg_seq_num, const struct tlv_ring_id *ring_id,
-    const struct node_list *nodes, enum tlv_vote *result_vote)
+    const struct node_list *nodes, enum tlv_heuristics heuristics, enum tlv_vote *result_vote)
 {
 
 	if (client->decision_algorithm >= QNETD_STATIC_SUPPORTED_DECISION_ALGORITHMS_SIZE ||
@@ -89,7 +89,7 @@ qnetd_algorithm_membership_node_list_received(struct qnetd_client *client,
 
 	return (qnetd_algorithm_array[client->decision_algorithm]->membership_node_list_received(
 		client, msg_seq_num,
-		ring_id, nodes, result_vote));
+		ring_id, nodes, heuristics, result_vote));
 }
 
 enum tlv_reply_error_code
@@ -154,6 +154,22 @@ qnetd_algorithm_vote_info_reply_received(struct qnetd_client *client, uint32_t m
 
 }
 
+enum tlv_reply_error_code
+qnetd_algorithm_heuristics_change_received(struct qnetd_client *client, uint32_t msg_seq_num,
+    enum tlv_heuristics heuristics, enum tlv_vote *result_vote)
+{
+
+	if (client->decision_algorithm >= QNETD_STATIC_SUPPORTED_DECISION_ALGORITHMS_SIZE ||
+	    qnetd_algorithm_array[client->decision_algorithm] == NULL) {
+		qnetd_log(LOG_CRIT, "qnetd_algorithm_ask_for_vote_received unhandled "
+		    "decision algorithm");
+		return (TLV_REPLY_ERROR_CODE_INTERNAL_ERROR);
+	}
+
+	return (qnetd_algorithm_array[client->decision_algorithm]->heuristics_change_received(
+		client, msg_seq_num, heuristics, result_vote));
+}
+
 enum tlv_reply_error_code
 qnetd_algorithm_timer_callback(struct qnetd_client *client, int *reschedule_timer,
     int *send_vote, enum tlv_vote *result_vote)

qdevices/qnetd-algorithm.h

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -54,7 +54,7 @@ extern enum tlv_reply_error_code	qnetd_algorithm_config_node_list_received(
 
 extern enum tlv_reply_error_code	qnetd_algorithm_membership_node_list_received(
     struct qnetd_client *client, uint32_t msg_seq_num, const struct tlv_ring_id *ring_id,
-    const struct node_list *nodes, enum tlv_vote *result_vote);
+     const struct node_list *nodes, enum tlv_heuristics heuristics, enum tlv_vote *result_vote);
 
 extern enum tlv_reply_error_code	qnetd_algorithm_quorum_node_list_received(
     struct qnetd_client *client, uint32_t msg_seq_num, enum tlv_quorate quorate,
@@ -69,6 +69,10 @@ extern enum tlv_reply_error_code	qnetd_algorithm_ask_for_vote_received(
 extern enum tlv_reply_error_code	qnetd_algorithm_vote_info_reply_received(
     struct qnetd_client *client, uint32_t msg_seq_num);
 
+extern enum tlv_reply_error_code	qnetd_algorithm_heuristics_change_received(
+    struct qnetd_client *client, uint32_t msg_seq_num, enum tlv_heuristics heuristics,
+    enum tlv_vote *result_vote);
+
 extern enum tlv_reply_error_code	qnetd_algorithm_timer_callback(
     struct qnetd_client *client, int *reschedule_timer, int *send_vote, enum tlv_vote *result_vote);
 
@@ -80,7 +84,7 @@ struct qnetd_algorithm {
 	enum tlv_reply_error_code (*membership_node_list_received)(
 	    struct qnetd_client *client, uint32_t msg_seq_num,
 	    const struct tlv_ring_id *ring_id,
-	    const struct node_list *nodes, enum tlv_vote *result_vote);
+	    const struct node_list *nodes, enum tlv_heuristics, enum tlv_vote *result_vote);
 
 	enum tlv_reply_error_code (*quorum_node_list_received)(
 	    struct qnetd_client *client, uint32_t msg_seq_num, enum tlv_quorate quorate,
@@ -97,6 +101,10 @@ struct qnetd_algorithm {
 	enum tlv_reply_error_code (*vote_info_reply_received)(struct qnetd_client *client,
 	    uint32_t msg_seq_num);
 
+	enum tlv_reply_error_code (*heuristics_change_received)(
+	    struct qnetd_client *client, uint32_t msg_seq_num, enum tlv_heuristics heuristics,
+	    enum tlv_vote *result_vote);
+
 	 enum tlv_reply_error_code (*timer_callback)(struct qnetd_client *client,
 	    int *reschedule_timer, int *send_vote, enum tlv_vote *result_vote);
 };

qdevices/qnetd-client-msg-received.c

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -490,7 +490,7 @@ qnetd_client_msg_received_init(struct qnetd_instance *instance, struct qnetd_cli
 	    instance->advanced_settings->max_client_receive_size,
 	    instance->advanced_settings->max_client_send_size,
 	    qnetd_static_supported_decision_algorithms,
-	    QNETD_STATIC_SUPPORTED_DECISION_ALGORITHMS_SIZE) == -1) {
+	    QNETD_STATIC_SUPPORTED_DECISION_ALGORITHMS_SIZE) == 0) {
 		qnetd_log(LOG_ERR, "Can't alloc init reply msg. Disconnecting client connection.");
 
 		send_buffer_list_discard_new(&client->send_buffer_list, send_buffer);
@@ -571,7 +571,7 @@ qnetd_client_msg_received_set_option(struct qnetd_instance *instance, struct qne
 	}
 
 	if (msg_create_set_option_reply(&send_buffer->buffer, msg->seq_number_set, msg->seq_number,
-	    client->heartbeat_interval) == -1) {
+	    client->heartbeat_interval) == 0) {
 		qnetd_log(LOG_ERR, "Can't alloc set option reply msg. "
 		    "Disconnecting client connection.");
 
@@ -624,7 +624,7 @@ qnetd_client_msg_received_echo_request(struct qnetd_instance *instance, struct q
 		return (-1);
 	}
 
-	if (msg_create_echo_reply(&send_buffer->buffer, msg_orig) == -1) {
+	if (msg_create_echo_reply(&send_buffer->buffer, msg_orig) == 0) {
 		qnetd_log(LOG_ERR, "Can't alloc echo reply msg. Disconnecting client connection.");
 
 		send_buffer_list_discard_new(&client->send_buffer_list, send_buffer);
@@ -721,10 +721,10 @@ qnetd_client_msg_received_node_list(struct qnetd_instance *instance, struct qnet
 		}
 
 		qnetd_log_debug_membership_node_list_received(client, msg->seq_number, &msg->ring_id,
-		    &msg->nodes);
+		    msg->heuristics, &msg->nodes);
 
 		reply_error_code = qnetd_algorithm_membership_node_list_received(client,
-		    msg->seq_number, &msg->ring_id, &msg->nodes, &result_vote);
+		    msg->seq_number, &msg->ring_id, &msg->nodes, msg->heuristics, &result_vote);
 		break;
 	case TLV_NODE_LIST_TYPE_QUORUM:
 		case_processed = 1;
@@ -801,6 +801,8 @@ qnetd_client_msg_received_node_list(struct qnetd_instance *instance, struct qnet
 			return (-1);
 		}
 		memcpy(&client->last_ring_id, &msg->ring_id, sizeof(struct tlv_ring_id));
+		client->last_membership_heuristics = msg->heuristics;
+		client->last_heuristics = msg->heuristics;
 		break;
 	case TLV_NODE_LIST_TYPE_QUORUM:
 		case_processed = 1;
@@ -841,7 +843,7 @@ qnetd_client_msg_received_node_list(struct qnetd_instance *instance, struct qnet
 	}
 
 	if (msg_create_node_list_reply(&send_buffer->buffer, msg->seq_number, msg->node_list_type,
-	    &client->last_ring_id, result_vote) == -1) {
+	    &client->last_ring_id, result_vote) == 0) {
 		qnetd_log(LOG_ERR, "Can't alloc node list reply msg. "
 		    "Disconnecting client connection.");
 
@@ -938,7 +940,7 @@ qnetd_client_msg_received_ask_for_vote(struct qnetd_instance *instance, struct q
 	}
 
 	if (msg_create_ask_for_vote_reply(&send_buffer->buffer, msg->seq_number,
-	    &client->last_ring_id, result_vote) == -1) {
+	    &client->last_ring_id, result_vote) == 0) {
 		qnetd_log(LOG_ERR, "Can't alloc ask for vote reply msg. "
 		    "Disconnecting client connection.");
 
@@ -1024,6 +1026,105 @@ qnetd_client_msg_received_vote_info_reply(struct qnetd_instance *instance,
 	return (0);
 }
 
+static int
+qnetd_client_msg_received_heuristics_change(struct qnetd_instance *instance, struct qnetd_client *client,
+    const struct msg_decoded *msg)
+{
+	int res;
+	struct send_buffer_list_entry *send_buffer;
+	enum tlv_reply_error_code reply_error_code;
+	enum tlv_vote result_vote;
+
+	reply_error_code = TLV_REPLY_ERROR_CODE_NO_ERROR;
+
+	if ((res = qnetd_client_msg_received_check_tls(instance, client, msg)) != 0) {
+		return (res == -1 ? -1 : 0);
+	}
+
+	if (!client->init_received) {
+		qnetd_log(LOG_ERR, "Received heuristics change message before init message. "
+		    "Sending error reply.");
+
+		if (qnetd_client_send_err(client, msg->seq_number_set, msg->seq_number,
+		    TLV_REPLY_ERROR_CODE_INIT_REQUIRED) != 0) {
+			return (-1);
+		}
+
+		return (0);
+	}
+
+	if (!msg->seq_number_set || msg->heuristics == TLV_HEURISTICS_UNDEFINED) {
+		qnetd_log(LOG_ERR, "Received heuristics change message without seq number set or "
+		    "with undefined heuristics. Sending error reply.");
+
+		if (qnetd_client_send_err(client, msg->seq_number_set, msg->seq_number,
+		    TLV_REPLY_ERROR_CODE_DOESNT_CONTAIN_REQUIRED_OPTION) != 0) {
+			return (-1);
+		}
+
+		return (0);
+	}
+
+	qnetd_log_debug_heuristics_change_received(client, msg->seq_number, msg->heuristics);
+
+	reply_error_code = qnetd_algorithm_heuristics_change_received(client, msg->seq_number,
+	    msg->heuristics, &result_vote);
+
+	if (reply_error_code != TLV_REPLY_ERROR_CODE_NO_ERROR) {
+		qnetd_log(LOG_ERR, "Algorithm returned error code. "
+		    "Sending error reply.");
+
+		if (qnetd_client_send_err(client, msg->seq_number_set, msg->seq_number,
+		    reply_error_code) != 0) {
+			return (-1);
+		}
+
+		return (0);
+	} else {
+		qnetd_log(LOG_DEBUG, "Algorithm result vote is %s", tlv_vote_to_str(result_vote));
+	}
+
+	/*
+	 * Store result vote and heuristics result
+	 */
+	client->last_sent_vote = result_vote;
+	if (result_vote == TLV_VOTE_ACK || result_vote == TLV_VOTE_NACK) {
+		client->last_sent_ack_nack_vote = result_vote;
+	}
+	client->last_regular_heuristics = msg->heuristics;
+	client->last_heuristics = msg->heuristics;
+
+	send_buffer = send_buffer_list_get_new(&client->send_buffer_list);
+	if (send_buffer == NULL) {
+		qnetd_log(LOG_ERR, "Can't alloc heuristics change reply msg from list. "
+		    "Disconnecting client connection.");
+
+		return (-1);
+	}
+
+	if (msg_create_heuristics_change_reply(&send_buffer->buffer, msg->seq_number,
+	    &client->last_ring_id, msg->heuristics, result_vote) == 0) {
+		qnetd_log(LOG_ERR, "Can't alloc heuristics change reply msg. "
+		    "Disconnecting client connection.");
+
+		send_buffer_list_discard_new(&client->send_buffer_list, send_buffer);
+
+		return (-1);
+	}
+
+	send_buffer_list_put(&client->send_buffer_list, send_buffer);
+
+	return (0);
+}
+
+static int
+qnetd_client_msg_received_heuristics_change_reply(struct qnetd_instance *instance, struct qnetd_client *client,
+    const struct msg_decoded *msg)
+{
+
+	return (qnetd_client_msg_received_unexpected_msg(client, msg, "heuristics change reply"));
+}
+
 int
 qnetd_client_msg_received(struct qnetd_instance *instance, struct qnetd_client *client)
 {
@@ -1121,6 +1222,15 @@ qnetd_client_msg_received(struct qnetd_instance *instance, struct qnetd_client *
 		msg_processed = 1;
 		ret_val = qnetd_client_msg_received_vote_info_reply(instance, client, &msg);
 		break;
+	case MSG_TYPE_HEURISTICS_CHANGE:
+		msg_processed = 1;
+		ret_val = qnetd_client_msg_received_heuristics_change(instance, client, &msg);
+		break;
+	case MSG_TYPE_HEURISTICS_CHANGE_REPLY:
+		msg_processed = 1;
+		ret_val = qnetd_client_msg_received_heuristics_change_reply(instance, client,
+		    &msg);
+		break;
 	/*
 	 * Default is not defined intentionally. Compiler shows warning when new
 	 * msg type is added.

qdevices/qnetd-client.h

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -87,6 +87,9 @@ struct qnetd_client {
 	uint32_t dpd_msg_received_since_last_check;
 	enum tlv_vote last_sent_vote;
 	enum tlv_vote last_sent_ack_nack_vote;
+	enum tlv_heuristics last_membership_heuristics; /* Passed in membership node list */
+	enum tlv_heuristics last_regular_heuristics; /* Passed in heuristics change callback */
+	enum tlv_heuristics last_heuristics; /* Latest heuristics both membership and regular */
 	TAILQ_ENTRY(qnetd_client) entries;
 	TAILQ_ENTRY(qnetd_client) cluster_entries;
 };

qdevices/qnetd-ipc-cmd.c

@@ -201,6 +201,25 @@ qnetd_ipc_cmd_list_add_client_info(const struct qnetd_client *client, struct dyn
 		}
 	}
 
+	if (client->last_heuristics != TLV_HEURISTICS_UNDEFINED || verbose) {
+		if (dynar_str_catf(outbuf, "        Heuristics:\t\t%s",
+		    tlv_heuristics_to_str(client->last_heuristics)) == -1) {
+			return (-1);
+		}
+
+		if (verbose) {
+			if (dynar_str_catf(outbuf, " (membership: %s, regular: %s)",
+			    tlv_heuristics_to_str(client->last_membership_heuristics),
+			    tlv_heuristics_to_str(client->last_regular_heuristics)) == -1) {
+				return (-1);
+			}
+		}
+
+		if (dynar_str_catf(outbuf, "\n") == -1) {
+			return (-1);
+		}
+	}
+
 	if (verbose) {
 		if (dynar_str_catf(outbuf, "        TLS active:\t\t%s",
 		    (client->tls_started ? "Yes" : "No")) == -1) {

qdevices/qnetd-log-debug.c

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -90,10 +90,10 @@ qnetd_log_debug_config_node_list_received(struct qnetd_client *client,
 	    "sent %s node list.", client->addr_str, client->cluster_name, client->node_id,
 	    (initial ? "initial" : "changed"));
 
-	qnetd_log(LOG_DEBUG, "  msg seq num "UTILS_PRI_MSG_SEQ, msg_seq_num);
+	qnetd_log(LOG_DEBUG, "  msg seq num = "UTILS_PRI_MSG_SEQ, msg_seq_num);
 
 	if (config_version_set) {
-		qnetd_log(LOG_DEBUG, "  config version " UTILS_PRI_CONFIG_VERSION, config_version);
+		qnetd_log(LOG_DEBUG, "  config version = " UTILS_PRI_CONFIG_VERSION, config_version);
 	}
 
 	qnetd_log_debug_dump_node_list(client, nodes);
@@ -102,15 +102,17 @@ qnetd_log_debug_config_node_list_received(struct qnetd_client *client,
 void
 qnetd_log_debug_membership_node_list_received(struct qnetd_client *client,
     uint32_t msg_seq_num, const struct tlv_ring_id *ring_id,
-    const struct node_list *nodes)
+    enum tlv_heuristics heuristics, const struct node_list *nodes)
 {
 	qnetd_log(LOG_DEBUG, "Client %s (cluster %s, node_id "UTILS_PRI_NODE_ID") "
 	    "sent membership node list.", client->addr_str, client->cluster_name, client->node_id);
 
-	qnetd_log(LOG_DEBUG, "  msg seq num "UTILS_PRI_MSG_SEQ, msg_seq_num);
+	qnetd_log(LOG_DEBUG, "  msg seq num = "UTILS_PRI_MSG_SEQ, msg_seq_num);
 
 	qnetd_log(LOG_DEBUG, "  ring id = (" UTILS_PRI_RING_ID ")", ring_id->node_id, ring_id->seq);
 
+	qnetd_log(LOG_DEBUG, "  heuristics = %s ", tlv_heuristics_to_str(heuristics));
+
 	qnetd_log_debug_dump_node_list(client, nodes);
 }
 
@@ -122,7 +124,7 @@ qnetd_log_debug_quorum_node_list_received(struct qnetd_client *client,
 	qnetd_log(LOG_DEBUG, "Client %s (cluster %s, node_id "UTILS_PRI_NODE_ID") "
 	    "sent quorum node list.", client->addr_str, client->cluster_name, client->node_id);
 
-	qnetd_log(LOG_DEBUG, "  msg seq num "UTILS_PRI_MSG_SEQ, msg_seq_num);
+	qnetd_log(LOG_DEBUG, "  msg seq num = "UTILS_PRI_MSG_SEQ, msg_seq_num);
 	qnetd_log(LOG_DEBUG, "  quorate = %u", quorate);
 
 	qnetd_log_debug_dump_node_list(client, nodes);
@@ -144,7 +146,7 @@ qnetd_log_debug_ask_for_vote_received(struct qnetd_client *client, uint32_t msg_
 
 	qnetd_log(LOG_DEBUG, "Client %s (cluster %s, node_id "UTILS_PRI_NODE_ID") "
 	    "asked for a vote", client->addr_str, client->cluster_name, client->node_id);
-	qnetd_log(LOG_DEBUG, "  msg seq num "UTILS_PRI_MSG_SEQ, msg_seq_num);
+	qnetd_log(LOG_DEBUG, "  msg seq num = "UTILS_PRI_MSG_SEQ, msg_seq_num);
 }
 
 void
@@ -154,7 +156,7 @@ qnetd_log_debug_vote_info_reply_received(struct qnetd_client *client, uint32_t m
 	qnetd_log(LOG_DEBUG, "Client %s (cluster %s, node_id "UTILS_PRI_NODE_ID") "
 	    "replied back to vote info message", client->addr_str, client->cluster_name,
 	    client->node_id);
-	qnetd_log(LOG_DEBUG, "  msg seq num "UTILS_PRI_MSG_SEQ, msg_seq_num);
+	qnetd_log(LOG_DEBUG, "  msg seq num = "UTILS_PRI_MSG_SEQ, msg_seq_num);
 }
 
 void
@@ -163,6 +165,17 @@ qnetd_log_debug_send_vote_info(struct qnetd_client *client, uint32_t msg_seq_num
 
 	qnetd_log(LOG_DEBUG, "Sending vote info to client %s (cluster %s, node_id "UTILS_PRI_NODE_ID") ",
 	    client->addr_str, client->cluster_name, client->node_id);
-	qnetd_log(LOG_DEBUG, "  msg seq num "UTILS_PRI_MSG_SEQ, msg_seq_num);
-	qnetd_log(LOG_DEBUG, "  vote %s", tlv_vote_to_str(vote));
+	qnetd_log(LOG_DEBUG, "  msg seq num = "UTILS_PRI_MSG_SEQ, msg_seq_num);
+	qnetd_log(LOG_DEBUG, "  vote = %s", tlv_vote_to_str(vote));
+}
+
+void
+qnetd_log_debug_heuristics_change_received(struct qnetd_client *client, uint32_t msg_seq_num,
+    enum tlv_heuristics heuristics)
+{
+
+	qnetd_log(LOG_DEBUG, "Client %s (cluster %s, node_id "UTILS_PRI_NODE_ID") "
+	    "sent heuristics change", client->addr_str, client->cluster_name, client->node_id);
+	qnetd_log(LOG_DEBUG, "  msg seq num = "UTILS_PRI_MSG_SEQ, msg_seq_num);
+	qnetd_log(LOG_DEBUG, "  heuristics = %s", tlv_heuristics_to_str(heuristics));
 }

qdevices/qnetd-log-debug.h

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -55,7 +55,7 @@ extern void		qnetd_log_debug_config_node_list_received(struct qnetd_client *clie
 
 extern void		qnetd_log_debug_membership_node_list_received(struct qnetd_client *client,
     uint32_t msg_seq_num, const struct tlv_ring_id *ring_id,
-    const struct node_list *nodes);
+    enum tlv_heuristics heuristics, const struct node_list *nodes);
 
 extern void		qnetd_log_debug_quorum_node_list_received(struct qnetd_client *client,
     uint32_t msg_seq_num, enum tlv_quorate quorate, const struct node_list *nodes);
@@ -72,6 +72,9 @@ extern void		qnetd_log_debug_vote_info_reply_received(struct qnetd_client *clien
 extern void		qnetd_log_debug_send_vote_info(struct qnetd_client *client,
     uint32_t msg_seq_num, enum tlv_vote vote);
 
+extern void		qnetd_log_debug_heuristics_change_received(struct qnetd_client *client,
+    uint32_t msg_seq_num, enum tlv_heuristics heuristics);
+
 #ifdef __cplusplus
 }
 #endif

qdevices/test-process-list.c

@@ -0,0 +1,517 @@
+/*
+ * Copyright (c) 2015-2017 Red Hat, Inc.
+ *
+ * All rights reserved.
+ *
+ * Author: Jan Friesse (jfriesse@redhat.com)
+ *
+ * This software licensed under BSD license, the text of which follows:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright notice,
+ *   this list of conditions and the following disclaimer.
+ * - Redistributions in binary form must reproduce the above copyright notice,
+ *   this list of conditions and the following disclaimer in the documentation
+ *   and/or other materials provided with the distribution.
+ * - Neither the name of the Red Hat, Inc. nor the names of its
+ *   contributors may be used to endorse or promote products derived from this
+ *   software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <stdio.h>
+#include <assert.h>
+#include <string.h>
+#include <poll.h>
+#include <signal.h>
+
+#include "process-list.h"
+
+static int no_executed;
+static int no_finished;
+
+static void
+signal_handlers_register(void)
+{
+	struct sigaction act;
+
+	act.sa_handler = SIG_DFL;
+	sigemptyset(&act.sa_mask);
+	act.sa_flags = SA_RESTART;
+
+	sigaction(SIGCHLD, &act, NULL);
+
+	act.sa_handler = SIG_IGN;
+	sigemptyset(&act.sa_mask);
+	act.sa_flags = SA_RESTART;
+
+	sigaction(SIGPIPE, &act, NULL);
+}
+
+static void
+plist_notify(enum process_list_notify_reason reason, const struct process_list_entry *entry,
+    void *user_data)
+{
+
+	assert(user_data == (void *)0x42);
+
+	switch (reason) {
+	case PROCESS_LIST_NOTIFY_REASON_EXECUTED:
+		no_executed++;
+		break;
+	case PROCESS_LIST_NOTIFY_REASON_FINISHED:
+		no_finished++;
+		break;
+	}
+}
+
+int
+main(void)
+{
+	struct process_list plist;
+	struct process_list_entry *plist_entry;
+	int i;
+	int timeout;
+	int no_repeats;
+
+	signal_handlers_register();
+
+	process_list_init(&plist, 10, 1, plist_notify, (void *)0x42);
+	plist_entry = process_list_add(&plist, "test name", "command");
+	assert(plist_entry != NULL);
+	assert(strcmp(plist_entry->name, "test name") == 0);
+	assert(plist_entry->state == PROCESS_LIST_ENTRY_STATE_INITIALIZED);
+	assert(plist_entry->exec_argc == 1);
+	assert(plist_entry->exec_argv[0] != NULL && strcmp(plist_entry->exec_argv[0], "command") == 0);
+	assert(plist_entry->exec_argv[1] == NULL);
+
+	plist_entry = process_list_add(&plist, "test name", "/bin/ping -c \"host wit\\\"h  space\"   notaspace");
+	assert(plist_entry != NULL);
+	assert(strcmp(plist_entry->name, "test name") == 0);
+	assert(plist_entry->state == PROCESS_LIST_ENTRY_STATE_INITIALIZED);
+	assert(plist_entry->exec_argc == 4);
+	assert(plist_entry->exec_argv[0] != NULL && strcmp(plist_entry->exec_argv[0], "/bin/ping") == 0);
+	assert(plist_entry->exec_argv[1] != NULL && strcmp(plist_entry->exec_argv[1], "-c") == 0);
+	assert(plist_entry->exec_argv[2] != NULL && strcmp(plist_entry->exec_argv[2], "host wit\"h  space") == 0);
+	assert(plist_entry->exec_argv[3] != NULL && strcmp(plist_entry->exec_argv[3], "notaspace") == 0);
+	assert(plist_entry->exec_argv[4] == NULL);
+
+	process_list_free(&plist);
+
+	/*
+	 * Test no process
+	 */
+	no_executed = 0;
+	no_finished = 0;
+	assert(process_list_exec_initialized(&plist) == 0);
+	assert(no_executed == 0);
+	assert(process_list_get_no_running(&plist) == 0);
+
+	/*
+	 * Wait to exit
+	 */
+	no_repeats = 10;
+	timeout = 1000 / no_repeats;
+	for (i = 0; i < no_repeats; i++) {
+		assert(process_list_waitpid(&plist) == 0);
+		if (process_list_get_no_running(&plist) > 0) {
+			poll(NULL, 0, timeout);
+		}
+	}
+
+	assert(process_list_waitpid(&plist) == 0);
+	assert(process_list_get_no_running(&plist) == 0);
+	assert(no_finished == 0);
+	assert(process_list_get_summary_result(&plist) == 0);
+	assert(process_list_get_summary_result_short(&plist) == 0);
+
+	process_list_free(&plist);
+
+	/*
+	 * Test two processes. /bin/true and /bin/false. Accumulated result should be fail
+	 */
+	plist_entry = process_list_add(&plist, "true", "/bin/true");
+	assert(plist_entry != NULL);
+
+	plist_entry = process_list_add(&plist, "false", "/bin/false");
+	assert(plist_entry != NULL);
+
+	no_executed = 0;
+	no_finished = 0;
+	assert(process_list_exec_initialized(&plist) == 0);
+	assert(no_executed == 2);
+	assert(process_list_get_no_running(&plist) == 2);
+
+	/*
+	 * Wait to exit
+	 */
+	no_repeats = 10;
+	timeout = 1000 / no_repeats;
+	for (i = 0; i < no_repeats; i++) {
+		assert(process_list_waitpid(&plist) == 0);
+		if (process_list_get_no_running(&plist) > 0) {
+			poll(NULL, 0, timeout);
+		}
+	}
+
+	assert(process_list_waitpid(&plist) == 0);
+	assert(process_list_get_no_running(&plist) == 0);
+	assert(no_finished == 2);
+	assert(process_list_get_summary_result(&plist) == 1);
+	assert(process_list_get_summary_result_short(&plist) == 1);
+
+	process_list_free(&plist);
+
+	/*
+	 * Test two processes. /bin/true and one non-existing. Accumulated result should be fail
+	 */
+	plist_entry = process_list_add(&plist, "true", "/bin/true");
+	assert(plist_entry != NULL);
+
+	plist_entry = process_list_add(&plist, "false", "/nonexistingdir/nonexistingfile");
+	assert(plist_entry != NULL);
+
+	no_executed = 0;
+	no_finished = 0;
+	assert(process_list_exec_initialized(&plist) == 0);
+	assert(no_executed == 2);
+	assert(process_list_get_no_running(&plist) == 2);
+
+	/*
+	 * Wait to exit
+	 */
+	no_repeats = 10;
+	timeout = 1000 / no_repeats;
+	for (i = 0; i < no_repeats; i++) {
+		assert(process_list_waitpid(&plist) == 0);
+		if (process_list_get_no_running(&plist) > 0) {
+			poll(NULL, 0, timeout);
+		}
+	}
+
+	assert(process_list_waitpid(&plist) == 0);
+	assert(process_list_get_no_running(&plist) == 0);
+	assert(no_finished == 2);
+	assert(process_list_get_summary_result(&plist) == 1);
+	assert(process_list_get_summary_result_short(&plist) == 1);
+
+	process_list_free(&plist);
+
+	/*
+	 * Test three processes /bin/true. Accumulated result should be success.
+	 */
+	plist_entry = process_list_add(&plist, "true", "/bin/true");
+	assert(plist_entry != NULL);
+
+	plist_entry = process_list_add(&plist, "true2", "/bin/true");
+	assert(plist_entry != NULL);
+
+	plist_entry = process_list_add(&plist, "true3", "/bin/true");
+	assert(plist_entry != NULL);
+
+	no_executed = 0;
+	no_finished = 0;
+	assert(process_list_exec_initialized(&plist) == 0);
+	assert(no_executed == 3);
+	assert(process_list_get_no_running(&plist) == 3);
+
+	/*
+	 * Wait to exit
+	 */
+	no_repeats = 10;
+	timeout = 1000 / no_repeats;
+	for (i = 0; i < no_repeats; i++) {
+		assert(process_list_waitpid(&plist) == 0);
+		if (process_list_get_no_running(&plist) > 0) {
+			poll(NULL, 0, timeout);
+		}
+	}
+
+	assert(process_list_waitpid(&plist) == 0);
+	assert(process_list_get_no_running(&plist) == 0);
+	assert(no_finished == 3);
+	assert(process_list_get_summary_result(&plist) == 0);
+	assert(process_list_get_summary_result_short(&plist) == 0);
+
+	process_list_free(&plist);
+
+	/*
+	 * Test two processes. /bin/true and cat. Waiting for maximum of 2 sec
+	 */
+	plist_entry = process_list_add(&plist, "true", "/bin/true");
+	assert(plist_entry != NULL);
+
+	plist_entry = process_list_add(&plist, "cat", "/bin/cat /dev/zero");
+	assert(plist_entry != NULL);
+
+	no_executed = 0;
+	no_finished = 0;
+	assert(process_list_exec_initialized(&plist) == 0);
+	assert(no_executed == 2);
+	assert(process_list_get_no_running(&plist) == 2);
+
+	poll(NULL, 0, 500);
+	assert(process_list_waitpid(&plist) == 0);
+	poll(NULL, 0, 500);
+	assert(process_list_waitpid(&plist) == 0);
+
+	assert(process_list_get_no_running(&plist) == 1);
+	assert(no_finished == 1);
+	assert(process_list_get_summary_result(&plist) == -1);
+	assert(process_list_get_summary_result_short(&plist) == -1);
+
+	process_list_move_active_entries_to_kill_list(&plist);
+	assert(process_list_process_kill_list(&plist) == 0);
+	poll(NULL, 0, 500);
+	assert(process_list_process_kill_list(&plist) == 0);
+	poll(NULL, 0, 500);
+	assert(process_list_waitpid(&plist) == 0);
+
+	assert(process_list_get_kill_list_items(&plist) == 0);
+
+	assert(process_list_process_kill_list(&plist) == 0);
+
+	process_list_free(&plist);
+
+	/*
+	 * Test two bash proceses. One ignores INT and second ignores INT and TERM. Waiting for maximum of 2 sec
+	 */
+	plist_entry = process_list_add(&plist, "ignoresig1", "bash -c \"trap 'echo trap' SIGINT;while true;do sleep 1;done\"");
+	assert(plist_entry != NULL);
+
+	plist_entry = process_list_add(&plist, "ignoresig2", "bash -c \"trap 'echo trap' SIGINT SIGTERM;while true;do sleep 1;done\"");
+	assert(plist_entry != NULL);
+
+	no_executed = 0;
+	no_finished = 0;
+	assert(process_list_exec_initialized(&plist) == 0);
+	assert(no_executed == 2);
+	assert(process_list_get_no_running(&plist) == 2);
+
+	poll(NULL, 0, 500);
+	assert(process_list_waitpid(&plist) == 0);
+
+	assert(process_list_get_no_running(&plist) == 2);
+	assert(no_finished == 0);
+	assert(process_list_get_summary_result(&plist) == -1);
+	assert(process_list_get_summary_result_short(&plist) == -1);
+
+	process_list_move_active_entries_to_kill_list(&plist);
+	assert(process_list_process_kill_list(&plist) == 0);
+	poll(NULL, 0, 500);
+	assert(process_list_waitpid(&plist) == 0);
+	assert(process_list_get_kill_list_items(&plist) == 1);
+
+	assert(process_list_process_kill_list(&plist) == 0);
+	poll(NULL, 0, 500);
+	assert(process_list_waitpid(&plist) == 0);
+	assert(process_list_get_kill_list_items(&plist) == 0);
+
+	process_list_free(&plist);
+
+	/*
+	 * Test 3 processes. Test if entries are properly deallocated
+	 */
+	process_list_init(&plist, 3, 1, plist_notify, (void *)0x42);
+	plist_entry = process_list_add(&plist, "true", "/bin/true");
+	assert(plist_entry != NULL);
+
+	plist_entry = process_list_add(&plist, "true2", "/bin/true");
+	assert(plist_entry != NULL);
+
+	plist_entry = process_list_add(&plist, "true3", "/bin/true");
+	assert(plist_entry != NULL);
+
+	plist_entry = process_list_add(&plist, "true4", "/bin/true");
+	assert(plist_entry == NULL);
+
+	no_executed = 0;
+	no_finished = 0;
+	assert(process_list_exec_initialized(&plist) == 0);
+	assert(no_executed == 3);
+	assert(process_list_get_no_running(&plist) == 3);
+
+	/*
+	 * Wait to exit
+	 */
+	no_repeats = 10;
+	timeout = 1000 / no_repeats;
+	for (i = 0; i < no_repeats; i++) {
+		assert(process_list_waitpid(&plist) == 0);
+		if (process_list_get_no_running(&plist) > 0) {
+			poll(NULL, 0, timeout);
+		}
+	}
+
+	assert(process_list_waitpid(&plist) == 0);
+	assert(process_list_get_no_running(&plist) == 0);
+	assert(no_finished == 3);
+	assert(process_list_get_summary_result(&plist) == 0);
+	assert(process_list_get_summary_result_short(&plist) == 0);
+
+	process_list_move_active_entries_to_kill_list(&plist);
+
+	plist_entry = process_list_add(&plist, "true", "/bin/true");
+	assert(plist_entry != NULL);
+
+	plist_entry = process_list_add(&plist, "ignoresig1", "bash -c \"trap 'echo trap' SIGINT;while true;do sleep 1;done\"");
+	assert(plist_entry != NULL);
+
+	plist_entry = process_list_add(&plist, "ignoresig2", "bash -c \"trap 'echo trap' SIGINT SIGTERM;while true;do sleep 1;done\"");
+	assert(plist_entry != NULL);
+
+	plist_entry = process_list_add(&plist, "true4", "/bin/true");
+	assert(plist_entry == NULL);
+
+	no_executed = 0;
+	no_finished = 0;
+	assert(process_list_exec_initialized(&plist) == 0);
+	assert(no_executed == 3);
+	assert(process_list_get_no_running(&plist) == 3);
+
+	poll(NULL, 0, 500);
+	assert(process_list_waitpid(&plist) == 0);
+
+	assert(process_list_get_no_running(&plist) == 2);
+	assert(no_finished == 1);
+	assert(process_list_get_summary_result(&plist) == -1);
+	assert(process_list_get_summary_result_short(&plist) == -1);
+
+	plist_entry = process_list_add(&plist, "true4", "/bin/true");
+	assert(plist_entry == NULL);
+
+	process_list_move_active_entries_to_kill_list(&plist);
+
+	plist_entry = process_list_add(&plist, "true4", "/bin/true");
+	assert(plist_entry != NULL);
+
+	plist_entry = process_list_add(&plist, "true5", "/bin/true");
+	assert(plist_entry == NULL);
+
+	assert(process_list_process_kill_list(&plist) == 0);
+	poll(NULL, 0, 500);
+	assert(process_list_waitpid(&plist) == 0);
+	assert(process_list_get_kill_list_items(&plist) == 1);
+
+	assert(process_list_process_kill_list(&plist) == 0);
+	poll(NULL, 0, 500);
+	assert(process_list_waitpid(&plist) == 0);
+	assert(process_list_get_kill_list_items(&plist) == 0);
+
+	process_list_move_active_entries_to_kill_list(&plist);
+	assert(process_list_get_summary_result(&plist) == 0);
+	assert(process_list_get_summary_result_short(&plist) == 0);
+
+	plist_entry = process_list_add(&plist, "true", "/bin/true");
+	assert(plist_entry != NULL);
+
+	plist_entry = process_list_add(&plist, "true2", "/bin/true");
+	assert(plist_entry != NULL);
+
+	plist_entry = process_list_add(&plist, "true3", "/bin/true");
+	assert(plist_entry != NULL);
+
+	plist_entry = process_list_add(&plist, "true4", "/bin/true");
+	assert(plist_entry == NULL);
+
+	process_list_free(&plist);
+
+	/*
+	 * Test 3 processes and difference between summary and short-circuit summary
+	 */
+	process_list_init(&plist, 3, 1, plist_notify, (void *)0x42);
+	plist_entry = process_list_add(&plist, "true", "/bin/true");
+	assert(plist_entry != NULL);
+
+	plist_entry = process_list_add(&plist, "false", "/bin/false");
+	assert(plist_entry != NULL);
+
+	plist_entry = process_list_add(&plist, "loop", "bash -c \"while true;do sleep 1;done\"");
+	assert(plist_entry != NULL);
+
+	plist_entry = process_list_add(&plist, "true4", "/bin/true");
+	assert(plist_entry == NULL);
+
+	no_executed = 0;
+	no_finished = 0;
+	assert(process_list_exec_initialized(&plist) == 0);
+	assert(no_executed == 3);
+	assert(process_list_get_no_running(&plist) == 3);
+
+	/*
+	 * Wait to exit
+	 */
+	no_repeats = 10;
+	timeout = 1000 / no_repeats;
+	for (i = 0; i < no_repeats; i++) {
+		assert(process_list_waitpid(&plist) == 0);
+		if (process_list_get_no_running(&plist) > 0) {
+			poll(NULL, 0, timeout);
+		}
+	}
+
+	assert(process_list_waitpid(&plist) == 0);
+	assert(process_list_get_no_running(&plist) == 1);
+	assert(no_finished == 2);
+	assert(process_list_get_summary_result(&plist) == -1);
+	assert(process_list_get_summary_result_short(&plist) == 1);
+
+	process_list_move_active_entries_to_kill_list(&plist);
+	assert(process_list_process_kill_list(&plist) == 0);
+	poll(NULL, 0, 500);
+	assert(process_list_waitpid(&plist) == 0);
+	assert(process_list_get_kill_list_items(&plist) == 0);
+
+	process_list_free(&plist);
+
+	/*
+	 * Test process_list_killall by running two bash proceses.
+	 * One ignores INT and second ignores INT and TERM. Waiting for maximum of 2 sec
+	 */
+	plist_entry = process_list_add(&plist, "ignoresig1", "bash -c \"trap 'echo trap' SIGINT;while true;do sleep 1;done\"");
+	assert(plist_entry != NULL);
+
+	plist_entry = process_list_add(&plist, "ignoresig2", "bash -c \"trap 'echo trap' SIGINT SIGTERM;while true;do sleep 1;done\"");
+	assert(plist_entry != NULL);
+
+	no_executed = 0;
+	no_finished = 0;
+	assert(process_list_exec_initialized(&plist) == 0);
+	assert(no_executed == 2);
+	assert(process_list_get_no_running(&plist) == 2);
+
+	poll(NULL, 0, 500);
+	assert(process_list_waitpid(&plist) == 0);
+
+	assert(process_list_get_no_running(&plist) == 2);
+	assert(no_finished == 0);
+	assert(process_list_get_summary_result(&plist) == -1);
+	assert(process_list_get_summary_result_short(&plist) == -1);
+
+	assert(process_list_killall(&plist, 2000) == 0);
+	assert(process_list_get_kill_list_items(&plist) == 0);
+
+	process_list_free(&plist);
+
+	/*
+	 * Empty killall exits with sucess result
+	 */
+	assert(process_list_killall(&plist, 2000) == 0);
+
+	process_list_free(&plist);
+
+	return (0);
+}

qdevices/timer-list.c

@@ -203,6 +203,21 @@ timer_list_time_to_expire(struct timer_list *tlist)
 	return (timer_list_entry_time_to_expire(entry, PR_IntervalNow()));
 }
 
+uint32_t
+timer_list_time_to_expire_ms(struct timer_list *tlist)
+{
+	struct timer_list_entry *entry;
+	uint32_t u32;
+
+	entry = TAILQ_FIRST(&tlist->list);
+	if (entry == NULL) {
+		u32 = ~((uint32_t)0);
+		return (u32);
+	}
+
+	return (PR_IntervalToMilliseconds(timer_list_entry_time_to_expire(entry, PR_IntervalNow())));
+}
+
 void
 timer_list_delete(struct timer_list *tlist, struct timer_list_entry *entry)
 {

qdevices/timer-list.h

@@ -85,6 +85,8 @@ extern void				 timer_list_expire(struct timer_list *tlist);
 
 extern PRIntervalTime			 timer_list_time_to_expire(struct timer_list *tlist);
 
+extern uint32_t				 timer_list_time_to_expire_ms(struct timer_list *tlist);
+
 extern void				 timer_list_free(struct timer_list *tlist);
 
 #ifdef __cplusplus

qdevices/tlv.c

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -35,6 +35,7 @@
 #include <sys/types.h>
 #include <arpa/inet.h>
 
+#include <assert.h>
 #include <inttypes.h>
 #include <stdlib.h>
 #include <string.h>
@@ -55,7 +56,7 @@
 #define TLV_TYPE_LENGTH		2
 #define TLV_LENGTH_LENGTH	2
 
-#define TLV_STATIC_SUPPORTED_OPTIONS_SIZE	22
+#define TLV_STATIC_SUPPORTED_OPTIONS_SIZE	23
 
 enum tlv_opt_type tlv_static_supported_options[TLV_STATIC_SUPPORTED_OPTIONS_SIZE] = {
     TLV_OPT_MSG_SEQ_NUMBER,
@@ -80,6 +81,7 @@ enum tlv_opt_type tlv_static_supported_options[TLV_STATIC_SUPPORTED_OPTIONS_SIZE
     TLV_OPT_VOTE,
     TLV_OPT_QUORATE,
     TLV_OPT_TIE_BREAKER,
+    TLV_OPT_HEURISTICS,
 };
 
 int
@@ -406,6 +408,17 @@ tlv_add_quorate(struct dynar *msg, enum tlv_quorate quorate)
 	return (tlv_add_u8(msg, TLV_OPT_QUORATE, quorate));
 }
 
+int
+tlv_add_heuristics(struct dynar *msg, enum tlv_heuristics heuristics)
+{
+
+	if (heuristics == TLV_HEURISTICS_UNDEFINED) {
+		return (-1);
+	}
+
+	return (tlv_add_u8(msg, TLV_OPT_HEURISTICS, heuristics));
+}
+
 void
 tlv_iter_init_str(const char *msg, size_t msg_len, size_t msg_header_len,
     struct tlv_iterator *tlv_iter)
@@ -953,6 +966,28 @@ tlv_iter_decode_quorate(struct tlv_iterator *tlv_iter, enum tlv_quorate *quorate
 	return (0);
 }
 
+int
+tlv_iter_decode_heuristics(struct tlv_iterator *tlv_iter, enum tlv_heuristics *heuristics)
+{
+	uint8_t u8;
+	enum tlv_heuristics tmp_heuristics;
+
+	if (tlv_iter_decode_u8(tlv_iter, &u8) != 0) {
+		return (-1);
+	}
+
+	tmp_heuristics = u8;
+
+	if (tmp_heuristics != TLV_HEURISTICS_PASS &&
+	    tmp_heuristics != TLV_HEURISTICS_FAIL) {
+		return (-4);
+	}
+
+	*heuristics = tmp_heuristics;
+
+	return (0);
+}
+
 void
 tlv_get_supported_options(enum tlv_opt_type **supported_options, size_t *no_supported_options)
 {
@@ -1035,3 +1070,52 @@ tlv_decision_algorithm_type_to_str(enum tlv_decision_algorithm_type algorithm)
 
 	return ("Unknown algorithm");
 }
+
+const char *
+tlv_heuristics_to_str(enum tlv_heuristics heuristics)
+{
+
+	switch (heuristics) {
+	case TLV_HEURISTICS_UNDEFINED: return ("Undefined"); break;
+	case TLV_HEURISTICS_PASS: return ("Pass"); break;
+	case TLV_HEURISTICS_FAIL: return ("Fail"); break;
+	}
+
+	return ("Unknown heuristics type");
+}
+
+int
+tlv_heuristics_cmp(enum tlv_heuristics h1, enum tlv_heuristics h2)
+{
+	int res;
+
+	res = -2;
+
+	switch (h1) {
+	case TLV_HEURISTICS_UNDEFINED:
+		switch (h2) {
+		case TLV_HEURISTICS_UNDEFINED: res = 0; break;
+		case TLV_HEURISTICS_PASS: res = -1; break;
+		case TLV_HEURISTICS_FAIL: res = 1; break;
+		}
+		break;
+	case TLV_HEURISTICS_PASS:
+		switch (h2) {
+		case TLV_HEURISTICS_UNDEFINED: res = 1; break;
+		case TLV_HEURISTICS_PASS: res = 0; break;
+		case TLV_HEURISTICS_FAIL: res = 1; break;
+		}
+		break;
+	case TLV_HEURISTICS_FAIL:
+		switch (h2) {
+		case TLV_HEURISTICS_UNDEFINED: res = -1; break;
+		case TLV_HEURISTICS_PASS: res = -1; break;
+		case TLV_HEURISTICS_FAIL: res = 0; break;
+		}
+		break;
+	}
+
+	assert(res == -1 || res == 0 || res == 1);
+
+	return (res);
+}

qdevices/tlv.h

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2015-2016 Red Hat, Inc.
+ * Copyright (c) 2015-2017 Red Hat, Inc.
  *
  * All rights reserved.
  *
@@ -67,6 +67,7 @@ enum tlv_opt_type {
 	TLV_OPT_VOTE = 19,
 	TLV_OPT_QUORATE = 20,
 	TLV_OPT_TIE_BREAKER = 21,
+	TLV_OPT_HEURISTICS = 22,
 };
 
 enum tlv_tls_supported {
@@ -155,6 +156,12 @@ struct tlv_node_info {
 	enum tlv_node_state node_state;		/* TLV_NODE_STATE_NOT_SET - state was not set */
 };
 
+enum tlv_heuristics {
+	TLV_HEURISTICS_UNDEFINED = 0,
+	TLV_HEURISTICS_PASS = 1,
+	TLV_HEURISTICS_FAIL = 2,
+};
+
 struct tlv_iterator {
 	const char *msg;
 	size_t msg_len;
@@ -243,6 +250,9 @@ extern int			 tlv_add_vote(struct dynar *msg, enum tlv_vote vote);
 
 extern int			 tlv_add_quorate(struct dynar *msg, enum tlv_quorate quorate);
 
+extern int			 tlv_add_heuristics(struct dynar *msg,
+    enum tlv_heuristics heuristics);
+
 extern void			 tlv_iter_init_str(const char *msg, size_t msg_len,
     size_t msg_header_len, struct tlv_iterator *tlv_iter);
 
@@ -315,6 +325,9 @@ extern int			 tlv_iter_decode_vote(struct tlv_iterator *tlv_iter,
 extern int			 tlv_iter_decode_quorate(struct tlv_iterator *tlv_iter,
     enum tlv_quorate *quorate);
 
+extern int			 tlv_iter_decode_heuristics(struct tlv_iterator *tlv_iter,
+    enum tlv_heuristics *heuristics);
+
 extern void			 tlv_get_supported_options(enum tlv_opt_type **supported_options,
     size_t *no_supported_options);
 
@@ -324,15 +337,22 @@ extern int			 tlv_ring_id_eq(const struct tlv_ring_id *rid1,
 extern int			 tlv_tie_breaker_eq(const struct tlv_tie_breaker *tb1,
     const struct tlv_tie_breaker *tb2);
 
-extern const char *		 tlv_vote_to_str(enum tlv_vote vote);
+extern const char		*tlv_vote_to_str(enum tlv_vote vote);
 
-extern const char *		 tlv_node_state_to_str(enum tlv_node_state state);
+extern const char		*tlv_node_state_to_str(enum tlv_node_state state);
 
-extern const char *		 tlv_tls_supported_to_str(enum tlv_tls_supported tls_supported);
+extern const char		*tlv_tls_supported_to_str(enum tlv_tls_supported tls_supported);
 
-extern const char *		 tlv_decision_algorithm_type_to_str(
+extern const char		*tlv_decision_algorithm_type_to_str(
     enum tlv_decision_algorithm_type algorithm);
 
+extern const char		*tlv_heuristics_to_str(enum tlv_heuristics heuristics);
+
+/*
+ * Compare h1 and h2. Return -1 if h1 < h2, 0 if h1 == h2 and 1 if h1 > h2
+ */
+extern int			 tlv_heuristics_cmp(enum tlv_heuristics h1, enum tlv_heuristics h2);
+
 #ifdef __cplusplus
 }
 #endif

qdevices/unix-socket-client.c

@@ -38,7 +38,7 @@
 #include "unix-socket-client.h"
 #include "unix-socket.h"
 
-#define UNIX_SOCKET_CLIENT_BUFFER	1024
+#define UNIX_SOCKET_CLIENT_BUFFER	256
 
 void
 unix_socket_client_init(struct unix_socket_client *client, int sock, size_t max_receive_size,
@@ -98,7 +98,7 @@ unix_socket_client_io_read(struct unix_socket_client *client)
 			goto exit_err;
 		}
 
-		for (zi = 0; zi < readed; zi++) {
+		for (zi = 0; zi < (size_t)readed; zi++) {
 			if (buf[zi] == '\n') {
 				res = 1;
 			}

qdevices/unix-socket.c

@@ -40,25 +40,7 @@
 #include <unistd.h>
 
 #include "unix-socket.h"
-
-static int
-unix_socket_set_non_blocking(int fd)
-{
-	int flags;
-
-	flags = fcntl(fd, F_GETFL, NULL);
-
-	if (flags < 0) {
-		return (-1);
-	}
-
-	flags |= O_NONBLOCK;
-	if (fcntl(fd, F_SETFL, flags) < 0) {
-		return (-1);
-	}
-
-	return (0);
-}
+#include "utils.h"
 
 int
 unix_socket_server_create(const char *path, int non_blocking, int backlog)
@@ -87,7 +69,7 @@ unix_socket_server_create(const char *path, int non_blocking, int backlog)
 	}
 
 	if (non_blocking) {
-		if (unix_socket_set_non_blocking(s) != 0) {
+		if (utils_fd_set_non_blocking(s) != 0) {
 			close(s);
 
 			return (-1);
@@ -124,7 +106,7 @@ unix_socket_client_create(const char *path, int non_blocking)
 	strncpy(sun.sun_path, path, strlen(path));
 
 	if (non_blocking) {
-		if (unix_socket_set_non_blocking(s) != 0) {
+		if (utils_fd_set_non_blocking(s) != 0) {
 			close(s);
 
 			return (-1);
@@ -173,7 +155,7 @@ unix_socket_server_accept(int sock, int non_blocking)
 	}
 
 	if (non_blocking) {
-		if (unix_socket_set_non_blocking(client_sock) != 0) {
+		if (utils_fd_set_non_blocking(client_sock) != 0) {
 			close(client_sock);
 
 			return (-1);