rpc.statd is single-threaded and runs its HA callout synchronously. If
it is too slow then latency accumulates and rpc.statd's backlog grows.

Running a pair of add-client/del-client events with the current code
averages ~0.030s in my test environment.  This mean that 1000 clients
reclaiming locks after failover can easily cause 10s of latency.  This
could cause rpc.statd to become unresponsive, resulting in a time out
for an rpcinfo-based health check of the status service.

Split the add-client/del-client events out to a standalone
statd_callout executable, written in C, to be used as the HA callout
for rpc.statd.  All other functions move to statd_callout_helper.
Now, running a pair of add-client/del-client events in my test
environment averages only ~0.002s.  This seems less likely to cause
latency problems.

The standalone statd_callout executable needs to read a configuration
file, which is generated by statd_callout_helper from the "startup"
event.  It also needs access to a list of currently assigned public
IPs.

For backward compatibility, during installation a symlink is created
from $CTDB_BASE/statd-callout to the new statd_callout, which is
installed in the helper directory.

Testing this as part of the eventscript unit tests starts to become
even more of a hack than it used to be.  However, the dependency on
stubs and the corresponding setup of fake state makes it hard to move
this elsewhere.

Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Tue Jun 25 04:24:57 UTC 2024 on atb-devel-224

Signed-off-by: Martin Schwenke <mschwenke@ddn.com>
Reviewed-by: Andreas Schneider <asn@samba.org>

Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Wed Jul  5 12:16:57 UTC 2023 on atb-devel-224

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

A problem can only occur if /etc/ctdb/ or an important subdirectory is
removed, which means the script itself would not be found.  Use && to
silence ShellCheck.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Mostly avoidance of quoting warnings.

Silencing warnings about unquoted $CTDB_TEST_CAT_RESULTS_OPTS is
handled by passing '-' to cat when that variable's value is empty.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Apart from the non-constant sourcing of include files.

Mostly avoidance of quoting warnings.

One subtle change is to simply pass "120" to wait_until_ready() to
stop warnings that it expects arguments but none are passed (both
SC2119 and SC2120).  There seems no way to indicate to structure
function argument handling so that shellcheck realises arguments are
optional.  In later shellcheck versions, disabling SC2120 for a
function also silences complaints about its callers... but not all of
our testing uses "later" shellcheck versions.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

ctdb_test_skip() will exit.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Avoid:

  .../UNIT/shellcheck/scripts/local.sh: line 14: type: shellcheck: not found

The "type" command in dash prints the "not found" message to stdout
but the bash version prints to stderr, so redirect stderr too.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Fix missing Reviewed-by: tag.

This reverts commit 847aa0e367c721944650aa34d67f8073461ae272.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>