path: root/ctdb/doc/ctdb.7.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE refentry
	PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN"
	"http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd">
<refentry id="ctdb.7">

<refmeta>
	<refentrytitle>ctdb</refentrytitle>
	<manvolnum>7</manvolnum>
	<refmiscinfo class="source">ctdb</refmiscinfo>
	<refmiscinfo class="manual">CTDB - clustered TDB database</refmiscinfo>
</refmeta>


<refnamediv>
	<refname>ctdb</refname>
        <refpurpose>Clustered TDB</refpurpose>
</refnamediv>

<refsect1>
  <title>DESCRIPTION</title>

  <para>
    CTDB is a clustered database component in clustered Samba that
    provides a high-availability load-sharing CIFS server cluster.
  </para>

  <para>
    The main functions of CTDB are:
  </para>

  <itemizedlist>
    <listitem>
      <para>
	Provide a clustered version of the TDB database with automatic
	rebuild/recovery of the databases upon node failures.
      </para>
    </listitem>

    <listitem>
      <para>
      Monitor nodes in the cluster and services running on each node.
      </para>
    </listitem>

    <listitem>
      <para>
	Manage a pool of public IP addresses that are used to provide
	services to clients.  Alternatively, CTDB can be used with
	LVS.
      </para>
    </listitem>
  </itemizedlist>

  <para>
    Combined with a cluster filesystem CTDB provides a full
    high-availablity (HA) environment for services such as clustered
    Samba, NFS and other services.
  </para>

  <para>
    In addition to the CTDB manual pages there is much more
    information available at
    <ulink url="https://wiki.samba.org/index.php/CTDB_and_Clustered_Samba"/>.
  </para>
</refsect1>

<refsect1>
  <title>ANATOMY OF A CTDB CLUSTER</title>

  <para>
    A CTDB cluster is a collection of nodes with 2 or more network
    interfaces.  All nodes provide network (usually file/NAS) services
    to clients.  Data served by file services is stored on shared
    storage (usually a cluster filesystem) that is accessible by all
    nodes.
  </para>
  <para>
    CTDB provides an "all active" cluster, where services are load
    balanced across all nodes.
  </para>
</refsect1>

  <refsect1>
    <title>Cluster leader</title>

    <para>
      CTDB uses a <emphasis>cluster leader and follower</emphasis>
      model of cluster management.  All nodes in a cluster elect one
      node to be the leader.  The leader node coordinates privileged
      operations such as database recovery and IP address failover.
    </para>

    <para>
      CTDB previously referred to the leader as the <emphasis>recovery
      master</emphasis> or <emphasis>recmaster</emphasis>.  References
      to these terms may still be found in documentation and code.
    </para>
  </refsect1>

  <refsect1>
    <title>Cluster Lock</title>

    <para>
      CTDB uses a cluster lock to assert its privileged role in the
      cluster.  This node takes the cluster lock when it becomes
      leader and holds the lock until it is no longer leader.  The
      <emphasis>cluster lock</emphasis> helps CTDB to avoid a
      <emphasis>split brain</emphasis>, where a cluster becomes
      partitioned and each partition attempts to operate
      independently.  Issues that can result from a split brain
      include file data corruption, because file locking metadata may
      not be tracked correctly.
    </para>

    <para>
      CTDB previously referred to the cluster lock as the
      <emphasis>recovery lock</emphasis>.  The abbreviation
      <emphasis>reclock</emphasis> is still used - just "clock" would
      be confusing.
    </para>

    <para>
      <emphasis>CTDB is unable configure a default cluster
      lock</emphasis>, because this would depend on factors such as
      cluster filesystem mountpoints.  However, <emphasis>running CTDB
      without a cluster lock is not recommended</emphasis> as there
      will be no split brain protection.
    </para>

    <para>
      When a cluster lock is configured it is used as the election
      mechanism.  Nodes race to take the cluster lock and the winner
      is the cluster leader.  This avoids problems when a node wins an
      election but is unable to take the lock - this can occur if a
      cluster becomes partitioned (for example, due to a communication
      failure) and a different leader is elected by the nodes in each
      partition, or if the cluster filesystem has a high failover
      latency.
    </para>

    <para>
      By default, the cluster lock is implemented using a file
      (specified by <parameter>cluster lock</parameter> in the
      <literal>[cluster]</literal> section of
      <citerefentry><refentrytitle>ctdb.conf</refentrytitle>
      <manvolnum>5</manvolnum></citerefentry>) residing in shared
      storage (usually) on a cluster filesystem.  To support a
      cluster lock the cluster filesystem must support lock
      coherence.  See
      <citerefentry><refentrytitle>ping_pong</refentrytitle>
      <manvolnum>1</manvolnum></citerefentry> for more details.
    </para>

    <para>
      The cluster lock can also be implemented using an arbitrary
      cluster mutex helper (or call-out).  This is indicated by using
      an exclamation point ('!') as the first character of the
      <parameter>cluster lock</parameter> parameter.  For example, a
      value of <command>!/usr/local/bin/myhelper cluster</command>
      would run the given helper with the specified arguments.  The
      helper will continue to run as long as it holds its mutex.  See
      <filename>ctdb/doc/cluster_mutex_helper.txt</filename> in the
      source tree, and related code, for clues about writing helpers.
    </para>

    <para>
      When a file is specified for the <parameter>cluster
      lock</parameter> parameter (i.e. no leading '!') the file lock
      is implemented by a default helper
      (<command>/usr/local/libexec/ctdb/ctdb_mutex_fcntl_helper</command>).
      This helper has arguments as follows:

      <!-- cmdsynopsis would not require long line but does not work :-( -->
      <synopsis>
<command>ctdb_mutex_fcntl_helper</command> <parameter>FILE</parameter> <optional><parameter>RECHECK-INTERVAL</parameter></optional>
      </synopsis>

      <command>ctdb_mutex_fcntl_helper</command> will take a lock on
      FILE and then check every RECHECK-INTERVAL seconds to ensure
      that FILE still exists and that its inode number is unchanged
      from when the lock was taken.  The default value for
      RECHECK-INTERVAL is 5.
    </para>

    <para>
      CTDB does sanity checks to ensure that the cluster lock is held
      as expected.
    </para>
  </refsect1>

  <refsect1>
    <title>Private vs Public addresses</title>

    <para>
      Each node in a CTDB cluster has multiple IP addresses assigned
      to it:

      <itemizedlist>
	<listitem>
	  <para>
	    A single private IP address that is used for communication
	    between nodes.
	  </para>
	</listitem>
	<listitem>
	  <para>
	    One or more public IP addresses that are used to provide
	    NAS or other services.
	  </para>
	</listitem>
      </itemizedlist>
    </para>

    <refsect2>
      <title>Private address</title>

      <para>
        Each node is configured with a unique, permanently assigned
        private address.  This address is configured by the operating
        system.  This address uniquely identifies a physical node in
        the cluster and is the address that CTDB daemons will use to
        communicate with the CTDB daemons on other nodes.
      </para>

      <para>
	Private addresses are listed in the file
	<filename>/usr/local/etc/ctdb/nodes</filename>).  This file
	contains the list of private addresses for all nodes in the
	cluster, one per line. This file must be the same on all nodes
	in the cluster.
      </para>

      <para>
	Some users like to put this configuration file in their
	cluster filesystem.  A symbolic link or the
	<parameter>nodes list</parameter> parameter can be used.
      </para>

      <para>
	Private addresses should not be used by clients to connect to
	services provided by the cluster.
      </para>

      <para>
	Note that CTDB is not resilient to private network interface
	failures.  CTDB will notice when a node becomes disconnected
	and will exclude it from the cluster.  Standard mechanisms
	such as bonding can be used to improve resilience of the
	private network.
      </para>

      <para>
        It is strongly recommended that the private addresses are
        configured on a private network that is separate from client
        networks.  This is because the CTDB protocol is both
        unauthenticated and unencrypted.  If clients share the private
        network then steps need to be taken to stop injection of
        packets to relevant ports on the private addresses.  It is
        also likely that CTDB protocol traffic between nodes could
        leak sensitive information if it can be intercepted.
      </para>

      <para>
	Example <filename>/usr/local/etc/ctdb/nodes</filename> for a four node
	cluster:
      </para>
      <screen format="linespecific">
192.168.1.1
192.168.1.2
192.168.1.3
192.168.1.4
      </screen>
    </refsect2>

    <refsect2>
      <title>Public addresses</title>

      <para>
	Public addresses are used to provide services to clients.
	Public addresses are not configured at the operating system
	level and are not permanently associated with a particular
	node.  Instead, they are managed by CTDB and are assigned to
	interfaces on physical nodes at runtime.
      </para>
      <para>
        The CTDB cluster will assign/reassign these public addresses
        across the available healthy nodes in the cluster. When one
        node fails, its public addresses will be taken over by one or
        more other nodes in the cluster.  This ensures that services
        provided by all public addresses are always available to
        clients, as long as there are nodes available capable of
        hosting each address.
      </para>

      <para>
	Note that public addresses must be on different networks to
	private addresses.  CTDB uses the private network to
	coordinate failover of public addresses.  This means that
	public addresses can not be failed over if the private network
	is not operational.
      </para>

      <para>
	The public address configuration is stored in
	<filename>/usr/local/etc/ctdb/public_addresses</filename> on
	each node.  This file contains a list of the public addresses
	that the node is capable of hosting, one per line.  Each entry
	also contains the netmask and the interface to which the
	address should be assigned.  If this file is missing then no
	public addresses are configured.
      </para>

      <para>
	Some users who have the same public addresses on all nodes
	like to put this configuration file in their cluster
	filesystem.  A symbolic link should be used in this case.
      </para>

      <para>
	Example <filename>/usr/local/etc/ctdb/public_addresses</filename> for a
	node that can host 4 public addresses, on 2 different
	interfaces:
      </para>
      <screen format="linespecific">
10.1.1.1/24 eth1
10.1.1.2/24 eth1
10.1.2.1/24 eth2
10.1.2.2/24 eth2
      </screen>

      <para>
	In many cases the public addresses file will be the same on
	all nodes.  However, it is possible to use different public
	address configurations on different nodes.
      </para>

      <para>
	Example: 4 nodes partitioned into two subgroups:
      </para>
      <screen format="linespecific">
Node 0:/usr/local/etc/ctdb/public_addresses
	10.1.1.1/24 eth1
	10.1.1.2/24 eth1

Node 1:/usr/local/etc/ctdb/public_addresses
	10.1.1.1/24 eth1
	10.1.1.2/24 eth1

Node 2:/usr/local/etc/ctdb/public_addresses
	10.1.2.1/24 eth2
	10.1.2.2/24 eth2

Node 3:/usr/local/etc/ctdb/public_addresses
	10.1.2.1/24 eth2
	10.1.2.2/24 eth2
      </screen>
      <para>
	In this example nodes 0 and 1 host two public addresses on the
	10.1.1.x network while nodes 2 and 3 host two public addresses
	for the 10.1.2.x network.
      </para>
      <para>
	Public address 10.1.1.1 can be hosted by either of nodes 0 or
	1 and will be available to clients as long as at least one of
	these two nodes are available.
      </para>
      <para>
	If both nodes 0 and 1 become unavailable then public address
	10.1.1.1 also becomes unavailable. 10.1.1.1 can not be failed
	over to nodes 2 or 3 since these nodes do not have this public
	address configured.
      </para>
      <para>
        The <command>ctdb ip</command> command can be used to view the
        current assignment of public addresses to physical nodes.
      </para>
    </refsect2>
  </refsect1>


  <refsect1>
    <title>Node status</title>

    <para>
      The current status of each node in the cluster can be viewed by the 
      <command>ctdb status</command> command.
    </para>

    <para>
      A node can be in one of the following states:
    </para>

    <variablelist>
      <varlistentry>
	<term>OK</term>
	<listitem>
	  <para>
	    This node is healthy and fully functional.  It hosts public
	    addresses to provide services.
	  </para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<term>DISCONNECTED</term>
	<listitem>
	  <para>
	    This node is not reachable by other nodes via the private
	    network.  It is not currently participating in the cluster.
	    It <emphasis>does not</emphasis> host public addresses to
	    provide services.  It might be shut down.
	  </para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<term>DISABLED</term>
	<listitem>
	  <para>
	    This node has been administratively disabled. This node is
	    partially functional and participates in the cluster.
	    However, it <emphasis>does not</emphasis> host public
	    addresses to provide services.
	  </para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<term>UNHEALTHY</term>
	<listitem>
	  <para>
	    A service provided by this node has failed a health check
	    and should be investigated.  This node is partially
	    functional and participates in the cluster.  However, it
	    <emphasis>does not</emphasis> host public addresses to
	    provide services.  Unhealthy nodes should be investigated
	    and may require an administrative action to rectify.
	  </para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<term>BANNED</term>
	<listitem>
	  <para>
	    CTDB is not behaving as designed on this node.  For example,
	    it may have failed too many recovery attempts.  Such nodes
	    are banned from participating in the cluster for a
	    configurable time period before they attempt to rejoin the
	    cluster.  A banned node <emphasis>does not</emphasis> host
	    public addresses to provide services.  All banned nodes
	    should be investigated and may require an administrative
	    action to rectify.
	  </para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<term>STOPPED</term>
	<listitem>
	  <para>
	    This node has been administratively exclude from the
	    cluster.  A stopped node does no participate in the cluster
	    and <emphasis>does not</emphasis> host public addresses to
	    provide services.  This state can be used while performing
	    maintenance on a node.
	  </para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<term>PARTIALLYONLINE</term>
	<listitem>
	  <para>
	    A node that is partially online participates in a cluster
	    like a healthy (OK) node.  Some interfaces to serve public
	    addresses are down, but at least one interface is up.  See
	    also <command>ctdb ifaces</command>.
	  </para>
	</listitem>
      </varlistentry>

    </variablelist>
  </refsect1>

  <refsect1>
    <title>CAPABILITIES</title>

    <para>
      Cluster nodes can have several different capabilities enabled.
      These are listed below.
    </para>

    <variablelist>

      <varlistentry>
	<term>LEADER</term>
	<listitem>
	  <para>
	    Indicates that a node can become the CTDB cluster leader.
	    The current leader is decided via an
	    election held by all active nodes with this capability.
	  </para>
	  <para>
	    Default is YES.
	  </para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<term>LMASTER</term>
	<listitem>
	  <para>
	    Indicates that a node can be the location master (LMASTER)
	    for database records.  The LMASTER always knows which node
	    has the latest copy of a record in a volatile database.
	  </para>
	  <para>
	    Default is YES.
	  </para>
	</listitem>
      </varlistentry>

    </variablelist>

    <para>
      The LEADER and LMASTER capabilities can be disabled when CTDB
      is used to create a cluster spanning across WAN links. In this
      case CTDB acts as a WAN accelerator.
    </para>

  </refsect1>

  <refsect1>
    <title>LVS</title>

    <para>