<!DOCTYPE book [
<!ENTITY % tnt SYSTEM "../tnt.ent">
%tnt;
]>
<chapter xmlns="http://docbook.org/ns/docbook" version="5.0"
         xmlns:xlink="http://www.w3.org/1999/xlink"
         xml:id="replication">

<title>Replication</title>
<blockquote><para>
  To set up replication, it's necessary to prepare the master,
  configure a replica, and establish procedures for recovery from
  a degraded state.
</para></blockquote>

<section xml:id="replication-architecture">
    <title>Replication architecture</title>
  <para>
    A replica gets all updates from the master by continuously
    fetching and applying its write ahead log (WAL).
    Each record in the WAL represents a single Tarantool
    command such as INSERT or UPDATE or DELETE, and is assigned
    a monotonically growing log sequence number (LSN).
    In essence, Tarantool replication is row-based:
    all data change commands are fully deterministic and operate
    on a single record.
  </para>
  <para>
    A stored program invocation
    <!-- , unless requested explicitly, -->
    does not enter the Write Ahead Log. Instead, log events
    for actual UPDATEs and DELETEs, performed by the Lua code,
    are written to the log. This ensures that possible
    non-determinism of Lua does not cause replication
    to go out of sync.
  </para>
<!--
  <para>
    It is still sometimes necessary to replicate stored program
    CALLs, rather than their effects: for example, when the
    procedure is fully deterministic and CALL representation in
    the WAL is known to be significantly more compact. Another
    example would be when a procedure is written to do one thing
    on the master, and another on a replica. BOX_RPL_STMT
    flag of the binary protocol can  be used 
    to replicate CALLs as statements.
  </para>
-->
  <para>
    For replication to work correctly, the latest LSN
    on the replica must match or fall behind the latest LSN
    on the master. If the replica had its own updates,
    this would lead to it getting out of sync, since
    updates from the master having identical LSNs would
    not be applied. In fact, if replication is ON, Tarantool
    does not accept updates, even on its <olink
    targetptr="primary_port"/>.
  </para>
</section>

<section xml:id="setting-up-the-master">
  <title>Setting up the master</title>
  <para>
    To prepare the master for connections from the replica, it's only
    necessary to enable <olink targetptr="replication_port"/> in
    the configuration file. An example configuration file can be
    found in <link
    xlink:href="http://github.com/tarantool/tarantool/blob/master/test/replication/cfg/master.cfg"><filename>test/replication/cfg/master.cfg</filename></link>. A master with enabled replication_port can accept connections
    from as many replicas as necessary on that port. Each replica
    has its own replication state.
  </para>
</section>
<section xml:id="settin-up-a-replica">
  <title>Setting up a replica</title>
  <para>
    A server, whether master or replica, always requires a valid
    snapshot file to boot from. For a master, a snapshot file is usually
    prepared with with the <olink targetptr="init-storage-option"/> option.
    For a replica, it's usually copied from the master.
  </para>
  <para>
    To start replication, configure <olink
    targetptr="replication_source"/>.
    Other parameters can also be changed, but existing spaces and
    their primary keys on the replica must be identical to the ones on the
    master.
  </para>
  <para>
    Once connected to the master, the replica requests all changes
    that happened after the latest local LSN. It is therefore
    necessary to keep WAL files on the master host as long as
    there are replicas that haven't applied them yet. An example
    configuration can be found in <link
    xlink:href="https://github.com/tarantool/tarantool/blob/master/test/replication/cfg/replica.cfg"><filename>test/replication/cfg/replica.cfg</filename></link>.
  </para>
  <para>
    If required WAL files are absent, a replica can be "re-seeded" at
    any time with a newer snapshot file, manually copied from the
    master.
  </para>
  <note><simpara>
    Replication parameters are "dynamic", which allows the
    replica to become a master and vice versa with the help of the
    <olink targetptr="reload-configuration"/> statement.
  </simpara></note>

</section>
<section xml:id="recovering-from-a-degraded-state">
  <title>Recovering from a degraded state</title>
  <para>
    "Degraded state" is a situation when the master becomes
    unavailable -- due to hardware or network failure, or due to a
    programming bug. There is no reliable way for a replica to detect
    that the master is gone for good, since sources of failure and
    replication environments vary significantly.
  </para>
  <para>
    A separate monitoring script (or scripts, if a decision-making
    quorum is desirable) is necessary to detect a master failure.
    Such a script would typically try to update a tuple in an
    auxiliary space on the master, and raise an alarm if a
    network or disk error persists for longer than is acceptable.
  </para>
  <para>
    When a master failure is detected, the following needs
    to be done:
    <itemizedlist>
      <listitem>
        <para>
          First and foremost, make sure that the master does not
          accept updates. This is necessary to prevent the
          situation when, should the master failure end up being
          transient, some updates still go to the master, while
          others already end up on the replica.
        </para>
        <para>
          If the master is available, the easiest way to turn
          on read-only mode is to turn Tarantool into a replica of
          itself. This can be done by setting the master's <olink
          targetptr="replication_source"/> to point to self.
        </para>
        <para>
          If the master is not available, best bet is to log into
          the machine and kill the server, or change the
          machine's network configuration (DNS, IP address).
        </para>
        <para>
          If the machine is not available, it's perhaps prudent
          to power it off.
        </para>
      </listitem>
      <listitem>
        <para>
          Record the replica's LSN, by issuing <olink
          targetptr="show-info"/>. This LSN may prove useful if
          there are updates on the master that never reached
          the replica.
        </para>
      </listitem>
      <listitem>
        <para>
          Propagate the replica to become a master. This is done
          by setting <olink targetptr="replication_source"/>
          on replica to an empty string.
        </para>
      </listitem>
      <listitem>
        <para>
          Change the application configuration to point to the new
          master. This can be done either by changing the
          application's internal routing table, or by setting up
          the old master's IP address on the new master's machine, or
          using some other approach.
        </para>
      </listitem>
      <listitem>
        <para>
          Recover the old master. If there are updates that didn't
          make it to the new master, they have to be applied
          manually. You can use the Tarantool command line client
          to read the server log files.
        </para>
      </listitem>
    </itemizedlist>
  </para>
</section>

</chapter>

<!--
vim: tw=66 syntax=docbk
vim: spell spelllang=en_us
-->
