<!DOCTYPE book [
<!ENTITY % tnt SYSTEM "../tnt.ent">
%tnt;
]>
<chapter xmlns="http://docbook.org/ns/docbook" version="5.0"
         xmlns:xlink="http://www.w3.org/1999/xlink"
         xmlns:xi="http://www.w3.org/2001/XInclude"
         xml:id="language-reference">

<title>Language reference</title>
<blockquote><para>
  This chapter provides a reference of Tarantool data operations
  and administrative commands.
</para></blockquote>
<bridgehead renderas="sect2">Digression: data and administrative ports</bridgehead>
<para>
  Unlike many other key/value servers, Tarantool uses
  different TCP ports and client/server protocols for data
  manipulation and administrative statements.
  During start up, the server can connect to up to five TCP ports:
  <itemizedlist>
    <listitem><para>
      Read/write data port, to handle INSERTs, UPDATEs,
      DELETEs, SELECTs and CALLs. This port speaks the
      <link xlink:href="https://github.com/tarantool/tarantool/blob/master/doc/box-protocol.txt" xlink:title="A complete BNF of Tarantool client/server protocol">native Tarantool protocol</link>, and provides full data access.
      </para>
      <para>
      The default value of the port is <literal>33013</literal>,
      as defined in <olink targetptr="primary_port"/>
      configuration option.
    </para></listitem>
    <listitem><para>
      Read only port, which only accepts SELECTs and CALLs,
      default  port number <literal>33014</literal>, as defined in
      <olink targetptr="secondary_port"/> configuration option.
    </para></listitem>
    <listitem><para>
      Administrative port, which defaults to <literal>33015</literal>,
      and is defined in <olink targetptr="admin_port"/>
      configuration option.
    </para></listitem>
    <listitem><para>Replication port (see <olink
        targetptr="replication_port"/>), by default set to
        <literal>33016</literal>, used to send updates to
        replicas. Replication is optional, and if this port is not
        set in the configuration file, the corresponding server process
        is not started.
      </para></listitem>
    <listitem><para>Memcached port. Optional, read-write data port
      that speaks Memcached text protocol. This port is off by default.
    </para></listitem>
  </itemizedlist>
  In absence of authentication, this approach allows system
  administrators to restrict access to read/write or
  administrative ports. The client, however, has to be aware of
  the separation, and <command>tarantool</command> command line
  client automatically selects the correct port for you
  with help of a simple regular expression. SELECTs, UPDATEs,
  INSERTs, DELETEs and CALLs are sent to the primary port.
  SHOW, RELOAD, SAVE and other statements are sent to the administrative port.
</para>

<section xml:id="data-manipulation">
  <title>Data manipulation</title>

  <para>
    Five basic request types are supported: INSERT, UPDATE,
    DELETE, SELECT and CALL. All requests, including
    INSERT, UPDATE and DELETE may return data. A SELECT can be
    requested to limit the number of returned tuples. This is
    useful when searching in a non-unique index or when 
    a special <quote>wildcard</quote> (zero-length string) value
    is supplied as search key or a key part.
 </para>
  <para>
    UPDATE statement supports operations on fields &mdash;
    assignment, arithmetic operations (the field must be numeric),
    cutting and pasting fragments of a field, &mdash; as well as
    operations on a tuple: push and pop of a field at the tail of
    a tuple, deletion and insertion of a field.  Multiple
    operations can be combined into a single update, and in this
    case they are performed atomically. Each operation expects
    field number as its first argument. When a sequence of changes
    is present, field identifier in each operation is assumed to
    be relative to the most recent state of the tuple, i.e. as if
    all previous operations in a multi-operation update have
    already been applied. In other words, it's always safe to
    merge multiple UPDATE statements into a single one, with no
    change in semantics.
  </para>
  <para>Tarantool protocol was designed with focus on asynchronous
    I/O and easy integration with proxies. Each client
    request starts with a 12-byte binary header, containing three
    fields: request type, length, and a numeric id.
  </para>
 <para>
    The mandatory length, present in request header simplifies
    client or proxy I/O.  A response to a request is sent to the
    client as soon as it is ready. It always carries in its header
    the same type and id as in the request. The id makes it
    possible to match a request to a response, even if the latter
    arrived out of order.
  </para>
  <para>Request type defines the format of the payload.
    INSERTs, UPDATEs and DELETEs can only be made by the primary key, so
    an index id and a key (possibly multipart) are always present
    in these requests. SELECTs can use secondary keys. UPDATE
    only needs to list the fields that are actually changed. With
    this one exception, all commands operate on whole tuple(s).
  </para>
  <para>Unless implementing a client driver, one needn't
    concern oneself with the complications of the binary
    protocol. <olink targetptr="connectors">Language-specific
    drivers</olink> provide a friendly way to store domain
    language data structures in Tarantool, and the command line
    client supports a subset of standard SQL.
    A complete description of both, the binary protocol and
    the supported SQL, is maintained in annotated Backus-Naur
    form in the source tree: please see
    <link xlink:href="https://github.com/tarantool/tarantool/blob/master/doc/box-protocol.txt"><filename>doc/box-protocol.txt</filename></link>
    and
    <link xlink:href="https://github.com/tarantool/tarantool/blob/master/doc/sql.txt"><filename>doc/sql.txt</filename></link> respectively.
  </para>
  <section xml:id="memcached-protocol">
    <title>Memcached protocol</title>
      <para>If full access to Tarantool functionality is not needed,
      or there is no readily available connector for the
      programming language in use, any existing client driver for
      Memcached will make do as a Tarantool connector.
      To enable text Memcached protocol, turn on
      <olink targetptr="memcached_port"/> in the configuration file.
      Since Memcached has no notion of spaces or secondary
      indexes, this port only makes it possible to access one
      dedicated space (see <olink
      targetptr="memcached_space"/>) via its primary key.
      Unless tuple expiration is enabled with <olink
      targetptr="memcached_expire"/>, TTL part of the message is
      stored but ignored.
     </para>
     <para>
       Notice, that <olink targetptr="memcached_space"/> is also
       accessible using the primary port or Lua. A common use
       of the Memcached port in Tarantool is when a Memcached default
       expiration algorithm is insufficient, and a custom Lua
       expiration procedure is used.
     </para>
     <para>
      Tarantool does not support the binary protocol of Memcached.
      If top performance is a must, Tarantool's own binary
      protocol should be used.
     </para>
  </section>
</section>

<section xml:id="administrative-console">
  <title>Administrative console</title>
  <para>
    The administrative console uses a simple text protocol.
    All commands are case-insensitive.
    You can connect to the administrative port using any
    <command>telnet</command> client, or a tool like
    <command>rlwrap</command>, if access to readline features is
    desired. Additionally, <command>tarantool</command>, the
    SQL-capable command line client,
    understands all administrative statements
    and automatically directs them to the administrative port.
    The server response to an administrative command, even though
    it is always in plain text, can be quite complex.
    It is encoded using YAML markup to simplify automated parsing.
  </para>
  <para>To learn about all supported administrative commands, you
    can type <emphasis role="tntadmin">help</emphasis> in the
    administrative console. A reference description also follows
    below:</para>

  <variablelist>

    <varlistentry>
      <term xml:id="save-snapshot" xreflabel="SAVE SNAPSHOT">
        <emphasis role="tntadmin">save snapshot</emphasis>
      </term>
      <listitem><para>
        Take a snapshot of all data and store it in
        <filename><olink
        targetptr="snap_dir"/>/&lt;latest-lsn&gt;.snap</filename>.
        To take a snapshot, Tarantool first enters the delayed
        garbage collection mode for all data. In this mode,
        tuples which were allocated before the snapshot has
        started are not freed until the snapshot has finished.
        To preserve consistency of the primary key, used to
        iterate over tuples, a copy-on-write technique is employed.
        If the master process changes part of a primary key, 
        the corresponding process page is split, and the snapshot
        process obtains an old copy of the page. Since a
        snapshot is written sequentially, one can expect a very
        high write performance (averaging to 80MB/second on modern
        disks), which means an average database instance gets
        saved in a matter of minutes.  Note, that as long as there
        are any changes to the parent index memory through concurrent
        updates, there are going to be page splits, and therefore
        one needs to have some extra free memory to run this
        command. 10% of <olink targetptr="slab_alloc_arena"/>
        is, on average, sufficient. This statement waits until a
        snapshot is taken and returns operation result. For
        example:
<programlisting>localhost> show info
---
info:
  version: "1.4.6"
  lsn: 843301
...
localhost> save snapshot
---
ok
...
localhost> save snapshot
---
fail: can't save snapshot, errno 17 (File exists)
...
</programlisting>
      </para>
      <para>
        Taking a snapshot does not cause the server to start a new
        write ahead log. Once a snapshot is taken, old WALs can be
        deleted as long as all replicas are up to date. But the
        WAL which was current at the time <emphasis
        role="tntadmin">save snapshot</emphasis> started must be
        kept for recovery, since it still contains log records
        written after the start of <emphasis role="tntadmin">save
        snapshot</emphasis>.
      </para>
      <para>
        An alternative way to save a snapshot is to send the server
        SIGUSR1 UNIX signal. While this approach could be handy, it
        is not recommended for use in automation: a signal provides no
        way to find out whether the snapshot was taken successfully
        or not.
      </para>
      </listitem>
    </varlistentry>


    <varlistentry>
      <term xml:id="reload-configuration" xreflabel="RELOAD CONFIGURATION">
        <emphasis role="tntadmin">reload configuration</emphasis>
      </term>
      <listitem><para>
        Re-read the configuration file. If the file contains
        changes to dynamic parameters, update the runtime
        settings. If configuration syntax is incorrect, or a
        read-only parameter is changed, produce an error and do
        nothing.
      </para></listitem>
    </varlistentry>

    <varlistentry>
      <term xml:id="show-configuration" xreflabel="SHOW CONFIGURATION">
        <emphasis role="tntadmin">show configuration</emphasis>
      </term>
      <listitem><para>
        Show the current settings. Displays all settings,
        including those that have default values and thus are not
        necessarily present in the configuration file.
      </para></listitem>
    </varlistentry>

    <varlistentry>
      <term xml:id="show-info" xreflabel="SHOW INFO">
        <emphasis role="tntadmin">show info</emphasis>
      </term>
      <listitem><para>
<programlisting>
<prompt>localhost&gt;</prompt> <userinput>show info</userinput>
---
info:
  version: "1.5.2-8-g54a279d"
  uptime: 441524
  pid: 12315
  logger_pid: 12316
  snapshot_pid: 0
  lsn: 15481913304
  recovery_lag: 0.000
  recovery_last_update: 1306964594.980
  status: primary
  config: "/usr/local/etc/tarantool.cfg"
</programlisting>
      </para>
      <para>
        <emphasis role="strong">recovery_lag</emphasis> holds the
        difference (in seconds) between the current time on the
        machine (wall clock time) and the time stamp of the last
        applied record.  In replication setup, this difference can
        indicate the delay taking place before a change is
        applied to a replica.
      </para>
      <para>
        <emphasis role="strong">recovery_last_update</emphasis> is
        the wall clock time of the last change recorded in the
        write ahead log. To convert it to human-readable time,
        you can use <command>date -d@<replaceable>1306964594.980</replaceable></command>.
      </para>
      <para>
        <emphasis role="strong">status</emphasis> is
        either "primary" or "replica/&lt;hostname&gt;".
      </para>

      </listitem>
    </varlistentry>

    <varlistentry>
      <term xml:id="show-index" xreflabel="SHOW INDEX">
       <emphasis role="tntadmin">show index</emphasis>
     </term>
     <listitem><para>
     Show the number of keys and the amount of memory used by each each index.
     </para></listitem>
    </varlistentry>

    <varlistentry>
      <term xml:id="show-stat" xreflabel="SHOW STAT">
        <emphasis role="tntadmin">show stat</emphasis>
      </term>
      <listitem><para>
      Show the average number of requests per second, and the
      total number of requests since startup, broken down by
      request type: INSERT or SELECT or UPDATE or DELETE."
<programlisting>
localhost> show stat
---
statistics:
  INSERT:        { rps:  139  , total:  48207694    }
  SELECT_LIMIT:  { rps:  0    , total:  0           }
  SELECT:        { rps:  1246 , total:  388322317   }
  UPDATE_FIELDS: { rps:  1874 , total:  743350520   }
  DELETE:        { rps:  147  , total:  48902544    }
</programlisting>
      </para></listitem>
    </varlistentry>

    <varlistentry>
      <term xml:id="show-slab" xreflabel="SHOW SLAB">
        <emphasis role="tntadmin">show slab</emphasis>
      </term>
      <listitem>
        <para>
          Show the statistics of the slab allocator. The slab
          allocator is the main allocator used to store tuples.
          This can be used to monitor the total memory use and
          memory fragmentation.
        </para>
        <para>
          <emphasis role="strong">items_used</emphasis> contains
          the % of <olink targetptr="slab_alloc_arena"/> already
          used to store tuples.
        </para>
        <para>
          <emphasis role="strong">arena_used</emphasis> contains
          the % of <olink targetptr="slab_alloc_arena"/> that is
          already distributed to the slab allocator.
        </para>
        <para>
          <emphasis role="strong">bytes_waste</emphasis> contains
          the amount of memory wasted by the slab allocator due to
          internal fragmentation.
        </para>
        <para>
        </para>
      </listitem>
    </varlistentry>

    <varlistentry>
      <term xml:id="show-palloc" xreflabel="SHOW PALLOC">
        <emphasis role="tntadmin">show palloc</emphasis>
      </term>
      <listitem><para>
        A pool allocator is used for temporary memory, when
        serving client requests. Every fiber has its own
        temporary pool. Shows the current state of pools of all fibers.
      </para></listitem>
    </varlistentry>

    <varlistentry>
      <term xml:id="save-coredump" xreflabel="SAVE COREDUMP">
        <emphasis role="tntadmin">save coredump</emphasis>
      </term>
      <listitem><para>
        Fork and dump a core. Since Tarantool stores all tuples
        in memory, it can take some time. Mainly useful for
        debugging.
      </para></listitem>
    </varlistentry>

    <varlistentry>
      <term xml:id="show-fiber" xreflabel="SHOW FIBER">
        <emphasis role="tntadmin">show fiber</emphasis>
      </term>
      <listitem><para>
        Show all running fibers, with their stack.
        Mainly useful for debugging.
      </para></listitem>
    </varlistentry>

    <varlistentry>
      <term xml:id="lua-command" xreflabel="LUA ...">
        <emphasis role="tntadmin">lua ...</emphasis>
      </term>
      <listitem><para>
        Execute a chunk of Lua code. This can be used to define,
        invoke, debug and drop <olink
        targetptr="stored-procedures">stored procedures</olink>,
        inspect server environment, perform automated
        administrative tasks.
      </para></listitem>
    </varlistentry>

  </variablelist>

</section>

<xi:include href="stored-procedures.xml" />
<xi:include href="triggers.xml" />

</chapter>

<!--
vim: tw=66 syntax=docbk
vim: spell spelllang=en_us
-->
