Рейтинг@Mail.ru

5. Server administration

5. Server administration

Typical server administration tasks include starting and stopping instances of the server, reloading configuration, taking snapshots, log rotation.

5.1. Using Tarantool as a client

If tarantool is started without an initialization file, or if the initialization file contains console.start(), then tarantool enters interactive mode. There will be a prompt (“tarantool>”) and it will be possible to enter requests. When used this way, tarantool can be a client for a remote server.

This section shows all legal syntax for the tarantool program, with short notes and examples. Other client programs may have similar options and request syntaxes. Some of the information in this section is duplicated in the chapter Configuration reference.

5.1.1. Conventions used in this section

Tokens are character sequences which are treated as syntactic units within requests. Square brackets [ and ] enclose optional syntax. Three dots in a row ... mean the preceding tokens may be repeated. A vertical bar | means the preceding and following tokens are mutually exclusive alternatives.

5.1.2. Options when starting client from the command line

General form:

$ tarantool
OR
$ tarantool options
OR
$ tarantool lua-initialization-file [ arguments ]

Here lua-initialization-file can be any script containing code for initializing. Effect: The code in the file is executed during startup. Example: init.lua.
Notes: If a script is used, there will be no prompt. The script should contain configuration information including box.cfg...listen=... or box.listen(...) so that a separate program can connect to the server via one of the ports.

Option is one of the following (in alphabetical order by the long form of the option):

-?, -h, --help

Client displays a help message including a list of options. Example: tarantool --help The program stops after displaying the help.

-V, --version

Client displays version information. Example: tarantool --version. The program stops after displaying the version.

5.1.3. Tokens, requests, and special key combinations

Procedure identifiers are: Any sequence of letters, digits, or underscores which is legal according to the rules for Lua identifiers. Procedure identifiers are also called function names.
Note: function names are case sensitive so insert and Insert are not the same thing.

String literals are: Any sequence of zero or more characters enclosed in single quotes. Double quotes are legal but single quotes are preferred. Enclosing in double square brackets is good for multi-line strings as described in Lua documentation.
Examples: ‘Hello, world’, ‘A’, [[A\B!]].

Numeric literals are: Any sequence of one or more digits, not enclosed in quotes, optionally preceded by - (minus sign). Large or floating-point numeric literals may include decimal points, exponential notation, or suffixes.|br| Examples: 500, -500, 5e2, 500.1, 5LL, 5ULL.

Single-byte tokens are: , or ( or ) or arithmetic operators.
Examples: * , ( ).

Tokens must be separated from each other by one or more spaces, except that spaces are not necessary around single-byte tokens or string literals.

5.1.4. Requests

Generally requests are entered following the prompt in interactive mode while tarantool is running. (A prompt will be the word ‘tarantool’ and a greater-than sign, for example tarantool>). The end-of-request marker is by default a newline (line feed).

For multi-line requests, it is possible to change the end-of-request marker. Syntax: console = require('console'); console.delimiter(string-literal). The string-literal must be a value in single quotes. Effect: string becomes end-of-request delimiter, so newline alone is not treated as end of request. To go back to normal mode: console.delimiter('')string-literal. Delimiters are usually not necessary because Tarantool can tell when a multi-line request has not ended (for example, if it sees that a function declaration does not have an end keyword). Example:

console = require('console'); console.delimiter('!')
function f ()
  statement_1 = 'a'
  statement_2 = 'b'
end!
console.delimiter('')!

See here a condensed Backus-Naur Form [BNF] description of the suggested form of client requests.

In interactive mode, one types requests and gets results. Typically the requests are typed in by the user following prompts. Here is an example of an interactive-mode Tarantool client session:

$ tarantool
[ tarantool will display an introductory message
  including version number here ]
tarantool> box.cfg{listen = 3301}
[ tarantool will display configuration information here ]
tarantool> s = box.schema.space.create('tester')
[ tarantool may display an in-progress message here ]
---
...
tarantool> s:create_index('primary', {type = 'hash', parts = {1, 'unsigned'}})
---
...
tarantool> box.space.tester:insert{1,'My first tuple'}
---
- [1, 'My first tuple']
...
tarantool> box.space.tester:select(1)
---
- - [1, 'My first tuple']
...
tarantool> box.space.tester:drop()
---
...
tarantool> os.exit()
2014-04-30 10:28:00.886 [20436] main/101/spawner I> Exiting: master shutdown
$

Explanatory notes about what Tarantool displayed in the above example:

  • Many requests return typed objects. In the case of box.cfglisten=3301, this result is displayed on the screen. If the request had assigned the result to a variable, for example c = box.cfglisten=3301, then the result would not have been displayed on the screen.
  • A display of an object always begins with “---” and ends with “...”.
  • The insert request returns an object of type = tuple, so the object display line begins with a single dash (‘-‘). However, the select request returns an object of type = table of tuples, so the object display line begins with two dashes (‘- -‘).

5.2. Utility tarantoolctl

With tarantoolctl, you can say: “start an instance of the Tarantool server which runs a single user-written Lua program, allocating disk resources specifically for that program, via a standardized deployment method.”

If Tarantool was installed with Debian or Red Hat installation packages, the script is in /usr/bin/tarantoolctl or /usr/local/bin/tarantoolctl. The script handles such things as: starting, stopping, rotating logs, logging in to the application’s console, and checking status.

Also, you can use tarantoolctl as a client to connect to another instance of Tarantool server and pass requests.

5.2.1. Configuration for tarantoolctl

The tarantoolctl script will look for a configuration file in the current directory ($PWD/.tarantoolctl). If that fails, it looks in the current user’s home directory ($HOME/.config/tarantool/tarantool). If that fails, it looks in the SYSCONFDIR directory (usually /etc/sysconfig/tarantool, but it may be different on some platforms). Most of the settings are similar to the settings used by box.cfg...; however, tarantoolctl adjusts some of them by adding an application name. A copy of usr/local/etc/default/tarantool, with defaults for all settings, would look like this:

default_cfg = {
    pid_file   = "/var/run/tarantool",
    wal_dir    = "/var/lib/tarantool",
    memtx_dir  = "/var/lib/tarantool",
    vinyl_dir  = "/var/lib/tarantool",
    log     = "/var/log/tarantool",
    username   = "tarantool",
}
instance_dir = "/etc/tarantool/instances.enabled"

The settings in the above script are:

pid_file
The directory for the pid file and control-socket file. The script will add “/instance-name” to the directory name.
wal_dir
The directory for the write-ahead *.xlog files. The script will add “/instance-name” to the directory-name.
memtx_dir
The directory for the snapshot *.snap files. The script will add “/instance-name” to the directory-name.
vinyl_dir
The directory for the vinyl-storage-engine files. The script will add “/vinyl/instance-name” to the directory-name.
log
The place where the application log will go. The script will add “/instance-name.log” to the name.
username
The user that runs the Tarantool instance. This is the operating-system user name rather than the Tarantool-client user name.
instance_dir
The directory where all applications for this host are stored. The user who writes an application for tarantoolctl must put the application’s source code in this directory, or a symbolic link. For examples in this section the application name my_app will be used, and its source will have to be in instance_dir/my_app.lua.

5.2.2. Commands for tarantoolctl

The command format is tarantoolctl operation application_name, where operation is one of: start, stop, enter, logrotate, status, eval. Thus ...

start <application>

Start application <application>

stop <application>

Stop application

enter <application>

Show application’s admin console

logrotate <application>

Rotate application’s log files (make new, remove old)

status <application>

Check application’s status

eval <application> <scriptname>

Execute code from <scriptname> on an instance of application

connect <URI>

Connect to a Tarantool instance running at the specified URI

5.2.3. Typical code snippets for tarantoolctl

A user can check whether my_app is running with these lines:

if tarantoolctl status my_app; then
...
fi

A user can initiate, for boot time, an init.d set of instructions:

for (each file mentioned in the instance_dir directory):
    tarantoolctl start `basename $ file .lua`

A user can set up a further configuration file for log rotation, like this:

/path/to/tarantool/*.log {
    daily
    size 512k
    missingok
    rotate 10
    compress
    delaycompress
    create 0640 tarantool adm
    postrotate
        /path/to/tarantoolctl logrotate basename $ 1 .log
    endscript
}

5.2.4. A detailed example for tarantoolctl

The example’s objective is to make a temporary directory where tarantoolctl can start a long-running application and monitor it.

The assumptions are: the root password is known, the computer is only being used for tests, the Tarantool server is ready to run but is not currently running, tarantoolctl is installed along the user’s path, and there currently is no directory named tarantool_test.

Create a directory named /tarantool_test:

$ sudo mkdir /tarantool_test

Edit /usr/local/etc/default/tarantool. It might be necessary to say sudo mkdir /usr/local/etc/default first. Let the new file contents be:

default_cfg = {
    pid_file  = "/tarantool_test/my_app.pid",
    wal_dir   = "/tarantool_test",
    memtx_dir = "/tarantool_test",
    vinyl_dir = "/tarantool_test",
    log       = "/tarantool_test/log",
    username  = "tarantool",
}
instance_dir = "/tarantool_test"

Make the my_app application file, that is, /tarantool_test/my_app.lua. Let the file contents be:

box.cfg{listen = 3301}
box.schema.user.passwd('Gx5!')
box.schema.user.grant('guest','read,write,execute','universe')
fiber = require('fiber')
box.schema.space.create('tester')
box.space.tester:create_index('primary',{})
i = 0
while 0 == 0 do
    fiber.sleep(5)
    i = i + 1
    print('insert ' .. i)
    box.space.tester:insert{i, 'my_app tuple'}
end

Tell tarantoolctl to start the application ...

$ cd /tarantool_test
$ sudo tarantoolctl start my_app

... expect to see messages indicating that the instance has started. Then ...

$ ls -l /tarantool_test/my_app

... expect to see the .snap file and the .xlog file. Then ...

$ sudo less /tarantool_test/log/my_app.log

... expect to see the contents of my_app‘s log, including error messages, if any. Then ...

$ cd /tarantool_test
$ # assume that 'tarantool' invokes a Tarantool instance
$ sudo tarantool
tarantool> box.cfg{}
tarantool> console = require('console')
tarantool> console.connect('localhost:3301')
tarantool> box.space.tester:select({0}, {iterator = 'GE'})

... expect to see several tuples that my_app has created.

Stop. The only clean way to stop my_app is with tarantoolctl, thus:

$ sudo tarantoolctl stop my_app

Clean up. Restore the original contents of /usr/local/etc/default/tarantool, and ...

$ cd /
$ sudo rm -R tarantool_test

5.2.5. An example for tarantoolctl connect

$ tarantoolctl connect username:password@127.0.0.1:3306

Note

There are alternatives to tarantoolctl connect – you can use the console module or the net.box module from a Tarantool server. Also, you can write your client programs with any of the Connectors. However, most of the examples in this manual illustrate usage with either tarantoolctl connect or with using the Tarantool server as a client.

5.3. Administrative ports

“Admin port”, “admin console”, and “text protocol” all refer to the same thing: a connection which is set up with console.listen(...) for entry of requests by administrators.

“Binary port”, “binary protocol”, and “primary port” all refer to a different thing: a connection which is set up with box.cfg{listen=...} for entry of requests by anyone.

Ordinary connections to the Tarantool instance should go via a binary port. But admin ports are useful for special cases involving security.

When you connect to an admin port:

  • No password is necessary
  • The user is automatically ‘admin’, a user with many privileges.

Therefore you must set up admin ports very cautiously. If it is a TCP port, it should only be opened for a specific IP. Ideally it should not be a TCP port at all, it should be a Unix domain socket, so that access to the server machine is required. Thus a typical setup for an admin port is:

console.listen('/var/lib/tarantool/socket_name.sock')

and a typical connection URI is:

admin:any_string@/var/lib/tarantool/socket_name.sock

if the listener has the privilege to write on /var/lib/tarantool and the connector has the privilege to read on /var/lib/tarantool. Alternatively both setup and connection can be done with tarantoolctl.

If no administrator password exists which could be given out to users, and admin ports are restricted or are sockets, then requests which require ‘admin’ privileges can only occur locally, and are subject to Unix security and monitoring.

For additional security, some requests are illegal. For example, “conn:eval” will result in the error message “- error: console does not support this request type” because conn:eval requires the binary protocol.

If security via admin ports is not necessary, it is still possible to be an admin user by using the tarantool server as a client, or by connecting to a binary port with a valid password.

To find out whether a TCP port is an admin port, use telnet. For example:

$ telnet 0 3303
Trying 0.0.0.0...
Connected to 0.
Escape character is '^]'.
Tarantool 1.7.2-70-gbc479ad (Lua console)
type 'help' for interactive help

In this example the response does not include the word “binary” and does include the words “Lua console”. Therefore it is clear that this is a successful connection to an admin port, and admin requests can now be entered on this terminal.

5.4. Administrative requests

To learn which functions are considered to be administrative, type help(). A reference description also follows below:

box.snapshot()

Take a snapshot of all data and store it in memtx_dir/<latest-lsn>.snap. To take a snapshot, Tarantool first enters the delayed garbage collection mode for all data. In this mode, tuples which were allocated before the snapshot has started are not freed until the snapshot has finished. To preserve consistency of the primary key, used to iterate over tuples, a copy-on-write technique is employed. If the master process changes part of a primary key, the corresponding process page is split, and the snapshot process obtains an old copy of the page. In effect, the snapshot process uses multi-version concurrency control in order to avoid copying changes which are superseded while it is running.

Since a snapshot is written sequentially, one can expect a very high write performance (averaging to 80MB/second on modern disks), which means an average database instance gets saved in a matter of minutes.

Note

As long as there are any changes to the parent index memory through concurrent updates, there are going to be page splits, and therefore you need to have some extra free memory to run this command. 10% of memtx_memory is, on average, sufficient. This statement waits until a snapshot is taken and returns operation result.

Note

Change notice: Prior to Tarantool version 1.6.6, the snapshot process caused a fork, which could cause occasional latency spikes. Starting with Tarantool version 1.6.6, the snapshot process creates a consistent read view and writes this view to the snapshot file from a separate thread.

Although box.snapshot() does not cause a fork, there is a separate fiber which may produce snapshots at regular intervals – see the discussion of the snapshot daemon.

Example:

tarantool> box.info.version
---
- 1.7.0-1216-g73f7154
...
tarantool> box.snapshot()
---
- ok
...
tarantool> box.snapshot()
---
- error: can't save snapshot, errno 17 (File exists)
...

Taking a snapshot does not cause the server to start a new write-ahead log. Once a snapshot is taken, old WALs can be deleted as long as all replicated data is up to date. But the WAL which was current at the time box.snapshot() started must be kept for recovery, since it still contains log records written after the start of box.snapshot().

An alternative way to save a snapshot is to send a SIGUSR1 signal to the instance. While this approach could be handy, it is not recommended for use in automation: a signal provides no way to find out whether the snapshot was taken successfully or not.

5.5. Server introspection

For server instance introspection, use the reports provided by functions in the following submodules:

  • box.cfg submodule (check and specify all configuration parameters for the Tarantool instance)
  • box.slab submodule (monitor the total use and fragmentation of memory allocated for storing data in Tarantool)
  • box.info submodule (introspect Tarantool’s server variables)
  • box.stat submodule (introspect Tarantool’s request and network statistics)

5.6. Backups

The exact procedure for backing up a database depends on: how up-to-date the database must be, how frequently backups must be taken, whether it is okay to disrupt other users, and whether the procedure should be optimized for size (saving disk space) or for speed (saving time). So there is a spectrum of possible policies, ranging from cold-and-simple to hot-and-difficult.

5.6.1. Cold backup

In essence: The last snapshot file is a backup of the entire database; and the WAL files that are made after the last snapshot are incremental backups. Therefore taking a backup is a matter of copying the snapshot and WAL files.

  1. Prevent all users from writing to the database. This can be done by shutting down the instance, or by saying box.cfg{read_only=true} and then ensuring that all earlier writes are complete (fsync can be used for this purpose).
  2. If this is a backup of the whole database, say box.snapshot().
  3. Use tar to make a (possibly compressed) copy of the latest .snap and .xlog files on the memtx_dir and wal_dir directories.
  4. If there is a security policy, encrypt the tar file.
  5. Copy the tar file to a safe place.

... Later, restoring the database is a matter of taking the tar file and putting its contents back in the memtx_dir and wal_dir directories.

5.6.2. Continuous remote backup

In essence: replication is useful for backup as well as for load balancing. Therefore taking a backup is a matter of ensuring that any given replica has an up-to-date copy of the data, and doing a cold backup on that. Since all the other replicas continue to operate, this is not a cold backup from the end user’s point of view. This could be done on a regular basis, with a cron job or with a Tarantool fiber.

5.6.3. Hot backup

In essence: The logged changes done since the last cold backup must be secured, while the system is running.

For this purpose you need a “file copy” utility that will do the copying remotely and continuously, copying only the parts of a file that are changing. One such utility is rsync.

Alternatively, you need an ordinary file copy utility, but there should be frequent production of new snapshot files or new WAL files as changes occur, so that only the new files need to be copied.

Note re storage engine: vinyl databases require additional steps.

5.7. Updates/upgrades

5.7.1. Updating Tarantool in production

First, put your application’s business logic in a Tarantool-Lua module that exports its functions for CALL.

For example, /usr/share/tarantool/myapp.lua:

local function start()
  -- Initial version
  box.once("myapp:.1.0", function()
  box.schema.space.create("somedata")
  box.space.somedata:create_index("primary")
  ...

  -- migration code from 1.0 to 1.1
  box.once("myapp:.v1.1", function()
  box.space.somedata.index.primary:alter(...)
  ...

  -- migration code from 1.1 to 1.2
  box.once("myapp:.v1.2", function()
  box.space.somedata.space:alter(...)
  box.space.somedata:insert(...)
  ...
end

-- start some background fibers if you need

local function stop()
  -- stop all background fibers and cleanup resources
end

local function api_for_call(xxx)
  -- do some business
end

return {
  start = start;
  stop = stop;
  api_for_call = api_for_call;
}

This file is maintained by the application’s developers. On its side, Tarantool Team provides templates for you to assemble deb/rpm packages and utilities to quickly assemble packages for specific platforms. If needed, you can split applications into standalone files and/or modules.

Second, put an initialization script to the /etc/tarantool/instances.available directory.

For example, /etc/tarantool/instances.available/myappcfg.lua:

#!/usr/bin/env tarantool

box.cfg {
  listen = 3301;
}

if myapp ~= nil then
  -- hot code reload using tarantoolctl or dofile()

  -- unload old application
  myapp.stop()
  -- clear cache for loaded modules and dependencies
  package.loaded['myapp'] = nil
  package.loaded['somedep'] = nil; -- dependency of 'myapp'
end

-- load a new version of app and all dependencies
myapp = require('myapp').start({some app options controlled by sysadmins})

As a more detailed example, you can take the example.lua script that ships with Tarantool and defines all configuration options.

This initialization script is actually a configuration file and should be maintained by system administrators, while developers only provide a template.

Now update your app file in /usr/share/tarantool. Replace your application file (for example, /usr/share/tarantool/myapp.lua) and manually reload the myappcfg.lua initialization script using tarantoolctl:

$ tarantoolctl eval /etc/tarantool/instance.enabled/myappcfg.lua

After that, you need to manually flush the cache of package.loaded modules.

For deb/rpm packages, you can add the tarantoolctl eval instruction directly into Tarantool’s specification in RPM.spec and the /debian directory.

Finally, clients make a CALL to myapp.api_for_call and other API functions.

In the case of tarantool-http, there is no need to start the binary protocol at all.

5.7.2. Upgrading a Tarantool database

This information applies for users who created databases with older versions of the Tarantool server, and have now installed a newer version. The request to make in this case is: box.schema.upgrade().

For example, here is what happens when one runs box.schema.upgrade() with a database that was created in early 2015. Only a small part of the output is shown.

tarantool> box.schema.upgrade()
alter index primary on _space set options to {"unique":true}, parts to [[0,"unsigned"]]
alter space _schema set options to {}
create view _vindex...
grant read access to 'public' role for _vindex view
set schema version to 1.7.0
---
...

5.8. Server signal handling

The server processes these signals during the main thread event loop:

SIGHUP
may cause log file rotation, see the example in section “Logging”.
SIGUSR1
may cause saving of a snapshot, see the description of box.snapshot.
SIGTERM
may cause graceful shutdown (information will be saved first).
SIGINT
(also known as keyboard interrupt) may cause graceful shutdown.
SIGKILL
causes shutdown.

Other signals will result in behavior defined by the operating system. Signals other than SIGKILL may be ignored, especially if the instance is executing a long-running procedure which prevents return to the main thread event loop.

5.9. Generating a core file

On rare occasions it may be necessary to generate a core file from the server. For example, the Tarantool developers may request one for a bug report. First make sure core dumps are enabled on the system (this may require some study of the system settings and administrative privileges).

Find out the process id of the instance. This is the value in box.info.pid. It can also be found with utilities such as ps -A | grep tarantool. We will refer to this process id as $PID.

Either run the debugger:

gdb -batch -ex "generate-core-file" -p $PID

or stop the instance with a SIGABRT signal:

kill -SIGABRT $PID

Since Tarantool stores tuples in memory, core files may be large.

5.10. Process title

Linux and FreeBSD operating systems allow a running process to modify its title, which otherwise contains the program name. Tarantool uses this feature to help meet the needs of system administration, such as figuring out what services are running on a host, their status, and so on.

A Tarantool instance’s process title has these components:

program_name [initialization_file_name] <role_name> [custom_proc_title]

  • program_name is typically “tarantool”.
  • initialization_file_name is the name of an initialization file, if one was specified.
  • role_name is:
    • “running” (ordinary node “ready to accept requests”),
    • “loading” (ordinary node recovering from old snap and wal files),
    • “orphan” (not in a cluster),
    • “hot_standby”, or
    • “dumper” + process-id (saving a snapshot).
  • custom_proc_title is taken from the custom_proc_title configuration parameter, if one was specified.

For example:

$ ps -AF | grep tarantool
1000     17337 16716  1 91362  6916   0 11:07 pts/5    00:00:13 tarantool script.lua <running>

5.11. System-specific administration notes

This section will contain information about issues or features which exist on some platforms but not others - for example, on certain versions of a particular Linux distribution.

5.11.1. Debian GNU/Linux and Ubuntu

Setting up an instance:

$ ln -s /etc/tarantool/instances.available/*instance-name.cfg* /etc/tarantool/instances.enabled/

Starting all instances:

$ service tarantool start

Stopping all instances:

$ service tarantool stop

Starting/stopping one instance:

$ service tarantool-instance-name start/stop

5.11.2. Fedora, RHEL, CentOS

There are no known permanent issues. For transient issues, go to http://github.com/tarantool/tarantool/issues and enter “RHEL” or “CentOS” or “Fedora” or “Red Hat” in the search box.

5.11.3. FreeBSD

There are no known permanent issues. For transient issues, go to http://github.com/tarantool/tarantool/issues and enter “FreeBSD” in the search box.

5.11.4. Mac OS X

There are no known permanent issues. For transient issues, go to http://github.com/tarantool/tarantool/issues and enter “OS X” in the search box.

5.12. Notes for systemd users

Tarantool fully supports systemd for managing instances and supervising database daemons.

5.12.1. Instance management

Tarantool was designed to have multiple running instances of Tarantool on the same machine. Use systemctl start|stop|restart|status tarantool@$MYAPP to manage your databases and Lua applications.

5.12.1.1. Creating instances

Simply put your Lua configuration to /etc/tarantool/instances.available/$MYAPP.lua:

box.cfg{listen = 3313}
require('myappcode').start()

(this minimal example is sufficient).

Another starting point could be the example.lua script that ships with Tarantool and defines all options.

5.12.1.2. Starting instances

Use systemctl start tarantool@$MYAPP to start ${MYAPP} instance:

$ systemctl start tarantool@example
$ ps axuf|grep exampl[e]
taranto+  5350  1.3  0.3 1448872 7736 ?        Ssl  20:05   0:28 tarantool example.lua <running>

(console examples here and further on are for Fedora).

Use systemctl enable tarantool@$MYAPP to enable ${MYAPP} instance for auto-load during system startup.

5.12.1.3. Monitoring instances

Use systemctl status tarantool@$MYAPP to check information about ${MYAPP} instance:

$ systemctl status tarantool@example
tarantool@example.service - Tarantool Database Server
Loaded: loaded (/etc/systemd/system/tarantool@.service; disabled; vendor preset: disabled)
Active: active (running)
Docs: man:tarantool(1)
Process: 5346 ExecStart=/usr/bin/tarantoolctl start %I (code=exited, status=0/SUCCESS)
Main PID: 5350 (tarantool)
Tasks: 11 (limit: 512)
CGroup: /system.slice/system-tarantool.slice/tarantool@example.service
+ 5350 tarantool example.lua <running>

Use journalctl -u tarantool@$MYAPP to check the boot log:

$ journalctl -u tarantool@example -n 5
-- Logs begin at Fri 2016-01-08 12:21:53 MSK, end at Thu 2016-01-21 21:17:47 MSK. --
Jan 21 21:17:47 localhost.localdomain systemd[1]: Stopped Tarantool Database Server.
Jan 21 21:17:47 localhost.localdomain systemd[1]: Starting Tarantool Database Server...
Jan 21 21:17:47 localhost.localdomain tarantoolctl[5969]: /usr/bin/tarantoolctl: Found example.lua in /etc/tarantool/instances.available
Jan 21 21:17:47 localhost.localdomain tarantoolctl[5969]: /usr/bin/tarantoolctl: Starting instance...
Jan 21 21:17:47 localhost.localdomain systemd[1]: Started Tarantool Database Server

5.12.1.4. Attaching to instances

You can attach to a running Tarantool instance and evaluate some Lua code using the tarantoolctl utility:

$ tarantoolctl enter example
/bin/tarantoolctl: Found example.lua in /etc/tarantool/instances.available
/bin/tarantoolctl: Connecting to /var/run/tarantool/example.control
/bin/tarantoolctl: connected to unix/:/var/run/tarantool/example.control
unix/:/var/run/tarantool/example.control> 1 + 1
---
- 2
...
unix/:/var/run/tarantool/example.control>

5.12.1.5. Checking logs

Tarantool logs important events to /var/log/tarantool/$MYAPP.log.

Let’s write something to the log file:

$ tarantoolctl enter example
/bin/tarantoolctl: Found example.lua in /etc/tarantool/instances.available
/bin/tarantoolctl: Connecting to /var/run/tarantool/example.control
/bin/tarantoolctl: connected to unix/:/var/run/tarantool/example.control
unix/:/var/run/tarantool/example.control> require('log').info("Hello for README.systemd readers")
---
...

Then check the logs:

$ tail /var/log/tarantool/example.log
2016-01-21 21:09:45.982 [5914] iproto I> binary: started
2016-01-21 21:09:45.982 [5914] iproto I> binary: bound to 0.0.0.0:3301
2016-01-21 21:09:45.983 [5914] main/101/tarantoolctl I> ready to accept requests
2016-01-21 21:09:45.983 [5914] main/101/example I> Run console at /var/run/tarantool/example.control
2016-01-21 21:09:45.984 [5914] main/101/example I> tcp_server: remove dead UNIX socket: /var/run/tarantool/example.control
2016-01-21 21:09:45.984 [5914] main/104/console/unix/:/var/run/tarant I> started
2016-01-21 21:09:45.985 [5914] main C> entering the event loop
2016-01-21 21:14:43.320 [5914] main/105/console/unix/: I> client unix/: connected
2016-01-21 21:15:07.115 [5914] main/105/console/unix/: I> Hello for README.systemd readers
2016-01-21 21:15:09.250 [5914] main/105/console/unix/: I> client unix/: disconnected

Log rotation is enabled by default if you have logrotate installed. Please configure /etc/logrotate.d/tarantool to change the default behavior.

5.12.1.6. Stopping instances

Use systemctl stop tarantool@$MYAPP to see information about the running ${MYAPP} instance.

$ systemctl stop tarantool@example

5.12.2. Daemon supervision

All instances are automatically restarted by systemd in case of failure.

Let’s try to destroy an instance:

$ systemctl status tarantool@example|grep PID
Main PID: 5885 (tarantool)
$ tarantoolctl enter example
/bin/tarantoolctl: Found example.lua in /etc/tarantool/instances.available
/bin/tarantoolctl: Connecting to /var/run/tarantool/example.control
/bin/tarantoolctl: connected to unix/:/var/run/tarantool/example.control
unix/:/var/run/tarantool/example.control> os.exit(-1)
/bin/tarantoolctl: unix/:/var/run/tarantool/example.control: Remote host closed connection

Now let’s make sure that systemd has revived our Tarantool instance:

$ systemctl status tarantool@example|grep PID
Main PID: 5914 (tarantool)

Finally, let’s check the boot logs:

$ journalctl -u tarantool@example -n 8
-- Logs begin at Fri 2016-01-08 12:21:53 MSK, end at Thu 2016-01-21 21:09:45 MSK. --
Jan 21 21:09:45 localhost.localdomain systemd[1]: tarantool@example.service: Unit entered failed state.
Jan 21 21:09:45 localhost.localdomain systemd[1]: tarantool@example.service: Failed with result 'exit-code'.
Jan 21 21:09:45 localhost.localdomain systemd[1]: tarantool@example.service: Service hold-off time over, scheduling restart.
Jan 21 21:09:45 localhost.localdomain systemd[1]: Stopped Tarantool Database Server.
Jan 21 21:09:45 localhost.localdomain systemd[1]: Starting Tarantool Database Server...
Jan 21 21:09:45 localhost.localdomain tarantoolctl[5910]: /usr/bin/tarantoolctl: Found example.lua in /etc/tarantool/instances.available
Jan 21 21:09:45 localhost.localdomain tarantoolctl[5910]: /usr/bin/tarantoolctl: Starting instance...
Jan 21 21:09:45 localhost.localdomain systemd[1]: Started Tarantool Database Server.

5.12.3. Customizing the service file

Please don’t modify the tarantool@.service file in-place, because it will be overwritten during package upgrades. It is recommended to copy this file to /etc/systemd/system and then modify the required settings. Alternatively, you can create a directory named unit.d/ within /etc/systemd/system and put there a drop-in file name.conf that only changes the required settings. Please see systemd.unit(5) manual page for additional information.

5.12.4. Debugging

coredumpctl automatically saves core dumps and stack traces in case of a crash. Here is how it works:

$ # !!! please never do this on the production system !!!
$ tarantoolctl enter example
/bin/tarantoolctl: Found example.lua in /etc/tarantool/instances.available
/bin/tarantoolctl: Connecting to /var/run/tarantool/example.control
/bin/tarantoolctl: connected to unix/:/var/run/tarantool/example.control
unix/:/var/run/tarantool/example.control> require('ffi').cast('char *', 0)[0] = 48
/bin/tarantoolctl: unix/:/var/run/tarantool/example.control: Remote host closed connection

coredumpctl list /usr/bin/tarantool displays the latest crashes of the Tarantool daemon:

$ coredumpctl list /usr/bin/tarantool
MTIME                            PID   UID   GID SIG PRESENT EXE
Sat 2016-01-23 15:21:24 MSK   20681  1000  1000   6   /usr/bin/tarantool
Sat 2016-01-23 15:51:56 MSK   21035   995   992   6   /usr/bin/tarantool

coredumpctl info <pid> shows the stack trace and other useful information:

$ coredumpctl info 21035
          PID: 21035 (tarantool)
          UID: 995 (tarantool)
          GID: 992 (tarantool)
       Signal: 6 (ABRT)
    Timestamp: Sat 2016-01-23 15:51:42 MSK (4h 36min ago)
 Command Line: tarantool example.lua <running>
   Executable: /usr/bin/tarantool
Control Group: /system.slice/system-tarantool.slice/tarantool@example.service
         Unit: tarantool@example.service
        Slice: system-tarantool.slice
      Boot ID: 7c686e2ef4dc4e3ea59122757e3067e2
   Machine ID: a4a878729c654c7093dc6693f6a8e5ee
     Hostname: localhost.localdomain
      Message: Process 21035 (tarantool) of user 995 dumped core.

               Stack trace of thread 21035:
               #0  0x00007f84993aa618 raise (libc.so.6)
               #1  0x00007f84993ac21a abort (libc.so.6)
               #2  0x0000560d0a9e9233 _ZL12sig_fatal_cbi (tarantool)
               #3  0x00007f849a211220 __restore_rt (libpthread.so.0)
               #4  0x0000560d0aaa5d9d lj_cconv_ct_ct (tarantool)
               #5  0x0000560d0aaa687f lj_cconv_ct_tv (tarantool)
               #6  0x0000560d0aaabe33 lj_cf_ffi_meta___newindex (tarantool)
               #7  0x0000560d0aaae2f7 lj_BC_FUNCC (tarantool)
               #8  0x0000560d0aa9aabd lua_pcall (tarantool)
               #9  0x0000560d0aa71400 lbox_call (tarantool)
               #10 0x0000560d0aa6ce36 lua_fiber_run_f (tarantool)
               #11 0x0000560d0a9e8d0c _ZL16fiber_cxx_invokePFiP13__va_list_tagES0_ (tarantool)
               #12 0x0000560d0aa7b255 fiber_loop (tarantool)
               #13 0x0000560d0ab38ed1 coro_init (tarantool)
               ...

coredumpctl -o filename.core info <pid> saves the core dump into a file.

coredumpctl gdb <pid> starts gdb on the core dump.

It is highly recommended to install the tarantool-debuginfo package to improve gdb experience. Example:

$ dnf debuginfo-install tarantool

gdb also provides information about the debuginfo packages you need to install:

$ # gdb -p <pid>
...
Missing separate debuginfos, use: dnf debuginfo-install
glibc-2.22.90-26.fc24.x86_64 krb5-libs-1.14-12.fc24.x86_64
libgcc-5.3.1-3.fc24.x86_64 libgomp-5.3.1-3.fc24.x86_64
libselinux-2.4-6.fc24.x86_64 libstdc++-5.3.1-3.fc24.x86_64
libyaml-0.1.6-7.fc23.x86_64 ncurses-libs-6.0-1.20150810.fc24.x86_64
openssl-libs-1.0.2e-3.fc24.x86_64

Symbol names are present in stack traces even if you don’t have the tarantool-debuginfo package installed.

For additional information, please refer to the documentation provided with your Linux distribution.

5.12.5. Precautions

  • Please don’t use tarantoolctl {start,stop,restart} to control instances started by systemd. It is still possible to use tarantoolctl to start and stop instances from your local directories (e.g. $HOME) without obtaining ROOT access.
  • tarantoolctl is configured to work properly with systemd. Please don’t modify system-wide settings of tarantoolctl, such as paths, directory permissions and usernames. Otherwise, you have a chance to shoot yourself in the foot.
  • systemd scripts are maintained by the Tarantool Team (http://tarantool.org). Please file tickets directly to the upstream’s bug tracker (https://github.com/tarantool/tarantool/issues/) rather than to your Linux distribution.

5.12.6. Limitations

These limitations exist due to decisions by packagers to support systemd alongside sysvinit.

/etc/init.d/tarantool start under systemd, or systemctl start tarantool (without an @instance argument), will start only those instances which were enabled before reboot or before the last time that systemd was reloaded with systemctl daemon-reload.

(systemctl start tarantool, without an @instance argument, is provided only for interoperability with sysvinit scripts. Please use systemctl start tarantool@instance instead.)

/etc/init.d/tarantool stop under systemd, or systemctl tarantool stop (without an @instance argument), will do nothing.

Starting with Tarantool version 1.7.1.42, a new version of tarantool-common is required. (tarantool-common is a downloadable package which provides scripts to work with tarantool configuration and log files.) An attempt to upgrade tarantool-common will cause restart of all instances.

5.12.7. sysvinit -> systemd conversion

These instructions apply only for Debian/Ubuntu distros where both sysvinit and systemd exist.

Install new systemd-enabled packages.

#For each instancename in /etc/tarantool/instances.enabled/:
#To enable the instance to be automatically loaded by systemd:
systemctl enable tarantool@instancename

#The following command does nothing but is recommended for consistency:
/etc/init.d/tarantool stop

#Disable sysvinit-compatible wrappers:
systemctl disable tarantool; update-rc.d tarantool remove

5.13. How to write a bug report

If it’s a bug, you’re doing us a favor by taking the time to tell us about it. To make sure that we repay you with a prompt and satisfactory result, you can help us by following some simple steps. Well, usually they’re simple.

Go to the github issues site. If the problem is with the Tarantool server, go to https://github.com/tarantool/tarantool/issues. If the problem is with the Tarantool manual, go to https://github.com/tarantool/doc/issues.

Look at the current list of bugs to see whether someone has already reported this or a similar problem.

Click “New Issue”.

Fill in the Title field. Usually this should be a synopsis and contain specific unique information. For example, instead of “I can’t get certain information”, say “box.space.x:delete() didn’t report what was deleted”.

Fill in the Description field. This normally should be in English, but we accept Russian. If you think it might make a difference, include: your operating system name and version, the Tarantool name and version, and any unusual details about your machine and its configuration. Then say what you did, and what Tarantool did. Usually this involves stating all the steps needed to reproduce, and an explanation why this differs from expected behavior according to our manual, but you can skip whatever is unnecessary or obvious. If this is a feature request or if it affects a special category of user, be sure to mention that.

Optionally, click on the word “Labels” on the right of the screen, to see a list of categories. Usually the category is filled in by the the Tarantool team, but if you know it in advance you can fill it in yourself.

Click on the words “Attach files by dragging & dropping, selecting them , or pasting from the clipboard.” – but only if the Description isn’t enough to reproduce the problem. Files should not contain data that’s not needed to reproduce the problem, and should not contain anything confidential. Sometimes a log file produced with Tarantool’s logger has been useful.

Click “Submit new issue”.

Watch for a response. Usually within one or two workdays a Tarantool team member will write an acknowledgment, or some questions, or suggestions for a workaround.

5.13.1. Core dumps and stack traces

If a bug is severe and is hard to reproduce, the developers may ask for files which show what execution path Tarantool was following when things went wrong.

Be sure in advance that a core dump will be generated if a crash happens. In Linux, this might mean that you need to say “ulimit -c unlimited”, but there are many other reasons why a core dump will not be produced, as explained by “man 5 core”. Often core dumps are not allowed by default because core dumps can be large; however, if you make sure to get rid of them, that’s not a problem.

Be sure you know in advance what directory core dumps would be written to, and be sure that that directory is writable.

Be sure in advance that a core dump will include stack trace information. If you use a binary distribution, this is automatic. If you build from source, you will not get good information if you pass -DCMAKE_BUILD_TYPE=Release to CMake.

Tarantool will make a core dump if it receives a SIGSEGV or SIGFPE or SIGABRT or SIGQUIT signal. This is automatic if Tarantool crashes. Otherwise you can send the signal yourself to a running program, thus:
Find out the process id with “ps -A | grep tarantool”.
Kill the process with “kill -SIGSEGV [process id]”.
This is also explained in the section Generating a core file.

Core dump files can be very large and usually the Tarantool team won’t want you to send the whole file – they’ll only need a “stack trace” or “backtrace”. To produce one, say gdb -se "tarantool" -ex "bt full" -ex "thread apply all bt" --batch -c core > /tmp/tarantool_trace.txt (replacing “tarantool” with the path to the Tarantool executable, and replacing “core” with the path to the core file, and replacing “/tmp/tarantool_trace.txt” with whatever path name you prefer).

If the /tmp/tarantool_trace.txt result file is small, add it in the “Description” field or in a comment. If the result is large, upload it – follow the github instruction “Attach files ...” as stated earlier.

Occasionally you will find that tarantool_trace.txt contains output without debug symbols – the lines will contain ”??” instead of names. If this happens, check the instructions on these Tarantool wiki pages: “How to debug core dump of stripped tarantool” and How to debug core from different OS”.