VoltDB Client Wire Protocol

VoltDB Team - 6/27/2011 - VoltDB Client Wire Protocol Version 0

Overview

A client connection to a VoltDB instance consists of a TCP connection on port 21212. After the initial login process the only exchange between the client library and the VoltDB server is the invocation of and response to stored procedures. All messages in the VoltDB wire protocol are big endian and all integers are signed. Every message is length preceded with a 4 byte integer and the length does not included the length value. The first value after the length preceding value is the version number of the wire protocol represented as a byte. These two items precede all messages.

The login message is the first message the client library sends to the VoltDB server and it is required even if authentication is disabled in the server's configuration. The login message consists of a service name string, a username string and a 160-bit SHA-1 hash of the password. The response from the server consists of a response byte. A value of 0 indicates successful authentication and all other values indicate failure. If authentication fails the server will close the connection. The client can safely send invocations before receiving an authentication response.

The procedure invocation request contains the procedure to be called by name, and the serialized parameters to the procedure. The message also includes an opaque 8 byte piece of client data that will be returned with the response, and can be used by the client to correlate requests with responses.

The returned response contains a byte status code, an integer measuring intra-cluster latency, a serialized exception if an error occured and the exception was serializable, an array of VoltTables (may be length 0, never null), a string value containing any extra information the server included, and the 8 byte piece of client data contained in the originating procedure invocation.

The following sections describe how values are serialized. The wire protocol is still under development and will be adapted to support new authentication methods and new data types as necessary.

Basic Data Types

Binary fields:

Binary fields do not have a wire type associated with them but they are present in certain messages. Binary fields (opacque client data, hashes) do not have any endianness, sign, or size other then what is specified in the message format.

Integer Types:

All integer types are signed, twos-compliment and big-endian.

• Byte - 1 Byte
• Short - 2 Bytes
• Integer - 4 Bytes
• Long - 8 Bytes

Floating Point Type

Only 8-Byte Double types are supported using the byte representation in IEEE 754 "double format." Positive and negative infinity, as well as NaN, are supported on the wire, but not guaranteed to work as SQL values.

String Type

Strings begin with a 4-byte integer storing the number of bytes of character data, followed by the character data. UTF-8 is the only supported character encoding. Note: Strings are artificially limited to 1 megabyte. The NULL string has a length preceded value of -1 (negative one) followed by 0 (zero) bytes of string data. The empty string is represented with a length preceding value of 0.

A String encoded into a parameter set has to be deserialized as a Java String before being passed to a stored procedure and this is a relatively slow operation. If the stored procedure does not need the Java String representation it can specify a byte array as its argument instead of String. The String should then be presented in the parameter set as a byte array (not preceded with the String wire type).

If an array of bytes is passed as a parameter to a SQL statement in a stored procedure, and that parameter expects a string, it will automatically be converted to a string on the native side without any Java deserialization. SQL statements do not support array parameters so this is not ambigous. In the current implementation this can be a signficant performance win, especially with large parameter sets with many strings.

The following table presents the byte by byte serialization of the string "foo"

Varbinary Type

Varbinary is a effectively a binary string, using the same serialization and storage.

Like strings, varbinary begin with a 4-byte integer storing the number of bytes of raw data, followed by the raw data itself. Note: Like strings, varbinary values are artificially limited to 1 megabyte. The NULL varbinary has a length preceded value of -1 (negative one) followed by 0 (zero) bytes of data. The value of zero bytes is represented with a length preceding value of 0.

The native java type for varbinary is byte[]. Stored procedures and SQL statements will accept byte[] as input for varbinary parameters. For compatibility with textual SQL and less binary-friendly clients, varbinary parameters may also be passed as hexidecimal strings using standard string serialization and the string type code indicator. For example, the string "aa" would represent a single byte of value 170. The hex-encoding is case-insensitive and will fail on invalid input, such as odd-length strings.

Date Type

All dates are represented on the wire as Long values. This signed number represents the number of microseconds (not the usual milliseconds before or after Jan. 1 1970 00:00:00 GMT, the Unix epoch.

Decimal Type

VoltDB implements a fixed precision and scale DECIMAL(38,12) type. This type is serialized as a 128 bit signed twos complement integer reperesenting the unscaled decimal value. The integer must be in big-endian byte order with the most significant bytes first. Null is serialized as the smallest representable value which is "-170141183460469231731687303715884105728." Serializing values (Null excluded) that are greater than "99999999999999999999999999999999999999" or less than "-99999999999999999999999999999999999999" will result in an error response.

Note that the serialization is given above in terms of a 128 bit integer. Since the logical value represented is actually scalled by 12 decimal places, the logical maximum value, minimum value and null values are "99999999999999999999999999.999999999999", "99999999999999999999999999.999999999999" and "170141183460469231731687303.715884105728" respectively.

The following table presents the byte by byte serialization of the number "-23325.23425"

Application Specific Data Types

Wire Type Info

A Byte value specifying the type of a serialized value on the wire.

• ARRAY = -99
• NULL = 1
• TINYINT = 3
• SMALLINT = 4
• INTEGER = 5
• BIGINT = 6
• FLOAT = 8
• STRING = 9
• TIMESTAMP = 11
• DECIMAL = 22
• VARBINARY = 25

Procedure Call Status Code

A Byte value specifying the success or failure of a remote stored procedure call.

• SUCCESS = 1
• USER_ABORT = -1
• GRACEFUL_FAILURE = -2
• UNEXPECTED_FAILURE = -3
• CONNECTION_LOST = -4

Compound Data Types

Array Types

Arrays are represented as Byte value indicating the wire type of the elements and a 2 byte Short value indicating the number of elements in the array, followed by the specified number of elements. The length preceding value for the TINYINT (byte) type is length preceded by a 4 byte integer instead of a 2 byte short. This important exception allows large quantities of binary or string data to be passed as a byte array. The size of byte arrays is artificially limited to 1 megabyte. Each array is serialized according to its type (Strings as Strings, VoltTables as VoltTables, Integers as Integers). Arrays are only present as parameters in parameter sets.

• Size is limited to 32,767 values due to the signed short length with the exception of TINYINT (byte) arrays which use a 4 byte integer length and are limited to 1 megabyte.
• All values must be homogeneous with respect to type.

The following example serialization shows an array with two String elements ("foo1", "foo2").

Complex API Data Types

VoltTable

On the wire a VoltTable is serialized as a header followed by tuple data. VoltTables, like all VoltDB serialized structures are stored in network byte order.

(Note: In the following description, the term "array" means a sequence of objects of the specified type, not a VoltDB array object as described in the previous section. Because the number of columns is fixed and is already part of the VoltTable descriptor, we know how many objects are in each array and a full array descriptor is not needed.)

It should also be noted that although a description of the VoltTable structure is being provided here for completeness, in most cases the client interface does not need to interpret the structure, but rather passes it unchanged between the server and the client application.

Notes on the Header Format:

• The "Table Metadata Length" stores the length in bytes of the contents of the table from byte 8 (the end of the metadata length field) all the way to the end of the "Column Names" array. NOTE: It does not include the row count value. See below for an example.
• The size of the "Column Types" and "Column Names" arrays is expected to equal the value stored in "Column Count".
• Column names are limited to the ASCII character set. Strings in row values are still UTF-8 encoded.
• Values with 4-byte (integer) length fields are signed and are limited to a max of 1 megabyte.

Row Data Format:

Each row is prefixed by a 4 byte integer that holds the non-inclusive length of the row. If a row is a single 4-byte integer column, the value of this length prefix will be 4. Row size is artifically restricted to 2 megabytes.

The body of the row is packed array of values. The value at index i is is of type specified by the column type field for index i. The values are serialized according to the serialization rules in "Basic Data Types" above.

Example VoltTable Serialization:

The following is a 35-byte long serialized table containing one column named "Test" of type BIGINT with one row containing the number 5.

Serializable exceptions

Currently serializable exceptions are not a part of the wire protocol although they are present in the invocation response message. The length every serialized exception (4 byte integer) is part of the invocation response message and it can be used to skip the exception. It is possible to retrieve the Byte ordinal that follows the exception's length preceding value. The ordinal will not be present if the exception's length is 0.

• 1 EEException Generic failure in Volt. Should indicate a failure in the server and not the application code. These should not occur in normal operation.
• 2 SQLException Base class for all exceptions that can occur during normal operation. Things like constraint failures (unique, string length, not null) that are caught and handled correct by Volt.
• 3 ConstraintFailureException Specialization of SQLException for contraint failures during the execution of a stored procedure.

In the future a set of serializable exceptions and their serialization format will be added to the wire protocol.

Parameter Set

A parameter set contains all the parameters to be passed to a stored procedure and it is one of the structures bundled inside a stored procedure invocation request. The first value of a parameter set is a Short indicating the number of parameters that follow. The following values are a series of <wire type, value> pairs. Each value is preceded by its wire type represented as a Byte. NULL is a valid wire type and value and it is not followed by any additional value. Arrays are preceded by the wire type -99 and the array value contains the type of the array elements as well as the number of elements (see Array type). A parameter set cannot contain a nested parameter set (there is no wire type for parameter set).

Note that varbinary values using type number 25 and arrays of bytes using type number -99, followed by type 3 are effectively interchangeable.

Parameter set

Parameter

The following example parameter set serialization shows a parameter set consisting of an array of two strings ("foo1, "foo2") and a decimal.

Message formats

Message header

The header that is included at the beginning of all messages. The length value includes the protocol version byte but not the 4 byte length value.

The following table shows and example header for a 140,000 byte message.

Login message

The login message is the first message a client can send to a server after opening a connection. A client does not need to wait for a response to the login message to begin sending invocation requests. The login message identifies a service to authenticate to. There are currently two supported services: the "database" service authenticates stored procedure callers; the "export" service authenticates export connector callers. Export connectors are extensible and future plugin connectors may handle other service string values.

The following table shows an example login message for username "scooby" password "doo"

Login response

A response is generated to a login request and success is indicated with a result code of 0. Any other value indicates authentication failure and will be followed by the server closing the connection. A response code of 1 indicates that the there are too many connections. A response code of 2 indicates that authentication failed because the client took too long to transmit credentials. A response code of 3 indicates a corrupt or invalid login message. If the response code is 0 the response will also contain additional information following the result code. A 4 byte integer specifying the host id of the Volt node . An 8 byte long specifying a connection id that is unique among connections to that node. An 8 byte long timestamp (milliseconds since Unix epoch) and a 4 byte IPV4 address representing the time the cluster was started and the address of the leader node. These two values uniquely identify a Volt cluster instance. And finally a string containing a textual description of the build the node being connected to is running.

The following table shows a login response indicating success

Invocation request

A request to invoke a stored procedure identifies the procedure to invoke by name, the parameters to pass to the procedure, and an 8 byte piece of client data that will be returned with the response to the invocation request. A client does not need to wait for a response to a request to continue sending requests. The server will use TCP backpressure to avoid running out of memory when a client sends too many invocations for the server to handle.

The following table shows the serialization for invoking a procedure called "proc" with a parameter set containing an array of two strings ("foo1", "foo2") and a decimal value "-23325.23425"

Invocation response

An invocation response contains the results of the server's attempt to execute the stored procedure. The response includes optional fields and the first byte after the header is used to indicate which optional fields are present. The status string, application status string, and serializable exception are all optional fields. Bit 7 indicates the presence of a serializable exception, bit 6 indicates the presence of a status string, and bit 8 indicates the presence of an app status string. The serializable exception that can be included in some responses is currently not a part of the wire protocol. The exception length value should be used to skip exceptions if they are present. The status string is used to return any human readable information the server or stored procedure wants to return with the response. The app status code and app status string can be set by application code from within stored procedures and is returned with the response.

The following table shows a response to a previous invocation. For demonstrational purposes two tables are returned, but in a real failure case there would be no tables returned.