VoltDB Enterprise Edition Release Notes

Swapping is an operating system feature that optimizes memory usage when running multiple processes. However, memory is a critical component of the VoltDB server process. Any contention for memory, including swapping, will have a very negative impact on performance and functionality.

We recommend using dedicated servers and disabling swapping when running the VoltDB database server process. Use the swapoff command to disable swapping on Linux systems. If swapping cannot be disabled for any reason, you can reduce the likelihood of VoltDB being swapped out by setting the kernel parameter vm.swappiness to zero.

Recent JDKs introduced an issue related to performance optimization and garbage collection. (See http://java.sun.com/javase/6/webnotes/6u18.html for details.) To use VoltDB (or any memory intensive application) with JDK 6 update 18 through 20, you should disable the card marking optimization, using the following command line argument:

java -XX:-ReduceInitialCardMarks

Note that this bug has been fixed in a more recent release, Sun JDK 6 Update 21. So the preceding workaround is only needed for Sun JDK and OpenJDK versions 6 update 18 through 20.

There is a bug in older Linux versions (prior to 2.6.25) that can cause TCP messaging queues to stall and time out. (See http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff; h=607bfbf2d55dd1cfe5368b41c2a81a8c9ccf4723 for details.)

If you are using CentOS or RedHat Enterprise Linux (RHEL) versions 5.4 or 5.5, you must either apply the preceding patch or turn off window scaling before running VoltDB in production mode. To turn off window scaling, issue the following shell command (or add the line "net.ipv4.tcp_window_scaling=0" to the file /etc/sysctl.conf to make the change persistent):

$ sudo /sbin/sysctl -w net.ipv4.tcp_window_scaling=0

Note that this bug is fixed for Linux releases after 2.6.25, including Ubuntu 9.10 and later, and backported to CentOS/RedHat 5.7 and 5.8. So no changes are needed for recent releases on those platforms or on the Macintosh OS.

There is an issue where, under certain conditions, the use of TCP segmentation offload (TSO) and generic receive offload (GRO) can cause nodes to randomly drop out of a cluster. The symptoms of this problem are that nodes timeout — that is, the rest of the cluster thinks they have failed — although the node is still running and no other network issues (such as a network partition) are the cause.

Disabling TSO and GRO is recommended for any VoltDB clusters that experience such instability. The commands to disable offloading are the following, where N is replaced by the number of the ethernet card:

ethtool -K ethN tso off
ethtool -K ethN gro off

Note that these commands disable offloading temporarily. You must issue these commands every time the node reboots.

The messages displayed on the console when compiling an application catalog or starting a VoltDB server have been improved. Fewer messages are displayed on the console by default, and their format has been simplified to make the messages clearer.

Note that a more thorough set of messages with an severity of INFO or above is still available and written, by default, to log files in the /log subfolder of the user's working directory. You can change both what is displayed and where logs are written by modifying the Log4J configuration file. See the chapter on logging in the Using VoltDB manual for details.

The messages returned by the VoltDB JDBC interface have been improved to provide more meaningful information concerning errors that occur at runtime.

The VoltDB compiler now warns you if any of the SQL queries in your stored procedures (or defined as statements in the project definition file) could result in non-deterministic output. It is important that all SQL queries are deterministic, especially when using durability features such as K-safety, command logging, or database replication. In most cases, non-determinism is a result of an unsorted query; that is, a query without an appropriate index or ORDER BY clause.

If VoltDB detects such queries, a warning is issued, listing the query and the reason it is non-deterministic. When you receive such warnings, it is recommended that you add the appropriate index to the table or specify a sort order to resolve the non-determinism.

The command for starting a single node VoltDB database has been simplified. If you are starting VoltDB on just one node (in other words, hostcount=1 running on localhost), specifying the deployment file and leader node are now optional. For example, the following command starts a database using the catalog myapp.jar and default settings:

$ voltdb catalog myapp.jar

When you leave off the deployment file, VoltDB assumes one node, two sites per host, and a K-safety value of zero. If you want to enable other features (such as export or automatic snapshots), you must create the appropriate deployment file and specify it on the command line.

The statistics package within the Java client API has been modified to improve reporting capabilities. The following classes have been added:

These classes provide a more compact, complete, and consistent approach to statistics than in previous releases. See the javadoc (in the /doc subfolder) that accompanies the VoltDB software for more details. The new classes replace the following methods that have been removed from the API:

Note, that these changes are specific to the Java Client API and do not affect other programming languages.

Several changes in this release improve the reliability of command logging. All users of this feature are strongly urged to upgrade.

Previously, if command logging was enabled, data restored manually (using the @SnapshotRestore system procedure) was not captured in the command logs until the next snapshot of the database was taken. Starting with this release of the Enterprise Edition, manual restores are immediately captured and made durable as part of the command logging process.

The major addition for this release is support for database replication. Database replication lets you create and maintain a separate and distinct copy of an entire database. Replication can be used for:

Database replication is a commercial feature that is available through the VoltDB Enterprise Edition. See Using VoltDB for more information..

All known issues related to running VoltDB on Java version 7 have been resolved. Java 7 is a supported development and production environment for VoltDB.

VoltDB does not support joining two partitioned tables on an arbitrary column in multi-partitioned stored procedures. However, it is now possible to join two tables that are partitioned on the same column, provided the join is based on the equality of the partitioning column. For example, the following is now a valid SQL statement in either single-partitioned or multi-partitioned stored procedures, as long as the tables EMPLOYEE and MANAGER are both partitioned on DEPT_ID:

SELECT E.EMP_NAME, M.MGR_NAME 
       FROM EMPLOYEE AS E, MANAGER AS M
       WHERE E.DEPT_ID = M.DEPT_ID;

When using a server with two or more network interfaces (and therefore multiple IP addresses), you can and should specify which interface to use for cluster communication (internal) and which interface to use for client connections (external). These settings are now available both in the REST interface as an attribute of the server resources and in the Enterprise Manager when adding a server. See the section on "Network Configuration" in the VoltDB Management Guide for details.

Previously, client requests would remain pending until they completed or the connection to the database server was lost. If there was a delay caused by network issues or a procedure taking too long, the client application was never notified.

Starting with V2.2.1, both the client procedure call and the database connection itself can and will timeout. Timeouts help the client application recognize and respond to performance or network issues that are not explicit errors. When a timeout occurs, the client application is notified through an error status returned by the procedure call and, optionally, invocation of a client status listener.

Note that the addition of timeouts is a change to previous behavior. Stored procedure calls that previously waited indefinitely will now time out after two minutes, by default. To emulate previous behavior, you can disable either or both timeout periods by setting the timeout value to zero using the setConnectionResponseTimeout and setProcedureCallTimeout methods on the ClientConfig object. See the section on "Handling Errors" in the Using VoltDB manual for more information about how to configure and identify time outs in procedure calls and client connections.

As part of the addition of timeouts to the Java client library, the client status listener has been improved. A new listener, lateProcedureResponse, has been added and the parameters to the connectionLost listener have been extended to provide additional information. To accommodate these changes, the original status listener ClientStatusListener has been deprecated and replaced with ClientStatusListenerExt (extended). This change allows existing applications that contain custom listeners to continue to work while providing extended capability to new or modified applications. Also, the replacement of the original Java interface with an abstract class will make future extensions easier.

The SQL statement planner is the VoltDB component that analyzes stored procedures when you compile the project definition file or at runtime when you issue an ad hoc query. Several changes have been made to the planner to improve performance. As a result, both compiling the application catalog and processing ad hoc queries should be significantly faster than in previous versions of VoltDB. Another improvement to the planner is a restructuring and simplification of the information it creates in the /debugoutput folder. An explanation of how to read and interpret the execution plans created by the planner is now available in a new book, the VoltDB Performance Guide.

Performance for the VoltOne product, which is a single server database, has been optimized. Certain communication overhead used in multi-node clusters has been removed for VoltOne. As a consequence, latency for transactions — particularly synchronous procedure invocations — on a single server database should be significantly improved over previous releases.

Support for the boolean operators IS NULL and NOT has been added to VoltDB. It is now possible to use WHERE conditions that test whether a column is null or not or if a boolean expression is true or not. For example, the following SELECT statement selects records where the column Email is not null.

SELECT Username, Email FROM Users 
WHERE NOT Email IS NULL;

This release provides improved performance while restoring a snapshot on a K-safe cluster. For most databases using K-safety, restore times should be noticeably shorter than in earlier versions of VoltDB.

There is a rare but critical bug in command logging that could result in the command log snapshot being erroneously deleted. This bug has been fixed. This release is recommended for anyone using command logging.

The following limitations that existed in the previous release of VoltDB (V2.1.1) have been resolved:

The LIMIT clause restricts the number of records that are returned by the SELECT statement. The new OFFSET option specifies the starting point within the selected record set. For example, in the set of alphabetic characters SELECT ... LIMIT 3 OFFSET 5 returns the characters F, G, and H. See the Using VoltDB manual for details.

Two new controls have been added to the configuration options for VoltDB databases. Command log size lets you specify the initial size (in megabytes) of the command log. Snapshot priority lets you control the relative priority of snapshots with relation to other database functions, including transactions. Both options have default values suitable for the majority of applications. However, the controls are provided for specific situations and workloads that may require fine tuning for optimal performance. See the VoltDB Management Guide for details.

The following limitation that existed in the previous release of VoltDB (V2.0) has been resolved:

Previously, you specified the identity of the leader node in the deployment file. Starting with V2.0, you specify the lead node on the command line, not in the deployment file. For existing applications, you will need to remove the leader attribute from the deployment file and add it to the command line.

In previous releases, the application catalog had to be available to every node in the cluster before the cluster could start. With V2.0, the application catalog only has to be available to the leader node. Once the cluster initializes, the lead node distributes the catalog to the rest of the cluster. In other words, you only need to specify the location of the catalog when starting the lead node. (However, specifying the catalog location to other nodes is not an error, the location is simply ignored.)

Similarly, you do not need to specify the catalog when rejoining a server to an existing database cluster or restarting a cluster with the recover startup option. Note, however, the deployment file still must exist and be available on all nodes when starting, rejoining, or recovering.

This change does not require any modification to existing applications.

Similar to the changes to the location of catalogs and deployment files on startup, the parameters to the @UpdateApplicationCatalog have changed. Previously, this stored procedure accepted two strings specifying the locations of the catalog and the deployment file. Starting with V2.0, the parameters to the stored procedure are the actual contents of the files, the catalog as a byte array and the deployment file as a string.

The following limitation that existed in the previous release of VoltDB (V1.3.6) has been resolved:

To ensure complete and accurate restoration of a database, recovery using command logs can only be performed to a cluster with the same number of unique partitions as the cluster that created the logs. If you restart and recover to the same cluster with the same deployment options, there is no problem. But if you change the deployment options for number of nodes, sites per host, or K-safety, recovery may not be possible.

For example, if a four node cluster is running with four sites per host and a K-safety value of one, the cluster has two copies of eight unique partitions (4 X 4 / 2). If one server fails, you cannot recover the command logs from the original cluster to a new cluster made up of the remaining three nodes, because the new cluster only has six unique partitions (3 X 4 / 2). You must either replace the failed server to reinstate the original hardware configuration or otherwise change the deployment options to match the number of unique partitions. (For example, increasing the site per host to eight and K-safety to two.)

VoltDB reserves the subfolder "segments" under the command log directory for storing the actual command log files. Do not add, remove, or modify any files in this directory. In particular, do not set the command log snapshot directory to a subfolder "segments" of the command log directory, or else the server will hang on startup.

Currently, replication and export do not work properly while a cluster is in replica mode. Export works properly on the master database and on the replica after it has been promoted to a normal writable database. But there are cases where not all data is exported while a cluster is on the receiving end of replication. To ensure completeness in export while using database replication:

Using the VoltDB Enterprise Manager, if a replica database was started with command logging, then stopped (intentionally or by accident), the Enterprise Manager cannot restart the database as a normal database using the recover action to reinstate the database's previous state. The Enterprise Manager can restore from a snapshot.

If you want to use the Enterprise Manager to stop a replica and restart it as a normal database, the recommended procedure is:

Note that this limitation is specific to the Enterprise Manager. Failed replica databases can be recovered manually using the command line.

If your SELECT statement includes two aggregates of the same column, VoltDB will return only one column value as part of the result set. For example, if your SQL statement is SELECT COUNT(ColumnA), COUNT(ColumnA) FROM MyTable, the resulting VoltTable will have only one column value per row. The workaround is to either not request the same value twice or aggregate on different columns (for example, SELECT COUNT(Column10), COUNT(Column2)).

In SELECT statements, the selection expression can include columns, aggregate functions (such as COUNT), or arithmetic expressions involving columns (such as Col1 + Col2). However, they cannot include arithmetic involving aggregate functions (such as SELECT SUM(Price) + 2). Using aggregate functions in an arithmetic expression results in an error when you try to compile the project definition file.

The workaround is to use the aggregate in the SELECT statement and then perform the additional arithmetic on the result set either in the stored procedure or in the client application after the transaction completes.

Use of SELECT DISTINCT is supported for a single column (such as SELECT DISTINCT Price FROM Inventory). However, using DISTINCT with multiple columns or arithmetic expressions is not currently supported. For example, the following SELECT DISTINCT statements should not be used:

SELECT DISTINCT Price, Discount FROM Inventory
SELECT DISTINCT (Price - Discount) FROM Inventory

Attempts to join two (or more) distinctly partitioned tables in a multi-partitioned procedure will result in an error. You can join one partitioned table and multiple replicated tables, or you can join two partitioned tables where both tables have the same partitioning column and are joined on equality of that column. But joining two partitioned tables on non-partition columns or using a range of values is not supported.

If a SELECT statement lists the same table twice (such as SELECT ... FROM Employee AS A, Employee AS B) VoltDB will generate an error at compile time.

VoltDB currently intercepts assertions as part of its handling of stored procedures. Attempts to use assertions in stored procedures for debugging or to find programmatic errors will not work as expected.

As part of the database schema, it is possible to assign a primary key or index as part of a table definition, even if that table is then defined as export-only as part of the project definition file. Since export-only tables are write-only (you cannot use them in SELECT statements), the index has no purpose. VoltDB silently ignores the index definition at compile time.

When creating the project definition file, the database name attribute, if specified must be "database". For example <database name="database">. Note, however, that the name attribute is optional.

There is a problem with how the math library used to build the C++ client library handles large decimal values on 32-bit operating systems. As a result, the C++ library cannot serialize and pass Decimal datatypes reliably on these systems.

Note that the C++ client interface can send and receive Decimal values properly on 64-bit platforms.

To ensure proper recovery on startup, either from command logs or the last database snapshot, make sure all snapshot files — or at least complete subsets of the snapshot files — are available on the nodes of the cluster. If you delete or move snapshot files (for example, copying all snapshot files to a single node) be sure to keep all of the files for each node together. Do not selectively delete or move individual files or else the recovery may fail.

Normally, manual snapshots (those created with the Take a Snapshot button) are copied to the management server. However, if automated snapshots are also being created and copied to the management server, it is possible for an automated snapshot to override the manual snapshot.

If this happens, the workaround is to turn off automated snapshots (and their copying) temporarily. To do this, uncheck the box for copying snapshots, set the frequency to zero, and click OK. Then re-open the Edit Snapshots dialog and take the manual snapshot. Once the snapshot is complete and copied to the management server (that is, the manual snapshot appears in the list on the dialog box), you can re-enable copying and automated snapshots.

When the Enterprise Manager starts, it unpacks files that the web server uses into a subfolder of the /tmp directory. It does not delete these files when it stops. Under normal operation, this is not a problem. However, if you upgrade to a new version of the Enterprise Edition, files for the new version become intermixed with the older files and can result in the Enterprise Manager starting databases using the wrong version of VoltDB. To avoid this situation, make sure these temporary files are deleted before starting a new version of VoltDB Enterprise Manager.

The /tmp directory is emptied every time the server reboots. So the simplest workaround is to reboot your management server after you upgrade VoltDB. Alternately, you can delete these temporary files manually by deleting the winstone subfolders in the /tmp directory:

$ rm -vr /tmp/winstone*

When upgrading VoltDB Enterprise Edition, please note that the configuration files for the Enterprise Manager are not upwardly compatible. New product features may make existing database and/or deployment definitions unusable. It is always a good idea to delete existing configuration information before upgrading. You can delete the configuration files by deleting the ~/.voltdb directory. For example:

$ rm -vr ~/.voltdb

The export-to-Hadoop client uses Hadoop and the Sqoop import application from Cloudera to automate the export of data from VoltDB to Hadoop. It is the user's responsibility to ensure that Hadoop (and Sqoop) are installed, configured, and running properly before using the export-to-Hadoop client. This includes making sure that the environment variables HADOOP_HOME and SQOOP_HOME are defined and that all of the required Hadoop and Sqoop JAR files are in the class path.

This is particularly important because the VoltDB export client does not attempt to access Sqoop or Hadoop until after the initial set of data has already been polled and acked from the VoltDB database. So any failure in connecting to Sqoop and/or Hadoop will result in that initial export data being lost. Future releases of VoltDB may attempt to detect and report this failure condition earlier. But for the current release, it is highly recommended that you perform a test run first to ensure all components are configured correctly.

For partitioned tables, the value of the column used to partition the table determines what partition the row belongs to. If you use UPDATE to change this value and the new value belongs in a different partition, the UPDATE request will fail and the stored procedure will be rolled back.

Updating the partition column value may or may not cause the record to be repartitioned (depending on the old and new values). However, since you cannot determine if the update will succeed or fail, you should not use UPDATE to change the value of partitioning columns.

The workaround, if you must change the value of the partitioning column, is to use both a DELETE and an INSERT statement to explicitly remove and then re-insert the desired rows.

If you refer to a table or column name that does not exist, the VoltDB compiler issues the error message "user lacks privilege or object not found". This can happen, for example, if you misspell a table or column name.

Another situation where this occurs is if you mistakenly use double quotation marks to enclose a string literal (such as WHERE ColumnA="True"). ANSI SQL requires single quotes for string literals and reserves double quotes for object names. In the preceding example, VoltDB interprets "True" as an object name, cannot resolve it, and issues the "user lacks privilege" error.

The workaround is, if you receive this error, to look for misspelled table or columns names or string literals delimited by double quotes in the offending SQL statement.

VoltDB opens a file descriptor for every client connection to the database. In normal operation, this use of file descriptors is transparent to the user. However, if there are an inordinate number of concurrent client connections, or clients open and close many connections in rapid succession, it is possible for VoltDB to exceed the process limit on file descriptors. When this happens, new connections may be rejected or other disk-based activities (such as snapshotting) may be disrupted.

In environments where there are likely to be an extremely large number of connections, you should consider increasing the operating system's per-process limit on file descriptors.

This is not a problem when looking at VoltDB logs separately. However, you should be aware of this distinction when integrating logging of VoltDB with logging of other system components that use the local time zone (rather than GMT). You may want to convert one or the other log streams so the time zones match.

The file voltdb/log4j.xml lists all of the VoltDB-specific logging categories. It also serves as a useful logging schema. The sample applications and the VoltDB shell commands use this file to configure logging and it is recommended for new application development.

The master and the replica databases must match in the following regards for replication to begin:

If not, the DR agent refuses to start.

The DR agent currently makes a connection to only one server on the replica database (the server named on the DR agent command line). If that server fails — even if the associated cluster is K-safe and continues to run — replication will stop and must be restarted manually.

The DR agent communicates with the master database through three special ports: 5555, 5556, and 5557. There are two methods for changing the ports that the master database uses for replication:

Adding the replication port to the deployment file is useful when setting the port for all nodes in the cluster. Using the command line option is useful for changing the default port for only one node in the cluster. If you specify the replication port in both the deployment file and on the command line, the command line argument takes precedence.

VoltDB uses the specified port and the next two numeric ports for replication. For example, if you specify the port number 6666, the server uses ports 6666, 6667, and 6668 for replication. If you specify an alternate port number for replication, you must also specify the starting port number as a suffix to the master database IP address

The master database initially queues replication data in memory. If the pending data exceeds the allocated queue size, data is overflowed to disk in the directory voltdbroot/dr_overflow. The values for the queue size and overflow directory are not currently configurable.

When using a multi-homed server (that is, a server with two or more network interface cards and network addresses) as part of a master database, you must explicitly identify which interface the server uses for both internal and external communication when you start the VoltDB server process. For example:

$ voltdb create deployment deployment.xml \
  catalog catalog.jar leader serverA \
  license license.xml \
  externalinterface 10.11.169.10 \
  internalinterface 10.12.171.14

The specified external interface is used for the client and admin ports and database replication. The internal interface is used for all inter-node cluster communication. If you do not explicitly specify which interface to use, replication will fail when the DR agent attempts to identify and connect to the master cluster nodes.

You cannot use @UpdateApplicationCatalog to modify the catalog on a master database that is being actively replicated. To update the catalog you must first save, shutdown, and restore/recover the master database with the new catalog, then restart replication.

Do not use CTRL-Z to place the DR agent process in the background, or the agent may crash.