Comparison of Rosbag2 Storage Plugins

James Smith
November 2022

Context

The ROS 2 bag recording framework supports pluggable storage layers, allowing users to choose different storage formats and recording libraries. When a user installs ROS 2 for the first time, they get a storage plugin by default which records bags with SQLite. This document compares write performance of the SQLite plugin against the MCAP storage plugin.

Why compare write performance?

Performance is most critical when writing bag data rather than reading it back. If a bag recorder can't keep up with the stream of messages, either the incoming message queue grows unbounded, resulting in an OOM, or some messages are dropped, resulting in data loss. Both of these are serious problems when running a robotics application.

Benchmark description

A new benchmark was created for this comparison, designed to directly measure the storage plugin with no other moving parts. Available here, this benchmark directly uses the storage plugin API to write a fixed number of bytes as fast as possible. This measures the raw write throughput of each storage plugin configuration.

Plugin Configurations

The following configurations of both plugins are tested:

• sqlite_default: The default configuration of the SQLite storage plugin, effectively measuring the performance that ROS 2 users get out-of-the-box. This is equivalent to running the following statements when opening a SQLite file:PRAGMA journal_mode=MEMORY; PRAGMA synchronous=OFF;. These settings produce the highest write performance available from SQLite at the risk of corruption in the case of an interruption or power loss.
• sqlite_resilient: The "resilient" preset profile of the SQLite storage plugin. This setting is equivalent to running the following statements when opening a SQLite file: PRAGMA journal_mode=WAL; PRAGMA synchronous=NORMAL. This removes the risk of an entire .db3 file being corrupted, at the cost of performance. By using a write-ahead log, only the messages in the last few transactions may be lost if the data from those transactions were not fully committed to disk. Therefore in this mode, SQLite3 offers similar guarantees against corruption as MCAP.
• mcap_default: The default recording configuration of the MCAP storage plugin. Produces un-compressed, chunked MCAPs with a message index for efficient read performance. Chunk size is 768KiB, though chunks expand to fit if a message is larger than the chunk size.
• mcap_nochunking: The highest-throughput configuration of the MCAP storage plugin. Does not write a message index to the MCAP, so bags recorded with this mode need to be reindexed later to be read efficiently.
• mcap_uncompressed_crc: Like mcap_default, but calculates CRCs for each chunk so that readers can identify if a chunk contains corrupted data. This feature does not have an equivalent in the SQLite3 storage plugin.
• mcap_compressed_nocrc: Like mcap_default, but compressed each chunk with zstd compression on default settings.

Software Versions

Some installed package versions were omitted for clarity.

ROS 2 Package Name	Version
mcap_vendor	0.5.0
rcl	5.4.0
rmw	6.3.0
rmw_dds_common	1.7.0
rmw_fastrtps_cpp	6.3.0
rmw_fastrtps_shared_cpp	6.3.0
rmw_implementation	2.9.0
rmw_implementation_cmake	6.3.0
ros_core	0.10.0
ros_environment	3.2.0
ros_workspace	1.0.2
rosbag2_compression	0.17.0
rosbag2_cpp	0.17.0
rosbag2_storage	0.17.0
rosbag2_storage_default_plugins	0.17.0
rosbag2_storage_mcap	0.5.0
rosbag2_storage_plugin_comparison	0.1.0
rosbag2_storage_sqlite3	0.17.0
rosbag2_test_common	0.17.0
shared_queues_vendor	0.17.0
sqlite3_vendor	0.17.0
zstd_vendor	0.17.0

Messages

Here the MiB suffix indicates that we are measuring Mebibytes.

Messages were written to the bag files in a variety of different size suites.

• All 1MiB
• All 10KiB
• All 100B
• Mixed: Messages from all of the above sizes at the following ratios (by number of bytes, not messages)
  • 1MiB: 70%
  • 10KiB: 20%
  • 100B: 10%

When testing write throughput, 250MiB of messages are stored as quickly as possible.

The message content was determined by taking sequential slices from a sample file containing uncompressed robotics data. For example, if a test needed two 100B messages, the first message would contain the bytes of the sample file from range [0, 100), and the second message [100, 200). An MCAP file containing [https://nuscenes.org] data in uncompressed form was used for the results presented here.

This strategy was chosen to provide non-zero, non-random message content that would be representative of a robotics application.

Cache sizes tested

The rosbag2::SequentialWriter collects messages in an internal cache before calling the storage plugin write() call with all messages at once. The cache size determines the size of each "batch" that is written to a storage plugin at once. This matters in particular for the SQLite storage plugin, because each batch constitutes a SQL transaction, and it's generally better for write performance to write many messages in one transaction.

Cache sizes tested were:

• 1KiB
• 10MiB

Benchmark hardware

These benchmarks were recorded on two hardware platforms:

Attribute	M1 Mac	Intel NUC7i5BNH
CPU	Apple M1 Pro	Core i5-7260U @ 2.20Ghz, 2 cores (4 threads)
RAM	32GB LPDDR5-6400, estimated 200GB/s bandwidth	8GiB DDR4-2133, estimated 34.1 GB/s bandwidth
Kernel	Linux 5.15.0-48-generic	Linux 5.15.0-52-generic
Distro	Ubuntu 22.04.1 Jammy aarch64 virtualized with Parallels 17	Ubuntu 22.04.1 Jammy x86_64

All bags were written to a ramdisk, to eliminate the effect of disk I/O speed from the tests.

Results

Results in CSV form are available here:

• Apple M1 Pro results
• Intel NUC7i5BNH results

The throughput values are represented below as bar charts. Error bars represent a 95% confidence interval.

Key Takeaways

• CRC calculation currently incurs a large performance penalty. MCAP currently uses an architecture-independent CRC calculation strategy with a 16KiB precalculated lookup table. See this issue for discussion on improving this.
• When dealing with many small messages, there is a clear advantage to using MCAP in un-chunked mode. This removes the overhead of calculating and writing message the message index to the file.
• MCAP either performs on-par with or significantly better than the SQLite storage plugin in their default configurations. It's also worth noting that in their default configurations, MCAP can offer better corruption resilience than the SQLite storage plugin as well.

Recommendations

Given the above results, we feel comfortable recommending MCAP as a replacement for SQLite as a general-purpose ROS 2 storage plugin.

How do I replicate these results?

Clone this branch into a ROS 2 workspace:

$ git clone -b plugin-comparison https://github.com/james-rms/rosbag2

Install dependencies:

$ rosdep install rosbag2_storage_plugin_comparison

build the test binaries:

$ colcon build --packages-select rosbag2_storage_plugin_comparison

Find some sample data to use as message content for the tests. A simple enough option is to download some public bag data and use that as your source of data. To ensure that any downloaded bags are uncompressed, you can use the mcap CLI:

mcap convert --compression none <input bag> sample.mcap

Launch the benchmark sweep.

$ ros2 run rosbag2_storage_plugin_comparison sweep.py --message-data sample.mcap output.csv

To produce bar charts like the ones on this page, use the plot.py script included:

$ pip install matplotlib numpy pandas
$ python3 rosbag2/rosbag2_performance/rosbag2_storage_plugin_comparison/scripts/plot.py output.csv