Storage Performance for Vultr Object Storage

We have conducted extensive benchmarking of Vultr Object Storage across all performance tiers. For additional context on the performance metrics referenced throughout this document and to better understand how storage performance is measured, benchmarked, and compared, see What Are the Fundamentals of Storage Performance?.

Vultr Object Storage is available in four performance tiers: Standard, Premium, Performance, and Accelerated.

The Standard and Premium tiers are primarily implemented on HDD storage with flash acceleration for metadata and write-ahead caching. This architecture improves metadata responsiveness and write performance while maintaining cost efficiency.

The Performance and Accelerated tiers are based on NVMe storage. These tiers are optimized for workloads running on Vultr Cloud Compute and GPU instances within the same data center and provide significantly higher throughput and operations per second than typical Internet-facing object storage.

Rate Limits

Tiers of Vultr Object Storage are limited at the object gateway to not allow a given subscription to exceed certain operations per second and throughput levels. These limits are imposed to avoid situations where one object storage subscription consumes all the available network throughput or processing power available on the object gateway for its storage workload, limiting what would be available for competing workloads.

It is important to understand how these limits interact with object sizes. For instance, object sizes smaller than 524,288 bytes will not reach the Accelerated tier’s throughput limit before reaching its operations limit.

Note also that metadata operations count toward the total operations per second. This means that they subtract from the available operations for reads and writes and so either reduce throughput for the remaining operations or require larger object sizes to make that throughput limit possible to reach.

Benchmark Results

As a rate limited service, both for operations per second and throughput, it is possible to reach one limit before reaching the other. Small object sizes will reach operations thresholds before throughput thresholds and large object sizes will reach throughput thresholds before operations thresholds. Moreover, metadata operations contribute toward the operations per second limits and so can negatively impact attainable throughput.

Standard Tier

The Standard tier is what most people would think of as conventional object storage. It is Internet-facing, usually with its clients accessing it from the internet. Access from Vultr Cloud Compute is possible, of course, but the general expectation is that it is low throughput and high latency. This makes it good for infrequently accessed objects and the accretion of large amounts of data retained over long periods.

Premium Tier

The Premium tier has a 25% increase in operations per second and a 33% increase in throughput relative to the Standard tier. It also has slightly better write performance than Standard. This makes it better suited to workloads that depend on better performance generally, particularly with writes, but still are geared towards clients accessing from the internet that expect relatively low throughput and high latency.

Performance Tier

The Performance tier is backed by NVMe storage but aims at a cost efficiency rather than maximum performance. It allows for significantly more operations per second and higher throughput than the Premium tier. It is also intended for access primarily by Vultr Cloud Compute in the same data center rather than clients over the Internet. This is not to say that it can’t be accessed over the Internet, but rather that in-data center clients will see benefits from local connectivity as compared to slower tiers.

Accelerated Tier

The Accelerated tier is designed to allow shared data access from within the data center at higher rates of both throughput and operations per second. Being accessible from the internet at the same time makes it ideally suited for data ingestion to cluster computing, such as AI training. Depositing data on the Accelerated tier of Object Storage makes it accessible for import onto, for example, Vultr File System, where it can then be used at very high throughputs required for keeping GPU clusters flush with data.

Replicating These Results

You can replicate these benchmarks using the warp utility.

1. Install the warp utility and any dependencies. It can be found at its official GitHub repository, but your distribution likely has it available as a package. In most distributions the package is simply called warp.

2. Warp operates by having a Warp Server that coordinates one or more Warp Clients, each of which generates load against one or more S3 servers. This allows synchronized workload generation across multiple client machines. In our testing, we used eight Vultr VPS in the same region as the object storage. To set up clients, on each server you’re using as a client you must run the command:

   console Copy

   $ warp client [listenaddress:port]
   

   Explain Code

   Where listenaddress:port is an optional specification of the port and address the Warp Client should listen on for connections from the Warp Server. Be sure that your firewall on the hosts is opened to allow communication to the ports you’ve configured your clients to use.

3. Once Warp Clients are running you can use the following command to execute the benchmark from the Warp Server. Note that the server is not generating any client activity itself. Rather, it is merely instructing the clients to generate the workload.

   console Copy

   $ warp get \
       --duration=2m \
       --warp-client=$WORKERS \
       --host=$S3HOST \
       --bucket=bucket-name-warp \
       --access-key=$S3ACCESS_KEY \
       --secret-key=$S3SECRET_KEY \
       --tls \
       --obj.size=512k \
       --rps-limit=$RATE
   

   Explain Code

   In the above command, get can be replaced by any of several benchmarks you may wish to run from amongst the list of available benchmark workloads. For a full list of available benchmark types, see the Warp documentation included at its GitHub repository. The options and variables above have the following meanings:

   • $WORKERS This is the list of Warp Clients to coordinate. The presence of the --warp-client option indicates that Warp is acting as a server.
   • $S3HOST The name of the S3 host to test against.
   • --bucket The name of the bucket that Warp should test against.
   • $S3ACCESS_KEY Your access key.
   • $S3SECRET_KEY Your secret key.
   • --obj.size The size of objects to be used in testing.
   • $RATE Sets the rate of requests per client to limit 429 status returns due to exceeding the object storage rate limit. This should be equal to or slightly above the published request rate of the storage tier divided by the number of Warp Clients in use.

   Refer to the official Warp documentation for complete details.

Tuning Tips for Best Performance with Vultr Object Storage

• Enable high levels of parallelism by making more simultaneous requests. Increase the number of clients making requests or the number of threads or workers per client. For instance, we used eight Vultr VPS as clients for our testing with warp rather than testing with a single s3cmd.
• Object Storage is inherently high latency, so also allow for more requests to be in flight waiting for replies so as not to allow high latency to lower your throughput artificially. For example, when writing code to connect to your Vultr Object Storage buckets, leverage your language’s object storage library’s support for queues to allow multiple requests across your connections.
• Honor 429 status codes by slowing down your own request rate. Remember that requests that go unfulfilled because you have exceeded your operations rate limit cost resources as well. Most client applications support some form of back-off after receiving 429 status, allowing them to avoid blasting the Object Storage with requests that will only result in more 429 status responses.
• If you are merely using s3cmd, consider moving to s5cmd instead. It has considerably improved performance, particularly with regard to highly parallelizable workloads. Refer to the official GitHub repository for further details (specifically the Configuring Concurrency section).
• If you’re using Loki with Vultr Object Storage, it benefits from 3 or more ingesters and queriers, increasing parallelism. Rclone has the --transfers flag to increase parallelism and the --retries-sleep to handle rate limiting.