Storage Performance for Vultr File System

Vultr File System is intended for clusters of CPU or GPU systems to share stored data between them, with an eye toward cache coherency so that writes made by any one client of the file system are seen by subsequent reads of the file system by other clients. Because Vultr File System tends to be used with large numbers of clients accessing the same file system at the same time, metadata operations have also been heavily optimized.

For additional context on the performance metrics referenced throughout this document and to better understand how storage performance is measured, benchmarked, and compared, see What Are the Fundamentals of Storage Performance?.

Rate Limits

To make performance tuning simple for large clusters, there are presently no rate limits imposed on Vultr File System by default. We reserve the option to impose such limits in the future if the lack of such limits is abused. Until such a time, this means that Vultr File System can provide very high throughput in use cases where workloads require lots of throughput, such as keeping data loading into GPU memory in AI Training clusters.

Benchmark Results

With VFS, we performed two different sets of benchmarks.

• To test generalized read/write performance we used the utility fio to measure performance for 4 KB, 64 KB, 512 KB, 1024 KB, and 4096 KB block sizes. We ran tests with 100% read (both random and sequential), 100% write (again both random and sequential), and a mixed workload of both reads and writes.

• We also used the benchmarking utility smallfile to measure file system performance, at 64 KB, 512 KB, 1024 KB and 4096 KB total file sizes. This tool exercises file system semantics and metadata operations, something a tool like fio cannot be used to measure properly.

We performed these tests across a wide variety of VM instance plans so as to see break points where insufficient CPU or memory in the plan could impact storage performance. We did not find such a break point with block performance even on plans with only 1 core and 1 GB of RAM.

Fio

There are three different classes of hypervisor hosts at Vultr and they each have different performance characteristics:

• Vultr Cloud GPU plans
• VX1 Plans
• All other CPU plans

Because of how these different plan types consume host resources, they may or may not be able to optimally support VFS. Both Vultr Cloud GPU and VX1 plans will perform optimally, whereas older GPU plans have less performance at the time of this writing.

VCG plans and VX1 plans have the following performance profile.

The following table shows the performance characteristics of the “other” plan types.

Smallfile

The smallfile benchmark exercises file system metadata operations. Generally, repeatedly opening and closing small file sizes result in significant metadata operations overhead. Many metadata operations have a significant cost when compared to simply reading and writing data. Using fio alone to gauge the performance of your file system storage would ignore this, essentially only exercising read and write operations without accounting for metadata operations overhead.

We did a significant amount of performance optimization work to dial in the right balance between impact to CPU and memory resources on the client hosts and the overall performance of VFS as a result of our testing. We paid particular attention to the performance when small files are opened, read, and then closed on after another. The result of these optimizations at the hypervisor level should lead to the best possible performance even when faced with what is colloquially known as “the small file problem”.

The following table can give you an idea of how VFS will perform when these metadata operations become a significant portion of your overall workload. For example, stat operations on files can be particularly taxing, so being able to perform them on a large number of files per second has been a significant focus for us.

Some interesting things to note about the table above:

• Some of the IO Types do not result in transfer of data, only metadata, as a result they do not have a corresponding data size or throughput associated with the test. In those cases, the results represent the number of files processed per second rather than the number of IOPS.
• The 4096 KB file size aligns with the object size of the underlying storage, so it tends to perform better than the trend with smaller file sizes might predict because it leads to less back-end storage system communication between storage cluster nodes.

Replicating These Results

Vultr File System is benchmarked using two different methods.

Using fio

You can replicate these results for yourself by using fio at any of the blocksizes mentioned at any operations mix you would like.

1. Install the fio utility and any dependencies. It can be found on git.kernel.org, but your distribution likely has it available as a package. In most distributions the package is simply called fio. You should also install libaio so that it is available to fio. The package is usually called either libaio_dev or libaio_devel, depending on your distribution.

2. When running fio, you will need to create a job configuration file that you can reference and then run a command line that points at the job file.

   ini Copy

   [FIOJOB]
   filename=/mnt/vbs/fio.raw
   size=500G
   random_generator=lfsr
   buffered=0
   direct=1
   invalidate=0
   ioengine=libaio
   rw=randwrite
   bs=4k
   iodepth=4
   numjobs=16
   runtime=900
   loops=1
   time_based=1
   

   Explain Code

   Key values to change to match the workload you are testing are:

   • filename= This should be a file on the file system where you are testing. If you are testing directly on the block device itself, understand that the test is destructive to any data contained in the file and will destroy any file system on the raw device.
   • direct= 1 enables O_DIRECT, 0 disables it. Use direct=1 with ioengine=libaio.
   • ioengine= We recommend libaio for best results, but you may wish to compare it with sync or psync.
   • rw= randread, randwrite, and randrw are the most useful options.
   • bs= Block size.
   • iodepth= The queue depth per job.
   • numjobs= The number of simultaneous jobs to run.

3. Then you can reference the job config from the command line:

   console Copy

   $ fio \
       --eta=never \
       --status-interval=5000ms \
       --output-format=json+ \
       $FIOJOBFILE
   

   Explain Code

   Where $FIOJOBFILE is the path to the job file created above. See the fio documentation for more details.

Using smallfile

1. To obtain smallfile you can download it from its GitHub repository. It requires Python.

2. You need to create a YAML jobfile describing the job parameters. Refer to the smallfile documentation on its GitHub repository for full details, but the jobfile takes the following form.

   ini Copy

   top: /mnt/vfs/smallfile
   operation: create
   output-json: /tmp/smallfile/smallfileresults.json
   files: 1000
   threads: 100
   auto-pause: true
   file-size: 512
   files-per-dir: 1000
   dirs-per-dir: 1000
   hash-into-dirs: true
   xattr-size: 128
   finish: true
   

   Explain Code

   Key values to change to match the workload you are testing are:

   • top: This is the path to the filesystem you’re testing. Change it to match your VFS mount point.
   • operation: This is the operation benchmark type. See the smallfile documentation for a full list of available benchmarks.
   • output-json: This is the path to the output file for your results.
   • threads: The number of threads to execute the benchmark. We used 100.
   • file-size: The size of the file used for testing. We used 4k, 64k, 512k, 1M, and 4M in our testing.

3. Once the job file is created, you can execute the benchmark with the command:

   console Copy

   $ python3 smallfile_cli.py \
       --yaml-input-file=$SMALLFILEJOBFILE
   

   Explain Code

   In the above command, $SMALLFILEJOBFILE is the path to the jobfile created above.

Tuning Tips for Best Performance with Vultr File System

• Run the latest kernel possible in your instance. Recent kernels have significant optimizations for the virtio-fs filesystem type used by VFS.

• Be sure to mount the file system with the relatime or noatime mount options if your application permits. Updating atime is expensive if your application doesn’t require it. See the documentation for the mount command and the /etc/fstab configuration file for your OS distribution for more details.

• Metadata operations are expensive. Avoid opening and closing files repeatedly whenever possible. Instead, open files and keep them open until they are no longer needed. Consolidating data into larger files that can be opened once and kept open (such as databases and big data file formats) can make a huge difference in file system performance (for all file systems, not just VFS).

• Avoid synchronous IO such as psync under most workloads. Instead, use libaio for the benefit of greater queue depths. Test both to determine best performance. Some applications may be able to leverage libaio via a configurable option. If you’re writing your own application, see the documentation for libaio as used in the language of your choice. For fio, enabling libaio is achieved with the option ioengine=libaio.

• Direct I/O allows you to bypass Linux caches. Consider using O_DIRECT for increased performance. In fio this is enabled with the combination of two options: direct=1 and buffered=0.

• Enable high levels of parallelism by making more simultaneous requests. Increase the number of processes, threads or workers making requests. How to do this in your specific application will vary. Many applications will have configuration options, and many libraries will have methods for doing this. In the fio benchmarking utility you can increase parallelism by increasing numjobs.

  If you are trying to discover best possible performance limits on a system where you do not already know the potential bottlenecks you can try repeatedly increasing levels of parallelism until you discover the point where increasing parallelism does not result in improved performance. Typically this will be some multiple of the number of available threads on your CPU. Start with 1 x vCPU and work up from there.

• Increasing queue depths will allow for more requests to be in flight waiting for replies so as not to allow high latency to lower your throughput artificially. How to do this in your specific application will vary. Many applications will have configuration options, and many libraries will have methods for doing this. In the fio benchmarking utility you can increase the number of requests each job will allow to be in flight without a response by increasing iodepth.

  If you are trying to discover best possible performance limits on a system where you do not know the potential bottlenecks you can try repeatedly increasing the queue depth until you discover the point where a deeper queue does not result in improved performance. Typically this is a function of average latency and the number of requests you can queue while waiting for a typical response. The goal is to have a queue that is always full.