Hands on with StarWind NVMe-oF Initiator for Windows

Windows is great for a lot of things, there’s a reason it’s the number one operating system in the world. That said, it’s not perfect, especially when it comes to embracing new storage standards. Therefore, there is an enormous opportunity for enterprising companies to develop solutions for Windows shops. As NVMe SSDs continue to dominate the business and become the standard for SSD server storage, the demand for sharing this storage increases. Unfortunately, this was a problem under Windows until recently. Earlier this year, StarWind commercialized an NVMe-oF initiator for Windows.

Windows is great for a lot of things, there’s a reason it’s the number one operating system in the world. That said, it’s not perfect, especially when it comes to embracing new storage standards. Therefore, there is an enormous opportunity for enterprising companies to develop solutions for Windows shops. As NVMe SSDs continue to dominate the business and become the standard for SSD server storage, the demand for sharing this storage increases. Unfortunately, this was a problem under Windows until recently. Earlier this year, StarWind commercialized an NVMe-oF initiator for Windows.

Up until commercialization, StarWind had offered its NVMe-oF Initiator for Windows as a free tool for development and PoC use cases. They still offer the free version for those who want to play, but we’re looking at the GA shipping version today. If you’re looking at an NVMe-oF initiator for Windows, you’re probably using StarWind IP. They offer OEM solutions for a large number of partners who need to expand their offerings.

NVMe-oF initiator for the Windows configuration

The StarWind NVMe-oF Initiator for Windows can be easily installed on any Windows host. No special hardware or additional Windows components are required. The software is Windows-certified (Server 2019 and Windows 10) and tested for compatibility with the most important NVMe-oF storage providers. In our scenario, we have a very simple configuration of one storage host and four clients.

starwind NVMe-oF initiator for Windows in the device manager

Each of the four clients are Dell PowerEdge R740 servers. They each run with two Intel Xeon Gold 6130 CPUs with 2.1 GHz and 256 GB DRAM. We use NVIDIA ConnectX-5 EN 100GbE NICs (MCX516A-CCAT) for connectivity. The servers are installed with Windows Server 2019 Standard Edition, using the StarWind NVMe-oF initiator for Windows version 1.9.0.455. For the Linux tests we used CentOS 8.4.2105 (Kernel – 4.18.0-305.10.2) with nvme-cli 1.12. The servers are directly connected to the storage host.

StarWind NVMe-oF initiator for the Windows architecture

The memory host is an Intel OEM server (M50CYP2SB2U), which is equipped with two Intel Xeon 8380 CPUs with 2.3 GHz with 512 GB DRAM. We used NVIDIA ConnectX-5 EN 100GbE NICs (MCX516A-CDAT) again, this time we had four in the host. In this case we are using CentOS 8.4.2105 (Kernel – 5.13.7-1.el8.elrepo) and SPDK v21.07.

NVMe-oF initiator for Windows P5510

We use eight Intel P5510 Gen4 NVMe SSDs inside the host. The SSDs are split into two batches of four for NUMA alignment with the CPUs. They are configured in RAID0 for maximum performance.

NVMe oF initiator for Windows performance

For these tests, we performed the following benchmarks via FIO, using both Linux and Windows initiators.

  • Random Read 4K – 16 threads, 32 queue depth
  • 4K Random Writing – 8 threads, 4 queue depth
  • Random Read 64K – 4 threads, 32 queue depth
  • 64K Random Write – 4 threads, 1 queue depth
  • Sequential Read 1M – 2 Threads, 8 Queue Depth
  • Sequential Write 1M – 1 thread, 8 queue depth

A single test duration is 3600 seconds (1 hour). Before benchmarking the writes, the memory was initially warmed up for 3600 seconds (1 hour). All tests were carried out three times and the average value was used as the final result.

In our first group, which examined the performance of the Linux NVMeoF initiator across 4 clients, we measured 5.54 million IOPS at a bandwidth of 21.6 GB / s with 0.369 ms latency with 4K random read. The 4K random write performance measures 1.58 million IOPS at a bandwidth of 6.2 GB / s with a latency of 0.08 ms.

Moving to large block transfers, we measured both a random 64K and a sequential 1M test that focused on bandwidth across the fabric. With 64K random read, we measured 46.6 GB / s with 0.69 ms latency and 7.2 GB / s with 0.14 ms latency when writing. 1M sequential came in at 42.9 GB / s reading with 1.48 ms latency and 25.4 GB / s with 1.26 ms latency writing.

Next we switched to Windows, where we used the StarWind NVMeoF initiator for the same four clients. Here we measured 4.17 million IOPS in 4K random read or 16.3 GB / s with 0.35 ms latency. 4K random write came with 1.54M IOPS or 6GB / s with 0.07ms latency.

We then moved on to a larger 64K transfer size with the same random access profile. Here we measured 46.6 GB / s reading at 0.68 ms latency and 7.2 GB / s writing at 0.13 ms latency. When switching to our last workload profile with a transfer size of 1 million with a sequential access pattern, we measured 42.9 GB / s read with 1.38 ms latency and 25.2 GB / s write with 1.14 ms latency.

In a direct comparison of the numbers, both Windows and Linux performance were very close to each other with the exception of 4K Random Read. In all other tests, the performance gap was less than 3%. The main difference is really in the CPU overhead that is added as it traverses the Windows memory stack. This gives a difference of 2.7-3.7, with increased I / O itself contributing the most to CPU usage.

Going from 16% CPU usage in Linux to 44% in Windows is a pretty big jump, but an increase from 3.5% to 9% wouldn’t be felt to the same extent. For applications that need to run on Windows or the IT shop in general is more Windows-focused, StarWind’s primary goal was to provide NVMeoF features and performance that could clearly be achieved.

diploma

The aim of this analysis is not to determine the best or fastest way to implement your own NVMe-oF solution. Most storage deployments follow the application as opposed to the application that follows the storage. However, there are a number of reasons a company might want to use Windows. It could be specific applications, existing infrastructures, cost reasons, or any other number of problems that make Windows the preferred platform. At least now, with StarWinds NVMe-oF Initiator for Windows, we have a way of sharing NVMe SSDs and bringing them as close as possible to application systems.

NVMe-oF initiator for Windows front enclosures

If we ignore the client operating system for a moment, the main limitation in our tests really lies in the network connections between the systems. In our case, we saturated the network with 100 Gb NICs and hit it at 46.6 GB / s in both Linux and Windows environments. Even the maximum 4K random read test in Windows got 16.3 GB / s, which would give more than six 25 GbE links for random I / O. The network plays a more important role for NVMe-oF as the NVMe performance can absorb a lot of traffic in any way.

But at the end of the day, our goal was to evaluate how well the StarWind initiator worked. It works really damn well. Given the alternative, “No NVMe-oF for you!” under Windows we like to have all options. Yes, there is a CPU hit for the privilege, but while the Linux to Windows percentage deltas are scary, the perceived impact outside of 4K random reads is minimal. If you’re not sure if this is the right solution, you can try StarWind for free. It’s so easy to install that there is every reason to try it and see how well NVMe-oF can work for your applications in Windows.

StarWind product page

StarWind demo request form

Participate in StorageReview

Newsletter | YouTube | Podcast iTunes / Spotify | Instagram | Twitter | Facebook | Rss feed

About Willie Ash

Check Also

Windows 11: The Ars Technica Test

Microsoft Microsoft wanted everyone to use Windows 10. Given the slow adoption of Windows 8 …

Leave a Reply

Your email address will not be published. Required fields are marked *