What is NVMe?

NVMe is a software protocol to connect non-volatile mass storage devices via PCI Express hardware interfaces. The abbreviation NVMe stands for “non-volatile memory express”. A non-volatile memory is a storage medium which contains contents that are not lost when the power is switched off.

NVMe enables fast data access even for persistent memory technology. But how does it work, and what other advantages does NVM Express offer?

Traditionally, a distinction has been made between volatile main memory (random access memory, RAM) of a computer and non-volatile mass storage. While the RAM of a computer loses its contents when it is switched off, contents of the non-volatile mass storage device are retained. Following the use of punch cards during the early days of computer technology and the subsequent widespread tape storage systems, rotating hard disks (hard disk drives, HDDs) have become the standard device for mass storage.

The separation into main memory and hard disk was accompanied by drastically different access times. The time it took to access sequentially stored data in RAM was about a hundredth of the time it took to read the same data from a rotating hard disk. Only with the widespread use of mobile devices and the associated emergence of solid-state drives (SSD) was this dynamic extended.

Now, SSDs are widely installed in laptop and desktop computers and are referred to as continuous storage media. The data storage is completely electronic, free from moving parts – from the processor register, to the L1 and L2 caches, to the SSD mass storage device. However, data access to SSDs has largely been reliant on protocols optimized for rotating HDDs such as Serial ATA (SATA).

NVM Express is the first protocol specifically developed for non-rotating, SSD-based mass storage devices. It succeeds the long-established software interfaces SATA and AHCI. SATA in particular, which was initially used for the internal communication with SSDs, quickly developed into a bottleneck: Modern SSDs were able to achieve a much higher data throughput than the SATA protocol allowed. With the non-volatile memory express interface, which has been specifically optimized for SSD media, SSDs can exploit their full potential.

The existing PCI Express (Peripheral Component Interconnect Express, PCIe) standard is preferably used to connect SSD mass storage devices to the mainboard. Graphics cards and other hardware, for example, are connected via PCIe. PCIe is a physical interface. As a protocol, NVMe presents as a logical counterpart that is optimized for non-volatile SSDs. This means that a single driver on the operating system side is sufficient to address all NVM Express compatible SSDs by different manufacturers.

A software protocol is used to structure digital communication, i.e. the exchange of data. The protocol defines a set of commands that determine the language used between individual communication partners. The non-volatile memory express protocol makes do with a lean instruction set comprising only 13 commands.

Only ten admin commands are mandatory:

• Create I/O Submission Queue
• Delete I/O Submission Queue
• Create I/O Completion Queue
• Delete I/O Completion Queue
• Get Log Page
• Identify
• Abort
• Set Features
• Get Features
• Async Event Requests

There are also three I/O commands that are required:

• Read
• Write
• Flush

The modern hard drive has several rotating disks. Two read/write heads are usually used per disk. The read/write heads are all attached to a single, movable read/write arm. Due to this physical arrangement, neither HDDs nor their protocols are optimized for mass parallel access to data.

The well-established SATA protocol has a single command queue to execute hard disk commands one after the other. The non-volatile memory express protocol, on the other hand, supports 64,000 queues, with up to 64,000 commands per queue. This ensures a high degree of data access parallelism, which leads to high data throughput rates and low latencies.

Compared to HDDs, SSDs do not read and write data sequentially; the data access to SSDs is just as fast at every position as it is with random access main memory.

Protocol commands with NVMe are created and processed according to the following pattern:

1. Host transmits I/O Submission Queues
2. Controller recalls I/O Submission Queues
3. Controller runs I/O Submission Queues
4. Controller transmits I/O Completion Queues
5. Host receives I/O Completion Queues

NVM Express is advantageous because it uses the existing PCIe standard and the optimized protocol for modern solid-state storage. The lean protocol command set, tailored for the operation of SSDs, leads to low overhead when reading and writing data. The direct use of PCIe means there is no need for a protocol translator, which in turn results in lower latencies.

Because command queues are processed in parallel, SSDs connected to NVMe can develop their full performance potential. NVMe devices can also be connected via multi-pathing in professional hosting and high-performance computing environments. Here, the NVMe device is connected via multiple, parallel data lines within the computer infrastructure. If a line fails, the device can still be accessed. NVMe devices also support hot swapping; if a storage medium fails completely, it can be exchanged during operation.

NVM Express is used in almost all computer systems nowadays. From laptop and desktop computers to servers and workstations to supercomputers: fast SSD storage media connected via NVMe are installed everywhere. The different form factors used include:

• PCI Express cards
• M.2 cards
• SFF (small form factor) or 2,5" devices with U.2 connection
• E1 or NGSFF (next generation small form factor)

Another interesting development of the non-volatile memory express protocol is “NVMe over Fabrics”. Here, instead of a direct physical connection, a network connection is used to address NVMe solid-state memory via the network. In this way, collections of SSD media can be implemented providing enormous amounts of storage with short access times.