AGIO

This is the repository for AGIO v0.1, from the paper Asynchrony and GPUs: Bridging this Dichotomy for I/O with AGIO.

Software Requirements

• Linux kernel 6.8
• Nvidia driver 570 (open version)
• CMake 3.31
• CUDA 12.8 and its nvcc compiler
• GCC 12 and a C++ compiler which supports C++17

Different versions may work, but not tested

System Configurations

• Disable IOMMU (both in BIOS and kernel)
  To disable in the kernel, have iommu=off and intel_iommu=off / amd_iommu=off in the kernel command line
  (Check /etc/default/grub if your system uses grub, and verify using dmesg)
• Build kernel symbols for the Nvidia driver
  (sudo make inside the Nvidia driver source directory, usually located in /usr/src)
• Enable PCI P2PDMA and enable PCI BAR1
  Add parameters for the nvidia kernel module:

  $ sudo cat /etc/modprobe.d/nvidia.conf
  options nvidia NVreg_RegistryDwords="RMForceStaticBar1=1;ForceP2P=0;RmForceDisableIomapWC=1;PeerMappingOverride=1;"

• Lock GPU frequency using nvidia-smi
  (e.g. to use 1410Mhz, $ sudo nvidia-smi -lgc 1410,1410)

Build

$ mkdir build; cd build
$ cmake ..
$ make
$ cd module; make

You will have AGIO applications (main-appname) and the necessary kernel module (libnvm.ko) file ready. This repository includes all dependencies as a git subtree.

AGIO Configurations (via .toml files)

On initial cmake build, cmake will check if the build folder has the AGIO configuration files. If not, it will copy from the sources to the build directory, with a _in.toml suffix.

• options_in.toml: contains configurations for AGIO
• work_in-appname.toml: contains configurations for appname application

Please refer to the comments in the configuration files for additional details.

Application Input Datasets

Please refer to app/README.md.

Bind NVMe devices with AGIO

1. Unbind NVMe device(s) from the default kernel module (nvme). Assuming the PCIe BDF of the NVMe device is 0000:e5:00.0,
   $ echo "0000:e5:00.0" | sudo tee /sys/bus/pci/drivers/nvme/unbind
2. Load the libnvm module, and all NVMe devices not bound to the default kernel module will be bound to libnvm, with a device file /dev/libnvm* exposed.
   $ sudo insmod libnvm.ko

Bind NVMe devices back with default kernel module

1. Unload and unbind the NVMe devices from the libnvm kernel module
   $ sudo rmmod libnvm
2. For each unbound NVMe drive, bind the drive back to its default kernel module
   $ echo "0000:e5:00.0" | sudo tee /sys/bus/pci/drivers/nvme/bind

Acknowledgement

AGIO uses libnvm and BaM. AGIO also includes JSON for Modern C++ (json.hpp) and toml++ (toml.hpp).