Pci passthrough vfio two identical GPUs, AMD Threadripper 1950x, X399
Receipt for vfio PCI Passthrough - pass physical videocard to qemu machine (virtmanager).
Firstly - use video instruction from youtube ( Ubuntu 20.04 ) : [https://www.youtube.com/watch?v=tDMoEvf8Q18](https://www.youtube.com/watch?v=tDMoEvf8Q18)
0.Setup.
Motherboard: ASUS ROG Zenith Extreme ( X399 Chipset )
Processor: AMD Ryzen Threadripper 1950X
HDD: NVMe Intel SSDSCKKW010X6X1
Memory: 32 GB Kingston HyperX Predator
GPU: ROG-POSEIDON-GTX1080TI-P11G-GAMING X2
Step 1. See currently installed drivers
root@SWARM# lspci -k | grep -A3 "NVIDIA Corporation GP102"
09:00.0 VGA compatible controller: NVIDIA Corporation GP102 \[GeForce GTX 1080 Ti\] (rev a1)
Subsystem: ASUSTeK Computer Inc. GP102 [GeForce GTX 1080 Ti]
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
09:00.1 Audio device: NVIDIA Corporation GP102 HDMI Audio Controller (rev a1)
Subsystem: ASUSTeK Computer Inc. GP102 HDMI Audio Controller
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
42:00.0 VGA compatible controller: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1)
Subsystem: ASUSTeK Computer Inc. GP102 [GeForce GTX 1080 Ti]
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
42:00.1 Audio device: NVIDIA Corporation GP102 HDMI Audio Controller (rev a1)
Subsystem: ASUSTeK Computer Inc. GP102 HDMI Audio Controller
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
2. Use Udevd to set another driver based by PCI address
Show current divers and identificators for device:
udevadm test /sys/bus/pci/devices/0000:09:00.0
Fix udevd rules based on ID\_PATH\_TAG (use nvidia driver only for pci-0000\_42\_00\_0 device ) :
root@SWARM:/home/okkk# cat /usr/lib/udev/rules.d/71-nvidia.rules
# Tag the device as master-of-seat so that logind is happy
# (see LP: #1365336)
#SUBSYSTEM=="pci", ATTRS{vendor}=="0x10de", DRIVERS=="nvidia", TAG+="seat", TAG+="master-of-seat"
SUBSYSTEM=="pci", ID_PATH_TAG="pci-0000_42_00_0", ATTRS{vendor}=="0x10de", DRIVERS=="nvidia", TAG+="seat", TAG+="master-of-seat"
3. Errors after try to create new virtual machine
Error: Failed to set iommu for container: Operation not permitted
libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2020-05-22T14:42:35.441512Z qemu-system-x86_64: -device vfio-pci,host=0000:09:00.0,id=hostdev0,bus=pci.4,addr=0x0: vfio 0000:09:00.0: failed to setup container for group 14: Failed to set iommu for container: Operation not permitted
SWARM:~$ dmesg | grep 'remapping'
[ 0.000000] AMD-Vi: Disabling interrupt remapping
[ 3704.964705] vfio_iommu_type1_attach_group: No interrupt remapping support. Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
[ 4129.622027] vfio_iommu_type1_attach_group: No interrupt remapping support. Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
[ 4327.845813] vfio_iommu_type1_attach_group: No interrupt remapping support. Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
Workaround:
sudo echo "Y" > /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts
Or simply add param for module to grub:
root@SWARM:/home/okkk# cat /etc/default/grub | grep "CMDLINE_LINUX"
GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on iommu=pt iommu=1 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1 apparmor=0"
GRUB_CMDLINE_LINUX=""
And don't forget:
root@SWARM:/home/okkk# update-grub
3. Things doesnot helping.
All this thing does not work for GPUs because at boot time nvidia driver takes control over the GPU device ( 0000:09:00.0 for example), and any commands listed below will be freeze.
Audio device driver ( 0000:09:00.1 ) can be overrided by hands.
Override drive by hands (for example):
echo "vfio-pci" > /sys/bus/pci/devices/0000\:09\:00.1/driver_override
or:
driverctl set-override 0000:09:00.0 vfio-pci
driverctl set-override 0000:09:00.1 vfio-pci
For unbind drivers you can do:
echo 0000:09:00.1 > /sys/bus/pci/drivers/nvidia/unbind
Firstly - use video instruction from youtube ( Ubuntu 20.04 ) : [https://www.youtube.com/watch?v=tDMoEvf8Q18](https://www.youtube.com/watch?v=tDMoEvf8Q18)
0.Setup.
Motherboard: ASUS ROG Zenith Extreme ( X399 Chipset )
Processor: AMD Ryzen Threadripper 1950X
HDD: NVMe Intel SSDSCKKW010X6X1
Memory: 32 GB Kingston HyperX Predator
GPU: ROG-POSEIDON-GTX1080TI-P11G-GAMING X2
Step 1. See currently installed drivers
root@SWARM# lspci -k | grep -A3 "NVIDIA Corporation GP102"
09:00.0 VGA compatible controller: NVIDIA Corporation GP102 \[GeForce GTX 1080 Ti\] (rev a1)
Subsystem: ASUSTeK Computer Inc. GP102 [GeForce GTX 1080 Ti]
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
09:00.1 Audio device: NVIDIA Corporation GP102 HDMI Audio Controller (rev a1)
Subsystem: ASUSTeK Computer Inc. GP102 HDMI Audio Controller
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
42:00.0 VGA compatible controller: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1)
Subsystem: ASUSTeK Computer Inc. GP102 [GeForce GTX 1080 Ti]
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
42:00.1 Audio device: NVIDIA Corporation GP102 HDMI Audio Controller (rev a1)
Subsystem: ASUSTeK Computer Inc. GP102 HDMI Audio Controller
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
2. Use Udevd to set another driver based by PCI address
Show current divers and identificators for device:
udevadm test /sys/bus/pci/devices/0000:09:00.0
Fix udevd rules based on ID\_PATH\_TAG (use nvidia driver only for pci-0000\_42\_00\_0 device ) :
root@SWARM:/home/okkk# cat /usr/lib/udev/rules.d/71-nvidia.rules
# Tag the device as master-of-seat so that logind is happy
# (see LP: #1365336)
#SUBSYSTEM=="pci", ATTRS{vendor}=="0x10de", DRIVERS=="nvidia", TAG+="seat", TAG+="master-of-seat"
SUBSYSTEM=="pci", ID_PATH_TAG="pci-0000_42_00_0", ATTRS{vendor}=="0x10de", DRIVERS=="nvidia", TAG+="seat", TAG+="master-of-seat"
3. Errors after try to create new virtual machine
Error: Failed to set iommu for container: Operation not permitted
libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2020-05-22T14:42:35.441512Z qemu-system-x86_64: -device vfio-pci,host=0000:09:00.0,id=hostdev0,bus=pci.4,addr=0x0: vfio 0000:09:00.0: failed to setup container for group 14: Failed to set iommu for container: Operation not permitted
SWARM:~$ dmesg | grep 'remapping'
[ 0.000000] AMD-Vi: Disabling interrupt remapping
[ 3704.964705] vfio_iommu_type1_attach_group: No interrupt remapping support. Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
[ 4129.622027] vfio_iommu_type1_attach_group: No interrupt remapping support. Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
[ 4327.845813] vfio_iommu_type1_attach_group: No interrupt remapping support. Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
Workaround:
sudo echo "Y" > /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts
Or simply add param for module to grub:
root@SWARM:/home/okkk# cat /etc/default/grub | grep "CMDLINE_LINUX"
GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on iommu=pt iommu=1 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1 apparmor=0"
GRUB_CMDLINE_LINUX=""
And don't forget:
root@SWARM:/home/okkk# update-grub
3. Things doesnot helping.
All this thing does not work for GPUs because at boot time nvidia driver takes control over the GPU device ( 0000:09:00.0 for example), and any commands listed below will be freeze.
Audio device driver ( 0000:09:00.1 ) can be overrided by hands.
Override drive by hands (for example):
echo "vfio-pci" > /sys/bus/pci/devices/0000\:09\:00.1/driver_override
or:
driverctl set-override 0000:09:00.0 vfio-pci
driverctl set-override 0000:09:00.1 vfio-pci
For unbind drivers you can do:
echo 0000:09:00.1 > /sys/bus/pci/drivers/nvidia/unbind