This is a project I have been wanting to do for nearly two years now. I had originally tried to do this on two different machines with Intel Processors one I7 and one Xenon E5, but I had ran into a few issues with older motherboards and its IOMMU Groupings of PCI-E Video Devices, also the ACS Override just did not work with my CPUs so it was a bust.

I recently built a new computer for my Birthday with doing the Passthrough somewhat in mind but mainly as a machine to to test out some of the Ryzen CPU's so I never picked and chose specific parts, in retrospect I should have gotten less/quicker RAM. Much thanks to powerhouse from the linuxmint forums for posting an EXCELLENT guide: https://forums.linuxmint.com/viewtopic.php?t=212692

This is what I largely followed during my time with some minor differences (pci stubbing, probably unnecessary)

I have been an AMD CPU fan for quite sometime, one of my favorite machines I ever owned had AMD 64 Athlon x2. After I built my newest machine I quickly inspected the IOMMU Groupings with two video cards and noticed they were on the same PCI Grouping to my dismay :( I was a bit bummed because in order to get the Pass-through working when your cards are on the same Grouping is to patch your kernel to allow "ACS Override". After some research the risk associated with using the kernel patch to allow ACS override was not an option.

To learn more about IOMMU see this article: iommu-groups-inside-and-out

I considered buying a new motherboard, I had one in my Cart and was about to pull the trigger. Then I noticed someone mentioned updating the Motherboards BIOS, as after a specific version they run better with AMD processors. I decided to grab this and give it a go, the update came out on my Birthday so their was no way my BIOS had the firmware. I did some research on it and the versions of BIOS 1401 and above have AGSEA Code > 1.0.0.6.

To my surprise after rebooting the machine and inspecting my groupings, the Video Cards got split into different groups. With one card on 12 and one on 13, this meant it was now possible to do the Passthrough.

Machine Specifications
  • Host OS: Linux Mint 18.2 (xenial 16.04.1)
  • Kernel: 4.10.0-33-generic
  • Guest OS: Windows 10 x64
  • CPU: AMD Ryzen 7 1800x
  • Mobo: ASUS Crosshair VI Hero - 1501 BIOS Version
  • Host Card: NVIDIA GTX 1050TI SC
  • Guest: NVIDIA GTX 970
  • RAM: 64GBD
  • Disks
    • Solid State: 500GB, 1TB
    • 3TB, 2TB, 2TB
Setup

First step is to turn on SVM in your BIOS Settings. This is AMD's Virtualization technology similar to Intels vt-d. It could also be labeled something to the effect of IOMMU depending on the Motherboard. Reboot your machine.

Edit your default grub command line default
sudo nano /etc/default/grub

Locate Line:

GRUB_CMDLINE_LINUX_DEFAULT="quiet"  

Change it to:

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"  

Save and Close the grub file and update grub
sudo update-grub

Reboot your machine.

Lets figure out the device ID's of our cards:
lspci -nn | grep VGA

29:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:1c82] (rev a1)  
2a:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)  

These are the important bits from the listing of our video cards write them down:

  • 10de:13c2 - Device ID of the card I want to passthrough to the VM
  • 2a:00.0 - Bus Number for card you want to pass through
  • 29:00.0 - You also want to get the Bus number of your other card as well

Now we can list our IOMMU Groups and look at the Bus numbers assigned to each groups and determine if your video cards are on separate groupings.

for a in /sys/kernel/iommu_groups/*; do find $a -type l; done

/sys/kernel/iommu_groups/12/devices/0000:29:00.0
/sys/kernel/iommu_groups/12/devices/0000:29:00.1
/sys/kernel/iommu_groups/13/devices/0000:2a:00.1
/sys/kernel/iommu_groups/13/devices/0000:2a:00.0`

(I left out a bunch of the other groups, and just copied the video cards groups)

We can see both of our video cards are on different groups, this is perfect! One is on 12 and one is on 13. If both were on 12 or 13 then this would not be as "easy" to accomplish and it is much less secure. Unfortunately if your groups are similar this guide going forward will not be much help as I don't show how to apply the ACS Override patch.

Now that we know our setup is good, have IOMMU on, have our device numbers we can go about disabling the card on boot.

We want to stub both id's for our video card, one is a sound and one is a video device. To get the ID's it can be done by inspecting the card we want to passthrough further.

lspci -nn | grep 29:00.

29:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:1c82] (rev a1)  
29:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fb9] (rev a1)  

What we want from here is the 10de:0fb9 sound id. Now we have both ids to stub up.

pci-stub.ids=10de:13c2,10de:0bff  

Edit your default grub command line default again...
sudo nano /etc/default/grub

Locate Line:

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"  

Make sure to remove nomodeset if this is in this line, you should have a video driver installed on your Host os now and it should be used by the card you want the host machine to have. Example: My setup at this point has the proprietary NVIDIA driver in use on my GTX 1050 ti sc. By default it was using the open source nouveau driver. The secondary card plugged into the machine the GTX 970 is still using the nouveau driver. At this point your video cards have different drivers assigned to them.

Change it to:

GRUB_CMDLINE_LINUX_DEFAULT="modprobe.blacklist=nouveau quiet amd_iommu=on iommu=pt pci-stub.ids=10de:13c2,10de:0bff"  

modprobe.blacklist stops the nouveau driver from loading, make sure if your using 2 NVIDIA Cards before blacklisting to have the Host machines card on a different driver than the nouveau driver. You will have many issues on your next reboot otherwise lol.

If your blacklisting an AMD card: modprobe.blacklist=radeon or modprobe.blacklist=amdgpu

Save and Close the grub file and update grub
sudo update-grub

Next thing I do is setup Jacob's PPA

sudo add-apt-repository ppa:jacob/virtualisation  
sudo apt-get update  
sudo apt-get install qemu-kvm seabios qemu-utils hugepages bridge-utils  

To make the video card available to the VM Edit your local.conf file to add the
sudo -i

nano /etc/modprobe.d/local.conf at the bottom of the file add:

options vfio-pci ids=10de:13c2,10de:0fbb  

Your also going to want to have your kvm ignore msrs, I tried to run some software that uses these processor debug functions this somehow allows it to work/disables I dont know exactly...cant hurt.

echo "options kvm ignore_msrs=1" >> /etc/modprobe.d/kvm.conf

sudo nano /etc/initramfs-tools/modules

add to the bottom of the file:

vfio  
vfio_iommu_type1  
vfio_pci  
vfio_virqfd  
vhost-net  
pci-stub  

Update your initramfs modules
update-initramfs -u

Reboot your machine

Setup hugepages
sudo -i sudo nano /etc/default/qemu-kvm

KVM_HUGEPAGES=1  

You need to Reboot again...
sudo nano /etc/sysctl.conf

The hugepages value will differ based on how much ram you want to allocate,
the rule is like half the number of pages than ram you want to allocate minus like 10%. so 8GB would be 4096 - 400 = 4596, rounded down to like 4500.

vm.nr_hugepages = 4500  

If you want to allocate 8GB of ram you want roughly 4096 huge pages, you can cut a few off and round up. Myself I wanted 32gb of ram so my huge pages is 16384 and I have an excess of ram so I am not worried about rounding the page number down.
You will Reboot again...

sudo hugeadm --set-recommended-shmmax

Install ovmf bios

sudo apt-get install ovmf

Download the latest VIRTIO windows driver iso (This is used so Windows 10's installer can find our virtio drive we pass to it, otherwise you see no disks trying to install Windows 10, this disk is also used once your in windows to install the network drivers)

Your in the home stretch your basically done now!

From here all you need to do is create a disk image and make the VM and pass your devices to it. You also need to make a network bridge to use with your VM. The original guide I linked has good steps on setting up a network bridge.

Here is an example qemu launch script I found/modified for installing Windows 10.
10_install.sh

Here is the script I use to launch the VM after the subsequent install.

10w.sh

Your going to want to use the scripts as a base to create what works for your setup. With my configuration, I pass through both my USB keyboard and mouse, because I have two plugged into the machine. I also pass through a USB sound card. Your going to need to inspect your usb devices using
lsusb and determine which devices you want to pass.

Benchmarks

User Benchmark - http://www.userbenchmark.com/UserRun/4829650
Heaven Benchmark - Ultra/Settings - 54.4fps overall and a score of 1371

Been playing games like Quake Champions (In Beta, no issues whatsoever) Elder Scrolls online. The machine performs so good, nothing like a VM your expirenced to using. THe video cards acceleration helps massively. I read somewhere that the overhead you lose using the hypervisor is something like .2, so your hardware is essentially running at 99.8 % efficiency.

Todo
  • Problem Section
  • Week 1 Experiences
  • Week 2 Experiences