Intel Arc PCI Passthru using bhyve on illumos

A while ago I bought a second hand NVIDIA Quadro P1000 on eBay, it being a Low-Profile card, means it fits well in my 2U servers in my home lab.

This card has been used in my Plex Media Server VM for a while, although it is working well, it is a few generations old and does not support newer codecs like h265 and AV1.

Intel recently launched the Intel Arc A310¹ which does support all the newer codecs, additionally one can avoid the horrible mess that is NVIDIA drivers on Linux! The i915 driver in kernel supports Arc, assuming one is running a recent enough version.

Getting it to work was however not as easy as I had hoped, bhyve does not support ReBAR, but I am only interested in using the card for transcoding so leaving some performance on the table should not be a big issue. Seems I was wrong on that front, I did manage to get it to work by setting some additional driver options in the guest.

Hardware Preparation

1. disable auto start of the plex bhyve vm
2. remove the current ppt config
3. shutdown the host system
4. swap the quadro for the A310
5. boot the host system

Updating ppt_matches and ppt_aliases

On illumos to use PCIe Passthru we need to tell the kernel to bind the ppt driver to it, finding the correct IDs can be a bit tricky. However the pciadm utility can help.

NOTE: I initially grepped on A310, but that meant I missed the audio devices, grepping for DG2 makes both show up

sjorge@jupiter:~$ pfexec pkg install diagnostic/pci
sjorge@jupiter:~$ /usr/lib/pci/pcieadm show-devs -o device,driver,vid,did,path | grep -i DG2
DG2 [Arc A310]                ppt            8086  56a6  /pci@a3,0/pci8086,2030@0/pci8086,4fa1@0/pci8086,4fa4@1/display@0
DG2 Audio Controller          ppt            8086  4f92  /pci@a3,0/pci8086,2030@0/pci8086,4fa1@0/pci8086,4fa4@4/pci1849,6007@0

In the output above I already did the steps to bind ppt to the devices, if this is not the case the driver column will be empty as there is no driver support for in illumos. For the ppt_matches file we are interested in the PID and DID values, for ppt_aliases we are interested in the PATH².

First we update /etc/ppt_matches by adding the following:

# Intel Arc A310
pciex8086,56a6
pciex8086,4f92

Next we update /etc/ppt_aliases by adding either one of the following:

# Inte Arc A310
# WARN: bind all Arc A310 cards in the system
#ppt "pciex8086,56a6"
#ppt "pciex8086,4f92"
# NOTE: bind a specific Arc A310 based on PATH
ppt "/pci@a3,0/pci8086,2030@0/pci8086,4fa1@0/pci8086,4fa4@1/display@0"
ppt "/pci@a3,0/pci8086,2030@0/pci8086,4fa1@0/pci8086,4fa4@4/pci1849,6007@0"

Binding to just PID,DID is easier and has the benefit that one can move the card between slots and it will still work. In the case of an Arc A310 this is probably fine as illumos doesn’t have drivers anyway. Alternatively you can also specify the full PATH from the pciadm output from earlier to bind a single card.

Finally, we update the boot-archive and reboot the system.

sjorge@jupiter:~$ pfexec bootadm update-archive
updating /platform/i86pc/amd64/boot_archive (CPIO)
sjorge@jupiter:~$ pfexec shutdown -i6 -g0 -y

Booting the VM

In my case the VM is running Ubuntu 22.04 LTS, the shipped kernel is too old to support the Arc (A310) but we’ll take care of that later.

Update the VM to pass through both the GPU and Audio device:

sjorge@jupiter:~$ pptadm list -a
DEV        VENDOR DEVICE PATH
<additional devices redacted>
/dev/ppt6  8086   56a6   /pci@a3,0/pci8086,2030@0/pci8086,4fa1@0/pci8086,4fa4@1/display@0
/dev/ppt7  8086   4f92   /pci@a3,0/pci8086,2030@0/pci8086,4fa1@0/pci8086,4fa4@4/pci1849,6007@0

Now use zonecfg or zadm to pass both ppt6 and ppt7 to the VM.

My final zone config looks like this:

sjorge@jupiter:~$ pfexec zadm show artemis
---
acpi: on
autoboot: 'false'
bootargs: ''
bootdisk:
  blocksize: 4K
  path: rpool/vms/artemis/disk0
  size: 35G
  sparse: 'false'
bootorder:
- cd
bootrom: BHYVE_RELEASE
brand: bhyve
cloud-init: off
cpu-shares: '25'
diskif: nvme
fs-allowed: ''
hostbridge: i440fx
hostid: ''
ip-type: exclusive
limitpriv: default
memreserve: on
net:
- global-nic: ixgbe0
  mac-addr: 00:00:00:00:00:00
  physical: artemis0
  vlan-id: '100'
  vqsize: '32768'
netif: virtio
pool: ''
ppt:
- device: ppt6
  state: on
- device: ppt7
  state: on
ram: 8G
rng: off
scheduling-class: ''
type: generic
uefivars: on
vcpus: 8,sockets=1,cores=4,threads=2
vga: off
vnc:
  enabled: off
xhci: off
zonename: artemis
zonepath: /zones/artemis

You can now boot the VM:

sjorge@jupiter:~$ pfexec zadm boot -c artemis

Once the VM is booted (it might hang for a long time, we’ll fix this later), install the required kernel and packages:

# NOTE: When using Ubuntu 22.04 LTS we need to install the hardware enablement (HWE) kernel.
sjorge@artemis:~$ sudo apt install -y --install-suggests linux-generic-hwe-22.04
# NOTE: Additionally install the required libraries for transcoding and the tools to monitor the GPU
sjorge@artemis:~$ sudo apt install -y intel-gpu-tools intel-opencl-icd

Reboot the VM, you might see this hang on the console:

[    3.628444] i915 0000:00:09.0: vgaarb: deactivate vga console
[    3.629218] i915 0000:00:09.0: BAR 0: releasing [mem 0xc0000000-0xc0ffffff 64bit]
[    3.630134] i915 0000:00:09.0: BAR 2: releasing [mem 0xd000000000-0xd00fffffff 64bit pref]
[    3.631132] i915 0000:00:09.0: [drm] Failed to resize BAR2 to 4096M (-EBUSY)
[    3.631966] i915 0000:00:09.0: BAR 2: assigned [mem 0xd000000000-0xd00fffffff 64bit pref]
[    3.642053] i915 0000:00:09.0: BAR 0: assigned [mem 0xd010000000-0xd010ffffff 64bit]
[    3.651860] i915 0000:00:09.0: [drm] Local memory IO size: 0x0000000010000000
[    3.652751] i915 0000:00:09.0: [drm] Local memory available: 0x00000000fd000000
[    3.653630] i915 0000:00:09.0: [drm] Using a reduced BAR size of 256MiB. Consider enabling 'Resizable BAR' or similar, if available in the BIOS.
[  183.655945] ------------[ cut here ]------------
[  183.656478] i915 0000:00:09.0: drm_WARN_ON_ONCE(timeout_base_ms > 3)
[  183.656495] WARNING: CPU: 0 PID: 139 at drivers/gpu/drm/i915/intel_pcode.c:196 skl_pcode_request+0x26e/0x280 [i915]
[  183.658569] Modules linked in: i915(+) drm_buddy video wmi i2c_algo_bit ttm drm_display_helper cec rc_core drm_kms_helper syscopyarea sysfillrect aesni_intel sysimgblt crypto_simd drm psmouse cryptd nvme input_leds serio_raw nvme_core mac_hid nvme_common
[  183.661087] CPU: 0 PID: 139 Comm: systemd-udevd Not tainted 6.2.0-37-generic #38~22.04.1-Ubuntu
[  183.661987] Hardware name: OmniOS OmniOS HVM/BHYVE, BIOS 14.0 10/10/2021
[  183.662681] RIP: 0010:skl_pcode_request+0x26e/0x280 [i915]
[  183.663378] Code: 4c 8b 7f 50 4d 85 ff 75 03 4c 8b 3f e8 cb ce c6 fa 48 c7 c1 f0 53 a7 c0 4c 89 fa 48 c7 c7 6c ff aa c0 48 89 c6 e8 d2 b5 29 fa <0f> 0b e9 07 ff ff ff e8 56 a7 24 fb 66 0f 1f 44 00 00 90 90 90 90
[  183.665422] RSP: 0018:ffffa2578070b8d0 EFLAGS: 00010246
[  183.666037] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[  183.666833] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[  183.667566] RBP: ffffa2578070b920 R08: 0000000000000000 R09: 0000000000000000
[  183.668308] R10: 0000000000000000 R11: 0000000000000000 R12: ffff94a118459ca8
[  183.669040] R13: 000000000000007e R14: 0000000000000000 R15: ffff94a100dcd0f0
[  183.669776] FS:  00007f12caca58c0(0000) GS:ffff94a237c00000(0000) knlGS:0000000000000000
[  183.670712] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  183.671361] CR2: 00007fe20c71cae0 CR3: 0000000102b42004 CR4: 00000000003706f0
[  183.672113] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  183.672847] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  183.673592] Call Trace:
[  183.673864]  <TASK>
[  183.674105]  ? show_regs+0x72/0x90
[  183.674476]  ? skl_pcode_request+0x26e/0x280 [i915]
[  183.675115]  ? __warn+0x8d/0x160
[  183.675471]  ? skl_pcode_request+0x26e/0x280 [i915]
[  183.676107]  ? report_bug+0x1bb/0x1d0
[  183.676500]  ? handle_bug+0x46/0x90
[  183.676881]  ? exc_invalid_op+0x19/0x80
[  183.677302]  ? asm_exc_invalid_op+0x1b/0x20
[  183.677817]  ? skl_pcode_request+0x26e/0x280 [i915]
[  183.678482]  intel_pcode_init+0x48/0x60 [i915]
[  183.679059]  i915_pcode_init+0x3c/0xb0 [i915]
[  183.679627]  i915_driver_hw_probe+0x2bb/0x340 [i915]
[  183.680266]  i915_driver_probe+0xfa/0x450 [i915]
[  183.680866]  ? acpi_dev_found+0x66/0x80
[  183.681277]  i915_pci_probe+0x111/0x1f0 [i915]
[  183.681855]  local_pci_probe+0x4b/0xb0
[  183.682252]  pci_call_probe+0x55/0x190
[  183.682721]  pci_device_probe+0x84/0x120
[  183.683223]  really_probe+0x1ed/0x450
[  183.683716]  __driver_probe_device+0x8a/0x190
[  183.684290]  driver_probe_device+0x23/0xd0
[  183.684837]  __driver_attach+0x10f/0x220
[  183.685321]  ? __pfx___driver_attach+0x10/0x10
[  183.685909]  bus_for_each_dev+0x83/0xe0
[  183.686418]  driver_attach+0x1e/0x30
[  183.686889]  bus_add_driver+0x152/0x250
[  183.687403]  driver_register+0x83/0x160
[  183.687894]  __pci_register_driver+0x68/0x80
[  183.688483]  i915_pci_register_driver+0x23/0x30 [i915]
[  183.689283]  i915_init+0x37/0x120 [i915]
[  183.689916]  ? __pfx_init_module+0x10/0x10 [i915]
[  183.690644]  do_one_initcall+0x49/0x240
[  183.691116]  ? kmalloc_trace+0x2a/0xb0
[  183.691575]  do_init_module+0x52/0x240
[  183.692045]  load_module+0xb96/0xd60
[  183.692518]  __do_sys_finit_module+0xcc/0x150
[  183.693045]  ? __do_sys_finit_module+0xcc/0x150
[  183.693560]  __x64_sys_finit_module+0x18/0x30
[  183.694025]  do_syscall_64+0x5c/0x90
[  183.694414]  ? do_syscall_64+0x69/0x90
[  183.694816]  ? do_syscall_64+0x69/0x90
[  183.695220]  ? do_syscall_64+0x69/0x90
[  183.695621]  ? sysvec_call_function+0x4e/0xb0
[  183.696097]  entry_SYSCALL_64_after_hwframe+0x73/0xdd
[  183.696635] RIP: 0033:0x7f12cb39ca7d
[  183.697016] Code: 5b 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 83 a3 0f 00 f7 d8 64 89 01 48
[  183.698928] RSP: 002b:00007ffca0e465a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[  183.699895] RAX: ffffffffffffffda RBX: 000055796c98e8c0 RCX: 00007f12cb39ca7d
[  183.700746] RDX: 0000000000000000 RSI: 00007f12cb535441 RDI: 0000000000000018
[  183.701484] RBP: 0000000000020000 R08: 0000000000000000 R09: 0000000000000002
[  183.702226] R10: 0000000000000018 R11: 0000000000000246 R12: 00007f12cb535441
[  183.703073] R13: 000055796c9857d0 R14: 000055796c98c410 R15: 000055796c98b5b0
[  183.703923]  </TASK>
[  183.704215] ---[ end trace 0000000000000000 ]---
[  183.755510] i915 0000:00:09.0: [drm] *ERROR* gt0: intel_pcode_init failed -11
[  183.772290] i915 0000:00:09.0: Device initialization failed (-11)
[  183.772937] i915 0000:00:09.0: Please file a bug on drm/i915; see https://gitlab.freedesktop.org/drm/intel/-/wikis/How-to-file-i915-bugs for details.
[  183.774466] i915: probe of 0000:00:09.0 failed with error -11

We can fix this by setting some options for the i915 driver, you can use modinfo -p i915 to see all options but the following seems to work:

root@artemis:~# echo "options i915 disable_display=true lmem_bar_size=256 reset=1" > /etc/modprobe.d/i915.conf
root@artemis:~# update-initramfs -k all -u

This will stop the driver from trying to ReBAR from 256M -> 4G, it will also do a full reset of the device, when dealing with PCIe passthru this is generally a good idea as the device might not be in the expected state that it would be on a cold boot. Finally, we also disable the displays, not 100% sure this is needed but since I’m using it for transcoding it prevents the kernel from trying to set up KMS and other things.

You can now reboot the VM and select the card in Plex Media Server.