Hi Alex, Alex Williamson <alex.william...@redhat.com> wrote: > On Wed, 2014-02-05 at 22:10 +0100, Maik Broemme wrote: > > Hi Alex, > > > > Alex Williamson <alex.william...@redhat.com> wrote: > > > On Wed, 2014-02-05 at 19:59 +0100, Maik Broemme wrote: > > > > Hi, > > > > > > > > currently VFIO with multi GPU passthrough is working partially and > > > > hopefully somebody has a hint about the problem. I'm doing passthrough > > > > of an AMD Radeon R9 290X and AMD Radeon 7870 GHz Edition to a single VM. > > > > > > > > If the VM is running Linux this works quite well with radeon or fglrx > > > > driver. Please see 'dmesg' log attached, when using the radeon driver. > > > > If needed I can also post one with fglrx driver. > > > > > > > > If I do the exact same passthrough to a Windows VM and use latest AMD > > > > Catalyst 14.1 (2/1/2014) or AMD Catalyst 13.12 (12/18/2013) I can get > > > > only the first device working (AMD R9 290X) with 'x-vga=on'. I don't > > > > enable 'x-vga=on' on second device as this should never work. :) > > > > > > Why not? The guest is able to change the VGA enable bit in the emulated > > > bridge registers and access VGA space of each device, just like happens > > > on bare metal. You'll only get one device initialized from seabios, but > > > that's the same as would happen on bare metal as well. > > > > > > > Well it was just my guess as it would behave like most physical boxes > > in this case. :) > > > > > > I see > > > > BIOS boot screen and everything works fine except for the second GPU. > > > > The windows device manager just show me "Code 12" for the second GPU > > > > and its HD Audio device. Code 12 means: "This device cannot find enough > > > > free resources that it can use". > > > > > > I've seen the same using Nvidia GRID GPUs (w/o x-vga=on), but only with > > > the Q35 chipset model, Linux works, Windows reports Code 12. I have no > > > idea why as all the PCI resources appear to be properly sized and > > > mapped. FWIW, 2 GRID GPUs assigned to a guest do work with the 440FX > > > chipset model. Beyond 2 we run out of MMIO resources below 4G and > > > something bad happens. > > > > > > > Interesting. I will try 440FX a bit later and see if this works. What I > > can also do is to post system resource conflicts from Windows, the AMD > > Catalyst Center has it integrated. Maybe this will help? > > If you actually see conflicts, then yes. The Code 12 I've seen I was > never able to identify a conflict. The trouble with 440FX is that > you'll need to use pci-bridges to isolate VGA space of each GPU. > Otherwise one card would need to be disabled to ensure the VGA accesses > go to the other. >
Okay I've collected all necessary information (hopefully). Some are in German but if needed I can translate it. Please find it below: - Conflicts: E/A-Port 0x000003C0-0x000003DF AMD Radeon R9 200 Series E/A-Port 0x000003C0-0x000003DF Intel(R) 5520/5500/X58 I/O Hub PCI Express Root Port 0 - 3420 IRQ 10 AMD Radeon HD 7800 Series IRQ 10 Intel(R) ICH9 Family SMBus Controller - 2930 Speicheradresse 0xFE800000-0xFE83FFFF AMD Radeon R9 200 Series Speicheradresse 0xFE800000-0xFE83FFFF Intel(R) 5520/5500/X58 I/O Hub PCI Express Root Port 0 - 3420 Speicheradresse 0xE0000000-0xEFFFFFFF AMD Radeon HD 7800 Series Speicheradresse 0xE0000000-0xEFFFFFFF Intel(R) 5520/5500/X58 I/O Hub PCI Express Root Port 0 - 3420 Speicheradresse 0xA0000-0xBFFFF AMD Radeon R9 200 Series Speicheradresse 0xA0000-0xBFFFF PCI-Bus Speicheradresse 0xA0000-0xBFFFF Intel(R) 5520/5500/X58 I/O Hub PCI Express Root Port 0 - 3420 Speicheradresse 0xC0000000-0xCFFFFFFF AMD Radeon R9 200 Series Speicheradresse 0xC0000000-0xCFFFFFFF PCI-Bus Speicheradresse 0xC0000000-0xCFFFFFFF Intel(R) 5520/5500/X58 I/O Hub PCI Express Root Port 0 - 3420 E/A-Port 0x000003B0-0x000003BB AMD Radeon R9 200 Series E/A-Port 0x000003B0-0x000003BB Intel(R) 5520/5500/X58 I/O Hub PCI Express Root Port 0 - 3420 Speicheradresse 0xFE600000-0xFE63FFFF AMD Radeon HD 7800 Series Speicheradresse 0xFE600000-0xFE63FFFF Intel(R) 5520/5500/X58 I/O Hub PCI Express Root Port 0 - 3420 E/A-Port 0x0000C000-0x0000C0FF AMD Radeon HD 7800 Series E/A-Port 0x0000C000-0x0000C0FF Intel(R) 5520/5500/X58 I/O Hub PCI Express Root Port 0 - 3420 E/A-Port 0x0000D000-0x0000D0FF AMD Radeon R9 200 Series E/A-Port 0x0000D000-0x0000D0FF Intel(R) 5520/5500/X58 I/O Hub PCI Express Root Port 0 - 3420 - Display devices: Name AMD Radeon R9 200 Series PNP-Gerätekennung PCI\VEN_1002&DEV_67B0&SUBSYS_0B001002&REV_00\4&2122111D&0&0018 Adaptertyp AMD Radeon Graphics Processor (0x67B0), Advanced Micro Devices, Inc.-kompatibel Adapterbeschreibung AMD Radeon R9 200 Series Adapter-RAM (1.048.576) Bytes Installierte Treiber aticfx64.dll,aticfx64.dll,aticfx64.dll,aticfx32,aticfx32,aticfx32,atiumd64.dll,atidxx64.dll,atidxx64.dll,atiumdag,atidxx32,atidxx32,atiumdva,atiumd6a.cap,atitmm64.dll Treiberversion 13.350.1005.0 INF-Datei oem7.inf (Abschnitt ati2mtag_Hawaii) Farbebenen Nicht verfügbar Farbtabelleneinträge 4294967296 Auflösung 1920 x 1080 x 60 Hz Bits/Pixel 32 Speicheradresse 0xC0000000-0xCFFFFFFF Speicheradresse 0xD0000000-0xD07FFFFF E/A-Port 0x0000D000-0x0000D0FF Speicheradresse 0xFE800000-0xFE83FFFF IRQ-Kanal IRQ 4294967287 E/A-Port 0x000003B0-0x000003BB E/A-Port 0x000003C0-0x000003DF Speicheradresse 0xA0000-0xBFFFF Treiber c:\windows\system32\drivers\atikmpag.sys (8.14.1.6367, 622,00 KB (636.928 Bytes), 31.01.2014 20:28) Name AMD Radeon HD 7800 Series PNP-Gerätekennung PCI\VEN_1002&DEV_6818&SUBSYS_32511682&REV_00\4&49049C7&0&0820 Adaptertyp Nicht verfügbar, Advanced Micro Devices, Inc.-kompatibel Adapterbeschreibung AMD Radeon HD 7800 Series Adapter-RAM Nicht verfügbar Installierte Treiber aticfx64.dll,aticfx64.dll,aticfx64.dll,aticfx32,aticfx32,aticfx32,atiumd64.dll,atidxx64.dll,atidxx64.dll,atiumdag,atidxx32,atidxx32,atiumdva,atiumd6a.cap,atitmm64.dll Treiberversion 13.350.1005.0 INF-Datei oem7.inf (Abschnitt ati2mtag_R575B) Farbebenen Nicht verfügbar Farbtabelleneinträge Nicht verfügbar Auflösung Nicht verfügbar Bits/Pixel Nicht verfügbar Speicheradresse 0xE0000000-0xEFFFFFFF Speicheradresse 0xFE600000-0xFE63FFFF E/A-Port 0x0000C000-0x0000C0FF IRQ-Kanal IRQ 10 Treiber c:\windows\system32\drivers\atikmpag.sys (8.14.1.6367, 622,00 KB (636.928 Bytes), 31.01.2014 20:28) - I/O: 0x00000000-0x00000CD7 PCI-Bus OK 0x00000060-0x00000060 Standardtastatur (PS/2) OK 0x00000064-0x00000064 Standardtastatur (PS/2) OK 0x00000070-0x00000071 System CMOS/Echtzeituhr OK 0x00000072-0x00000077 System CMOS/Echtzeituhr OK 0x000002F8-0x000002FF Kommunikationsanschluss (COM2) OK 0x00000378-0x0000037F Druckeranschluss (LPT1) OK 0x000003B0-0x000003BB AMD Radeon R9 200 Series OK 0x000003B0-0x000003BB Intel(R) 5520/5500/X58 I/O Hub PCI Express Root Port 0 - 3420 OK 0x000003C0-0x000003DF AMD Radeon R9 200 Series OK 0x000003C0-0x000003DF Intel(R) 5520/5500/X58 I/O Hub PCI Express Root Port 0 - 3420 OK 0x000003F2-0x000003F5 Standard-Diskettenlaufwerkcontroller OK 0x000003F7-0x000003F7 Standard-Diskettenlaufwerkcontroller OK 0x000003F8-0x000003FF Kommunikationsanschluss (COM1) OK 0x00000CD8-0x00000CF7 ACPI-Modulgerät OK 0x00000D00-0x0000FFFF PCI-Bus OK 0x0000B100-0x0000B13F Intel(R) ICH9 Family SMBus Controller - 2930 OK 0x0000C000-0x0000C0FF AMD Radeon HD 7800 Series OK 0x0000C000-0x0000C0FF Intel(R) 5520/5500/X58 I/O Hub PCI Express Root Port 0 - 3420 OK 0x0000D000-0x0000D0FF AMD Radeon R9 200 Series OK 0x0000D000-0x0000D0FF Intel(R) 5520/5500/X58 I/O Hub PCI Express Root Port 0 - 3420 OK 0x0000E000-0x0000E03F Red Hat VirtIO SCSI controller OK 0x0000E080-0x0000E09F Red Hat VirtIO Ethernet Adapter OK 0x0000E0A0-0x0000E0BF Standard AHCI 1.0 Serieller-ATA-Controller OK - Memory: 0xC0000000-0xCFFFFFFF AMD Radeon R9 200 Series OK 0xC0000000-0xCFFFFFFF PCI-Bus OK 0xC0000000-0xCFFFFFFF Intel(R) 5520/5500/X58 I/O Hub PCI Express Root Port 0 - 3420 OK 0xD0000000-0xD07FFFFF AMD Radeon R9 200 Series OK 0xFE800000-0xFE83FFFF AMD Radeon R9 200 Series OK 0xFE800000-0xFE83FFFF Intel(R) 5520/5500/X58 I/O Hub PCI Express Root Port 0 - 3420 OK 0xFEA40000-0xFEA40FFF Red Hat VirtIO Ethernet Adapter OK 0xFED00000-0xFED003FF Hochpräzisionsereigniszeitgeber OK 0xFEA41000-0xFEA41FFF Red Hat VirtIO SCSI controller OK 0xE0000000-0xEFFFFFFF AMD Radeon HD 7800 Series OK 0xE0000000-0xEFFFFFFF Intel(R) 5520/5500/X58 I/O Hub PCI Express Root Port 0 - 3420 OK 0xFE600000-0xFE63FFFF AMD Radeon HD 7800 Series OK 0xFE600000-0xFE63FFFF Intel(R) 5520/5500/X58 I/O Hub PCI Express Root Port 0 - 3420 OK 0xFEA42000-0xFEA42FFF Standard AHCI 1.0 Serieller-ATA-Controller OK 0xFE860000-0xFE863FFF High Definition Audio-Controller OK 0xA0000-0xBFFFF AMD Radeon R9 200 Series OK 0xA0000-0xBFFFF PCI-Bus OK 0xA0000-0xBFFFF Intel(R) 5520/5500/X58 I/O Hub PCI Express Root Port 0 - 3420 OK - IRQ: IRQ 1 Standardtastatur (PS/2) OK IRQ 3 Kommunikationsanschluss (COM2) OK IRQ 4 Kommunikationsanschluss (COM1) OK IRQ 6 Standard-Diskettenlaufwerkcontroller OK IRQ 8 System CMOS/Echtzeituhr OK IRQ 10 AMD Radeon HD 7800 Series OK IRQ 10 Intel(R) ICH9 Family SMBus Controller - 2930 OK IRQ 12 PS/2-kompatible Maus OK IRQ 16 Standard AHCI 1.0 Serieller-ATA-Controller OK IRQ 20 High Definition Audio-Controller OK IRQ 81 Microsoft ACPI-konformes System OK IRQ 82 Microsoft ACPI-konformes System OK IRQ 83 Microsoft ACPI-konformes System OK IRQ 84 Microsoft ACPI-konformes System OK IRQ 85 Microsoft ACPI-konformes System OK IRQ 86 Microsoft ACPI-konformes System OK IRQ 87 Microsoft ACPI-konformes System OK IRQ 88 Microsoft ACPI-konformes System OK IRQ 89 Microsoft ACPI-konformes System OK IRQ 90 Microsoft ACPI-konformes System OK IRQ 91 Microsoft ACPI-konformes System OK IRQ 92 Microsoft ACPI-konformes System OK IRQ 93 Microsoft ACPI-konformes System OK IRQ 94 Microsoft ACPI-konformes System OK IRQ 95 Microsoft ACPI-konformes System OK IRQ 96 Microsoft ACPI-konformes System OK IRQ 97 Microsoft ACPI-konformes System OK IRQ 98 Microsoft ACPI-konformes System OK IRQ 99 Microsoft ACPI-konformes System OK IRQ 100 Microsoft ACPI-konformes System OK IRQ 101 Microsoft ACPI-konformes System OK IRQ 102 Microsoft ACPI-konformes System OK IRQ 103 Microsoft ACPI-konformes System OK IRQ 104 Microsoft ACPI-konformes System OK IRQ 105 Microsoft ACPI-konformes System OK IRQ 106 Microsoft ACPI-konformes System OK IRQ 107 Microsoft ACPI-konformes System OK IRQ 108 Microsoft ACPI-konformes System OK IRQ 109 Microsoft ACPI-konformes System OK IRQ 110 Microsoft ACPI-konformes System OK IRQ 111 Microsoft ACPI-konformes System OK IRQ 112 Microsoft ACPI-konformes System OK IRQ 113 Microsoft ACPI-konformes System OK IRQ 114 Microsoft ACPI-konformes System OK IRQ 115 Microsoft ACPI-konformes System OK IRQ 116 Microsoft ACPI-konformes System OK IRQ 117 Microsoft ACPI-konformes System OK IRQ 118 Microsoft ACPI-konformes System OK IRQ 119 Microsoft ACPI-konformes System OK IRQ 120 Microsoft ACPI-konformes System OK IRQ 121 Microsoft ACPI-konformes System OK IRQ 122 Microsoft ACPI-konformes System OK IRQ 123 Microsoft ACPI-konformes System OK IRQ 124 Microsoft ACPI-konformes System OK IRQ 125 Microsoft ACPI-konformes System OK IRQ 126 Microsoft ACPI-konformes System OK IRQ 127 Microsoft ACPI-konformes System OK IRQ 128 Microsoft ACPI-konformes System OK IRQ 129 Microsoft ACPI-konformes System OK IRQ 130 Microsoft ACPI-konformes System OK IRQ 131 Microsoft ACPI-konformes System OK IRQ 132 Microsoft ACPI-konformes System OK IRQ 133 Microsoft ACPI-konformes System OK IRQ 134 Microsoft ACPI-konformes System OK IRQ 135 Microsoft ACPI-konformes System OK IRQ 136 Microsoft ACPI-konformes System OK IRQ 137 Microsoft ACPI-konformes System OK IRQ 138 Microsoft ACPI-konformes System OK IRQ 139 Microsoft ACPI-konformes System OK IRQ 140 Microsoft ACPI-konformes System OK IRQ 141 Microsoft ACPI-konformes System OK IRQ 142 Microsoft ACPI-konformes System OK IRQ 143 Microsoft ACPI-konformes System OK IRQ 144 Microsoft ACPI-konformes System OK IRQ 145 Microsoft ACPI-konformes System OK IRQ 146 Microsoft ACPI-konformes System OK IRQ 147 Microsoft ACPI-konformes System OK IRQ 148 Microsoft ACPI-konformes System OK IRQ 149 Microsoft ACPI-konformes System OK IRQ 150 Microsoft ACPI-konformes System OK IRQ 151 Microsoft ACPI-konformes System OK IRQ 152 Microsoft ACPI-konformes System OK IRQ 153 Microsoft ACPI-konformes System OK IRQ 154 Microsoft ACPI-konformes System OK IRQ 155 Microsoft ACPI-konformes System OK IRQ 156 Microsoft ACPI-konformes System OK IRQ 157 Microsoft ACPI-konformes System OK IRQ 158 Microsoft ACPI-konformes System OK IRQ 159 Microsoft ACPI-konformes System OK IRQ 160 Microsoft ACPI-konformes System OK IRQ 161 Microsoft ACPI-konformes System OK IRQ 162 Microsoft ACPI-konformes System OK IRQ 163 Microsoft ACPI-konformes System OK IRQ 164 Microsoft ACPI-konformes System OK IRQ 165 Microsoft ACPI-konformes System OK IRQ 166 Microsoft ACPI-konformes System OK IRQ 167 Microsoft ACPI-konformes System OK IRQ 168 Microsoft ACPI-konformes System OK IRQ 169 Microsoft ACPI-konformes System OK IRQ 170 Microsoft ACPI-konformes System OK IRQ 171 Microsoft ACPI-konformes System OK IRQ 172 Microsoft ACPI-konformes System OK IRQ 173 Microsoft ACPI-konformes System OK IRQ 174 Microsoft ACPI-konformes System OK IRQ 175 Microsoft ACPI-konformes System OK IRQ 176 Microsoft ACPI-konformes System OK IRQ 177 Microsoft ACPI-konformes System OK IRQ 178 Microsoft ACPI-konformes System OK IRQ 179 Microsoft ACPI-konformes System OK IRQ 180 Microsoft ACPI-konformes System OK IRQ 181 Microsoft ACPI-konformes System OK IRQ 182 Microsoft ACPI-konformes System OK IRQ 183 Microsoft ACPI-konformes System OK IRQ 184 Microsoft ACPI-konformes System OK IRQ 185 Microsoft ACPI-konformes System OK IRQ 186 Microsoft ACPI-konformes System OK IRQ 187 Microsoft ACPI-konformes System OK IRQ 188 Microsoft ACPI-konformes System OK IRQ 189 Microsoft ACPI-konformes System OK IRQ 190 Microsoft ACPI-konformes System OK IRQ 4294967287 AMD Radeon R9 200 Series OK IRQ 4294967288 Red Hat VirtIO Ethernet Adapter OK IRQ 4294967289 Red Hat VirtIO Ethernet Adapter OK IRQ 4294967290 Red Hat VirtIO Ethernet Adapter OK IRQ 4294967291 Red Hat VirtIO SCSI controller OK IRQ 4294967292 Red Hat VirtIO SCSI controller OK IRQ 4294967293 Intel(R) 5520/5500/X58 I/O Hub PCI Express Root Port 0 - 3420 OK IRQ 4294967294 Intel(R) 5520/5500/X58 I/O Hub PCI Express Root Port 0 - 3420 OK I'm no expert but it looks like Windows never enabled MSI for the second card as qemu shell with 'info pci' show me both with IRQ 10. I'll hope it helps. > > > > QEMU is called in both cases via the following. I just replace the > > > > '-drive' accordingly. > > > > > > > > /usr/bin/taskset -c 0,1,2,3 /usr/bin/qemu-system-x86_64 \ > > > > -machine q35,accel=kvm \ > > > > -enable-kvm \ > > > > -nodefaults \ > > > > -nographic \ > > > > -vga none \ > > > > -boot order=nc \ > > > > -cpu host \ > > > > -smp cores=4,threads=1,sockets=1 \ > > > > -m 8192 \ > > > > -rtc base=localtime \ > > > > -k de \ > > > > -drive > > > > file=/srv/kvm/linux-drive0.img,id=drive0,if=none,cache=none,aio=threads > > > > \ > > > > -mon chardev=monitor0 \ > > > > -chardev socket,id=monitor0,path=/tmp/linux.monitor,nowait,server \ > > > > -netdev tap,id=net0,vhost=on,helper=/usr/lib/qemu/qemu-bridge-helper \ > > > > -device virtio-net-pci,netdev=net0,mac=00:00:00:02:01:04 \ > > > > -device virtio-blk-pci,drive=drive0,ioeventfd=on \ > > > > -device ioh3420,bus=pcie.0,id=pcie0,port=1,chassis=1,multifunction=on > > > > \ > > > > -device ioh3420,bus=pcie.0,id=pcie1,port=2,chassis=2,multifunction=on > > > > \ > > > > -device > > > > vfio-pci,host=01:00.0,addr=00.0,bus=pcie0,multifunction=on,x-vga=on \ > > > > -device vfio-pci,host=01:00.1,addr=00.1,bus=pcie0 \ > > > > -device vfio-pci,host=02:00.0,addr=00.0,bus=pcie1,multifunction=on \ > > > > -device vfio-pci,host=02:00.1,addr=00.1,bus=pcie1 \ > > > > -no-reboot > > > > > > > > My setup is the following: > > > > > > > > Kernel: linux-3.13.1 > > > > Seabios: seabios-git-rel.1.7.4.r51.g151d034 (5/2/2014) > > > > QEMU: qemu-git-2.0.r30666.g31db5b3 (5/2/2014) > > > > > > > > Below is the 'lspci' output and I'm using the AMD Radeon HD 5430 as > > > > device > > > > for my local X server: > > > > > > > > 00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI > > > > to PCI bridge (external gfx0 port B) (rev 02) > > > > 00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD/ATI] RD990 I/O Memory > > > > Management Unit (IOMMU) > > > > 00:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to > > > > PCI bridge (PCI express gpp port B) > > > > 00:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to > > > > PCI bridge (PCI express gpp port D) > > > > 00:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to > > > > PCI bridge (PCI express gpp port H) > > > > 00:0d.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to > > > > PCI bridge (external gfx1 port B) > > > > 00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD/ATI] > > > > SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] (rev 40) > > > > 00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] > > > > SB7x0/SB8x0/SB9x0 USB OHCI0 Controller > > > > 00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] > > > > SB7x0/SB8x0/SB9x0 USB EHCI Controller > > > > 00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] > > > > SB7x0/SB8x0/SB9x0 USB OHCI0 Controller > > > > 00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] > > > > SB7x0/SB8x0/SB9x0 USB EHCI Controller > > > > 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus > > > > Controller (rev 42) > > > > 00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 > > > > Azalia (Intel HDA) (rev 40) > > > > 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD/ATI] > > > > SB7x0/SB8x0/SB9x0 LPC host controller (rev 40) > > > > 00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to > > > > PCI Bridge (rev 40) > > > > 00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] > > > > SB7x0/SB8x0/SB9x0 USB OHCI2 Controller > > > > 00:15.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] > > > > SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0) > > > > 00:15.1 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] > > > > SB700/SB800/SB900 PCI to PCI bridge (PCIE port 1) > > > > 00:15.2 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB900 PCI to > > > > PCI bridge (PCIE port 2) > > > > 00:15.3 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB900 PCI to > > > > PCI bridge (PCIE port 3) > > > > 00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] > > > > SB7x0/SB8x0/SB9x0 USB OHCI0 Controller > > > > 00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] > > > > SB7x0/SB8x0/SB9x0 USB EHCI Controller > > > > 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h > > > > Processor Function 0 > > > > 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h > > > > Processor Function 1 > > > > 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h > > > > Processor Function 2 > > > > 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h > > > > Processor Function 3 > > > > 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h > > > > Processor Function 4 > > > > 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h > > > > Processor Function 5 > > > > 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. > > > > [AMD/ATI] Hawaii XT [Radeon HD 8970] > > > > 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device aac8 > > > > 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. > > > > [AMD/ATI] Pitcairn XT [Radeon HD 7870 GHz Edition] > > > > 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape > > > > Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] > > > > 03:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host > > > > Controller (rev 01) > > > > 04:00.0 VGA compatible controller: Advanced Micro Devices, Inc. > > > > [AMD/ATI] Park [Mobility Radeon HD 5430] > > > > 04:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI > > > > Audio [Radeon HD 5400/6300 Series] > > > > 06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. > > > > RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06) > > > > 07:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host > > > > Controller (rev 01) > > > > > > > > Another minor issue is that the R9 290X is not reset during shutdown of > > > > VM (neither Linux nor Windows) but it can be tricked with doing > > > > "suspend-to-ram" between two starts. That's why I use '-no-reboot' > > > > option > > > > in QEMU. The 7870 is doing the reset properly. > > > > > > > > > Is the NoSoftRst "-" on the 290X vs "+" on the 7870 in lspci -vvv by > > > chance? Thanks, > > > > > > > Here are both. It is funny it is opposite as you described. :) > > > Oops, yes. Does this help? > > --- a/hw/misc/vfio.c > +++ b/hw/misc/vfio.c > @@ -3136,7 +3136,7 @@ static void vfio_pci_reset_handler(void *opaque) > > QLIST_FOREACH(group, &group_list, next) { > QLIST_FOREACH(vdev, &group->device_list, next) { > - if (!vdev->reset_works || (!vdev->has_flr && > vdev->has_pm_reset)) { > + if (!vdev->reset_works || !vdev->has_flr) { > vdev->needs_reset = true; > } > } > > I can't figure out why I coded it the way that I did. Probably overly > targeting a specific device. Thanks, > This patch works absolutely fine. After applying it to my 'qemu-git', the device resets works flawlessly. So it would be great to push it upstream as it looks good. > Alex > > > root@homer:~# lspci -vvv -s 01:00.0 | grep NoSoftRst > > Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- > > > > root@homer:~# lspci -vvv -s 02:00.0 | grep NoSoftRst > > Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- > > > > root@homer:~# lspci -vvv -s 01:00.0 > > 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] > > Hawaii XT [Radeon HD 8970] (prog-if 00 [VGA controller]) > > Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 0b00 > > Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- > > Stepping- SERR+ FastB2B- DisINTx+ > > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- > > <MAbort- >SERR- <PERR- INTx- > > Latency: 0, Cache Line Size: 64 bytes > > Interrupt: pin A routed to IRQ 49 > > Region 0: Memory at c0000000 (64-bit, prefetchable) [size=256M] > > Region 2: Memory at df800000 (64-bit, prefetchable) [size=8M] > > Region 4: I/O ports at be00 [size=256] > > Region 5: Memory at fdd80000 (32-bit, non-prefetchable) [size=256K] > > [virtual] Expansion ROM at d0000000 [disabled] [size=128K] > > Capabilities: [48] Vendor Specific Information: Len=08 <?> > > Capabilities: [50] Power Management version 3 > > Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA > > PME(D0-,D1+,D2+,D3hot+,D3cold-) > > Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- > > Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00 > > DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 > > unlimited > > ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- > > DevCtl: Report errors: Correctable- Non-Fatal- Fatal- > > Unsupported- > > RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ > > MaxPayload 128 bytes, MaxReadReq 512 bytes > > DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- > > TransPend- > > LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit > > Latency L0s <64ns, L1 <1us > > ClockPM- Surprise- LLActRep- BwNot- > > LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- CommClk+ > > ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- > > LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk+ > > DLActive- BWMgmt- ABWMgmt- > > DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, > > OBFF Not Supported > > DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, > > OBFF Disabled > > LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- > > Transmit Margin: Normal Operating Range, > > EnterModifiedCompliance- ComplianceSOS- > > Compliance De-emphasis: -6dB > > LnkSta2: Current De-emphasis Level: -3.5dB, > > EqualizationComplete-, EqualizationPhase1- > > EqualizationPhase2-, EqualizationPhase3-, > > LinkEqualizationRequest- > > Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+ > > Address: 00000000fee00000 Data: 0000 > > Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 > > Len=010 <?> > > Capabilities: [150 v2] Advanced Error Reporting > > UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- > > RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > > UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- > > RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > > UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- > > RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- > > CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ > > CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ > > AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- > > Capabilities: [270 v1] #19 > > Capabilities: [2b0 v1] Address Translation Service (ATS) > > ATSCap: Invalidate Queue Depth: 00 > > ATSCtl: Enable+, Smallest Translation Unit: 00 > > Capabilities: [2c0 v1] #13 > > Capabilities: [2d0 v1] #1b > > Kernel driver in use: vfio-pci > > Kernel modules: radeon > > > > root@homer:~# lspci -vvv -s 02:00.0 > > 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] > > Pitcairn XT [Radeon HD 7870 GHz Edition] (prog-if 00 [VGA controller]) > > Subsystem: XFX Pine Group Inc. Device 3251 > > Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- > > Stepping- SERR+ FastB2B- DisINTx+ > > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- > > <MAbort- >SERR- <PERR- INTx- > > Latency: 0, Cache Line Size: 64 bytes > > Interrupt: pin A routed to IRQ 48 > > Region 0: Memory at a0000000 (64-bit, prefetchable) [size=256M] > > Region 2: Memory at fda80000 (64-bit, non-prefetchable) [size=256K] > > Region 4: I/O ports at ee00 [size=256] > > [virtual] Expansion ROM at fda00000 [disabled] [size=128K] > > Capabilities: [48] Vendor Specific Information: Len=08 <?> > > Capabilities: [50] Power Management version 3 > > Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA > > PME(D0-,D1+,D2+,D3hot+,D3cold-) > > Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- > > Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00 > > DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 > > unlimited > > ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- > > DevCtl: Report errors: Correctable- Non-Fatal- Fatal- > > Unsupported- > > RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ > > MaxPayload 128 bytes, MaxReadReq 512 bytes > > DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- > > TransPend- > > LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit > > Latency L0s <64ns, L1 <1us > > ClockPM- Surprise- LLActRep- BwNot- > > LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- CommClk+ > > ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- > > LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- > > BWMgmt- ABWMgmt- > > DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, > > OBFF Not Supported > > DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, > > OBFF Disabled > > LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- > > Transmit Margin: Normal Operating Range, > > EnterModifiedCompliance- ComplianceSOS- > > Compliance De-emphasis: -6dB > > LnkSta2: Current De-emphasis Level: -3.5dB, > > EqualizationComplete-, EqualizationPhase1- > > EqualizationPhase2-, EqualizationPhase3-, > > LinkEqualizationRequest- > > Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+ > > Address: 00000000fee00000 Data: 0000 > > Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 > > Len=010 <?> > > Capabilities: [150 v2] Advanced Error Reporting > > UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- > > RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > > UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- > > RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > > UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- > > RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- > > CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- > > CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ > > AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- > > Capabilities: [270 v1] #19 > > Capabilities: [2b0 v1] Address Translation Service (ATS) > > ATSCap: Invalidate Queue Depth: 00 > > ATSCtl: Enable+, Smallest Translation Unit: 00 > > Capabilities: [2c0 v1] #13 > > Capabilities: [2d0 v1] #1b > > Kernel driver in use: vfio-pci > > Kernel modules: radeon > > > > > Alex > > > > > > > --Maik > > > --Maik