Re: We are not able to virtualize FreeBSD using xen 4.17 on Arm 32 bit

[Chuck Zmudzinski]

On 11/27/2023 9:28 PM, Stefano Stabellini wrote:
> On Mon, 27 Nov 2023, Chuck Zmudzinski wrote:
>> On 11/27/2023 10:22 AM, Chuck Zmudzinski wrote:
>> > On 11/27/2023 7:45 AM, Mario Marietto wrote:
>> >> @Chuck Zmudzinski <mailto:brchu...@netscape.net> : Stay tuned. They want 
>> >> to help us. The xen developers are great. Very good support for us. I'm 
>> >> sure that you can give a good contribution to understand what's our 
>> >> problem and how to implement a fix with the help of all those good guys.
>> >> 
>> >> On Mon, Nov 27, 2023 at 11:56 AM Roger Pau Monné > >> <mailto:roger@citrix.com>> wrote:
>> >> 
>> >> On Mon, Nov 27, 2023 at 10:28:13AM +, Henry Wang wrote:
>> >> > +(xen-devel and Arm maintainers, including Julien)
>> >> >
>> >> > > On Nov 27, 2023, at 18:03, Mario Marietto > >> <mailto:marietto2...@gmail.com>>
>> >> > > wrote:
>> >> > >
>> >> > > Hello.  We have just virtualized Debian 12 on our arm (32 bit)
>> >> > > Chromebook model xe303c12 . As host / dom0 we have chosen Devuan
>> >> > > 5,and for guest / domU,Debian 12. It works great. But our goal is
>> >> > > different. We want to virtualize FreeBSD as domU. Can we have a
>> >> > > working Xen PV network driver for a FreeBSD arm guest ?. I found
>> >> > > that Julien Grall has ported the Xen drivers to FreeBSD on arm. I
>> >> > > would like to know if Julien's work was accepted upstream by
>> >> > > FreeBSD, in which case FreeBSD as a Xen guest on arm should work
>> >> > > if we enable the Xen PV drivers in the FreeBSD on arm kernel. If
>> >> > > Julien's work was not accepted upstream by FreeBSD, we will have
>> >> > > to find his patches and apply them ourselves to the FreeBSD on arm
>> >> > > kernel.
>> >> 
>> >> I've added Elliot on Cc as he is working on upstreaming the patches to
>> >> FreeBSD.  He will be able to provide a better update than myself.
>> >> 
>> >> Regards, Roger.
>> > 
>> > I have been collaborating with Mario, and I can explain what we have done 
>> > so far :
>> > 
>> > We are using Julien's patch set against an old development version of 
>> > FreeBSD 11
>> > from 2014-12-03 :
>> > 
>> > https://xenbits.xen.org/gitweb/?p=people/julieng/freebsd.git;a=shortlog;h=refs/heads/xen-arm-v2.2
>> > 
>> > We successfully built the XENVIRT kernel and FreeBSD world, and created the
>> > FreeBSD rootfs according to Julien's instructions here :
>> > 
>> > https://lists.freebsd.org/pipermail/freebsd-xen/2014-November/002202.html
>> > 
>> > There were some adjustments to the instructions :
>> > 
>> > To build the kernel, we used :
>> > 
>> > $ sudo make TARGET_ARCH=armv6 KERNCONF=XENVIRT buildkernel
>> > 
>> > instead of
>> > 
>> > $ sudo make TARGET_ARCH=armv6 KERNCONF=XENHVM buildkernel
>> > 
>> > The FreeBSD 'kernel' file is in ELF format and did not work, and we spent
>> > some time trying to convert it to the zImage format without realizing the
>> > build of the FreeBSD kernel creates the 'kernel.bin' file in the zImage 
>> > format.
>> > So when booting with the 'kernel.bin' file instead, it actually boots :
>> > 
>> > user@devuan-bunsen ~ % sudo xl create freebsd.cfg
>> > Parsing config from freebsd.cfg
>> > user@devuan-bunsen ~ % sudo xl li
>> > NameID   Mem VCPUs State   Time(s)
>> > Domain-0 0   768 2 r-
>> > 1439.4
>> > freebsd  1  1152 1 r-  
>> >  3.0
>> > user@devuan-bunsen ~ %
>> > 
>> > However, the guest is still not working correctly :
>> > 
>> > 1. Attaching the console with the -c option at creation or with
>> >'xl console freebsd' results in no output to the console.
>> > 
>> > 2. The timestamp on the virtual disk image file shows that the filesystem
>> >was at best mounted read-only, if it was mounted at all by the guest
>> >FreeBSD kernel.
>> > 
>> > 3. The 'xl shutdown freebsd' command does not work, 

Re: xc_dom_guest_type: image not capable of booting inside a HV M container: Invalid kernel

[Chuck Zmudzinski]

On 12/11/2023 12:59 PM, Mario Marietto wrote:
> root@marietto:/mnt/zroot2/zroot2/OS/Chromebook/domU/freebsd-xen/boot-xen/kernel
>  # file 
> /mnt/zroot2/zroot2/OS/Chromebook/domU/freebsd-xen/boot-xen/kernel/kernel
> 
> ELF 32-bit LSB executable,ARM, EABI5 version 1 (FreeBSD), dynamically linked, 
> interpreter /red/herring, 
> BuildID[sha1]=5e6982c9cb67d9c94571b76419142a8c495388d0, for FreeBSD 13.2, not 
> stripped
> 
> root@marietto:/mnt/zroot2/zroot2/OS/Chromebook/domU/freebsd-xen/boot-xen/kernel
>  # file 
> /mnt/zroot2/zroot2/OS/Chromebook/domU/freebsd-xen/boot-xen/kernel/kernel.bin
> 
> kernel.bin: data 

This needs to be :

kernel.bin: Linux kernel ARM boot executable zImage (little-endian)

> 
> It does not boot from the kernel.bin file.

I suggest not trying to direct boot a kernel in Xen on arm unless the file 
command reports the kernel image is a Linux kernel ARM boot executable zImage 
(little endian).

Did you try applying Julien's patch (link is in my earlier message) to add 
zImage support to FreeBSD? Maybe after applying the patch the kernel.bin file 
will be in the correct zImage format.

The patch I linked in the earlier 

> 
> 
> On Mon, Dec 11, 2023 at 6:23 PM Chuck Zmudzinski  <mailto:brchu...@netscape.net>> wrote:
> 
> On 12/11/2023 9:02 AM, Mario Marietto wrote:
> > Hello.
> >
> > Finally I tried to recompile the FreeBSD kernel using the @Elliott 
> Mitchell <mailto:ehem+free...@m5p.com <mailto:ehem%2bfree...@m5p.com>> code 
> because I want to boot FreeBSD as domU with Xen installed on my Arm 32 bit 
> Chromebook. Unfortunately it didn't work at all. Maybe I've missed something 
> / I haven't understood well what to do. Please give me some suggestions.
> >
> > Basically this is what I did :
> >
> > $ created a vm called FreeBSD-13.2-RELEASE-armv7.img with qemu / kvm / 
> libvirt / virt-manager
> >
> > $ within the vm : mkdir /build-xen
> >
> > $ cd /usr
> >
> > $ git clone https://gitlab.com/ehem/freebsd-src.git
> >
> > $ cd freebsd-src
> >
> > $ make KERNCONF=GENERIC TARGET=arm TARGET_ARCH=armv7 buildkernel
> >
> > $ make KERNCONF=GENERIC TARGET=arm TARGET_ARCH=armv7 DESTDIR=/build-xen 
> installkernel
> >
> > $ echo "/dev/xbd0 / ufs rw 1 1" > /mnt/etc/fstab
> >
> > $ nano /etc/ttys (add the line 'xc0 "/usr/libexec/getty Pc" xterm on 
> secure")
> >
> > $ renamed the directories dtb to dtb_ and kernel to kernel_ that are 
> inside the /boot dir of the vm
> >
> > $ copied the directory dtb and kernel from the directory /build-xen to 
> the directory /boot inside the vm
> >
> > $ shut down the vm
> >
> > $ copied the directory /build-xen outside of the vm using this method 
> (in this case I used Linux installed on the Host OS,because the kernel that 
> I'm using on the Chromebook has the kernel parameter related to the ufs2 fs 
> set to off) :
> >
> > on my X64 workstation :
> >
> > # modprobe ufs
> >
> > # sudo losetup -fP FreeBSD-13.2-RELEASE-armv7.img
> >
> > # ls /dev/loop0*
> >
> > /dev/loop0 /dev/loop0p1 /dev/loop0p2 /dev/loop0p5
> >
> > # mount -t ufs -o ufstype=ufs2 /dev/loop0p5 ./FreeBSD-xen
> >
> > then :
> >
> > # nano freebsd.cfg
> >
> > 
> kernel="/mnt/zroot2/zroot2/OS/Chromebook/domU/freebsd-xen/boot-xen/kernel/kernel"
> > memory=64
> > name="freebsd"
> > vcpus=1
> > autoballon="off"
> > disk=[ 'phy:/dev/loop0,xvda,w' ]
> > # nano start-freebsd
> > losetup -fP FreeBSD-13.2-RELEASE-armv7.img
> > xl create freebsd.cfg
> > xl console freebsd
> >
> > # ./start-freebsd
> >
> > Parsing config from freebsd.cfg
> > xc: error: panic: xg_dom_elfloader.c:63: xc_dom_guest_type: image not 
> capable of booting inside a HV
> > M container: Invalid kernel
> 
> It is detecting the kernel as an elf binary. IIUC, Xen on arm guests 
> should have zImage kernels, not elf.
> 
> > libxl: error: libxl_dom.c:571:libxl__build_dom: xc_dom_parse_image 
> failed
> > libxl: error: libxl_create.c:1640:domcreate_rebuild_done: Domain 
> 1:cannot (re-)build domain: -3
> > libxl: error: libxl_domain.c:1183:libxl__destroy_domid: Domain 
> 1:Non-existent domain
> > libxl: error: libxl_domain.c:1137:domain_destroy_callback: Do

Re: xc_dom_guest_type: image not capable of booting inside a HV M container: Invalid kernel

[Chuck Zmudzinski]

On 12/11/2023 9:02 AM, Mario Marietto wrote:
> Hello.
> 
> Finally I tried to recompile the FreeBSD kernel using the @Elliott Mitchell 
>  code because I want to boot FreeBSD as domU 
> with Xen installed on my Arm 32 bit Chromebook. Unfortunately it didn't work 
> at all. Maybe I've missed something / I haven't understood well what to do. 
> Please give me some suggestions.
> 
> Basically this is what I did :
> 
> $ created a vm called FreeBSD-13.2-RELEASE-armv7.img with qemu / kvm / 
> libvirt / virt-manager
> 
> $ within the vm : mkdir /build-xen
> 
> $ cd /usr
> 
> $ git clone https://gitlab.com/ehem/freebsd-src.git 
> 
> 
> $ cd freebsd-src
> 
> $ make KERNCONF=GENERIC TARGET=arm TARGET_ARCH=armv7 buildkernel
> 
> $ make KERNCONF=GENERIC TARGET=arm TARGET_ARCH=armv7 DESTDIR=/build-xen 
> installkernel
> 
> $ echo "/dev/xbd0 / ufs rw 1 1" > /mnt/etc/fstab
> 
> $ nano /etc/ttys (add the line 'xc0 "/usr/libexec/getty Pc" xterm on secure")
> 
> $ renamed the directories dtb to dtb_ and kernel to kernel_ that are inside 
> the /boot dir of the vm
> 
> $ copied the directory dtb and kernel from the directory /build-xen to the 
> directory /boot inside the vm
> 
> $ shut down the vm
> 
> $ copied the directory /build-xen outside of the vm using this method (in 
> this case I used Linux installed on the Host OS,because the kernel that I'm 
> using on the Chromebook has the kernel parameter related to the ufs2 fs set 
> to off) :
> 
> on my X64 workstation :
> 
> # modprobe ufs
> 
> # sudo losetup -fP FreeBSD-13.2-RELEASE-armv7.img
> 
> # ls /dev/loop0*
> 
> /dev/loop0 /dev/loop0p1 /dev/loop0p2 /dev/loop0p5
> 
> # mount -t ufs -o ufstype=ufs2 /dev/loop0p5 ./FreeBSD-xen
> 
> then :
> 
> # nano freebsd.cfg
> 
> kernel="/mnt/zroot2/zroot2/OS/Chromebook/domU/freebsd-xen/boot-xen/kernel/kernel"
> memory=64
> name="freebsd"
> vcpus=1
> autoballon="off"
> disk=[ 'phy:/dev/loop0,xvda,w' ]
> # nano start-freebsd
> losetup -fP FreeBSD-13.2-RELEASE-armv7.img
> xl create freebsd.cfg
> xl console freebsd
> 
> # ./start-freebsd
> 
> Parsing config from freebsd.cfg
> xc: error: panic: xg_dom_elfloader.c:63: xc_dom_guest_type: image not capable 
> of booting inside a HV
> M container: Invalid kernel

It is detecting the kernel as an elf binary. IIUC, Xen on arm guests should 
have zImage kernels, not elf.

> libxl: error: libxl_dom.c:571:libxl__build_dom: xc_dom_parse_image failed
> libxl: error: libxl_create.c:1640:domcreate_rebuild_done: Domain 1:cannot 
> (re-)build domain: -3
> libxl: error: libxl_domain.c:1183:libxl__destroy_domid: Domain 1:Non-existent 
> domain
> libxl: error: libxl_domain.c:1137:domain_destroy_callback: Domain 1:Unable to 
> destroy guest
> libxl: error: libxl_domain.c:1064:domain_destroy_cb: Domain 1:Destruction of 
> domain failed
> freebsd is an invalid domain identifier (rc=-6)
> 
> I have also tried with kernel.bin :
> 
> # nano freebsd.cfg
> 
> kernel="/mnt/zroot2/zroot2/OS/Chromebook/domU/freebsd-xen/boot-xen/kernel/kernel.bin"
> memory=64
> name="freebsd"
> vcpus=1
> autoballon="off"
> disk=[ 'phy:/dev/loop0,xvda,w' ]
> 
> # ./start-freebsd
> 
> Parsing config from freebsd.cfg
> xc: error: panic: xg_dom_core.c:689: xc_dom_find_loader: no loader found: 
> Invalid kernel
> libxl: error: libxl_dom.c:571:libxl__build_dom: xc_dom_parse_image failed
> libxl: error: libxl_create.c:1640:domcreate_rebuild_done: Domain 2:cannot 
> (re-)build domain: -3
> libxl: error: libxl_domain.c:1183:libxl__destroy_domid: Domain 2:Non-existent 
> domain
> libxl: error: libxl_domain.c:1137:domain_destroy_callback: Domain 2:Unable to 
> destroy guest
> libxl: error: libxl_domain.c:1064:domain_destroy_cb: Domain 2:Destruction of 
> domain failed
> freebsd is an invalid domain identifier (rc=-6)
> 
> -- 
> Mario.

I would be interested to see the output of :

$ file /mnt/zroot2/zroot2/OS/Chromebook/domU/freebsd-xen/boot-xen/kernel/kernel

and

$ file 
/mnt/zroot2/zroot2/OS/Chromebook/domU/freebsd-xen/boot-xen/kernel/kernel.bin

I have been trying out Julien's old patch set from 2014, and in there was this 
patch :

> arm: Add zImage support
> 
> Currently Xen on ARM is only supported zImage for guest kernel. Adding support
> for ARM ELF in the toolstack looks a bit complicate for ARM (though there is
> an x86 support).

Link to Julien's 2014 patch to provide zImage support for FreeBSD :

https://xenbits.xen.org/gitweb/?p=people/julieng/freebsd.git;a=commit;h=12a7cb346b88c6d3f52a20b98f361dc62797fbcd

When using Julien's patches, from 'file' I find that the kernel file is in
the elf format, and the kernel.bin file is in the zImage format, so I have
been trying to boot the kernel.bin file.

Re: We are not able to virtualize FreeBSD using xen 4.17 on Arm 32 bit

[Chuck Zmudzinski]

On 11/27/2023 10:22 AM, Chuck Zmudzinski wrote:
> On 11/27/2023 7:45 AM, Mario Marietto wrote:
>> @Chuck Zmudzinski <mailto:brchu...@netscape.net> : Stay tuned. They want to 
>> help us. The xen developers are great. Very good support for us. I'm sure 
>> that you can give a good contribution to understand what's our problem and 
>> how to implement a fix with the help of all those good guys.
>> 
>> On Mon, Nov 27, 2023 at 11:56 AM Roger Pau Monné > <mailto:roger@citrix.com>> wrote:
>> 
>> On Mon, Nov 27, 2023 at 10:28:13AM +, Henry Wang wrote:
>> > +(xen-devel and Arm maintainers, including Julien)
>> >
>> > > On Nov 27, 2023, at 18:03, Mario Marietto > <mailto:marietto2...@gmail.com>>
>> > > wrote:
>> > >
>> > > Hello.  We have just virtualized Debian 12 on our arm (32 bit)
>> > > Chromebook model xe303c12 . As host / dom0 we have chosen Devuan
>> > > 5,and for guest / domU,Debian 12. It works great. But our goal is
>> > > different. We want to virtualize FreeBSD as domU. Can we have a
>> > > working Xen PV network driver for a FreeBSD arm guest ?. I found
>> > > that Julien Grall has ported the Xen drivers to FreeBSD on arm. I
>> > > would like to know if Julien's work was accepted upstream by
>> > > FreeBSD, in which case FreeBSD as a Xen guest on arm should work
>> > > if we enable the Xen PV drivers in the FreeBSD on arm kernel. If
>> > > Julien's work was not accepted upstream by FreeBSD, we will have
>> > > to find his patches and apply them ourselves to the FreeBSD on arm
>> > > kernel.
>> 
>> I've added Elliot on Cc as he is working on upstreaming the patches to
>> FreeBSD.  He will be able to provide a better update than myself.
>> 
>> Regards, Roger.
> 
> I have been collaborating with Mario, and I can explain what we have done so 
> far :
> 
> We are using Julien's patch set against an old development version of FreeBSD 
> 11
> from 2014-12-03 :
> 
> https://xenbits.xen.org/gitweb/?p=people/julieng/freebsd.git;a=shortlog;h=refs/heads/xen-arm-v2.2
> 
> We successfully built the XENVIRT kernel and FreeBSD world, and created the
> FreeBSD rootfs according to Julien's instructions here :
> 
> https://lists.freebsd.org/pipermail/freebsd-xen/2014-November/002202.html
> 
> There were some adjustments to the instructions :
> 
> To build the kernel, we used :
> 
> $ sudo make TARGET_ARCH=armv6 KERNCONF=XENVIRT buildkernel
> 
> instead of
> 
> $ sudo make TARGET_ARCH=armv6 KERNCONF=XENHVM buildkernel
> 
> The FreeBSD 'kernel' file is in ELF format and did not work, and we spent
> some time trying to convert it to the zImage format without realizing the
> build of the FreeBSD kernel creates the 'kernel.bin' file in the zImage 
> format.
> So when booting with the 'kernel.bin' file instead, it actually boots :
> 
> user@devuan-bunsen ~ % sudo xl create freebsd.cfg
> Parsing config from freebsd.cfg
> user@devuan-bunsen ~ % sudo xl li
> NameID   Mem VCPUsState   Time(s)
> Domain-0 0   768 2 r-
> 1439.4
> freebsd  1  1152 1 r-   
> 3.0
> user@devuan-bunsen ~ %
> 
> However, the guest is still not working correctly :
> 
> 1. Attaching the console with the -c option at creation or with
>'xl console freebsd' results in no output to the console.
> 
> 2. The timestamp on the virtual disk image file shows that the filesystem
>was at best mounted read-only, if it was mounted at all by the guest
>FreeBSD kernel.
> 
> 3. The 'xl shutdown freebsd' command does not work, it just hangs. To stop
>the guest, you need to do 'xl destroy freebsd'.
> 
> However, I think we can get the console to work and the rootfs to mount 
> because I
> just realized I forgot to do the steps from Julien's instructions of editing 
> the
> /etc/fstab and /etc/ttys files in the FreeBSD rootfs :
> 
> $ echo "/dev/xbd0   /   ufs rw  1   1" > /mnt/etc/fstab
> $ vi /mnt/etc/ttys (add the line 'xc0 "/usr/libexec/getty Pc" xterm on 
> secure")
> 
> I will add those and see if the console and disk are working.

Unfortunately, adding xc0 to /etc/ttys and /dev/xbd0 as the root device in
/etc/fstab did not make the console or disk work. Still no output on the
xen console from the guest kernel, and the timestamp on the rootfs image
file did not change

Re: We are not able to virtualize FreeBSD using xen 4.17 on Arm 32 bit

[Chuck Zmudzinski]

On 11/27/2023 7:45 AM, Mario Marietto wrote:
> @Chuck Zmudzinski <mailto:brchu...@netscape.net> : Stay tuned. They want to 
> help us. The xen developers are great. Very good support for us. I'm sure 
> that you can give a good contribution to understand what's our problem and 
> how to implement a fix with the help of all those good guys.
> 
> On Mon, Nov 27, 2023 at 11:56 AM Roger Pau Monné  <mailto:roger@citrix.com>> wrote:
> 
> On Mon, Nov 27, 2023 at 10:28:13AM +, Henry Wang wrote:
> > +(xen-devel and Arm maintainers, including Julien)
> >
> > > On Nov 27, 2023, at 18:03, Mario Marietto  <mailto:marietto2...@gmail.com>>
> > > wrote:
> > >
> > > Hello.  We have just virtualized Debian 12 on our arm (32 bit)
> > > Chromebook model xe303c12 . As host / dom0 we have chosen Devuan
> > > 5,and for guest / domU,Debian 12. It works great. But our goal is
> > > different. We want to virtualize FreeBSD as domU. Can we have a
> > > working Xen PV network driver for a FreeBSD arm guest ?. I found
> > > that Julien Grall has ported the Xen drivers to FreeBSD on arm. I
> > > would like to know if Julien's work was accepted upstream by
> > > FreeBSD, in which case FreeBSD as a Xen guest on arm should work
> > > if we enable the Xen PV drivers in the FreeBSD on arm kernel. If
> > > Julien's work was not accepted upstream by FreeBSD, we will have
> > > to find his patches and apply them ourselves to the FreeBSD on arm
> > > kernel.
> 
> I've added Elliot on Cc as he is working on upstreaming the patches to
> FreeBSD.  He will be able to provide a better update than myself.
> 
> Regards, Roger.

I have been collaborating with Mario, and I can explain what we have done so 
far :

We are using Julien's patch set against an old development version of FreeBSD 11
from 2014-12-03 :

https://xenbits.xen.org/gitweb/?p=people/julieng/freebsd.git;a=shortlog;h=refs/heads/xen-arm-v2.2

We successfully built the XENVIRT kernel and FreeBSD world, and created the
FreeBSD rootfs according to Julien's instructions here :

https://lists.freebsd.org/pipermail/freebsd-xen/2014-November/002202.html

There were some adjustments to the instructions :

To build the kernel, we used :

$ sudo make TARGET_ARCH=armv6 KERNCONF=XENVIRT buildkernel

instead of

$ sudo make TARGET_ARCH=armv6 KERNCONF=XENHVM buildkernel

The FreeBSD 'kernel' file is in ELF format and did not work, and we spent
some time trying to convert it to the zImage format without realizing the
build of the FreeBSD kernel creates the 'kernel.bin' file in the zImage format.
So when booting with the 'kernel.bin' file instead, it actually boots :

user@devuan-bunsen ~ % sudo xl create freebsd.cfg
Parsing config from freebsd.cfg
user@devuan-bunsen ~ % sudo xl li
NameID   Mem VCPUs  State   Time(s)
Domain-0 0   768 2 r-1439.4
freebsd  1  1152 1 r-   3.0
user@devuan-bunsen ~ %

However, the guest is still not working correctly :

1. Attaching the console with the -c option at creation or with
   'xl console freebsd' results in no output to the console.

2. The timestamp on the virtual disk image file shows that the filesystem
   was at best mounted read-only, if it was mounted at all by the guest
   FreeBSD kernel.

3. The 'xl shutdown freebsd' command does not work, it just hangs. To stop
   the guest, you need to do 'xl destroy freebsd'.

However, I think we can get the console to work and the rootfs to mount because 
I
just realized I forgot to do the steps from Julien's instructions of editing the
/etc/fstab and /etc/ttys files in the FreeBSD rootfs :

$ echo "/dev/xbd0   /   ufs rw  1   1" > /mnt/etc/fstab
$ vi /mnt/etc/ttys (add the line 'xc0 "/usr/libexec/getty Pc" xterm on secure")

I will add those and see if the console and disk are working.

But it would be much better if we could have this working with FreeBSD 13 / 14
instead of the old FreeBSD 11. Also, Julien's patch set only supports one vcpu
and it would be great to get the smp support for FreeBSD also.

Thanks,

Chuck

> 
> 
> 
> -- 
> Mario.

Re: [PATCH] arm/mm: add option to prefer IOMMU ops for DMA on Xen

[Fr. Chuck Zmudzinski, C.P.M.]

On 11/14/2023 5:20 PM, Stefano Stabellini wrote:
> On Tue, 14 Nov 2023, Robin Murphy wrote:
>> On 11/11/2023 6:45 pm, Chuck Zmudzinski wrote:
>> > Enabling the new option, ARM_DMA_USE_IOMMU_XEN, fixes this error when
>> > attaching the Exynos mixer in Linux dom0 on Xen on the Chromebook Snow
>> > (and probably on other devices that use the Exynos mixer):
>> > 
>> > [drm] Exynos DRM: using 1440.fimd device for DMA mapping operations
>> > exynos-drm exynos-drm: bound 1440.fimd (ops 0xc0d96354)
>> > exynos-mixer 1445.mixer: [drm:exynos_drm_register_dma] *ERROR* Device
>> >   1445.mixer lacks support for IOMMU
>> > exynos-drm exynos-drm: failed to bind 1445.mixer (ops 0xc0d97554): -22
>> > exynos-drm exynos-drm: adev bind failed: -22
>> > exynos-dp: probe of 145b.dp-controller failed with error -22
>> > 
>> > Linux normally uses xen_swiotlb_dma_ops for DMA for all devices when
>> > xen_swiotlb is detected even when Xen exposes an IOMMU to Linux. Enabling
>> > the new config option allows devices such as the Exynos mixer to use the
>> > IOMMU instead of xen_swiotlb_dma_ops for DMA and this fixes the error.
>> > 
>> > The new config option is not set by default because it is likely some
>> > devices that use IOMMU for DMA on Xen will cause DMA errors and memory
>> > corruption when Xen PV block and network drivers are in use on the system.
>> > 
>> > Link:
>> > https://lore.kernel.org/xen-devel/acfab1c5-eed1-4930-8c70-8681e256c...@netscape.net/
>> > 
>> > Signed-off-by: Chuck Zmudzinski 
>> > ---
>> > The reported error with the Exynos mixer is not fixed by default by adding
>> > a second patch to select the new option in the Kconfig definition for the
>> > Exynos mixer if EXYNOS_IOMMU and SWIOTLB_XEN are enabled because it is
>> > not certain setting the config option is suitable for all cases. So it is
>> > necessary to explicitly select the new config option during the config
>> > stage of the Linux kernel build to fix the reported error or similar
>> > errors that have the same cause of lack of support for IOMMU on Xen. This
>> > is necessary to avoid any regressions that might be caused by enabling the
>> > new option by default for the Exynos mixer.
>> >   arch/arm/mm/dma-mapping.c |  6 ++
>> >   drivers/xen/Kconfig   | 16 
>> >   2 files changed, 22 insertions(+)
>> > 
>> > diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
>> > index 5409225b4abc..ca04fdf01be3 100644
>> > --- a/arch/arm/mm/dma-mapping.c
>> > +++ b/arch/arm/mm/dma-mapping.c
>> > @@ -1779,6 +1779,12 @@ void arch_setup_dma_ops(struct device *dev, u64
>> > dma_base, u64 size,
>> >if (iommu)
>> >arm_setup_iommu_dma_ops(dev, dma_base, size, iommu, coherent);
>> >   +#ifdef CONFIG_ARM_DMA_USE_IOMMU_XEN
>> 
>> FWIW I don't think this really needs a config option - if Xen *has* made an
>> IOMMU available, then there isn't really much reason not to use it, and if 
>> for
>> some reason someone really didn't want to then they could simply disable the
>> IOMMU driver anyway.
> 
> The fact that the Exynos IOMMU is exposed to Linux is a mistake. Xen
> doesn't recognize the Exynos IOMMU (it is not one of the IOMMUs Xen has
> a driver for) so it assigns the IOMMU to Dom0. It doesn't happen on
> purpose, it happens by accident. Certain things are going to break,
> specifically I am fairly certain PV drivers are going to break.
> 
> If Xen recognized the Exynos IOMMU as an IOMMU it would probably hide it
> from Dom0. (Today Xen doesn't have a list of IOMMUs Xen recognizes but
> doesn't have a driver for.)
> 
> I think it is OK for Chuck and others to play around with this
> configuration but I wouldn't add a new kconfig option to Linux to
> support it.
> 
> If we do want a kconfig option, I would add a kconfig option or Linux
> command line option to enable/disable swiotlb-xen. Basically a way to
> force-enable or force-disable xen_swiotlb_detect().That could be

I am trying to understand what you are proposing.

I tried disabling the CONFIG_SWIOTLB_XEN option that already
exists and it does not seem possible to disable that option without
also totally removing Xen dom0 support from Linux on arm. So do you
suggest keeping that option as is and adding a Linux command line
switch or new Linux Kconfig option that is ignored unless
CONFIG_SWIOTLB_XEN is enabled and would make xen_swiotlb_detect()
always return false or always return true, depending on the setting
of the new option?

Thanks,

Chuck

> generally useful for debugging and would also solve the problem here as
> it could be used to force-disable swiotlb-xen. I would imagine that the
> end result is the same: the default ops (iommu_ops) are used.
> 

Re: [PATCH] arm/mm: add option to prefer IOMMU ops for DMA on Xen

[Chuck Zmudzinski]

On 11/15/2023 12:56 PM, Chuck Zmudzinski wrote:
> On 11/14/2023 5:20 PM, Stefano Stabellini wrote:
>> On Tue, 14 Nov 2023, Robin Murphy wrote:
>>> On 11/11/2023 6:45 pm, Chuck Zmudzinski wrote:
>>> > Enabling the new option, ARM_DMA_USE_IOMMU_XEN, fixes this error when
>>> > attaching the Exynos mixer in Linux dom0 on Xen on the Chromebook Snow
>>> > (and probably on other devices that use the Exynos mixer):
>>> > 
>>> > [drm] Exynos DRM: using 1440.fimd device for DMA mapping operations
>>> > exynos-drm exynos-drm: bound 1440.fimd (ops 0xc0d96354)
>>> > exynos-mixer 1445.mixer: [drm:exynos_drm_register_dma] *ERROR* Device
>>> >   1445.mixer lacks support for IOMMU
>>> > exynos-drm exynos-drm: failed to bind 1445.mixer (ops 0xc0d97554): -22
>>> > exynos-drm exynos-drm: adev bind failed: -22
>>> > exynos-dp: probe of 145b.dp-controller failed with error -22
>>> > 
>>> > Linux normally uses xen_swiotlb_dma_ops for DMA for all devices when
>>> > xen_swiotlb is detected even when Xen exposes an IOMMU to Linux. Enabling
>>> > the new config option allows devices such as the Exynos mixer to use the
>>> > IOMMU instead of xen_swiotlb_dma_ops for DMA and this fixes the error.
>>> > 
>>> > The new config option is not set by default because it is likely some
>>> > devices that use IOMMU for DMA on Xen will cause DMA errors and memory
>>> > corruption when Xen PV block and network drivers are in use on the system.
>>> > 
>>> > Link:
>>> > https://lore.kernel.org/xen-devel/acfab1c5-eed1-4930-8c70-8681e256c...@netscape.net/
>>> > 
>>> > Signed-off-by: Chuck Zmudzinski 
>>> > ---
>>> > The reported error with the Exynos mixer is not fixed by default by adding
>>> > a second patch to select the new option in the Kconfig definition for the
>>> > Exynos mixer if EXYNOS_IOMMU and SWIOTLB_XEN are enabled because it is
>>> > not certain setting the config option is suitable for all cases. So it is
>>> > necessary to explicitly select the new config option during the config
>>> > stage of the Linux kernel build to fix the reported error or similar
>>> > errors that have the same cause of lack of support for IOMMU on Xen. This
>>> > is necessary to avoid any regressions that might be caused by enabling the
>>> > new option by default for the Exynos mixer.
>>> >   arch/arm/mm/dma-mapping.c |  6 ++
>>> >   drivers/xen/Kconfig   | 16 
>>> >   2 files changed, 22 insertions(+)
>>> > 
>>> > diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
>>> > index 5409225b4abc..ca04fdf01be3 100644
>>> > --- a/arch/arm/mm/dma-mapping.c
>>> > +++ b/arch/arm/mm/dma-mapping.c
>>> > @@ -1779,6 +1779,12 @@ void arch_setup_dma_ops(struct device *dev, u64
>>> > dma_base, u64 size,
>>> >   if (iommu)
>>> >   arm_setup_iommu_dma_ops(dev, dma_base, size, iommu, 
>>> > coherent);
>>> >   +#ifdef CONFIG_ARM_DMA_USE_IOMMU_XEN
>>> 
>>> FWIW I don't think this really needs a config option - if Xen *has* made an
>>> IOMMU available, then there isn't really much reason not to use it, and if 
>>> for
>>> some reason someone really didn't want to then they could simply disable the
>>> IOMMU driver anyway.
>> 
>> The fact that the Exynos IOMMU is exposed to Linux is a mistake. Xen
>> doesn't recognize the Exynos IOMMU (it is not one of the IOMMUs Xen has
>> a driver for) so it assigns the IOMMU to Dom0. It doesn't happen on
>> purpose, it happens by accident. Certain things are going to break,
>> specifically I am fairly certain PV drivers are going to break.
>> 
>> If Xen recognized the Exynos IOMMU as an IOMMU it would probably hide it
>> from Dom0. (Today Xen doesn't have a list of IOMMUs Xen recognizes but
>> doesn't have a driver for.)
>> 
>> I think it is OK for Chuck and others to play around with this
>> configuration but I wouldn't add a new kconfig option to Linux to
>> support it.
>> 
>> If we do want a kconfig option, I would add a kconfig option or Linux
>> command line option to enable/disable swiotlb-xen. Basically a way to
>> force-enable or force-disable xen_swiotlb_detect(). That could be
>> generally useful for debugging and would also solve the problem here as
>>

Re: Values generated by the ViryaOS uboot-script-gen do not work correctly on the Chromebook Snow

[Chuck Zmudzinski]

On 11/15/2023 7:07 PM, Mario Marietto wrote:
> It didn't work. This is the scr file generated.
> 
> ext2load mmc 1:3 0x5100 zImage-6.6.0-xen-iommu-dma-on-xen
> ext2load mmc 1:3 0x6000 xen-4.17-armhf.ub

When you created the xen-4.17-armhf.ub file, do you remember what
the LOADADDR and entry point was? In my case those needed to be offset by 
0x4000.

So the uboot version of Xen would created with with the offset of 0x4000 for
the LOADADDR and entry point from where the script loads it:

mkimage -A arm -T kernel -C none -a 0x60004000 -e 0x60004000 -d xen-4.17-armhf 
xen-4.17-armhf.ub

That is what has been working in my case.

> ext2load mmc 1:3 0x6100 exynos5250-snow.dtb
> fdt addr 0x6100
> fdt resize 1024
> fdt set /chosen \#address-cells <0x2>
> fdt set /chosen \#size-cells <0x2>
> fdt set /chosen xen,xen-bootargs "console=dtuart dtuart=serial0 dom0_mem=768 
> dom0_max_vcpus=2 bootscrub=0 vwfi=native sched=null"
> fdt mknod /chosen dom0
> fdt set /chosen/dom0 compatible  "xen,linux-zimage" "xen,multiboot-module" 
> "multiboot,module"
> fdt set /chosen/dom0 reg <0x0 0x5100 0x0 0x87C200 >
> fdt set /chosen xen,dom0-bootargs "console=tty earlycon=xen earlyprintk=xen 
> root=/dev/mmcblk1p4 rw rootwait clk_ignore_unused"
> setenv fdt_high 0x
> bootm 0x6000 - 0x6100
> 
> So,I ran :
> 
> bash /boot/./uboot-script-gen -c /boot/xen-config -d .
> 
> it says :
> 
> Image Name:    
> Created:  Wed Nov 15 23:55:40 2023
> Image Type:   ARM Linux Kernel Image (uncompressed)
> Data Size:    884744 Bytes = 864.01 KiB = 0.84 MiB
> Load Address: 6000
> Entry Point:  6000
> Generated uboot script xen-stef.scr, to be loaded at address 0x4200:
> ext2load mmc 1:3 0x4200 xen-stef.scr; source 0x4200
> 
> ok,I've booted xen with the suggested address :
> 
> ext2load mmc 1:3 0x4200 xen-stef.scr; source 0x4200
> 
> but it rebooted to the verification screen.
> 
> NB : I have applied both your suggestions (offset + your new start and end 
> memory address. Maybe they auto exclude each other ?)
> 
> On Thu, Nov 16, 2023 at 12:49 AM Stefano Stabellini  <mailto:sstabell...@kernel.org>> wrote:
> 
> On Wed, 15 Nov 2023, Chuck Zmudzinski wrote:
> > On 11/14/2023 6:43 PM, Mario Marietto wrote:
> > > I hope that the informations below are correct :
> >
> > I don't know that they are correct. I have not spent the necessary time 
> to
> > determine what the correct values for MEMORY_START and MEMORY_END are 
> for
> > the Chromebook we are using. I just presumed, probably incorrectly, that
> > the entire 2 GB memory is safe, but obviously that is not the case with
> > this Chromebook. Most likely, it requires a good understanding of the
> > particular way booting is done on a Chromebook, which seems to be 
> different
> > from other devices.
> >
> > I plan to eventually look into finding values for MEMORY_START and 
> MEMORY_END
> > sothe uboot-script-gen script computes usable values for loading Xen 
> and dom0
> > on this Chromebook in the script, but I might not get to that task 
> immediately.
> > I plan to look at it within the next week or so.
> 
> A couple of suggestions. I noticed that the addresses you chose have a
> higher alignment compared to the one chosen by Imagebuilder.
> Imagebuilder uses 2MB:
> 
> offset=$((2*1024*1024))
> 
> I would think that a 2MB alignment should be sufficient, but you can
> increase the alignment chosen by Imagebuilder simply by changing
> "offset" at the top of uboot-script-gen. You seem to be used a 240MB
> offset:
> 
> offset=$((240*1024*1024))
> 
> The other suggestion is about MEMORY_START and MEMORY_END. Looking at
> the addresses you picked by hand, the following you should give you very
> similar results:
> 
> MEMORY_START=0x3300
> MEMORY_END=0x8000
> 
> 
> > > - the imagebuilder config file :
> > >
> > > MEMORY_START="0x0"
> > > MEMORY_END="0x8000"
> > > LOAD_CMD="ext2load mmc 1:3"
> > > BOOT_CMD="bootm"
> > > DEVICE_TREE="exynos5250-snow.dtb"
> > > XEN="xen-4.17-armhf"
> > > XEN_CMD="console=dtuart dtuart=serial0 dom0_mem=1152M 
> dom0_max_vcpus=2 bootscrub=0 vwfi=native sched=null"
> > > DOM0_KERNEL="zImage-6.6.0-xen-iommu-dma-on-xen"
> > > DOM0_CMD="console=tty

Re: [PATCH] arm/mm: add option to prefer IOMMU ops for DMA on Xen

[Chuck Zmudzinski]

On 11/14/2023 5:20 PM, Stefano Stabellini wrote:
> On Tue, 14 Nov 2023, Robin Murphy wrote:
>> On 11/11/2023 6:45 pm, Chuck Zmudzinski wrote:
>> > Enabling the new option, ARM_DMA_USE_IOMMU_XEN, fixes this error when
>> > attaching the Exynos mixer in Linux dom0 on Xen on the Chromebook Snow
>> > (and probably on other devices that use the Exynos mixer):
>> > 
>> > [drm] Exynos DRM: using 1440.fimd device for DMA mapping operations
>> > exynos-drm exynos-drm: bound 1440.fimd (ops 0xc0d96354)
>> > exynos-mixer 1445.mixer: [drm:exynos_drm_register_dma] *ERROR* Device
>> >   1445.mixer lacks support for IOMMU
>> > exynos-drm exynos-drm: failed to bind 1445.mixer (ops 0xc0d97554): -22
>> > exynos-drm exynos-drm: adev bind failed: -22
>> > exynos-dp: probe of 145b.dp-controller failed with error -22
>> > 
>> > Linux normally uses xen_swiotlb_dma_ops for DMA for all devices when
>> > xen_swiotlb is detected even when Xen exposes an IOMMU to Linux. Enabling
>> > the new config option allows devices such as the Exynos mixer to use the
>> > IOMMU instead of xen_swiotlb_dma_ops for DMA and this fixes the error.
>> > 
>> > The new config option is not set by default because it is likely some
>> > devices that use IOMMU for DMA on Xen will cause DMA errors and memory
>> > corruption when Xen PV block and network drivers are in use on the system.
>> > 
>> > Link:
>> > https://lore.kernel.org/xen-devel/acfab1c5-eed1-4930-8c70-8681e256c...@netscape.net/
>> > 
>> > Signed-off-by: Chuck Zmudzinski 
>> > ---
>> > The reported error with the Exynos mixer is not fixed by default by adding
>> > a second patch to select the new option in the Kconfig definition for the
>> > Exynos mixer if EXYNOS_IOMMU and SWIOTLB_XEN are enabled because it is
>> > not certain setting the config option is suitable for all cases. So it is
>> > necessary to explicitly select the new config option during the config
>> > stage of the Linux kernel build to fix the reported error or similar
>> > errors that have the same cause of lack of support for IOMMU on Xen. This
>> > is necessary to avoid any regressions that might be caused by enabling the
>> > new option by default for the Exynos mixer.
>> >   arch/arm/mm/dma-mapping.c |  6 ++
>> >   drivers/xen/Kconfig   | 16 
>> >   2 files changed, 22 insertions(+)
>> > 
>> > diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
>> > index 5409225b4abc..ca04fdf01be3 100644
>> > --- a/arch/arm/mm/dma-mapping.c
>> > +++ b/arch/arm/mm/dma-mapping.c
>> > @@ -1779,6 +1779,12 @@ void arch_setup_dma_ops(struct device *dev, u64
>> > dma_base, u64 size,
>> >if (iommu)
>> >arm_setup_iommu_dma_ops(dev, dma_base, size, iommu, coherent);
>> >   +#ifdef CONFIG_ARM_DMA_USE_IOMMU_XEN
>> 
>> FWIW I don't think this really needs a config option - if Xen *has* made an
>> IOMMU available, then there isn't really much reason not to use it, and if 
>> for
>> some reason someone really didn't want to then they could simply disable the
>> IOMMU driver anyway.
> 
> The fact that the Exynos IOMMU is exposed to Linux is a mistake. Xen
> doesn't recognize the Exynos IOMMU (it is not one of the IOMMUs Xen has
> a driver for) so it assigns the IOMMU to Dom0. It doesn't happen on
> purpose, it happens by accident. Certain things are going to break,
> specifically I am fairly certain PV drivers are going to break.
> 
> If Xen recognized the Exynos IOMMU as an IOMMU it would probably hide it
> from Dom0. (Today Xen doesn't have a list of IOMMUs Xen recognizes but
> doesn't have a driver for.)
> 
> I think it is OK for Chuck and others to play around with this
> configuration but I wouldn't add a new kconfig option to Linux to
> support it.
> 
> If we do want a kconfig option, I would add a kconfig option or Linux
> command line option to enable/disable swiotlb-xen. Basically a way to
> force-enable or force-disable xen_swiotlb_detect(). That could be
> generally useful for debugging and would also solve the problem here as
> it could be used to force-disable swiotlb-xen. I would imagine that the
> end result is the same: the default ops (iommu_ops) are used.

I will try this. It isn't exactly what I have tested until now because
in all my tests so far all the DMA capable devices on the Chromebook use
swioltlb-xen except for the two devices that need to use the Exynos IOMMU
to fix the error with the

Re: Values generated by the ViryaOS uboot-script-gen do not work correctly on the Chromebook Snow

[Chuck Zmudzinski]

On 11/14/2023 6:43 PM, Mario Marietto wrote:
> I hope that the informations below are correct :

I don't know that they are correct. I have not spent the necessary time to
determine what the correct values for MEMORY_START and MEMORY_END are for
the Chromebook we are using. I just presumed, probably incorrectly, that
the entire 2 GB memory is safe, but obviously that is not the case with
this Chromebook. Most likely, it requires a good understanding of the
particular way booting is done on a Chromebook, which seems to be different
from other devices.

I plan to eventually look into finding values for MEMORY_START and MEMORY_END
sothe uboot-script-gen script computes usable values for loading Xen and dom0
on this Chromebook in the script, but I might not get to that task immediately.
I plan to look at it within the next week or so.

> 
> - the imagebuilder config file :
> 
> MEMORY_START="0x0"
> MEMORY_END="0x8000"
> LOAD_CMD="ext2load mmc 1:3"
> BOOT_CMD="bootm"
> DEVICE_TREE="exynos5250-snow.dtb"
> XEN="xen-4.17-armhf"
> XEN_CMD="console=dtuart dtuart=serial0 dom0_mem=1152M dom0_max_vcpus=2 
> bootscrub=0 vwfi=native sched=null"
> DOM0_KERNEL="zImage-6.6.0-xen-iommu-dma-on-xen"
> DOM0_CMD="console=tty earlycon=xen earlyprintk=xen root=/dev/mmcblk1p4 rw 
> rootwait clk_ignore_unused"
> UBOOT_SOURCE="xen.source"
> UBOOT_SCRIPT="xen.scr"
> 
> xen.source : (that does not work)
> 
> mmc dev 1
> ext2load mmc 1:3 0xE0 zImage-6.6.0-xen-iommu-dma-on-xen
> ext2load mmc 1:3 0x180 xen-4.17-armhf.ub
> ext2load mmc 1:3 0x1A0 exynos5250-snow.dtb
> fdt addr 0x1A0
> fdt resize 1024
> fdt set /chosen \#address-cells <0x2>
> fdt set /chosen \#size-cells <0x2>
> fdt set /chosen xen,xen-bootargs "console=dtuart dtuart=serial0 
> dom0_mem=1152M dom0_max_vcpus=2 bootscrub=0 vwfi=native sched=null"
> fdt mknod /chosen dom0
> fdt set /chosen/dom0 compatible  "xen,linux-zimage" "xen,multiboot-module" 
> "multiboot,module"
> fdt set /chosen/dom0 reg <0x0 0xE0 0x0 0x87C200 >
> fdt set /chosen xen,dom0-bootargs "console=tty earlycon=xen earlyprintk=xen 
> root=/dev/mmcblk1p4 rw rootwait clk_ignore_unused"
> setenv fdt_high 0x
> bootm 0x180 - 0x1A0
> 
> xen.source : (created by chuck and that works)
> 
> mmc dev 1
> ext2load mmc 1:3 0x4200 zImage-6.6.0-xen-iommu-dma-on-xen
> ext2load mmc 1:3 0x5100 xen-4.17-armhf-armmp-0x51004000.ub
> ext2load mmc 1:3 0x5ffec000 exynos5250-snow.dtb
> fdt addr 0x5ffec000
> fdt resize 1024
> fdt set /chosen \#address-cells <0x2>
> fdt set /chosen \#size-cells <0x2>
> fdt set /chosen xen,xen-bootargs "console=dtuart dtuart=serial0 
> dom0_mem=1152M dom0_max_vcpus=2 bootscrub=0 vwfi=native sched=null"
> fdt mknod /chosen dom0
> fdt set /chosen/dom0 compatible  "xen,linux-zimage" "xen,multiboot-module" 
> "multiboot,module"
> fdt set /chosen/dom0 reg <0x0 0x4200 0x0 0x87C200 >
> fdt set /chosen xen,dom0-bootargs "console=tty1 root=/dev/mmcblk1p4 rw 
> rootwait clk_ignore_unused --no-log"
> bootm 0x5100 - 0x5ffec000
> 
> all the values that you see in this conf. files have been calculated by chuck 
> by hand,because the values generated by the imagebuilder are wrong. The only 
> value that's well calculated by the imagebuilder is 0x87C200
> 
> - the size of all the binaries specified in the imagebuilder config file :
> 
> exynos5250-snow.dtb = 46.6 KiB (47,769 byte)
> zImage-6.6.0-xen-iommu-dma-on-xen = 8.5 MiB (8,897,024 byte)
> 
> 
> 
> On Wed, Nov 15, 2023 at 12:17 AM Stefano Stabellini  > wrote:
> 
> Hi Mario,
> 
> I think we misunderstood each other :-)
> 
> MEMORY_START-MEMORY_END is not supposed to be computed: it is supposed
> to come from the memory node in device tree tree (/memory) of the
> platform. The idea is that you should not have to do any computations,
> but only reuse the same address range specified there.
> 
> Similarly in regards to "please post the size of all the binaries",
> this is just for debugging, so that I can see if there are any bugs with
> uboot-script-gen. I cannot debug the script unless I figure out what the
> problem is and the only way I can do that is with the binary sizes and
> redoing all the steps by hand.
> 
> The expected outcome is that once we resolve the problem you should be
> able to use uboot-script-gen without any additional computation needed.
> 
> Of course using static values is also OK.
> 
> 
> On Wed, 15 Nov 2023, Mario Marietto wrote:
> > ---> uboot-script-gen assumes that the memory range specified by 
> MEMORY_START-MEMORY_END is valid and correct.
> >
> > Actually Chuck chose 0 as MEMORY_START and 0x80 as MEMORY_END and 
> these are stable values,they don't change. If you ask me to calculate
> > those values,it means that we need to compute these values. I imagine 
> that to calculate these values is not easy.
> >
> > ---> To debug this kind of issues please post 

[PATCH] arm/mm: add option to prefer IOMMU ops for DMA on Xen

[Chuck Zmudzinski]

Enabling the new option, ARM_DMA_USE_IOMMU_XEN, fixes this error when
attaching the Exynos mixer in Linux dom0 on Xen on the Chromebook Snow
(and probably on other devices that use the Exynos mixer):

[drm] Exynos DRM: using 1440.fimd device for DMA mapping operations
exynos-drm exynos-drm: bound 1440.fimd (ops 0xc0d96354)
exynos-mixer 1445.mixer: [drm:exynos_drm_register_dma] *ERROR* Device
 1445.mixer lacks support for IOMMU
exynos-drm exynos-drm: failed to bind 1445.mixer (ops 0xc0d97554): -22
exynos-drm exynos-drm: adev bind failed: -22
exynos-dp: probe of 145b.dp-controller failed with error -22

Linux normally uses xen_swiotlb_dma_ops for DMA for all devices when
xen_swiotlb is detected even when Xen exposes an IOMMU to Linux. Enabling
the new config option allows devices such as the Exynos mixer to use the
IOMMU instead of xen_swiotlb_dma_ops for DMA and this fixes the error.

The new config option is not set by default because it is likely some
devices that use IOMMU for DMA on Xen will cause DMA errors and memory
corruption when Xen PV block and network drivers are in use on the system.

Link: 
https://lore.kernel.org/xen-devel/acfab1c5-eed1-4930-8c70-8681e256c...@netscape.net/

Signed-off-by: Chuck Zmudzinski 
---
The reported error with the Exynos mixer is not fixed by default by adding
a second patch to select the new option in the Kconfig definition for the
Exynos mixer if EXYNOS_IOMMU and SWIOTLB_XEN are enabled because it is
not certain setting the config option is suitable for all cases. So it is
necessary to explicitly select the new config option during the config
stage of the Linux kernel build to fix the reported error or similar
errors that have the same cause of lack of support for IOMMU on Xen. This
is necessary to avoid any regressions that might be caused by enabling the
new option by default for the Exynos mixer.
 arch/arm/mm/dma-mapping.c |  6 ++
 drivers/xen/Kconfig   | 16 
 2 files changed, 22 insertions(+)

diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
index 5409225b4abc..ca04fdf01be3 100644
--- a/arch/arm/mm/dma-mapping.c
+++ b/arch/arm/mm/dma-mapping.c
@@ -1779,6 +1779,12 @@ void arch_setup_dma_ops(struct device *dev, u64 
dma_base, u64 size,
if (iommu)
arm_setup_iommu_dma_ops(dev, dma_base, size, iommu, coherent);
 
+#ifdef CONFIG_ARM_DMA_USE_IOMMU_XEN
+   if (dev->dma_ops == _ops) {
+   dev->archdata.dma_ops_setup = true;
+   return;
+   }
+#endif
xen_setup_dma_ops(dev);
dev->archdata.dma_ops_setup = true;
 }
diff --git a/drivers/xen/Kconfig b/drivers/xen/Kconfig
index d5989871dd5d..44e1334b6acd 100644
--- a/drivers/xen/Kconfig
+++ b/drivers/xen/Kconfig
@@ -181,6 +181,22 @@ config SWIOTLB_XEN
select DMA_OPS
select SWIOTLB
 
+config ARM_DMA_USE_IOMMU_XEN
+   bool "Prefer IOMMU DMA ops on Xen"
+   depends on SWIOTLB_XEN
+   depends on ARM_DMA_USE_IOMMU
+   help
+ Normally on Xen, the IOMMU is used by Xen and not exposed to
+ Linux. Some Arm systems such as Exynos have an IOMMU that
+ Xen does not use so the IOMMU is exposed to Linux in those
+ cases. This option enables Linux to use the IOMMU instead of
+ using the Xen swiotlb_dma_ops for DMA on Xen.
+
+ Say N here unless support for one or more devices that use
+ IOMMU ops instead of Xen swiotlb ops for DMA is needed and the
+ devices that use the IOMMU do not cause any problems on the
+ Xen system in use.
+
 config XEN_PCI_STUB
bool
 
-- 
2.39.2

Re: exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU

[Chuck Zmudzinski]

Hi everyone,

This reply is intended to clarify the latest test results and bring the
clarifications and other relevant discussion to the xen-devel mailing list.

On 11/1/2023 5:14 AM, Julien Grall wrote:
> 
> 
> On 01/11/2023 08:45, Chuck Zmudzinski wrote:
>> On 11/1/2023 4:27 AM, Julien Grall wrote:
>>> Hi,
>>>
>>> @Stefano, as you pointed out, there is already a thread on xen-users for
>>> this discussion. Could we use this thread for any discussion? This would
>>> make easier to follow.
>>>
>>> Some high level comment below.
>> 
>> I agree to keep the discussion here and not at other places.
> 
> I was meant to suggest the other thread :). But either is fine with me. 
> I just want to avoid avoid multiple seperate threads for the discussion.
> 
>> 
>> I just want to add that the best results for Xen dom0 so far are
>> by implementing Marek's suggestion to disable these two settings
>> in the 6.1.59 kernel config, but leaving everything else the same,
>> including keeping the EXYNOS_IOMMU support enabled:

I got even better results with a small patch in arch/arm/mm to disable
the overwriting of dma_ops with xen_swiotlb_dma_ops for a device when
the dma_ops are already set to use the iommu_ops, otherwise, the
current behavior of overwriting or setting dma_ops for the first time
with the xen_swiotlb_dma_ops is done. This totally fixes the error,
and also allows the HDMI port to work with Linux dom0 on Xen!

> That's good news! I would be interested to hear how this works once you 
> start to have PV backend in dom0 (I expect that the IOMMU will get 
> confused with grant mapping).

I did lots of tests such as building a kernel in a domU with PV block
and network frontend drivers connected to dom0 on the backend while
also building the qemu device model in dom0 using a Linux kernel in dom0
with the aforementioned patch to not overwrite dma_ops if dma_ops is
already set to iommu_ops, and on this Chromebook IOMMU had no confusion
and the feared DMA errors and memory corruption did not materialize!

So I am preparing to submit a patch to lkml to fix the exynos mixer.
on Xen. I just finished testing a version of the patch implemented as
a new config option that is set when support for the device causing
the trouble, the exynos mixer, is present in Linux and EXYNOS_IOMMU
config option is also enabled. I think this is a conservative approach
to add a new config option that can be set for cases like this
Chromebook when the devices that need to use IOMMU are behaving nicely
and do not cause any trouble on Xen. The default will continue to be
that Linux will overwrite IOMMU dma_ops with xen_swiotlb_dma_ops on
Xen unless the new config option is set.

> 
> Also, do you plan to passthrough any of the devices protected by IOMMU?

No. On this Chromebook, the only two devices using IOMMMU in the system on
dom0 with the soon-to-be proposed patch are the exynos-fimd and the
exynos-mixer, which support video for dom0. All other devices in the system
are using the xen_swiotlb_dma_ops. These facts, I think, explain why the
feared DMA errors and IOMMU confusion with the PV drivers for other guests
on the system did not materialize in this case. 

> 
>> 
>> # CONFIG_DRM_EXYNOS_MIXER is not set
>> 
>> Disabling the mixer also makes this unavailable:
>> 
>> # CONFIG_HDMI
>> 
>> With this change, the GPU is working well enough to allow the display
>> manager and an X11 session to run normally on the built-in display of the
>> Chromebook. The Wifi also works well.

As mentioned earlier, these settings worked also, but with the disadvantage
of disabling support for the HDMI port on the Chromebook. My latest tests
indicate we can fix this on Xen without giving up support for the HDMI!

> 
> I saw your other answer about the Wifi not working when the IOMMU is not 
> used. I was about to reply there, but instead I will do it here.
> 
> TBH, I am quite surprised this is the case. The only difference with 
> baremetal would be the RAM regions. Do you know if the Wifi dongle only 
> accept certain physical address?

The Wifi device worked well enough to associate with a Wifi access point
without EXYNOS_IOMMU enabled, it just failed to get IP addresses from
DHCP. I don't know if the exynos Wifi device requires a certain physical
address for the function of acquiring IP addresses from DHCP to work. Marek
might be able to answer that question.

In any case, since the Chromebook works fine on Xen, including Wifi, when
the EXYNOS_IOMMU is used by Linux dom0 as long as Linux does not overwrite
the dma_ops with xen_swiotlb_dma_ops when they had previously been set to
iommu_ops in arch/arm/mm/dma-mapping.c, finding the answer to the problem of
Wifi not working when the IOMMU is not used is not essential because the
default for both exynos systems and multi_v7 arm systems in Linux is to use
the exynos IOMMU when it is available, both on bare metal and on Xen / dom0.

Cheers,

> 
> Cheers,
> 

Re: exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU

[Chuck Zmudzinski]

On 11/1/2023 4:27 AM, Julien Grall wrote:
> Hi,
> 
> @Stefano, as you pointed out, there is already a thread on xen-users for 
> this discussion. Could we use this thread for any discussion? This would 
> make easier to follow.
> 
> Some high level comment below.

I agree to keep the discussion here and not at other places.

I just want to add that the best results for Xen dom0 so far are
by implementing Marek's suggestion to disable these two settings
in the 6.1.59 kernel config, but leaving everything else the same,
including keeping the EXYNOS_IOMMU support enabled:

# CONFIG_DRM_EXYNOS_MIXER is not set

Disabling the mixer also makes this unavailable:

# CONFIG_HDMI

With this change, the GPU is working well enough to allow the display
manager and an X11 session to run normally on the built-in display of the
Chromebook. The Wifi also works well.

The patch from Linux 6.2 and above that fixes the exynos IOMMU initialization
did not help at all, and the same error is reported that the mixer lacks
support for IOMMU.


> 
> On 01/11/2023 02:50, Chuck Zmudzinski wrote:
>> On 10/31/2023 7:45 PM, Stefano Stabellini wrote:
>>> Unfortunately there is no easy solution.
>>>
>>> Do you know the version of the SMMU available on the platform?
>> 
>> I am trying to discern, but I doubt we have v3 because we are
>> working on a very old chromebook from 2012, and I am finding
>> patches for smmv3 in Linux not starting until 2015. It is good to
>> know about this option, though, for future work we might do on newer
>> devices.
> 
> The chromebook is using the Exynos Sysmmu. So there is no SMMU.
> 
>> 
>>> If it is a SMMUv3 you can try to use the nested SMMU patch series to
>>> enable a virtual SMMU in Dom0: 
>>> https://marc.info/?l=xen-devel=166991020831005
>>> That way, Xen can use the SMMU to protect VMs, and Dom0 can also use the
>>> SMMU for its own purposes at the same time.
>>>
>>> Alternatively, you can dig into the details of the exynos-drm driver to
>>> see what exactly is the dependency on the IOMMU framework in Linux and
>>> remove the dependency. Unfortunately none of us in this thread are
>>> expert on exynos-drm so it would be difficult to advise on how to do
>>> that. For example, I don't know how you could debug the x11 problem you
>>> described because I don't typically work with x11 or with the exynos. If
>>> there is an open source mailing list for exynos-drm development they
>>> might be able to advise on how to remove the IOMMU dependency there.
>> 
>> We have received this message from Marek Szyprowski of Samsung:
>> 
>> https://lore.kernel.org/lkml/7a71e348-f892-4fd4-8857-b72f35ab5...@samsung.com/
>> 
>> Marek recommends this patch to possibly help with this issue:
>> 
>> https://github.com/torvalds/linux/commit/bbc4d205d93f52ee18dfa7858d51489c0506547f
>> 
>> and also these kernel config settings:
>> 
>> On 10/31/2023 8:08 AM, Marek Szyprowski wrote:
>>> If not, then as a temporary workaround please disable
>>> CONFIG_DRM_EXYNOS_MIXER and CONFIG_DRM_EXYNOS_HDMI in your kernel config
>>> and check what will happen (You will lose the HDMI output, but maybe
>>> this won't a big issue).
>> 
>> Mario and I have preliminary evidence that applying both of Marek's
>> recommendations to the 6.1.59 kernel have improved the situation to
>> the point where now the Chromebook can run X.org on Xen. We are doing
>> further tests to see how Marek's patch and/or the kernel config settings
>> to disable the mixer and the HDMI affect the behavior of the GPU in dom0
>> on Xen.
>> 
>>>
>>> The final option, which is a gross hack, would be to let Dom0 use the
>>> IOMMU for its own gain. Xen doesn't use the IOMMU. If you do that you
>>> lose freedom from interference between the VMs and you cannot run driver
>>> domains or directly assign devices to DomUs. But if you are running a
>>> research project you might be OK with that. To get it to work, you need
>>> to hack Xen so that it remaps the IOMMU to Dom0 to let Dom0 program it
>>> directly. The attached patch (untested) would be a place to start. You
>>> also need to pass iommu=false to the Xen command line to prevent Xen
>>> from using the iommu itself.
> 
> This is actually one of the reason why I am suggesting to do all the 
> investigation in one thread. There, we already discovered that the IOMMU 
> was assigned to dom0 because Xen doesn't have a driver and we don't hide 
> them by default.
> 
>> 
>> I am interested to investigate if only the mixer and 

Re: exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU

[Chuck Zmudzinski]

On 10/31/2023 8:08 AM, Marek Szyprowski wrote:
> Hi,
> 
> On 31.10.2023 00:03, Mario Marietto wrote:
>> We are a team of linux enthusiasts who are trying to boot Xen on a 
>> Samsung XE303C12 Chromebook aka "snow" following the suggestions in 
>> the slide show presentation here: 
>> https://www.slideshare.net/xen_com_mgr/xpds16-porting-xen-on-arm-to-a-new-soc-julien-grall-arm
>>  
>> This device uses an exynos5250 SOC dual core 1.7 GHz with 2 MB RAM, it 
>> is a Samsung armv7 chip with virtualization extensions. In particular, 
>> we have it working fairly well both on the bare metal with a recent 
>> 6.1.59 Linux LTS kernel and also with a recent 5.4.257 LTS kernel with 
>> KVM, the older LTS kernel version is used to test KVM because support 
>> for KVM on arm v7 was removed from Linux around kernel version 5.7. So 
>> we know we have the hypervisor mode enabled because we were able to 
>> use it with KVM. For Xen, we are using the latest Debian build of Xen 
>> 4.17 for the Debian armhf architecture: (XEN) Xen version 4.17.2-pre 
>> (Debian 4.17.1+2-gb773c48e36-1) 
>> (pkg-xen-devel@xxx) (arm-linux-gnueabihf-gcc 
>> (Debian 12.2.0-14) 12.2.0) debug=n Thu May 18 19:26:30 UTC 2023 The 
>> Linux kernel is a custom build that adds the Xen config kernel options 
>> (CONFIG_XEN_DOM0, etc) on top of a kernel that works well on the same 
>> Chromebook model on the bare metal. I can provide the config options 
>> of the kernel that was used if that is helpful. Our method of booting 
>> is to have u-boot boot the Xen hypervisor and load the device tree 
>> after adding the dom0 to the otherwise unaltered device tree from the 
>> Linux kernel using u-boot fdt commands to add a /chosen node, as 
>> described on the Xen wiki and in the pages linked from there. We have 
>> also tried adding and loading an initrd.img using the device tree 
>> /chosen node but that made no difference in our tests. We actually 
>> have the Linux LTS kernel version 6.1.59 working as dom0 with Xen 
>> using the same version of u-boot that we used for KVM, but with a big 
>> problem. The problem we see is that when booting the 6.1.59 kernel 
>> version as dom0 with Xen, the screen is totally dark and the only way 
>> to access the system is remotely through ssh. Logs indicate most 
>> everything else is working, such as the wifi card so we can access it 
>> remotely via ssh and a USB optical mouse lights up when connected so 
>> USB is also working. Obviously, the disk is also working. The 
>> Chromebook is configured to boot from the device's SD card slot by 
>> turning on Chrome OS developer mode options to enable booting from the 
>> SD card slot. The mystery is that when booting the exact same 6.1.59 
>> kernel on the bare metal instead of booting it as dom0 on Xen, it 
>> boots up with full access to the screen and we can interact with the 
>> system using the X.org windows system. But booting as dom0 with Xen, 
>> the screen is totally dark and the only access we have to the system 
>> is through the network via ssh. Also, when booting the 5.4.257 kernel 
>> with KVM in hypervisor mode, the screen works and we can interact with 
>> the system through the X.org windows system. Exploring the log file,we 
>> have seen the errors below :
>>
>> Without Xen (or in bare metal):
>>
>> devuan-bunsen kernel: [drm] Exynos DRM: using 1440.fimd device for 
>> DMA mapping operations
>> devuan-bunsen kernel: exynos-drm exynos-drm: bound 1440.fimd (ops 
>> 0xc0d96354)
>> devuan-bunsen kernel: exynos-drm exynos-drm: bound 1445.mixer (ops 
>> 0xc0d97554)
>> devuan-bunsen kernel: exynos-drm exynos-drm: bound 
>> 145b.dp-controller (ops 0xc0d97278)
>> devuan-bunsen kernel: exynos-drm exynos-drm: bound 1453.hdmi (ops 
>> 0xc0d97bd0)
>> ...
>> devuan-bunsen kernel: Console: switching to colour frame buffer device 
>> 170x48
>> devuan-bunsen kernel: exynos-drm exynos-drm: [drm] fb0: exynosdrmfb 
>> frame buffer device
>> devuan-bunsen kernel: [drm] Initialized exynos 1.1.0 20180330 for 
>> exynos-drm on minor 0
>>
>> In this case,the kernel is able to use the exynos-drm kernel to start 
>> the fb0 device. But with Xen we get this error with exynos-drm:
>>
>> devuan-bunsen kernel: [drm] Exynos DRM: using 1440.fimd device for 
>> DMA mapping operations
>> devuan-bunsen kernel: exynos-drm exynos-drm: bound 1440.fimd (ops 
>> 0xc0d96354)
>> devuan-bunsen kernel: exynos-mixer 1445.mixer: 
>> [drm:exynos_drm_register_dma] *ERROR* Device 1445.mixer lacks 
>> support for IOMMU
>> devuan-bunsen kernel: exynos-drm exynos-drm: failed to bind 
>> 1445.mixer (ops 0xc0d97554): -22
>> devuan-bunsen kernel: exynos-drm exynos-drm: adev bind failed: -22
>> devuan-bunsen kernel: exynos-dp: probe of 145b.dp-controller 
>> failed with error -22
>>
>> I'm trying to find for a solution and I've googled a little bit and I 
>> found this web site : 
>> 

Re: exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU

[Chuck Zmudzinski]

On 10/31/2023 7:45 PM, Stefano Stabellini wrote:
> Unfortunately there is no easy solution.
> 
> Do you know the version of the SMMU available on the platform? 

I am trying to discern, but I doubt we have v3 because we are
working on a very old chromebook from 2012, and I am finding
patches for smmv3 in Linux not starting until 2015. It is good to
know about this option, though, for future work we might do on newer
devices.

> If it is a SMMUv3 you can try to use the nested SMMU patch series to
> enable a virtual SMMU in Dom0: 
> https://marc.info/?l=xen-devel=166991020831005
> That way, Xen can use the SMMU to protect VMs, and Dom0 can also use the
> SMMU for its own purposes at the same time.
> 
> Alternatively, you can dig into the details of the exynos-drm driver to
> see what exactly is the dependency on the IOMMU framework in Linux and
> remove the dependency. Unfortunately none of us in this thread are
> expert on exynos-drm so it would be difficult to advise on how to do
> that. For example, I don't know how you could debug the x11 problem you
> described because I don't typically work with x11 or with the exynos. If
> there is an open source mailing list for exynos-drm development they
> might be able to advise on how to remove the IOMMU dependency there.

We have received this message from Marek Szyprowski of Samsung:

https://lore.kernel.org/lkml/7a71e348-f892-4fd4-8857-b72f35ab5...@samsung.com/

Marek recommends this patch to possibly help with this issue:

https://github.com/torvalds/linux/commit/bbc4d205d93f52ee18dfa7858d51489c0506547f

and also these kernel config settings:

On 10/31/2023 8:08 AM, Marek Szyprowski wrote:
> If not, then as a temporary workaround please disable 
> CONFIG_DRM_EXYNOS_MIXER and CONFIG_DRM_EXYNOS_HDMI in your kernel config 
> and check what will happen (You will lose the HDMI output, but maybe 
> this won't a big issue).

Mario and I have preliminary evidence that applying both of Marek's
recommendations to the 6.1.59 kernel have improved the situation to
the point where now the Chromebook can run X.org on Xen. We are doing
further tests to see how Marek's patch and/or the kernel config settings
to disable the mixer and the HDMI affect the behavior of the GPU in dom0
on Xen.

> 
> The final option, which is a gross hack, would be to let Dom0 use the
> IOMMU for its own gain. Xen doesn't use the IOMMU. If you do that you
> lose freedom from interference between the VMs and you cannot run driver
> domains or directly assign devices to DomUs. But if you are running a
> research project you might be OK with that. To get it to work, you need
> to hack Xen so that it remaps the IOMMU to Dom0 to let Dom0 program it
> directly. The attached patch (untested) would be a place to start. You
> also need to pass iommu=false to the Xen command line to prevent Xen
> from using the iommu itself.

I am interested to investigate if only the mixer and the HDMI is causing
the trouble. Based on what you are telling me about Xen not exposing the
IOMMU to dom0, I don't fully understand the original log messages I was
getting when I followed Julien's suggestion to find the symbols associated
with each address in the original message, and those seemed to indicate that
the exynos_drm device was using the IOMMU in dom0, but the mixer was not,
and the fact that they both were not using the IOMMU is what caused the
test to fail and Linux refuse to initialize the GPU on dom0, whereas on
bare metal, the logs indicated both the exynos mixer, which I think is a
sub device of the exynos_drm, and the exynos_drm, use the IOMMU on bare
metal.

I also found this patch which suggests if we can get the devices to work
we will be compromising the security and isolation between guests:

https://patchwork.kernel.org/project/linux-arm-kernel/patch/20190301192017.39770-1-diand...@chromium.org/

There are plenty of unanswered questions here in my mind,

Cheers,

Chuck

> 
> Cheers,
> 
> Stefano
> 
> 
> On Wed, 1 Nov 2023, Mario Marietto wrote:
>> I'm aware of the presence of that post. I'm working on the same
>> project with the guy who explained the problem. Unfortunately,the
>> solution proposed does not work well. Xen is working,but the screen is
>> still black.
>> 
>> On Wed, Nov 1, 2023 at 12:04 AM Stefano Stabellini
>>  wrote:
>> >
>> > Hi Mario,
>> >
>> > I am adding xen-devel and a couple of other Xen maintainers that might
>> > know how to help make progress on this issues.
>> >
>> > Replies inline below.
>> >
>> >
>> > On Tue, 31 Oct 2023, Mario Marietto wrote:
>> > > Hello,
>> > >
>> > > We are a team of linux enthusiasts who are trying to boot Xen on a
>> > > Samsung XE303C12 Chromebook aka "snow"
>> > > following the suggestions in the slide show presentation here:
>> > >
>> > > https://www.slideshare.net/xen_com_mgr/xpds16-porting-xen-on-arm-to-a-new-soc-julien-grall-arm
>> > >
>> > > This device uses an exynos5250 SOC dual core 1.7 GHz with 2 MB RAM, it is
>> > > a Samsung armv7 chip with 

Re: [PATCH v3 2/6] hw/isa/piix3: Reuse piix3_realize() in piix3_xen_realize()

[Chuck Zmudzinski]

On 9/19/2023 4:02 PM, Bernhard Beschow wrote:
> 
> 
> Am 3. April 2023 12:27:14 UTC schrieb Jason Andryuk :
>>On Mon, Apr 3, 2023 at 5:33 AM Anthony PERARD  
>>wrote:
>>>
>>> On Sat, Apr 01, 2023 at 10:36:45PM +, Bernhard Beschow wrote:
>>> >
>>> >
>>> > Am 30. März 2023 13:00:25 UTC schrieb Anthony PERARD 
>>> > :
>>> > >On Sun, Mar 12, 2023 at 01:02:17PM +0100, Bernhard Beschow wrote:
>>> > >> This is a preparational patch for the next one to make the following
>>> > >> more obvious:
>>> > >>
>>> > >> First, pci_bus_irqs() is now called twice in case of Xen where the
>>> > >> second call overrides the pci_set_irq_fn with the Xen variant.
>>> > >
>>> > >pci_bus_irqs() does allocates pci_bus->irq_count, so the second call in
>>> > >piix3_xen_realize() will leak `pci_bus->irq_count`. Could you look if
>>> > >pci_bus_irqs_cleanup() can be called before the second pci_bus_irqs()
>>> > >call, or maybe some other way to avoid the leak?
>>> >
>>> > Thanks for catching this! I'll post a v4.
>>> >
>>> > I think the most fool-proof way to fix this is to free irq_count just 
>>> > before the assignment. pci_bus_irqs_cleanup() would then have to NULL the 
>>> > attribute such that pci_bus_irqs() can be called afterwards.
>>> >
>>> > BTW: I tried running qemu-system-x86_64 with PIIX4 rather than PIIX3 as 
>>> > Xen guest with my pc-piix4 branch without success. This branch 
>>> > essentially just provides slightly different PCI IDs for PIIX. Does xl or 
>>> > something else in Xen check these? If not then this means I'm still 
>>> > missing something. Under KVM this branch works just fine. Any idea?
>>>
>>> Maybe the ACPI tables provided by libxl needs to be updated.
>>> Or maybe something in the firmware (SeaBIOS or OVMF/OvmfXen) check the
>>> id (I know that the PCI id of the root bus is checked, but I don't know
>>> if that's the one that's been changed).
>>
>>Xen also has hvmloader, which runs before SeaBIOS/OVMF.  Looking at
>>tools/firmware/hvmloader/pci.c, it has
>>ASSERT((devfn != PCI_ISA_DEVFN) ||
>>   ((vendor_id == 0x8086) && (device_id == 0x7000)));
>>
>>From QEMU, it looks like 0x7000 is PCI_DEVICE_ID_INTEL_82371SB_0, but
>>PIIX4 uses 0x7110 (PCI_DEVICE_ID_INTEL_82371AB_0).  Maybe try removing
>>that check?
> 
> I was finally able to build Xen successfully (without my distribution 
> providing too recent dependencies that prevent compilation). With 0x7110 
> added in the line above I could indeed run a Xen guest with PIIX4. Yay!
> 
> Now I just need to respin my PIIX consolidation series...

Welcome to the world of running guests on Xen! I am the one who tested your 
earlier patches with Xen guests, and I just wanted to say thanks for keeping me 
in the loop. Please Cc me when you post your respin of the PIIX consolidation 
series since I would like to also test it in my Xen environment. I understand I 
will also need to patch hvmloader.c on the Xen side to set the correct device 
id.

Kind regards,

Chuck

> 
> Best regards,
> Bernhard
> 
>>
>>Regards,
>>Jason

Re: xl dmesg buffer too small for Xen 4.18?

[Chuck Zmudzinski]

On 9/19/2023 12:00 PM, Jan Beulich wrote:
> On 19.09.2023 17:56, Julien Grall wrote:
>> On 19/09/2023 16:09, Chuck Zmudzinski wrote:
>>> On 9/19/2023 8:10 AM, Julien Grall wrote:
>>>> On 19/09/2023 08:02, Roger Pau Monné wrote:
>>>>> On Mon, Sep 18, 2023 at 07:49:26PM +0100, Julien Grall wrote:
>>>>>> (+Roger and moving to xen-devel)
>>>>>> On 18/09/2023 19:17, Chuck Zmudzinski wrote:
>>>>>>> On 9/18/2023 9:00 AM, Chuck Zmudzinski wrote:
>>>>>>>> I tested Xen 4.18~rc0 on Alma Linux 9 and my first tests indicate it 
>>>>>>>> works fine for starting the guests I manage but I notice that 
>>>>>>>> immediately after boot and with only dom0 running on the system, I get:
>>>>>>>>
>>>>>>>> [user@Malmalinux ~]$ sudo xl dmesg
>>>>>>>> 00bee72000-0bee72fff type=7 attr=000f
>>>>>>>> (XEN)  0bee73000-0bef49fff type=4 attr=000f
>>>>>>>> (XEN)  0bef4a000-0bef4bfff type=7 attr=000f
>>>>>>>> (XEN)  0bef4c000-0befbafff type=4 attr=000f
>>>>>>>> (XEN)  0befbb000-0befbbfff type=7 attr=000f
>>>>>>>> ...
>>>>>>>>
>>>>>>>> I have noticed the buffer fills up quickly on earlier Xen versions, 
>>>>>>>> but never have I seen it fill up during boot and with only dom0 
>>>>>>>> running.
>>>>>>>>
>>>>>>>> Can increasing the buffer fix this? How would one do that?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>
>>>>>>> I see the setting is the command line option conring_size:
>>>>>>>
>>>>>>> https://xenbits.xen.org/docs/unstable/misc/xen-command-line.html#conring_size
>>>>>>>
>>>>>>> The default is 16k, I tried 48k and that was big enough to capture all 
>>>>>>> the messages at boot for 4.18 rc0. This is probably not an issue if the 
>>>>>>> release candidate is being more verbose than the actual release will 
>>>>>>> be. But if the release is still this verbose, maybe the default of 16k 
>>>>>>> should be increased.
>>>>>>
>>>>>> Thanks for the report. This remind me the series [1] from Roger which 
>>>>>> tries
>>>>>> to increase the default size to 32K. @Roger, I am wondering if we should
>>>>>> revive it?
>>>>>
>>>>> I think the relevant patch (2/2) will still apply as-is, it's just a
>>>>> Kconfig one line change.  I'm however thinking it might be better to
>>>>> bump it even further, to 128K.  From a system point of view it's still
>>>>> a very small amount of memory.
>>>>
>>>> I don't have a strong opinion about 128K vs 32K.
>>>
>>> I am sure 32k will be big enough on my system, and based on Jan's comment
>>> about release builds being less verbose, the current default of 16k may
>>> still work on my system once the release is out. 
>> 
>> I think it is quite (actually more) important to capture all the logs 
>> even in non-release build. So it would makes sense to increase the 
>> buffer to 32KB.
>> 
>> An alternative option would be to have a different limit for debug and 
>> production build. Not sure what the others thinks.
> 
> I would certainly like a two-way default better than the uniform bumping
> to 128k.
> 
> Jan

I think for release builds (production) minimize it as much as possible without
losing the less verbose logs. For debug builds, make it as big as needed for
the convenience of developers and the more verbose logs.

Re: xl dmesg buffer too small for Xen 4.18?

[Chuck Zmudzinski]

On 9/19/2023 8:10 AM, Julien Grall wrote:
> Hi Roger,
> 
> On 19/09/2023 08:02, Roger Pau Monné wrote:
>> On Mon, Sep 18, 2023 at 07:49:26PM +0100, Julien Grall wrote:
>>> (+Roger and moving to xen-devel)
>>>
>>> Hi,
>>>
>>> On 18/09/2023 19:17, Chuck Zmudzinski wrote:
>>>> On 9/18/2023 9:00 AM, Chuck Zmudzinski wrote:
>>>>> Hello,
>>>>>
>>>>> I tested Xen 4.18~rc0 on Alma Linux 9 and my first tests indicate it 
>>>>> works fine for starting the guests I manage but I notice that immediately 
>>>>> after boot and with only dom0 running on the system, I get:
>>>>>
>>>>> [user@Malmalinux ~]$ sudo xl dmesg
>>>>> 00bee72000-0bee72fff type=7 attr=000f
>>>>> (XEN)  0bee73000-0bef49fff type=4 attr=000f
>>>>> (XEN)  0bef4a000-0bef4bfff type=7 attr=000f
>>>>> (XEN)  0bef4c000-0befbafff type=4 attr=000f
>>>>> (XEN)  0befbb000-0befbbfff type=7 attr=000f
>>>>> ...
>>>>>
>>>>> I have noticed the buffer fills up quickly on earlier Xen versions, but 
>>>>> never have I seen it fill up during boot and with only dom0 running.
>>>>>
>>>>> Can increasing the buffer fix this? How would one do that?
>>>>>
>>>>> Thanks
>>>>>
>>>>
>>>> I see the setting is the command line option conring_size:
>>>>
>>>> https://xenbits.xen.org/docs/unstable/misc/xen-command-line.html#conring_size
>>>>
>>>> The default is 16k, I tried 48k and that was big enough to capture all the 
>>>> messages at boot for 4.18 rc0. This is probably not an issue if the 
>>>> release candidate is being more verbose than the actual release will be. 
>>>> But if the release is still this verbose, maybe the default of 16k should 
>>>> be increased.
>>>
>>> Thanks for the report. This remind me the series [1] from Roger which tries
>>> to increase the default size to 32K. @Roger, I am wondering if we should
>>> revive it?
>> 
>> I think the relevant patch (2/2) will still apply as-is, it's just a
>> Kconfig one line change.  I'm however thinking it might be better to
>> bump it even further, to 128K.  From a system point of view it's still
>> a very small amount of memory.
> 
> I don't have a strong opinion about 128K vs 32K.

I am sure 32k will be big enough on my system, and based on Jan's comment
about release builds being less verbose, the current default of 16k may
still work on my system once the release is out. I am willing to defer to
whatever the developers decide according to the ordinary process for deciding
such questions.

> 
>> 
>> I'm happy to repost with an increased buffer size, but only if there's
>> someone willing to Ack it, otherwise it's not worth spending time on
>> it.
> 
> Sorry that patch fell through the cracks. I would be happy to ack the patch.
> 
> Cheers,
> 

Re: xl dmesg buffer too small for Xen 4.18?

[Chuck Zmudzinski]

On 9/18/2023 2:49 PM, Julien Grall wrote:
> (+Roger and moving to xen-devel)
> 
> Hi,
> 
> On 18/09/2023 19:17, Chuck Zmudzinski wrote:
>> On 9/18/2023 9:00 AM, Chuck Zmudzinski wrote:
>>> Hello,
>>>
>>> I tested Xen 4.18~rc0 on Alma Linux 9 and my first tests indicate it works 
>>> fine for starting the guests I manage but I notice that immediately after 
>>> boot and with only dom0 running on the system, I get:
>>>
>>> [user@Malmalinux ~]$ sudo xl dmesg
>>> 00bee72000-0bee72fff type=7 attr=000f
>>> (XEN)  0bee73000-0bef49fff type=4 attr=000f
>>> (XEN)  0bef4a000-0bef4bfff type=7 attr=000f
>>> (XEN)  0bef4c000-0befbafff type=4 attr=000f
>>> (XEN)  0befbb000-0befbbfff type=7 attr=000f
>>> ...
>>>
>>> I have noticed the buffer fills up quickly on earlier Xen versions, but 
>>> never have I seen it fill up during boot and with only dom0 running.
>>>
>>> Can increasing the buffer fix this? How would one do that?
>>>
>>> Thanks
>>>
>> 
>> I see the setting is the command line option conring_size:
>> 
>> https://xenbits.xen.org/docs/unstable/misc/xen-command-line.html#conring_size
>> 
>> The default is 16k, I tried 48k and that was big enough to capture all the 
>> messages at boot for 4.18 rc0. This is probably not an issue if the release 
>> candidate is being more verbose than the actual release will be. But if the 
>> release is still this verbose, maybe the default of 16k should be increased.
> 
> Thanks for the report. This remind me the series [1] from Roger which 
> tries to increase the default size to 32K. @Roger, I am wondering if we 
> should revive it?
> 
> Cheers,
> 
> [1] 
> https://lore.kernel.org/xen-devel/20220630082330.82706-1-roger@citrix.com/
> 

I just tested with 24k, and that is also big enough. So 32k would also be good. 
But the default of 16k appears to be too small for Xen 4.18 rc0 on my machine.

Re: Community Manager update - August 2023

[Chuck Zmudzinski]

On 8/18/2023 6:55 AM, Kelly Choi wrote:
> Hi everyone! :) 
> 
> I hope you're all well. 
> 
> If we haven't met before, I'd like to introduce myself. I'm Kelly, the 
> Community Manager for The Xen Project. My role is to support everyone and 
> make sure the project is healthy and thriving. 
> 
> *The latest update below requires your attention:*
> *
> *
> 
>   * *We will be moving IRC channels fully to Matrix in September 2023. Once 
> the channels have been created, further information will be shared. *
>   * *New Mission Statement, goals, and purpose is attached to this email and 
> will be shared publicly.*
> 
> *Should anyone have any concerns or feedback, please let me know*
> 
> Many thanks,
> Kelly Choi
> 
> Open Source Community Manager, XenServer
> Cloud Software Group

This looks good, but I thought IBM rebranded Softlayer as IBM Cloud several 
years ago. Maybe IBM Softlayer should be changed to IBM Cloud? Thanks.

Chuck

Re: [PATCH] xen/pt: fix igd passthrough for pc machine with xen accelerator

[Chuck Zmudzinski]

On 5/17/2023 2:39 AM, Michael Tokarev wrote:
> 08.02.2023 05:03, Chuck Zmudzinski wrote:
> > Commit 998250e97661 ("xen, gfx passthrough: register host bridge specific
> > to passthrough") uses the igd-passthrough-i440FX pci host device with
> > the xenfv machine type and igd-passthru=on, but using it for the pc
> > machine type, xen accelerator, and igd-passtru=on was omitted from that
> > commit.
> > 
> > The igd-passthru-i440FX pci host device is also needed for guests
> > configured with the pc machine type, the xen accelerator, and
> > igd-passthru=on. Specifically, tests show that not using the igd-specific
> > pci host device with the Intel igd passed through to the guest results
> > in slower startup performance and reduced resolution of the display
> > during startup. This patch fixes this issue.
> > 
> > To simplify the logic that is needed to support both the --enable-xen
> > and the --disable-xen configure options, introduce the boolean symbol
> > pc_xen_igd_gfx_pt_enabled() whose value is set appropriately in the
> > sysemu/xen.h header file as the test to determine whether or not
> > to use the igd-passthrough-i440FX pci host device instead of the
> > normal i440FX pci host device.
> > 
> > Fixes: 998250e97661 ("xen, gfx passthrough: register host bridge specific 
> > to passthrough")
> > Signed-off-by: Chuck Zmudzinski 
>
> Has this change been forgotten?  Is it not needed anymore?

Short answer:

After 4f67543b ("xen/pt: reserve PCI slot 2 for Intel igd-passthru ") was
applied, I was inclined to think this change is not needed anymore, but
it would not hurt to add this change also, and now I think it might be
more correct to also add this change.

Longer explanation:

I strongly desired that at least one of the patches I proposed to improve
support for Intel IGD passthrough with xen be committed. Since
4f67543b ("xen/pt: reserve PCI slot 2 for Intel igd-passthru ") that fixed
Intel IGD passthrough for the xenfv machine type has been committed,
I reasoned that there is not such a great need to also fix Intel IGD
passthrough for the pc machine type with xen so I did not push hard for
this patch to also be applied.

My requirement was that either the xenfv machine be fixed or the pc
machine be fixed. I did not think it was necessary to fix them both, and
4f67543b ("xen/pt: reserve PCI slot 2 for Intel igd-passthru ") fixed the
xenfv machine. But this patch provides the additional fix for the pc machine,
a fix that is distinct from the fix that has already been committed for the
xenfv machine, and it probably should also be applied so pc and xenfv
machines will work equally well with Intel IGD passthrough.

In other words, it is good to fix at least one of the two broken machines 
configured
for Intel IGD passthrough and xen, it is better to fix them both. We already 
fixed
one of them with 4f67543b ("xen/pt: reserve PCI slot 2 for Intel igd-passthru 
"),
this patch would fix the other one.

If you want to add this change also, let's make sure recent changes to the
xen header files do not require the patch to be rebased before committing
it.

Chuck

[PATCH v4 2/2] pci: introduce slot_reserved_auto_mask and slot_reserved_manual_mask

[Chuck Zmudzinski]

Commit 4f67543bb8c5 ("xen/pt: reserve PCI slot 2 for Intel igd-passthru")
uses slot_reserved_mask to reserve slot 2 for the Intel igd for the
xenfv machine when the guest is configured for igd-passthru.

Prior to that commit, a single 32-bit mask was sufficient to meet the
needs of the only machine that used the 32-bit slot_reserved_mask, the
sun4u machine. However, the requirements of the xenfv machine with
igd-passthru is somewhat different from the requirements of the sun4u
machine.

First, the sun4u machine reserves slots in such a way that no device
can be assigned to a reserved slot, but the xenfv machine needs to
reserve a single slot that is reserved for a particular device, the
Intel igd. The necessary logic to ensure that the reserved slot is used
by the Intel igd was recently added by the aforementioned commit.

Second, it is useful to limit slot reservation in the case of the xenfv
machine with the Intel igd to devices configured for automatic slot
assignment so an administrator can assign a device to the reserved slot
by manually specifying the reserved slot address on the command line,
but the sun4u machine requires slot reservation for all devices, whether
or not the device is configured for automatic slot assignment or
configured manually by specifying a slot address on the command line. In
other words, for the sun4u machine, the required behavior is that an
attempt to assign a reserved slot to a device must always result in an
error, but it is useful to allow manual assignment of a reserved slot to
succeed for the xenfv machine with the Intel igd.

The necessary logic to implement the desired behavior of reserving one
or more slots only for the case of automatic slot allocation has not yet
been implemented, and that is the purpose of this patch.

The implementation is simple: the xenfv machine only sets
slot_reserved_auto_mask and the sun4u machine sets both
slot_reserved_manual_mask and slot_reserved_auto_mask. A single
"set" accessor function allows xenfv and sun4u machines to set the
value of the two masks appropriately for each use case.

Since the xenfv machine needs to implement additional logic to detect
the Intel igd and clear the bit in the mask to allow a particular device
to use the reserved slot, there is a need for a "get" and "clear" accessor
function for slot_reserved_auto_mask, but these accessor functions are
not needed for slot_reserved_manual_mask because the sun4u machine has
no need to get the value of the mask or clear bits in the mask.

Link: 
https://lore.kernel.org/qemu-devel/20230106064838-mutt-send-email-...@kernel.org/
Signed-off-by: Chuck Zmudzinski 
---
Changelog

v4: Style corrections, 3 lines in v3 were too long:

1. In hw/pci/pci.c:

void pci_bus_set_slot_reserved_masks(PCIBus *bus, uint32_t auto_mask, uint32_t 
manual_mask)

is changed to

void pci_bus_set_slot_reserved_masks(PCIBus *bus, uint32_t auto_mask,
 uint32_t manual_mask)
...
2. In hw/xen/xen_pt.c:

if (!(pci_bus_get_slot_reserved_auto_mask(pci_bus) & 
XEN_PCI_IGD_SLOT_MASK)) {

is changed to

if (!(pci_bus_get_slot_reserved_auto_mask(pci_bus) &
  XEN_PCI_IGD_SLOT_MASK)) {
...
3. In include/hw/pci/pci.h:

void pci_bus_set_slot_reserved_masks(PCIBus *bus, uint32_t auto_mask, uint32_t 
manual_mask);

is changed to

void pci_bus_set_slot_reserved_masks(PCIBus *, uint32_t, uint32_t);

No other changes since v3

v3: Change Subject of patch from
"pci: allow slot_reserved_mask to be ignored with manual slot assignment" To
"pci: introduce slot_reserved_auto_mask and slot_reserved_manual_mask"

Substantially reword the commit message to clearly explain the reasons
this patch is needed

Apply changes in response to comments on v2:
   - slot_reserved_mask -> slot_reserved_auto_mask
   - remove enforce_slot_reserved_mask_manual
   - remove pci_bus_ignore_slot_reserved_mask_manual
   - add slot_reserved_manual_mask
   - pci_bus_devfn_reserved -> pci_bus_devfn_reserved_auto
   - change code in pci_bus_devfn_reserved_manual appropriately
   - pci_bus_set_slot_reserved_mask -> pci_bus_set_slot_reserved_masks
   - use renamed "set" function to set value of both masks for sun4u 
and xenfv cases
   - pci_bus_get_slot_reserved_mask -> pci_bus_get_slot_reserved_auto_mask
   - pci_bus_clear_slot_reserved_mask -> 
pci_bus_clear_slot_reserved_auto_mask

v2: Change Subject of patch from
"pci: add enforce_slot_reserved_mask_manual property" To
"pci: allow slot_reserved_mask to be ignored with manual slot assignment"

Add pci_bus_ignore_slot_reserved_mask_manual function

Call pci_bus_ignore_slot_reserved_mask_manual at appropriate place
in hw/pci-host/i440fx.c

 hw/pci/pci.c | 30 +++---
 hw/sparc64/sun4u.c   |  6 +

[PATCH v4 0/2] pci: slot_reserved_mask improvements

[Chuck Zmudzinski]

This patch series consists of two patches. The first provides accessor
functions in pci.h to avoid direct access of slot_reserved_mask
according to the comment at the top of include/hw/pci/pci_bus.h. No
functional change is intended with this patch.

The second patch replaces slot_reserved_mask with two new masks,
slot_reserved_auto_mask and slot_reserved_manual_mask so the current
behavior of reserving slot 2 for the Intel IGD for the xenfv machine
will be ignored if an administrator manually configures a device to use
the reserved slot.

The current behavior of always reserving slots in the sun4u machine is
preserved by this patch series; the patch series only changes how
slot_reserved_mask works in the xenfv machine. Although the patch
series can affect xenfv machines configured for igd-passthru if an
administrator assigns some of the pci slot addresses manually, it
does not affect the libxl default configuration for igd-passthru because
libxl uses automatic slot assignment by default.

Testing:
   - Tested xenfv/igd with both manual and auto slot allocation - behaves as 
expected
   - Verified that qemu-system-sparc64 still compiles with the patches to 
sun4u.c
   - xen4u machine not tested -- Mark, can you do this?

Link: 
https://lore.kernel.org/qemu-devel/20230106064838-mutt-send-email-...@kernel.org/

Chuck Zmudzinski (2):
  pci: avoid accessing slot_reserved_mask directly outside of pci.c
  pci: introduce slot_reserved_auto_mask and slot_reserved_manual_mask

Changelog

v4: I forgot to check the patches in v3 for style corrections (sorry about
that), and the second patch had three lines that were too long. Other
than correcting the style problems, no changes since v3.

v3: Re-work second patch in response to comments/discussion of v2

v2: Add first patch and cover letter to make this a 2-patch series
Make changes to the second patch (see second patch for changelog)

 hw/pci/pci.c | 33 -
 hw/sparc64/sun4u.c   |  7 +++
 hw/xen/xen_pt.c  |  8 
 include/hw/pci/pci.h |  3 +++
 include/hw/pci/pci_bus.h |  3 ++-
 5 files changed, 40 insertions(+), 14 deletions(-)

-- 
2.39.2

[PATCH v4 1/2] pci: avoid accessing slot_reserved_mask directly outside of pci.c

[Chuck Zmudzinski]

This patch provides accessor functions as replacements for direct
access to slot_reserved_mask according to the comment at the top
of include/hw/pci/pci_bus.h which advises that data structures for
PCIBus should not be directly accessed but instead be accessed using
accessor functions in pci.h.

Three accessor functions can conveniently replace all direct accesses
of slot_reserved_mask. With this patch, the new accessor functions are
used in hw/sparc64/sun4u.c and hw/xen/xen_pt.c and pci_bus.h is removed
from the included header files of the same two files.

No functional change intended.

Signed-off-by: Chuck Zmudzinski 
---
Changelog

v4: This patch is unchanged since v2.

v3: This patch is unchanged since v2.

v2: This is the first version of this patch, it did not exist in v1.


 hw/pci/pci.c | 15 +++
 hw/sparc64/sun4u.c   |  7 +++
 hw/xen/xen_pt.c  |  7 +++
 include/hw/pci/pci.h |  3 +++
 4 files changed, 24 insertions(+), 8 deletions(-)

diff --git a/hw/pci/pci.c b/hw/pci/pci.c
index def5000e7b..8a87ccc8b0 100644
--- a/hw/pci/pci.c
+++ b/hw/pci/pci.c
@@ -1116,6 +1116,21 @@ static bool pci_bus_devfn_reserved(PCIBus *bus, int 
devfn)
 return bus->slot_reserved_mask & (1UL << PCI_SLOT(devfn));
 }
 
+uint32_t pci_bus_get_slot_reserved_mask(PCIBus *bus)
+{
+return bus->slot_reserved_mask;
+}
+
+void pci_bus_set_slot_reserved_mask(PCIBus *bus, uint32_t mask)
+{
+bus->slot_reserved_mask |= mask;
+}
+
+void pci_bus_clear_slot_reserved_mask(PCIBus *bus, uint32_t mask)
+{
+bus->slot_reserved_mask &= ~mask;
+}
+
 /* -1 for devfn means auto assign */
 static PCIDevice *do_pci_register_device(PCIDevice *pci_dev,
  const char *name, int devfn,
diff --git a/hw/sparc64/sun4u.c b/hw/sparc64/sun4u.c
index a25e951f9d..eae7589462 100644
--- a/hw/sparc64/sun4u.c
+++ b/hw/sparc64/sun4u.c
@@ -31,7 +31,6 @@
 #include "hw/irq.h"
 #include "hw/pci/pci.h"
 #include "hw/pci/pci_bridge.h"
-#include "hw/pci/pci_bus.h"
 #include "hw/pci/pci_host.h"
 #include "hw/qdev-properties.h"
 #include "hw/pci-host/sabre.h"
@@ -608,9 +607,9 @@ static void sun4uv_init(MemoryRegion *address_space_mem,
 /* Only in-built Simba APBs can exist on the root bus, slot 0 on busA is
reserved (leaving no slots free after on-board devices) however slots
0-3 are free on busB */
-pci_bus->slot_reserved_mask = 0xfffc;
-pci_busA->slot_reserved_mask = 0xfff1;
-pci_busB->slot_reserved_mask = 0xfff0;
+pci_bus_set_slot_reserved_mask(pci_bus, 0xfffc);
+pci_bus_set_slot_reserved_mask(pci_busA, 0xfff1);
+pci_bus_set_slot_reserved_mask(pci_busB, 0xfff0);
 
 ebus = pci_new_multifunction(PCI_DEVFN(1, 0), true, TYPE_EBUS);
 qdev_prop_set_uint64(DEVICE(ebus), "console-serial-base",
diff --git a/hw/xen/xen_pt.c b/hw/xen/xen_pt.c
index 2d33d178ad..a540149639 100644
--- a/hw/xen/xen_pt.c
+++ b/hw/xen/xen_pt.c
@@ -57,7 +57,6 @@
 #include 
 
 #include "hw/pci/pci.h"
-#include "hw/pci/pci_bus.h"
 #include "hw/qdev-properties.h"
 #include "hw/qdev-properties-system.h"
 #include "xen_pt.h"
@@ -951,7 +950,7 @@ void xen_igd_reserve_slot(PCIBus *pci_bus)
 }
 
 XEN_PT_LOG(0, "Reserving PCI slot 2 for IGD\n");
-pci_bus->slot_reserved_mask |= XEN_PCI_IGD_SLOT_MASK;
+pci_bus_set_slot_reserved_mask(pci_bus, XEN_PCI_IGD_SLOT_MASK);
 }
 
 static void xen_igd_clear_slot(DeviceState *qdev, Error **errp)
@@ -971,7 +970,7 @@ static void xen_igd_clear_slot(DeviceState *qdev, Error 
**errp)
 return;
 }
 
-if (!(pci_bus->slot_reserved_mask & XEN_PCI_IGD_SLOT_MASK)) {
+if (!(pci_bus_get_slot_reserved_mask(pci_bus) & XEN_PCI_IGD_SLOT_MASK)) {
 xpdc->pci_qdev_realize(qdev, errp);
 return;
 }
@@ -982,7 +981,7 @@ static void xen_igd_clear_slot(DeviceState *qdev, Error 
**errp)
 s->real_device.dev == XEN_PCI_IGD_DEV &&
 s->real_device.func == XEN_PCI_IGD_FN &&
 s->real_device.vendor_id == PCI_VENDOR_ID_INTEL) {
-pci_bus->slot_reserved_mask &= ~XEN_PCI_IGD_SLOT_MASK;
+pci_bus_clear_slot_reserved_mask(pci_bus, XEN_PCI_IGD_SLOT_MASK);
 XEN_PT_LOG(pci_dev, "Intel IGD found, using slot 2\n");
 }
 xpdc->pci_qdev_realize(qdev, errp);
diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h
index d5a40cd058..935b4b91b4 100644
--- a/include/hw/pci/pci.h
+++ b/include/hw/pci/pci.h
@@ -287,6 +287,9 @@ void pci_bus_irqs(PCIBus *bus, pci_set_irq_fn set_irq,
 void pci_bus_map_irqs(PCIBus *bus, pci_map_irq_fn map_irq);
 void pci_bus_irqs_cleanup(PCIBus *bus);
 int pci_bus_get_irq_level(PCIBus *bus, int irq_num);
+uint32_t pci_bus_get_slot_reserved_mask(PCIBus *bus);
+void pci_bus_set_slot_reserved_mask(PCIBus *bus, uint32_t mask);
+void pci_bus_clear_slot_reserved_mask(PCIBus *bus, uint32_t mask);
 /* 0 <= pin <= 3 0 = INTA, 1 = INTB, 2 = INTC, 3 = INTD */
 static inline int pci_swizzle(int slot, int pin)
 {
-- 
2.39.2

[PATCH v3 1/2] pci: avoid accessing slot_reserved_mask directly outside of pci.c

[Chuck Zmudzinski]

This patch provides accessor functions as replacements for direct
access to slot_reserved_mask according to the comment at the top
of include/hw/pci/pci_bus.h which advises that data structures for
PCIBus should not be directly accessed but instead be accessed using
accessor functions in pci.h.

Three accessor functions can conveniently replace all direct accesses
of slot_reserved_mask. With this patch, the new accessor functions are
used in hw/sparc64/sun4u.c and hw/xen/xen_pt.c and pci_bus.h is removed
from the included header files of the same two files.

No functional change intended.

Signed-off-by: Chuck Zmudzinski 
---
Changelog

v3: This patch is unchanged since v2.

v2: This is the first version of this patch, it did not exist in v1.

 hw/pci/pci.c | 15 +++
 hw/sparc64/sun4u.c   |  7 +++
 hw/xen/xen_pt.c  |  7 +++
 include/hw/pci/pci.h |  3 +++
 4 files changed, 24 insertions(+), 8 deletions(-)

diff --git a/hw/pci/pci.c b/hw/pci/pci.c
index def5000e7b..8a87ccc8b0 100644
--- a/hw/pci/pci.c
+++ b/hw/pci/pci.c
@@ -1116,6 +1116,21 @@ static bool pci_bus_devfn_reserved(PCIBus *bus, int 
devfn)
 return bus->slot_reserved_mask & (1UL << PCI_SLOT(devfn));
 }
 
+uint32_t pci_bus_get_slot_reserved_mask(PCIBus *bus)
+{
+return bus->slot_reserved_mask;
+}
+
+void pci_bus_set_slot_reserved_mask(PCIBus *bus, uint32_t mask)
+{
+bus->slot_reserved_mask |= mask;
+}
+
+void pci_bus_clear_slot_reserved_mask(PCIBus *bus, uint32_t mask)
+{
+bus->slot_reserved_mask &= ~mask;
+}
+
 /* -1 for devfn means auto assign */
 static PCIDevice *do_pci_register_device(PCIDevice *pci_dev,
  const char *name, int devfn,
diff --git a/hw/sparc64/sun4u.c b/hw/sparc64/sun4u.c
index a25e951f9d..eae7589462 100644
--- a/hw/sparc64/sun4u.c
+++ b/hw/sparc64/sun4u.c
@@ -31,7 +31,6 @@
 #include "hw/irq.h"
 #include "hw/pci/pci.h"
 #include "hw/pci/pci_bridge.h"
-#include "hw/pci/pci_bus.h"
 #include "hw/pci/pci_host.h"
 #include "hw/qdev-properties.h"
 #include "hw/pci-host/sabre.h"
@@ -608,9 +607,9 @@ static void sun4uv_init(MemoryRegion *address_space_mem,
 /* Only in-built Simba APBs can exist on the root bus, slot 0 on busA is
reserved (leaving no slots free after on-board devices) however slots
0-3 are free on busB */
-pci_bus->slot_reserved_mask = 0xfffc;
-pci_busA->slot_reserved_mask = 0xfff1;
-pci_busB->slot_reserved_mask = 0xfff0;
+pci_bus_set_slot_reserved_mask(pci_bus, 0xfffc);
+pci_bus_set_slot_reserved_mask(pci_busA, 0xfff1);
+pci_bus_set_slot_reserved_mask(pci_busB, 0xfff0);
 
 ebus = pci_new_multifunction(PCI_DEVFN(1, 0), true, TYPE_EBUS);
 qdev_prop_set_uint64(DEVICE(ebus), "console-serial-base",
diff --git a/hw/xen/xen_pt.c b/hw/xen/xen_pt.c
index 2d33d178ad..a540149639 100644
--- a/hw/xen/xen_pt.c
+++ b/hw/xen/xen_pt.c
@@ -57,7 +57,6 @@
 #include 
 
 #include "hw/pci/pci.h"
-#include "hw/pci/pci_bus.h"
 #include "hw/qdev-properties.h"
 #include "hw/qdev-properties-system.h"
 #include "xen_pt.h"
@@ -951,7 +950,7 @@ void xen_igd_reserve_slot(PCIBus *pci_bus)
 }
 
 XEN_PT_LOG(0, "Reserving PCI slot 2 for IGD\n");
-pci_bus->slot_reserved_mask |= XEN_PCI_IGD_SLOT_MASK;
+pci_bus_set_slot_reserved_mask(pci_bus, XEN_PCI_IGD_SLOT_MASK);
 }
 
 static void xen_igd_clear_slot(DeviceState *qdev, Error **errp)
@@ -971,7 +970,7 @@ static void xen_igd_clear_slot(DeviceState *qdev, Error 
**errp)
 return;
 }
 
-if (!(pci_bus->slot_reserved_mask & XEN_PCI_IGD_SLOT_MASK)) {
+if (!(pci_bus_get_slot_reserved_mask(pci_bus) & XEN_PCI_IGD_SLOT_MASK)) {
 xpdc->pci_qdev_realize(qdev, errp);
 return;
 }
@@ -982,7 +981,7 @@ static void xen_igd_clear_slot(DeviceState *qdev, Error 
**errp)
 s->real_device.dev == XEN_PCI_IGD_DEV &&
 s->real_device.func == XEN_PCI_IGD_FN &&
 s->real_device.vendor_id == PCI_VENDOR_ID_INTEL) {
-pci_bus->slot_reserved_mask &= ~XEN_PCI_IGD_SLOT_MASK;
+pci_bus_clear_slot_reserved_mask(pci_bus, XEN_PCI_IGD_SLOT_MASK);
 XEN_PT_LOG(pci_dev, "Intel IGD found, using slot 2\n");
 }
 xpdc->pci_qdev_realize(qdev, errp);
diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h
index d5a40cd058..935b4b91b4 100644
--- a/include/hw/pci/pci.h
+++ b/include/hw/pci/pci.h
@@ -287,6 +287,9 @@ void pci_bus_irqs(PCIBus *bus, pci_set_irq_fn set_irq,
 void pci_bus_map_irqs(PCIBus *bus, pci_map_irq_fn map_irq);
 void pci_bus_irqs_cleanup(PCIBus *bus);
 int pci_bus_get_irq_level(PCIBus *bus, int irq_num);
+uint32_t pci_bus_get_slot_reserved_mask(PCIBus *bus);
+void pci_bus_set_slot_reserved_mask(PCIBus *bus, uint32_t mask);
+void pci_bus_clear_slot_reserved_mask(PCIBus *bus, uint32_t mask);
 /* 0 <= pin <= 3 0 = INTA, 1 = INTB, 2 = INTC, 3 = INTD */
 static inline int pci_swizzle(int slot, int pin)
 {
-- 
2.39.2

[PATCH v3 0/2] pci: slot_reserved_mask improvements

[Chuck Zmudzinski]

This patch series consists of two patches. The first provides accessor
functions in pci.h to avoid direct access of slot_reserved_mask
according to the comment at the top of include/hw/pci/pci_bus.h. No
functional change is intended with this patch.

The second patch replaces slot_reserved_mask with two new masks,
slot_reserved_auto_mask and slot_reserved_manual_mask so the current
behavior of reserving slot 2 for the Intel IGD for the xenfv machine
will be ignored if an administrator manually configures a device to use
the reserved slot.

The current behavior of always reserving slots in the sun4u machine is
preserved by this patch series; the patch series only changes how
slot_reserved_mask works in the xenfv machine. Although the patch
series can affect xenfv machines configured for igd-passthru if an
administrator assigns some of the pci slot addresses manually, it
does not affect the libxl default configuration for igd-passthru because
libxl uses automatic slot assignment by default.

Testing:
   - Tested xenfv/igd with both manual and auto slot allocation - behaves as 
expected
   - Verified that qemu-system-sparc64 still compiles with the patches to 
sun4u.c
   - xen4u machine not tested -- Mark, can you do this?

Link: 
https://lore.kernel.org/qemu-devel/20230106064838-mutt-send-email-...@kernel.org/

Chuck Zmudzinski (2):
  pci: avoid accessing slot_reserved_mask directly outside of pci.c
  pci: introduce slot_reserved_auto_mask and slot_reserved_manual_mask

Changelog

v3: Re-work second patch in response to comments/discussion of v2

v2: Add first patch and cover letter to make this a 2-patch series
Make changes to the second patch (see second patch for changelog)

 hw/pci/pci.c | 32 +++-
 hw/sparc64/sun4u.c   |  7 +++
 hw/xen/xen_pt.c  |  7 +++
 include/hw/pci/pci.h |  3 +++
 include/hw/pci/pci_bus.h |  3 ++-
 5 files changed, 38 insertions(+), 14 deletions(-)

-- 
2.39.2

[PATCH v3 2/2] pci: introduce slot_reserved_auto_mask and slot_reserved_manual_mask

[Chuck Zmudzinski]

Commit 4f67543bb8c5 ("xen/pt: reserve PCI slot 2 for Intel igd-passthru")
uses slot_reserved_mask to reserve slot 2 for the Intel igd for the
xenfv machine when the guest is configured for igd-passthru.

Prior to that commit, a single 32-bit mask was sufficient to meet the
needs of the only machine that used the 32-bit slot_reserved_mask, the
sun4u machine. However, the requirements of the xenfv machine with
igd-passthru is somewhat different from the requirements of the sun4u
machine.

First, the sun4u machine reserves slots in such a way that no device
can be assigned to a reserved slot, but the xenfv machine needs to
reserve a single slot that is reserved for a particular device, the
Intel igd. The necessary logic to ensure that the reserved slot is used
by the Intel igd was recently added by the aforementioned commit.

Second, it is useful to limit slot reservation in the case of the xenfv
machine with the Intel igd to devices configured for automatic slot
assignment so an administrator can assign a device to the reserved slot
by manually specifying the reserved slot address on the command line,
but the sun4u machine requires slot reservation for all devices, whether
or not the device is configured for automatic slot assignment or
configured manually by specifying a slot address on the command line. In
other words, for the sun4u machine, the required behavior is that an
attempt to assign a reserved slot to a device must always result in an
error, but it is useful to allow manual assignment of a reserved slot to
succeed for the xenfv machine with the Intel igd.

The necessary logic to implement the desired behavior of reserving one
or more slots only for the case of automatic slot allocation has not yet
been implemented, and that is the purpose of this patch.

The implementation is simple: the xenfv machine only sets
slot_reserved_auto_mask and the sun4u machine sets both
slot_reserved_manual_mask and slot_reserved_auto_mask. A single
"set" accessor function allows xenfv and sun4u machines to set the
value of the two masks appropriately for each use case.

Since the xenfv machine needs to implement additional logic to detect
the Intel igd and clear the bit in the mask to allow a particular device
to use the reserved slot, there is a need for a "get" and "clear" accessor
function for slot_reserved_auto_mask, but these accessor functions are
not needed for slot_reserved_manual_mask because the sun4u machine has
no need to get the value of the mask or clear bits in the mask.

Link: 
https://lore.kernel.org/qemu-devel/20230106064838-mutt-send-email-...@kernel.org/
Signed-off-by: Chuck Zmudzinski 
---
Changelog

v3: Change Subject of patch from
"pci: allow slot_reserved_mask to be ignored with manual slot assignment" To
"pci: introduce slot_reserved_auto_mask and slot_reserved_manual_mask"

Substantially reword the commit message to clearly explain the reasons
this patch is needed

Apply changes in response to comments on v2:
   - slot_reserved_mask -> slot_reserved_auto_mask
   - remove enforce_slot_reserved_mask_manual
   - remove pci_bus_ignore_slot_reserved_mask_manual
   - add slot_reserved_manual_mask
   - pci_bus_devfn_reserved -> pci_bus_devfn_reserved_auto
   - change code in pci_bus_devfn_reserved_manual appropriately
   - pci_bus_set_slot_reserved_mask -> pci_bus_set_slot_reserved_masks
   - use renamed "set" function to set value of both masks for sun4u 
and xenfv cases
   - pci_bus_get_slot_reserved_mask -> pci_bus_get_slot_reserved_auto_mask
   - pci_bus_clear_slot_reserved_mask -> 
pci_bus_clear_slot_reserved_auto_mask

v2: Change Subject of patch from
"pci: add enforce_slot_reserved_mask_manual property" To
"pci: allow slot_reserved_mask to be ignored with manual slot assignment"

Add pci_bus_ignore_slot_reserved_mask_manual function

Call pci_bus_ignore_slot_reserved_mask_manual at appropriate place
in hw/pci-host/i440fx.c

 hw/pci/pci.c | 29 ++---
 hw/sparc64/sun4u.c   |  6 +++---
 hw/xen/xen_pt.c  |  6 +++---
 include/hw/pci/pci.h |  6 +++---
 include/hw/pci/pci_bus.h |  3 ++-
 5 files changed, 29 insertions(+), 21 deletions(-)

diff --git a/hw/pci/pci.c b/hw/pci/pci.c
index 8a87ccc8b0..58a530a651 100644
--- a/hw/pci/pci.c
+++ b/hw/pci/pci.c
@@ -500,7 +500,8 @@ static void pci_root_bus_internal_init(PCIBus *bus, 
DeviceState *parent,
 {
 assert(PCI_FUNC(devfn_min) == 0);
 bus->devfn_min = devfn_min;
-bus->slot_reserved_mask = 0x0;
+bus->slot_reserved_auto_mask = 0x0;
+bus->slot_reserved_manual_mask = 0x0;
 bus->address_space_mem = address_space_mem;
 bus->address_space_io = address_space_io;
 bus->flags |= PCI_BUS_IS_ROOT;
@@ -,24 +1112,30 @@ static bool pci_bus_devfn_available(PCIBus *bus,

Re: [PATCH v2 2/2] pci: allow slot_reserved_mask to be ignored with manual slot assignment

[Chuck Zmudzinski]

On 3/14/2023 10:39 AM, Mark Cave-Ayland wrote:
> On 14/03/2023 14:21, Chuck Zmudzinski wrote:
>
> > On 3/14/2023 9:41 AM, Mark Cave-Ayland wrote:
> >> On 14/03/2023 13:26, Chuck Zmudzinski wrote:
> >>
> >>> On 3/14/2023 9:17 AM, Michael S. Tsirkin wrote:
> >>>> On Tue, Mar 14, 2023 at 12:43:12PM +, Mark Cave-Ayland wrote:
> >>>>> On 14/03/2023 06:33, Michael S. Tsirkin wrote:
> >>>>>
> >>>>>> On Tue, Mar 14, 2023 at 12:01:09AM -0400, Chuck Zmudzinski wrote:
> >>>>>>> Commit 4f67543bb8c5 ("xen/pt: reserve PCI slot 2 for Intel 
> >>>>>>> igd-passthru")
> >>>>>>> uses slot_reserved_mask to reserve slot 2 for the Intel IGD for the
> >>>>>>> xenfv machine when the guest is configured for igd-passthru.
> >>>>>>>
> >>>>>>> A desired extension to that commit is to allow use of the reserved 
> >>>>>>> slot
> >>>>>>> if the administrator manually configures a device to use the reserved
> >>>>>>> slot. Currently, slot_reserved_mask is enforced unconditionally. With
> >>>>>>> this patch, the pci bus can be configured so the slot is only reserved
> >>>>>>> if the pci device to be added to the bus is configured for automatic
> >>>>>>> slot assignment.
> >>>>>>>
> >>>>>>> To enable the desired behavior of slot_reserved_mask machine, add a
> >>>>>>> boolean member enforce_slot_reserved_mask_manual to struct PCIBus and
> >>>>>>> add a function pci_bus_ignore_slot_reserved_mask_manual which can be
> >>>>>>> called to change the default behavior of always enforcing
> >>>>>>> slot_reserved_mask so, in that case, slot_reserved_mask is only 
> >>>>>>> enforced
> >>>>>>> when the pci device being added is configured for automatic slot
> >>>>>>> assignment.
> >>>>>>>
> >>>>>>> Call the new pci_bus_ignore_slot_reserved_mask_manual function after
> >>>>>>> creating the pci bus for the pc/i440fx/xenfv machine type to implement
> >>>>>>> the desired behavior of causing slot_reserved_mask to only apply when
> >>>>>>> the pci device to be added to a pc/i440fx/xenfv machine is configured
> >>>>>>> for automatic slot assignment.
> >>>>>>>
> >>>>>>> Link: 
> >>>>>>> https://lore.kernel.org/qemu-devel/20230106064838-mutt-send-email-...@kernel.org/
> >>>>>>> Signed-off-by: Chuck Zmudzinski 
> >>>>>>
> >>>>>> I really dislike this.
> >>>>>> It seems that xen should not have used slot_reserved_mask,
> >>>>>> and instead needs something new like slot_manual_mask.
> >>>>>> No?
> >>>>>
> >>>>> My suggestion was to move the validation logic to a separate callback
> >>>>> function in PCIBus (see
> >>>>> https://lists.gnu.org/archive/html/qemu-devel/2023-03/msg03988.html) but
> >>>>> perhaps I wasn't clear enough in pointing out that I was thinking this 
> >>>>> could
> >>>>> *replace* the existing slot_reserved_mask mechanism, rather than 
> >>>>> providing a
> >>>>> hook to allow it to be manipulated.
> >>>>>
> >>>>> Here's a very rough patch put together over lunch that attempts this for
> >>>>> pci_bus_devfn_reserved(): the idea is that sun4u and Xen would call
> >>>>> pci_bus_set_slot_reserved_fn() with a suitable pci_slot_reserved_fn
> >>>>> implementation, and slot_reserved_mask gets removed completely i.e.:
> >>>>>
> >>>>>
> >>>>> diff --git a/hw/pci/pci.c b/hw/pci/pci.c
> >>>>> index def5000e7b..30b856499a 100644
> >>>>> --- a/hw/pci/pci.c
> >>>>> +++ b/hw/pci/pci.c
> >>>>> @@ -493,6 +493,13 @@ bool pci_bus_bypass_iommu(PCIBus *bus)
> >>>>>return host_bridge->bypass_iommu;
> >>>>>}
> >>>>>
> >>>>> +static bool pci_bus_default_slot_reserved(PCISlotReservationType 
> >>>>> restype,
> >>>

Re: [PATCH v2 2/2] pci: allow slot_reserved_mask to be ignored with manual slot assignment

[Chuck Zmudzinski]

On 3/14/2023 9:41 AM, Mark Cave-Ayland wrote:
> On 14/03/2023 13:26, Chuck Zmudzinski wrote:
>
> > On 3/14/2023 9:17 AM, Michael S. Tsirkin wrote:
> >> On Tue, Mar 14, 2023 at 12:43:12PM +, Mark Cave-Ayland wrote:
> >>> On 14/03/2023 06:33, Michael S. Tsirkin wrote:
> >>>
> >>>> On Tue, Mar 14, 2023 at 12:01:09AM -0400, Chuck Zmudzinski wrote:
> >>>>> Commit 4f67543bb8c5 ("xen/pt: reserve PCI slot 2 for Intel 
> >>>>> igd-passthru")
> >>>>> uses slot_reserved_mask to reserve slot 2 for the Intel IGD for the
> >>>>> xenfv machine when the guest is configured for igd-passthru.
> >>>>>
> >>>>> A desired extension to that commit is to allow use of the reserved slot
> >>>>> if the administrator manually configures a device to use the reserved
> >>>>> slot. Currently, slot_reserved_mask is enforced unconditionally. With
> >>>>> this patch, the pci bus can be configured so the slot is only reserved
> >>>>> if the pci device to be added to the bus is configured for automatic
> >>>>> slot assignment.
> >>>>>
> >>>>> To enable the desired behavior of slot_reserved_mask machine, add a
> >>>>> boolean member enforce_slot_reserved_mask_manual to struct PCIBus and
> >>>>> add a function pci_bus_ignore_slot_reserved_mask_manual which can be
> >>>>> called to change the default behavior of always enforcing
> >>>>> slot_reserved_mask so, in that case, slot_reserved_mask is only enforced
> >>>>> when the pci device being added is configured for automatic slot
> >>>>> assignment.
> >>>>>
> >>>>> Call the new pci_bus_ignore_slot_reserved_mask_manual function after
> >>>>> creating the pci bus for the pc/i440fx/xenfv machine type to implement
> >>>>> the desired behavior of causing slot_reserved_mask to only apply when
> >>>>> the pci device to be added to a pc/i440fx/xenfv machine is configured
> >>>>> for automatic slot assignment.
> >>>>>
> >>>>> Link: 
> >>>>> https://lore.kernel.org/qemu-devel/20230106064838-mutt-send-email-...@kernel.org/
> >>>>> Signed-off-by: Chuck Zmudzinski 
> >>>>
> >>>> I really dislike this.
> >>>> It seems that xen should not have used slot_reserved_mask,
> >>>> and instead needs something new like slot_manual_mask.
> >>>> No?
> >>>
> >>> My suggestion was to move the validation logic to a separate callback
> >>> function in PCIBus (see
> >>> https://lists.gnu.org/archive/html/qemu-devel/2023-03/msg03988.html) but
> >>> perhaps I wasn't clear enough in pointing out that I was thinking this 
> >>> could
> >>> *replace* the existing slot_reserved_mask mechanism, rather than 
> >>> providing a
> >>> hook to allow it to be manipulated.
> >>>
> >>> Here's a very rough patch put together over lunch that attempts this for
> >>> pci_bus_devfn_reserved(): the idea is that sun4u and Xen would call
> >>> pci_bus_set_slot_reserved_fn() with a suitable pci_slot_reserved_fn
> >>> implementation, and slot_reserved_mask gets removed completely i.e.:
> >>>
> >>>
> >>> diff --git a/hw/pci/pci.c b/hw/pci/pci.c
> >>> index def5000e7b..30b856499a 100644
> >>> --- a/hw/pci/pci.c
> >>> +++ b/hw/pci/pci.c
> >>> @@ -493,6 +493,13 @@ bool pci_bus_bypass_iommu(PCIBus *bus)
> >>>   return host_bridge->bypass_iommu;
> >>>   }
> >>>
> >>> +static bool pci_bus_default_slot_reserved(PCISlotReservationType restype,
> >>> +  int devfn)
> >>> +{
> >>> +/* All slots accessible by default */
> >>> +return false;
> >>> +}
> >>> +
> >>>   static void pci_root_bus_internal_init(PCIBus *bus, DeviceState *parent,
> >>>  MemoryRegion *address_space_mem,
> >>>  MemoryRegion *address_space_io,
> >>> @@ -500,7 +507,7 @@ static void pci_root_bus_internal_init(PCIBus *bus,
> >>> DeviceState *parent,
> >>>   {
> >>>   assert(PCI_FUNC(devfn_min) == 0);
> >>>   bus->devfn_min = devfn_

Re: [PATCH v2 2/2] pci: allow slot_reserved_mask to be ignored with manual slot assignment

[Chuck Zmudzinski]

On 3/14/2023 9:41 AM, Mark Cave-Ayland wrote:
> On 14/03/2023 13:26, Chuck Zmudzinski wrote:
>
> > On 3/14/2023 9:17 AM, Michael S. Tsirkin wrote:
> >> On Tue, Mar 14, 2023 at 12:43:12PM +, Mark Cave-Ayland wrote:
> >>> On 14/03/2023 06:33, Michael S. Tsirkin wrote:
> >>>
> >>>> On Tue, Mar 14, 2023 at 12:01:09AM -0400, Chuck Zmudzinski wrote:
> >>>>> Commit 4f67543bb8c5 ("xen/pt: reserve PCI slot 2 for Intel 
> >>>>> igd-passthru")
> >>>>> uses slot_reserved_mask to reserve slot 2 for the Intel IGD for the
> >>>>> xenfv machine when the guest is configured for igd-passthru.
> >>>>>
> >>>>> A desired extension to that commit is to allow use of the reserved slot
> >>>>> if the administrator manually configures a device to use the reserved
> >>>>> slot. Currently, slot_reserved_mask is enforced unconditionally. With
> >>>>> this patch, the pci bus can be configured so the slot is only reserved
> >>>>> if the pci device to be added to the bus is configured for automatic
> >>>>> slot assignment.
> >>>>>
> >>>>> To enable the desired behavior of slot_reserved_mask machine, add a
> >>>>> boolean member enforce_slot_reserved_mask_manual to struct PCIBus and
> >>>>> add a function pci_bus_ignore_slot_reserved_mask_manual which can be
> >>>>> called to change the default behavior of always enforcing
> >>>>> slot_reserved_mask so, in that case, slot_reserved_mask is only enforced
> >>>>> when the pci device being added is configured for automatic slot
> >>>>> assignment.
> >>>>>
> >>>>> Call the new pci_bus_ignore_slot_reserved_mask_manual function after
> >>>>> creating the pci bus for the pc/i440fx/xenfv machine type to implement
> >>>>> the desired behavior of causing slot_reserved_mask to only apply when
> >>>>> the pci device to be added to a pc/i440fx/xenfv machine is configured
> >>>>> for automatic slot assignment.
> >>>>>
> >>>>> Link: 
> >>>>> https://lore.kernel.org/qemu-devel/20230106064838-mutt-send-email-...@kernel.org/
> >>>>> Signed-off-by: Chuck Zmudzinski 
> >>>>
> >>>> I really dislike this.
> >>>> It seems that xen should not have used slot_reserved_mask,
> >>>> and instead needs something new like slot_manual_mask.
> >>>> No?
> >>>
> >>> My suggestion was to move the validation logic to a separate callback
> >>> function in PCIBus (see
> >>> https://lists.gnu.org/archive/html/qemu-devel/2023-03/msg03988.html) but
> >>> perhaps I wasn't clear enough in pointing out that I was thinking this 
> >>> could
> >>> *replace* the existing slot_reserved_mask mechanism, rather than 
> >>> providing a
> >>> hook to allow it to be manipulated.
> >>>
> >>> Here's a very rough patch put together over lunch that attempts this for
> >>> pci_bus_devfn_reserved(): the idea is that sun4u and Xen would call
> >>> pci_bus_set_slot_reserved_fn() with a suitable pci_slot_reserved_fn
> >>> implementation, and slot_reserved_mask gets removed completely i.e.:
> >>>
> >>>
> >>> diff --git a/hw/pci/pci.c b/hw/pci/pci.c
> >>> index def5000e7b..30b856499a 100644
> >>> --- a/hw/pci/pci.c
> >>> +++ b/hw/pci/pci.c
> >>> @@ -493,6 +493,13 @@ bool pci_bus_bypass_iommu(PCIBus *bus)
> >>>   return host_bridge->bypass_iommu;
> >>>   }
> >>>
> >>> +static bool pci_bus_default_slot_reserved(PCISlotReservationType restype,
> >>> +  int devfn)
> >>> +{
> >>> +/* All slots accessible by default */
> >>> +return false;
> >>> +}
> >>> +
> >>>   static void pci_root_bus_internal_init(PCIBus *bus, DeviceState *parent,
> >>>  MemoryRegion *address_space_mem,
> >>>  MemoryRegion *address_space_io,
> >>> @@ -500,7 +507,7 @@ static void pci_root_bus_internal_init(PCIBus *bus,
> >>> DeviceState *parent,
> >>>   {
> >>>   assert(PCI_FUNC(devfn_min) == 0);
> >>>   bus->devfn_min = devfn_

Re: [PATCH v2 2/2] pci: allow slot_reserved_mask to be ignored with manual slot assignment

[Chuck Zmudzinski]

On 3/14/2023 9:17 AM, Michael S. Tsirkin wrote:
> On Tue, Mar 14, 2023 at 12:43:12PM +, Mark Cave-Ayland wrote:
> > On 14/03/2023 06:33, Michael S. Tsirkin wrote:
> > 
> > > On Tue, Mar 14, 2023 at 12:01:09AM -0400, Chuck Zmudzinski wrote:
> > > > Commit 4f67543bb8c5 ("xen/pt: reserve PCI slot 2 for Intel 
> > > > igd-passthru")
> > > > uses slot_reserved_mask to reserve slot 2 for the Intel IGD for the
> > > > xenfv machine when the guest is configured for igd-passthru.
> > > > 
> > > > A desired extension to that commit is to allow use of the reserved slot
> > > > if the administrator manually configures a device to use the reserved
> > > > slot. Currently, slot_reserved_mask is enforced unconditionally. With
> > > > this patch, the pci bus can be configured so the slot is only reserved
> > > > if the pci device to be added to the bus is configured for automatic
> > > > slot assignment.
> > > > 
> > > > To enable the desired behavior of slot_reserved_mask machine, add a
> > > > boolean member enforce_slot_reserved_mask_manual to struct PCIBus and
> > > > add a function pci_bus_ignore_slot_reserved_mask_manual which can be
> > > > called to change the default behavior of always enforcing
> > > > slot_reserved_mask so, in that case, slot_reserved_mask is only enforced
> > > > when the pci device being added is configured for automatic slot
> > > > assignment.
> > > > 
> > > > Call the new pci_bus_ignore_slot_reserved_mask_manual function after
> > > > creating the pci bus for the pc/i440fx/xenfv machine type to implement
> > > > the desired behavior of causing slot_reserved_mask to only apply when
> > > > the pci device to be added to a pc/i440fx/xenfv machine is configured
> > > > for automatic slot assignment.
> > > > 
> > > > Link: 
> > > > https://lore.kernel.org/qemu-devel/20230106064838-mutt-send-email-...@kernel.org/
> > > > Signed-off-by: Chuck Zmudzinski 
> > > 
> > > I really dislike this.
> > > It seems that xen should not have used slot_reserved_mask,
> > > and instead needs something new like slot_manual_mask.
> > > No?
> > 
> > My suggestion was to move the validation logic to a separate callback
> > function in PCIBus (see
> > https://lists.gnu.org/archive/html/qemu-devel/2023-03/msg03988.html) but
> > perhaps I wasn't clear enough in pointing out that I was thinking this could
> > *replace* the existing slot_reserved_mask mechanism, rather than providing a
> > hook to allow it to be manipulated.
> > 
> > Here's a very rough patch put together over lunch that attempts this for
> > pci_bus_devfn_reserved(): the idea is that sun4u and Xen would call
> > pci_bus_set_slot_reserved_fn() with a suitable pci_slot_reserved_fn
> > implementation, and slot_reserved_mask gets removed completely i.e.:
> > 
> > 
> > diff --git a/hw/pci/pci.c b/hw/pci/pci.c
> > index def5000e7b..30b856499a 100644
> > --- a/hw/pci/pci.c
> > +++ b/hw/pci/pci.c
> > @@ -493,6 +493,13 @@ bool pci_bus_bypass_iommu(PCIBus *bus)
> >  return host_bridge->bypass_iommu;
> >  }
> > 
> > +static bool pci_bus_default_slot_reserved(PCISlotReservationType restype,
> > +  int devfn)
> > +{
> > +/* All slots accessible by default */
> > +return false;
> > +}
> > +
> >  static void pci_root_bus_internal_init(PCIBus *bus, DeviceState *parent,
> > MemoryRegion *address_space_mem,
> > MemoryRegion *address_space_io,
> > @@ -500,7 +507,7 @@ static void pci_root_bus_internal_init(PCIBus *bus,
> > DeviceState *parent,
> >  {
> >  assert(PCI_FUNC(devfn_min) == 0);
> >  bus->devfn_min = devfn_min;
> > -bus->slot_reserved_mask = 0x0;
> > +bus->slot_reserved_fn = pci_bus_default_slot_reserved;
> >  bus->address_space_mem = address_space_mem;
> >  bus->address_space_io = address_space_io;
> >  bus->flags |= PCI_BUS_IS_ROOT;
> > @@ -,9 +1118,15 @@ static bool pci_bus_devfn_available(PCIBus *bus, int 
> > devfn)
> >  return !(bus->devices[devfn]);
> >  }
> > 
> > -static bool pci_bus_devfn_reserved(PCIBus *bus, int devfn)
> > +static bool pci_bus_devfn_reserved(PCIBus *bus, int devfn,
> > +   PCISlotReservationType

Re: [PATCH v2 2/2] pci: allow slot_reserved_mask to be ignored with manual slot assignment

[Chuck Zmudzinski]

On 3/14/2023 2:33 AM, Michael S. Tsirkin wrote:
> On Tue, Mar 14, 2023 at 12:01:09AM -0400, Chuck Zmudzinski wrote:
> > Commit 4f67543bb8c5 ("xen/pt: reserve PCI slot 2 for Intel igd-passthru")
> > uses slot_reserved_mask to reserve slot 2 for the Intel IGD for the
> > xenfv machine when the guest is configured for igd-passthru.
> > 
> > A desired extension to that commit is to allow use of the reserved slot
> > if the administrator manually configures a device to use the reserved
> > slot. Currently, slot_reserved_mask is enforced unconditionally. With
> > this patch, the pci bus can be configured so the slot is only reserved
> > if the pci device to be added to the bus is configured for automatic
> > slot assignment.
> > 
> > To enable the desired behavior of slot_reserved_mask machine, add a
> > boolean member enforce_slot_reserved_mask_manual to struct PCIBus and
> > add a function pci_bus_ignore_slot_reserved_mask_manual which can be
> > called to change the default behavior of always enforcing
> > slot_reserved_mask so, in that case, slot_reserved_mask is only enforced
> > when the pci device being added is configured for automatic slot
> > assignment.
> > 
> > Call the new pci_bus_ignore_slot_reserved_mask_manual function after
> > creating the pci bus for the pc/i440fx/xenfv machine type to implement
> > the desired behavior of causing slot_reserved_mask to only apply when
> > the pci device to be added to a pc/i440fx/xenfv machine is configured
> > for automatic slot assignment.
> > 
> > Link: 
> > https://lore.kernel.org/qemu-devel/20230106064838-mutt-send-email-...@kernel.org/
> > Signed-off-by: Chuck Zmudzinski 
>
> I really dislike this. 
> It seems that xen should not have used slot_reserved_mask,
> and instead needs something new like slot_manual_mask.
> No?

Actually, xen would use something like slot_auto_mask, and
sun4u would use both slot_auto_mask and slot_manual_mask.

Is it just that this patch touches hw/pci-host/i440fx.c that you
don't like or is it that you don't like adding slot_reserved_mask_manual
and pci_bus_ignore_slot_reserved_mask_manual, or is it both
that you don't like?

If it's the former that you don't like, the call to
pci_bus_ignore_slot_reserved_mask_manual can be moved to
xen_igd_reserve_slot in hw/xen/xen_pt.c and this would
avoid touching hw/pci-host/i440fx.c.

If it's the latter that you don't like, both slot_reserved_mask_manual
and pci_bus_ignore_slot_reserved_mask_manual can be removed
and this can be implemented with two independent slot masks:

rename slot_reserved_mask as slot_auto_mask - used by both xen and sun4u
slot_manual_mask - new mask, used only by sun4u.

We would also need to have two sets of accessor functions in this case, one
set to access slot_auto_mask, and the other to access slot_manual_mask.
Since the sun4u machine does not need to either get the value of
slot_manual_mask or clear the slot_manual_mask, slot_manual_mask
would only need to have one accessor function to set the value of the
mask. slot_auto_mask would have all three accessor functions that xen
needs to use.

Would that be OK?

>
> > ---
> > Changelog
> > 
> > v2: Change Subject of patch from
> > "pci: add enforce_slot_reserved_mask_manual property" To
> > "pci: allow slot_reserved_mask to be ignored with manual slot 
> > assignment"
> > 
> > Add pci_bus_ignore_slot_reserved_mask_manual function
> > 
> > Call pci_bus_ignore_slot_reserved_mask_manual at appropriate place
> > in hw/pci-host/i440fx.c
> > 
> >  hw/pci-host/i440fx.c |  1 +
> >  hw/pci/pci.c | 14 +-
> >  include/hw/pci/pci.h |  1 +
> >  include/hw/pci/pci_bus.h |  1 +
> >  4 files changed, 16 insertions(+), 1 deletion(-)
> > 
> > diff --git a/hw/pci-host/i440fx.c b/hw/pci-host/i440fx.c
> > index 262f82c303..8e00b88926 100644
> > --- a/hw/pci-host/i440fx.c
> > +++ b/hw/pci-host/i440fx.c
> > @@ -257,6 +257,7 @@ PCIBus *i440fx_init(const char *pci_type,
> >  s = PCI_HOST_BRIDGE(dev);
> >  b = pci_root_bus_new(dev, NULL, pci_address_space,
> >   address_space_io, 0, TYPE_PCI_BUS);
> > +pci_bus_ignore_slot_reserved_mask_manual(b);
> >  s->bus = b;
> >  object_property_add_child(qdev_get_machine(), "i440fx", OBJECT(dev));
> >  sysbus_realize_and_unref(SYS_BUS_DEVICE(dev), _fatal);
> > diff --git a/hw/pci/pci.c b/hw/pci/pci.c
> > index 8a87ccc8b0..670ecc6986 100644
> > --- a/hw/pci/pci.c
> > +++ b/hw/pci/pci.c
> > @@ -501,6 +501,7 @@ static void pci_root_bus_internal_init(

[PATCH v2 0/2] pci: slot_reserved_mask improvements

[Chuck Zmudzinski]

This patch series consists of two patches. The first provides accessor
functions in pci.h to avoid direct access of slot_reserved_mask
according to the comment at the top of include/hw/pci/pci_bus.h. No
functional change is intended with this patch.

The second patch allows a pci bus to be configured so slot_reserved_mask
will only be enforced when the device to be added to the bus is
configured for automatic slot assignment. The second patch also uses the
new capability in the case of the pc/i440fx/xenfv machine types so
the current behavior of reserving slot 2 for the Intel IGD for the
xenfv machine will be ignored if an administrator manually configures
another device to use the reserved slot.

The current behavior of always reserving slots in the sun4u machine is
preserved by this patch series; the patch series only changes how
slot_reserved_mask works in the xenfv machine. Although the patch
series can affect xenfv machines configured for igd-passthru if an
administrator assigns some of the pci slot addresses manually, it
does not affect the libxl default configuration for igd-passthru because
libxl uses automatic slot assignment by default.

Link: 
https://lore.kernel.org/qemu-devel/20230106064838-mutt-send-email-...@kernel.org/

Chuck Zmudzinski (2):
  pci: avoid accessing slot_reserved_mask directly outside of pci.c
  pci: allow slot_reserved_mask to be ignored with manual slot
assignment

Changelog

v2: Add first patch and cover letter to make this a 2-patch series
Make changes to the second patch (see second patch for changelog)

 hw/pci-host/i440fx.c |  1 +
 hw/pci/pci.c | 29 -
 hw/sparc64/sun4u.c   |  7 +++
 hw/xen/xen_pt.c  |  7 +++
 include/hw/pci/pci.h |  4 
 include/hw/pci/pci_bus.h |  1 +
 6 files changed, 40 insertions(+), 9 deletions(-)

-- 
2.39.2

[PATCH v2 2/2] pci: allow slot_reserved_mask to be ignored with manual slot assignment

[Chuck Zmudzinski]

Commit 4f67543bb8c5 ("xen/pt: reserve PCI slot 2 for Intel igd-passthru")
uses slot_reserved_mask to reserve slot 2 for the Intel IGD for the
xenfv machine when the guest is configured for igd-passthru.

A desired extension to that commit is to allow use of the reserved slot
if the administrator manually configures a device to use the reserved
slot. Currently, slot_reserved_mask is enforced unconditionally. With
this patch, the pci bus can be configured so the slot is only reserved
if the pci device to be added to the bus is configured for automatic
slot assignment.

To enable the desired behavior of slot_reserved_mask machine, add a
boolean member enforce_slot_reserved_mask_manual to struct PCIBus and
add a function pci_bus_ignore_slot_reserved_mask_manual which can be
called to change the default behavior of always enforcing
slot_reserved_mask so, in that case, slot_reserved_mask is only enforced
when the pci device being added is configured for automatic slot
assignment.

Call the new pci_bus_ignore_slot_reserved_mask_manual function after
creating the pci bus for the pc/i440fx/xenfv machine type to implement
the desired behavior of causing slot_reserved_mask to only apply when
the pci device to be added to a pc/i440fx/xenfv machine is configured
for automatic slot assignment.

Link: 
https://lore.kernel.org/qemu-devel/20230106064838-mutt-send-email-...@kernel.org/
Signed-off-by: Chuck Zmudzinski 
---
Changelog

v2: Change Subject of patch from
"pci: add enforce_slot_reserved_mask_manual property" To
"pci: allow slot_reserved_mask to be ignored with manual slot assignment"

Add pci_bus_ignore_slot_reserved_mask_manual function

Call pci_bus_ignore_slot_reserved_mask_manual at appropriate place
in hw/pci-host/i440fx.c

 hw/pci-host/i440fx.c |  1 +
 hw/pci/pci.c | 14 +-
 include/hw/pci/pci.h |  1 +
 include/hw/pci/pci_bus.h |  1 +
 4 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/hw/pci-host/i440fx.c b/hw/pci-host/i440fx.c
index 262f82c303..8e00b88926 100644
--- a/hw/pci-host/i440fx.c
+++ b/hw/pci-host/i440fx.c
@@ -257,6 +257,7 @@ PCIBus *i440fx_init(const char *pci_type,
 s = PCI_HOST_BRIDGE(dev);
 b = pci_root_bus_new(dev, NULL, pci_address_space,
  address_space_io, 0, TYPE_PCI_BUS);
+pci_bus_ignore_slot_reserved_mask_manual(b);
 s->bus = b;
 object_property_add_child(qdev_get_machine(), "i440fx", OBJECT(dev));
 sysbus_realize_and_unref(SYS_BUS_DEVICE(dev), _fatal);
diff --git a/hw/pci/pci.c b/hw/pci/pci.c
index 8a87ccc8b0..670ecc6986 100644
--- a/hw/pci/pci.c
+++ b/hw/pci/pci.c
@@ -501,6 +501,7 @@ static void pci_root_bus_internal_init(PCIBus *bus, 
DeviceState *parent,
 assert(PCI_FUNC(devfn_min) == 0);
 bus->devfn_min = devfn_min;
 bus->slot_reserved_mask = 0x0;
+bus->enforce_slot_reserved_mask_manual = true;
 bus->address_space_mem = address_space_mem;
 bus->address_space_io = address_space_io;
 bus->flags |= PCI_BUS_IS_ROOT;
@@ -1116,6 +1117,17 @@ static bool pci_bus_devfn_reserved(PCIBus *bus, int 
devfn)
 return bus->slot_reserved_mask & (1UL << PCI_SLOT(devfn));
 }
 
+static bool pci_bus_devfn_reserved_manual(PCIBus *bus, int devfn)
+{
+return bus->enforce_slot_reserved_mask_manual &&
+(bus->slot_reserved_mask & (1UL << PCI_SLOT(devfn)));
+}
+
+void pci_bus_ignore_slot_reserved_mask_manual(PCIBus *bus)
+{
+bus->enforce_slot_reserved_mask_manual = false;
+}
+
 uint32_t pci_bus_get_slot_reserved_mask(PCIBus *bus)
 {
 return bus->slot_reserved_mask;
@@ -1164,7 +1176,7 @@ static PCIDevice *do_pci_register_device(PCIDevice 
*pci_dev,
"or reserved", name);
 return NULL;
 found: ;
-} else if (pci_bus_devfn_reserved(bus, devfn)) {
+} else if (pci_bus_devfn_reserved_manual(bus, devfn)) {
 error_setg(errp, "PCI: slot %d function %d not available for %s,"
" reserved",
PCI_SLOT(devfn), PCI_FUNC(devfn), name);
diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h
index 935b4b91b4..48d29ec234 100644
--- a/include/hw/pci/pci.h
+++ b/include/hw/pci/pci.h
@@ -287,6 +287,7 @@ void pci_bus_irqs(PCIBus *bus, pci_set_irq_fn set_irq,
 void pci_bus_map_irqs(PCIBus *bus, pci_map_irq_fn map_irq);
 void pci_bus_irqs_cleanup(PCIBus *bus);
 int pci_bus_get_irq_level(PCIBus *bus, int irq_num);
+void pci_bus_ignore_slot_reserved_mask_manual(PCIBus *bus);
 uint32_t pci_bus_get_slot_reserved_mask(PCIBus *bus);
 void pci_bus_set_slot_reserved_mask(PCIBus *bus, uint32_t mask);
 void pci_bus_clear_slot_reserved_mask(PCIBus *bus, uint32_t mask);
diff --git a/include/hw/pci/pci_bus.h b/include/hw/pci/pci_bus.h
index 5653175957..e0f15ee9be 100644
--- a/include/hw/pci/pci_bus.h
+++ b/include/hw/pci/pci_bus.h
@@ -37,6 +37,7 @@ struct P

[PATCH v2 1/2] pci: avoid accessing slot_reserved_mask directly outside of pci.c

[Chuck Zmudzinski]

This patch provides accessor functions as replacements for direct
access to slot_reserved_mask according to the comment at the top
of include/hw/pci/pci_bus.h which advises that data structures for
PCIBus should not be directly accessed but instead be accessed using
accessor functions in pci.h.

Three accessor functions can conveniently replace all direct accesses
of slot_reserved_mask. With this patch, the new accessor functions are
used in hw/sparc64/sun4u.c and hw/xen/xen_pt.c and pci_bus.h is removed
from the included header files of the same two files.

No functional change intended.

Signed-off-by: Chuck Zmudzinski 
---
v2: This is the first version of this patch, it did not exist in v1.

 hw/pci/pci.c | 15 +++
 hw/sparc64/sun4u.c   |  7 +++
 hw/xen/xen_pt.c  |  7 +++
 include/hw/pci/pci.h |  3 +++
 4 files changed, 24 insertions(+), 8 deletions(-)

diff --git a/hw/pci/pci.c b/hw/pci/pci.c
index def5000e7b..8a87ccc8b0 100644
--- a/hw/pci/pci.c
+++ b/hw/pci/pci.c
@@ -1116,6 +1116,21 @@ static bool pci_bus_devfn_reserved(PCIBus *bus, int 
devfn)
 return bus->slot_reserved_mask & (1UL << PCI_SLOT(devfn));
 }
 
+uint32_t pci_bus_get_slot_reserved_mask(PCIBus *bus)
+{
+return bus->slot_reserved_mask;
+}
+
+void pci_bus_set_slot_reserved_mask(PCIBus *bus, uint32_t mask)
+{
+bus->slot_reserved_mask |= mask;
+}
+
+void pci_bus_clear_slot_reserved_mask(PCIBus *bus, uint32_t mask)
+{
+bus->slot_reserved_mask &= ~mask;
+}
+
 /* -1 for devfn means auto assign */
 static PCIDevice *do_pci_register_device(PCIDevice *pci_dev,
  const char *name, int devfn,
diff --git a/hw/sparc64/sun4u.c b/hw/sparc64/sun4u.c
index a25e951f9d..eae7589462 100644
--- a/hw/sparc64/sun4u.c
+++ b/hw/sparc64/sun4u.c
@@ -31,7 +31,6 @@
 #include "hw/irq.h"
 #include "hw/pci/pci.h"
 #include "hw/pci/pci_bridge.h"
-#include "hw/pci/pci_bus.h"
 #include "hw/pci/pci_host.h"
 #include "hw/qdev-properties.h"
 #include "hw/pci-host/sabre.h"
@@ -608,9 +607,9 @@ static void sun4uv_init(MemoryRegion *address_space_mem,
 /* Only in-built Simba APBs can exist on the root bus, slot 0 on busA is
reserved (leaving no slots free after on-board devices) however slots
0-3 are free on busB */
-pci_bus->slot_reserved_mask = 0xfffc;
-pci_busA->slot_reserved_mask = 0xfff1;
-pci_busB->slot_reserved_mask = 0xfff0;
+pci_bus_set_slot_reserved_mask(pci_bus, 0xfffc);
+pci_bus_set_slot_reserved_mask(pci_busA, 0xfff1);
+pci_bus_set_slot_reserved_mask(pci_busB, 0xfff0);
 
 ebus = pci_new_multifunction(PCI_DEVFN(1, 0), true, TYPE_EBUS);
 qdev_prop_set_uint64(DEVICE(ebus), "console-serial-base",
diff --git a/hw/xen/xen_pt.c b/hw/xen/xen_pt.c
index 2d33d178ad..a540149639 100644
--- a/hw/xen/xen_pt.c
+++ b/hw/xen/xen_pt.c
@@ -57,7 +57,6 @@
 #include 
 
 #include "hw/pci/pci.h"
-#include "hw/pci/pci_bus.h"
 #include "hw/qdev-properties.h"
 #include "hw/qdev-properties-system.h"
 #include "xen_pt.h"
@@ -951,7 +950,7 @@ void xen_igd_reserve_slot(PCIBus *pci_bus)
 }
 
 XEN_PT_LOG(0, "Reserving PCI slot 2 for IGD\n");
-pci_bus->slot_reserved_mask |= XEN_PCI_IGD_SLOT_MASK;
+pci_bus_set_slot_reserved_mask(pci_bus, XEN_PCI_IGD_SLOT_MASK);
 }
 
 static void xen_igd_clear_slot(DeviceState *qdev, Error **errp)
@@ -971,7 +970,7 @@ static void xen_igd_clear_slot(DeviceState *qdev, Error 
**errp)
 return;
 }
 
-if (!(pci_bus->slot_reserved_mask & XEN_PCI_IGD_SLOT_MASK)) {
+if (!(pci_bus_get_slot_reserved_mask(pci_bus) & XEN_PCI_IGD_SLOT_MASK)) {
 xpdc->pci_qdev_realize(qdev, errp);
 return;
 }
@@ -982,7 +981,7 @@ static void xen_igd_clear_slot(DeviceState *qdev, Error 
**errp)
 s->real_device.dev == XEN_PCI_IGD_DEV &&
 s->real_device.func == XEN_PCI_IGD_FN &&
 s->real_device.vendor_id == PCI_VENDOR_ID_INTEL) {
-pci_bus->slot_reserved_mask &= ~XEN_PCI_IGD_SLOT_MASK;
+pci_bus_clear_slot_reserved_mask(pci_bus, XEN_PCI_IGD_SLOT_MASK);
 XEN_PT_LOG(pci_dev, "Intel IGD found, using slot 2\n");
 }
 xpdc->pci_qdev_realize(qdev, errp);
diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h
index d5a40cd058..935b4b91b4 100644
--- a/include/hw/pci/pci.h
+++ b/include/hw/pci/pci.h
@@ -287,6 +287,9 @@ void pci_bus_irqs(PCIBus *bus, pci_set_irq_fn set_irq,
 void pci_bus_map_irqs(PCIBus *bus, pci_map_irq_fn map_irq);
 void pci_bus_irqs_cleanup(PCIBus *bus);
 int pci_bus_get_irq_level(PCIBus *bus, int irq_num);
+uint32_t pci_bus_get_slot_reserved_mask(PCIBus *bus);
+void pci_bus_set_slot_reserved_mask(PCIBus *bus, uint32_t mask);
+void pci_bus_clear_slot_reserved_mask(PCIBus *bus, uint32_t mask);
 /* 0 <= pin <= 3 0 = INTA, 1 = INTB, 2 = INTC, 3 = INTD */
 static inline int pci_swizzle(int slot, int pin)
 {
-- 
2.39.2

Re: [PATCH v2 0/6] Resolve TYPE_PIIX3_XEN_DEVICE

[Chuck Zmudzinski]

On 3/12/23 5:22 AM, Bernhard Beschow wrote:
> 
> 
> Am 11. März 2023 22:20:29 UTC schrieb Chuck Zmudzinski :
>>On 2/9/2023 4:53 PM, Bernhard Beschow wrote:
>>> Am 1. Februar 2023 08:11:10 UTC schrieb Bernhard Beschow 
>>> :
>>> >
>>> >
>>> >Am 24. Januar 2023 17:07:30 UTC schrieb Bernhard Beschow 
>>> >:
>>> >>
>>> >>
>>> >>Am 24. Januar 2023 16:11:47 UTC schrieb Anthony PERARD 
>>> >>:
>>> >>>On Wed, Jan 18, 2023 at 05:13:03AM -0500, Michael S. Tsirkin wrote:
>>> >>>> On Wed, Jan 04, 2023 at 03:44:31PM +0100, Bernhard Beschow wrote:
>>> >>>> > This series first renders TYPE_PIIX3_XEN_DEVICE redundant and 
>>> >>>> > finally removes
>>> >>>> > it. The motivation is to 1/ decouple PIIX from Xen and 2/ to make 
>>> >>>> > Xen in the PC
>>> >>>> > machine agnostic to the precise southbridge being used. 2/ will 
>>> >>>> > become
>>> >>>> > particularily interesting once PIIX4 becomes usable in the PC 
>>> >>>> > machine, avoiding
>>> >>>> > the "Frankenstein" use of PIIX4_ACPI in PIIX3.
>>> >>>> 
>>> >>>> Looks ok to me.
>>> >>>> Reviewed-by: Michael S. Tsirkin 
>>> >>>> 
>>> >>>> Feel free to merge through Xen tree.
>>> >>>
>>> >>>Hi Bernhard,
>>> >>
>>> >>Hi Anthony,
>>> >>
>>> >>>The series currently doesn't apply on master. And a quick try at
>>> >>>applying the series it is based on also failed. Could you rebase it , or
>>> >>>maybe you would prefer to wait until the other series "Consolidate
>>> >>>PIIX..." is fully applied?
>>> >>
>>> >>Thanks for looking into it!
>>> >>
>>> >>You can get the compilable series from 
>>> >>https://patchew.org/QEMU/20230104144437.27479-1-shen...@gmail.com/ . If 
>>> >>it doesn't work for you let me know, then I can rebase onto master. All 
>>> >>necessary dependencies for the series are upstreamed meanwhile.
>>> >
>>> >Ping
>>>
>>> Ping^2
>>
>>Hi Bernhard,
> 
> Hi Chuck,
> 
>>I took a look at this today to see why it cannot be applied.
> 
> Thanks for looking at it!
> 
>>I can see clearly that
>>all the prerequisite patches have *not* been applied to master yet, so I can
>>understand why Anthony cannot pull this up yet. Specifically, the series that
>>consolidates PIIX3 and PIIX4 south bridges is not yet applied, and that is 
>>one of
>>the prerequisites. I think you said it was reviewed, but it apparently never 
>>got
>>pulled up into master.
> 
> Correct, the PIIX consolidation series isn't merged yet. This series 
> currently depends on it to avoid merge conflicts but doesn't need it 
> otherwise. Back then I anticipated that the consolidation series would land 
> in master soon since it was fully reviewed before this one. But that turned 
> out not to be the case.
> 
> This series depends on some necessary refactoring [1] which I did in the 
> context of PIIX consolidation which is already in master. So this series can 
> easily be rebased onto master and it even simplifies the consolidation series 
> a bit. I'll take this route now and I'll post a v3.

Thanks for posting v3, I was at a loss trying to figure out how to merge the 
30-patch piix
consolidation series into the master branch. I just tested your recent v3 (all 
6 patches)
on top of the current master branch and it works well on my Xen guests, so you 
can keep my
Tested-by tag on that patch series!

Kind regards,

Chuck

Re: [PATCH v2 0/6] Resolve TYPE_PIIX3_XEN_DEVICE

[Chuck Zmudzinski]

On 2/9/2023 4:53 PM, Bernhard Beschow wrote:
> Am 1. Februar 2023 08:11:10 UTC schrieb Bernhard Beschow :
> >
> >
> >Am 24. Januar 2023 17:07:30 UTC schrieb Bernhard Beschow :
> >>
> >>
> >>Am 24. Januar 2023 16:11:47 UTC schrieb Anthony PERARD 
> >>:
> >>>On Wed, Jan 18, 2023 at 05:13:03AM -0500, Michael S. Tsirkin wrote:
>  On Wed, Jan 04, 2023 at 03:44:31PM +0100, Bernhard Beschow wrote:
>  > This series first renders TYPE_PIIX3_XEN_DEVICE redundant and finally 
>  > removes
>  > it. The motivation is to 1/ decouple PIIX from Xen and 2/ to make Xen 
>  > in the PC
>  > machine agnostic to the precise southbridge being used. 2/ will become
>  > particularily interesting once PIIX4 becomes usable in the PC machine, 
>  > avoiding
>  > the "Frankenstein" use of PIIX4_ACPI in PIIX3.
>  
>  Looks ok to me.
>  Reviewed-by: Michael S. Tsirkin 
>  
>  Feel free to merge through Xen tree.
> >>>
> >>>Hi Bernhard,
> >>
> >>Hi Anthony,
> >>
> >>>The series currently doesn't apply on master. And a quick try at
> >>>applying the series it is based on also failed. Could you rebase it , or
> >>>maybe you would prefer to wait until the other series "Consolidate
> >>>PIIX..." is fully applied?
> >>
> >>Thanks for looking into it!
> >>
> >>You can get the compilable series from 
> >>https://patchew.org/QEMU/20230104144437.27479-1-shen...@gmail.com/ . If it 
> >>doesn't work for you let me know, then I can rebase onto master. All 
> >>necessary dependencies for the series are upstreamed meanwhile.
> >
> >Ping
>
> Ping^2

Hi Bernhard,

I took a look at this today to see why it cannot be applied. I can see clearly 
that
all the prerequisite patches have *not* been applied to master yet, so I can
understand why Anthony cannot pull this up yet. Specifically, the series that
consolidates PIIX3 and PIIX4 south bridges is not yet applied, and that is one 
of
the prerequisites. I think you said it was reviewed, but it apparently never got
pulled up into master.

For reference, here is the link to the prerequisite patch set I tested with
this patch set:

https://lore.kernel.org/qemu-devel/20221221170003.2929-1-shen...@gmail.com/

The patch set I tested is a 30-patch series, and I don't know if it has
been partially applied. The title of that patch set is:

This series consolidates the implementations of the PIIX3 and PIIX4 south

So before this patch set to resolve the TYPE_PIIX3_XEN_DEVICE can be
applied, the patch set to consolidate the PIIX3 and PIIX4 south bridges
needs to be applied.

I don't know if the feature freeze means these patches that do not add any
new features still need to wait until the next development cycle.

Kind regards,

Chuck

> >
> >>
> >>Thanks,
> >>Bernhard
> >>>
> >>>Thanks.
> >>>
>  > Testing done:
>  > None, because I don't know how to conduct this properly :(
>  > 
>  > Based-on: <20221221170003.2929-1-shen...@gmail.com>
>  >   "[PATCH v4 00/30] Consolidate PIIX south bridges"
> >>>

[RFC PATCH] libxl/dm: Stop using "xenfv" machine in the Qemu upstream device model

[Chuck Zmudzinski]

The "xenfv" machine is almost a clone of the "pc" machine with accel=xen
in Qemu upstream and is mostly redundant, so with this patch libxl
stops using the "xenfv" machine type in favor of the "pc" machine type
with accel=xen.

Looking at the Qemu upstream code in hw/i386/pc_piix.c, the obvious
differences between the "xenfv" machine and the "pc" machine are:
  - The "xenfv" machine unconditionally adds the xen-platform pci device.
  - The "xenfv" machine uses the igd-passthrough-i440FX host pci device
instead of the i440FX pci device when the guest is configured with
igd-passthru=on.

This patch adds the xen-platform pci device using a command line option
with the "pc" machine type instead of relying on upstream Qemu to add
it as part of the process of building the "xenfv" machine if the
administrator has not explicitly disabled the xen-platform pci device
in the domain configuration file. Therefore, the current behavior of
adding the xen-platform pci device unless the administrator configures
the guest with the xen_platform_pci option set to 0 is retained.

So this patch does not affect guest behavior except as follows:

  - If the guest is configured for Intel igd passthrough, the guest will
no longer be configured with the igd-passthrough-i440FX host pci
device. For this reason, Qemu upstream should be patched so the
igd-passthrough-i440FX host pci device will be used when the guest
is configured for igd-passthru.

  - Live migrations of guests configured to use the "xenfv" machine
when they were created to a host that configures the device model
to use the "pc" machine instead might be affected. For this reason,
it might be appropriate to deprecate the "xenfv" machine in upstream
Qemu and it might be necessary patch upstream Qemu to properly
handle these live migrations.

  - Testing reveals the display resolution can be affected in some
guests by the change from the "xenfv" machine to the "pc" machine
but there is no degredation in performance, just a possible need
to reset the display resolution to the desired value.

Signed-off-by: Chuck Zmudzinski 
---
Sorry about how long this discussion is, but it is helpful to again
bring together all the considerations about this patch and its
alternatives in one place.

I submit this as an RFC patch because some caveats might make this patch
more trouble than it is worth.

The first caveat is complications that might be involved for existing
guests that currently use the "xenfv" machine and are migrated to a host
with this patch that uses the "pc" machine instead.

The second caveat is the effect of the patch on display resolution in
the guest. Details of tests that reveal the display resolution can be
affected by this patch:

Testing existing guests that previously used the "xenfv" machine reveals
a subtle difference that is not obvious from reading the Qemu upstream
code in hw/i386/ppc_piix.c. The tested guests were configured to use the
Qemu stdvga (bochs) emulated video device with a vnc display. When the
guest is started using the "xenfv" machine, the resolution of the
display manager (either gdm3 or lightdm) is 1024x768, but when using
the "pc" machine with the xen accelerator, the resolution of the display
manager (either gdm3 or lightdm) is 1280x800. Also, the display
resolution of a user's graphical login session might be reset to a
default value when changing from the "xenfv" to "pc" machine type, but
the user can easily correct this by resetting the display resolution to
the desired value in the first graphical login session after changing
from the "xenfv" to the "pc" machine type.

On a more positive note, this patch will not introduce much of a
regression for users of the upstream Qemu igd-passthru feature because
that feature is already currently broken in upstream Qemu because
upstream Qemu currently fails to reserve pci slot 2 for the Intel IGD
and, in fact, with this patch, it is actually easier to patch upstream
Qemu to properly support the igd passthrough feature than it is
without this patch because, in that case, a more complicated alternative
patch to upstream Qemu that reserves pci slot 2 for the igd is required
to properly support the igd passthrough feature if we continue to use
the "xenfv" machine type:

http://patchew.org/QEMU/b1b4a21fe9a600b1322742dda55a40e9961daa57.1674346505.git.brchu...@aol.com/

This alternative patch, however, has been reviewed and could be merged
into Qemu anytime at the discretion of the Qemu Xen maintainers. Some of
the "pc" machine maintainers might want an additional patch to Qemu
upstream to be applied to change how upstream Qemu's slot_reserved_mask
works if this alternative patch is merged i

[PATCH] xen/pt: fix igd passthrough for pc machine with xen accelerator

[Chuck Zmudzinski]

Commit 998250e97661 ("xen, gfx passthrough: register host bridge specific
to passthrough") uses the igd-passthrough-i440FX pci host device with
the xenfv machine type and igd-passthru=on, but using it for the pc
machine type, xen accelerator, and igd-passtru=on was omitted from that
commit.

The igd-passthru-i440FX pci host device is also needed for guests
configured with the pc machine type, the xen accelerator, and
igd-passthru=on. Specifically, tests show that not using the igd-specific
pci host device with the Intel igd passed through to the guest results
in slower startup performance and reduced resolution of the display
during startup. This patch fixes this issue.

To simplify the logic that is needed to support both the --enable-xen
and the --disable-xen configure options, introduce the boolean symbol
pc_xen_igd_gfx_pt_enabled() whose value is set appropriately in the
sysemu/xen.h header file as the test to determine whether or not
to use the igd-passthrough-i440FX pci host device instead of the
normal i440FX pci host device.

Fixes: 998250e97661 ("xen, gfx passthrough: register host bridge specific to 
passthrough")
Signed-off-by: Chuck Zmudzinski 
---
This patch is intended to replace or complement a recently proposed
patch that modifies slot_reserved_mask for the xenfv machine with
igd-passthru=on in order to fix the problem of Qemu not reserving slot 2
for the Intel IGD for the xenfv machine type. This patch provides a
simple way to improve Qemu support for the Intel IGD with the xen
accelerator without needing to change how slot_reserved_mask functions.

For reference, the latest version of the patch to fix the xenfv machine
using slot_reserved_mask is at:

https://lore.kernel.org/qemu-devel/b1b4a21fe9a600b1322742dda55a40e9961daa57.1674346505.git.brchu...@aol.com/

Reason for introducing the new boolean pc_xen_igd_gfx_pt_enabled():

It is also possible to use xen_igd_gfx_pt_enabled() directly to check
if the igd-passthru-i440FX pci host device is needed in this patch,
but in that case it would be necessary to implement it in
accel/stubs/xen-stub.c like this:

bool xen_igd_gfx_pt_enabled(void)
{
return false;
}

to cover the case when Qemu is configured with --disable-xen. I thought
it was simpler to introduce the same boolean prefixed with pc_ and
set it to 0 when --disable-xen is the configure option, and that explains
why the proposed patch introduces pc_xen_igd_gfx_pt_enabled() instead of
using xen_igd_gfx_pt_enabled() directly.

Another reason to use pc_xen_igd_gfx_pt_enabled() is to distinguish it
from xen_igd_gfx_pt_enabled() in hw/i386/pc_piix.c, because the use of
xen_igd_gfx_pt_enabled() is guarded by CONFIG_XEN but this patch needs
to place the boolean in a position that is not guarded by CONFIG_XEN.
This approach will simplify any future effort to move the code in
pc_piix.c that is not guarded by CONFIG_XEN to a xen-specific file.

 hw/i386/pc_piix.c| 2 ++
 include/sysemu/xen.h | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c
index df64dd8dcc..fd5b9ae1eb 100644
--- a/hw/i386/pc_piix.c
+++ b/hw/i386/pc_piix.c
@@ -433,6 +433,8 @@ static void pc_xen_hvm_init(MachineState *machine)
 compat(machine); \
 } \
 pc_init1(machine, TYPE_I440FX_PCI_HOST_BRIDGE, \
+ pc_xen_igd_gfx_pt_enabled() ? \
+ TYPE_IGD_PASSTHROUGH_I440FX_PCI_DEVICE : \
  TYPE_I440FX_PCI_DEVICE); \
 } \
 DEFINE_PC_MACHINE(suffix, name, pc_init_##suffix, optionfn)
diff --git a/include/sysemu/xen.h b/include/sysemu/xen.h
index 0ca25697e4..99ae41e619 100644
--- a/include/sysemu/xen.h
+++ b/include/sysemu/xen.h
@@ -23,6 +23,7 @@
 extern bool xen_allowed;
 
 #define xen_enabled()   (xen_allowed)
+#define pc_xen_igd_gfx_pt_enabled()xen_igd_gfx_pt_enabled()
 
 #ifndef CONFIG_USER_ONLY
 void xen_hvm_modified_memory(ram_addr_t start, ram_addr_t length);
@@ -33,6 +34,7 @@ void xen_ram_alloc(ram_addr_t ram_addr, ram_addr_t size,
 #else /* !CONFIG_XEN_IS_POSSIBLE */
 
 #define xen_enabled() 0
+#define pc_xen_igd_gfx_pt_enabled() 0
 #ifndef CONFIG_USER_ONLY
 static inline void xen_hvm_modified_memory(ram_addr_t start, ram_addr_t length)
 {
-- 
2.39.1

Re: [XEN PATCH v2 0/3] Configure qemu upstream correctly by default for igd-passthru

[Chuck Zmudzinski]

On 1/29/23 7:38 PM, Chuck Zmudzinski wrote:
> On 1/25/23 6:19 PM, Chuck Zmudzinski wrote:
>> On 1/25/2023 6:37 AM, Anthony PERARD wrote:
>>> On Tue, Jan 10, 2023 at 02:32:01AM -0500, Chuck Zmudzinski wrote:
>>> > I call attention to the commit message of the first patch which points
>>> > out that using the "pc" machine and adding the xen platform device on
>>> > the qemu upstream command line is not functionally equivalent to using
>>> > the "xenfv" machine which automatically adds the xen platform device
>>> > earlier in the guest creation process. As a result, there is a noticeable
>>> > reduction in the performance of the guest during startup with the "pc"
>>> > machne type even if the xen platform device is added via the qemu
>>> > command line options, although eventually both Linux and Windows guests
>>> > perform equally well once the guest operating system is fully loaded.
>>>
>>> There shouldn't be a difference between "xenfv" machine or using the
>>> "pc" machine while adding the "xen-platform" device, at least with
>>> regards to access to disk or network.
>>>
>>> The first patch of the series is using the "pc" machine without any
>>> "xen-platform" device, so we can't compare startup performance based on
>>> that.
>>>
>>> > Specifically, startup time is longer and neither the grub vga drivers
>>> > nor the windows vga drivers in early startup perform as well when the
>>> > xen platform device is added via the qemu command line instead of being
>>> > added immediately after the other emulated i440fx pci devices when the
>>> > "xenfv" machine type is used.
>>>
>>> The "xen-platform" device is mostly an hint to a guest that they can use
>>> pv-disk and pv-network devices. I don't think it would change anything
>>> with regards to graphics.
>>>
>>> > For example, when using the "pc" machine, which adds the xen platform
>>> > device using a command line option, the Linux guest could not display
>>> > the grub boot menu at the native resolution of the monitor, but with the
>>> > "xenfv" machine, the grub menu is displayed at the full 1920x1080
>>> > native resolution of the monitor for testing. So improved startup
>>> > performance is an advantage for the patch for qemu.
>>>
>>> I've just found out that when doing IGD passthrough, both machine
>>> "xenfv" and "pc" are much more different than I though ... :-(
>>> pc_xen_hvm_init_pci() in QEMU changes the pci-host device, which in
>>> turns copy some informations from the real host bridge.
>>> I guess this new host bridge help when the firmware setup the graphic
>>> for grub.
> 
> Yes, it is needed - see below for the very simple patch to Qemu
> upstream that fixes it for the "pc" machine!
> 
>> 
>> I am surprised it works at all with the "pc" machine, that is, without the
>> TYPE_IGD_PASSTHROUGH_I440FX_PCI_DEVICE that is used in the "xenfv"
>> machine. This only seems to affect the legacy grub vga driver and the legacy
>> Windows vga driver during early boot. Still, I much prefer keeping the 
>> "xenfv"
>> machine for Intel IGD than this workaround of patching libxl to use the "pc"
>> machine.
>> 
>>>
>>> > I also call attention to the last point of the commit message of the
>>> > second patch and the comments for reviewers section of the second patch.
>>> > This approach, as opposed to fixing this in qemu upstream, makes
>>> > maintaining the code in libxl__build_device_model_args_new more
>>> > difficult and therefore increases the chances of problems caused by
>>> > coding errors and typos for users of libxl. So that is another advantage
>>> > of the patch for qemu.
>>>
>>> We would just needs to use a different approach in libxl when generating
>>> the command line. We could probably avoid duplications.
> 
> I was thinking we could also either write a test to verify the correctness
> of the second patch to ensure it generates the correct Qemu command line
> or take the time to verify the second patch's accuracy before committing it.
> 
>>> I was hopping to
>>> have patch series for libxl that would change the machine used to start
>>> using "p

Re: [XEN PATCH v2 0/3] Configure qemu upstream correctly by default for igd-passthru

[Chuck Zmudzinski]

On 1/25/23 6:19 PM, Chuck Zmudzinski wrote:
> On 1/25/2023 6:37 AM, Anthony PERARD wrote:
>> On Tue, Jan 10, 2023 at 02:32:01AM -0500, Chuck Zmudzinski wrote:
>> > I call attention to the commit message of the first patch which points
>> > out that using the "pc" machine and adding the xen platform device on
>> > the qemu upstream command line is not functionally equivalent to using
>> > the "xenfv" machine which automatically adds the xen platform device
>> > earlier in the guest creation process. As a result, there is a noticeable
>> > reduction in the performance of the guest during startup with the "pc"
>> > machne type even if the xen platform device is added via the qemu
>> > command line options, although eventually both Linux and Windows guests
>> > perform equally well once the guest operating system is fully loaded.
>>
>> There shouldn't be a difference between "xenfv" machine or using the
>> "pc" machine while adding the "xen-platform" device, at least with
>> regards to access to disk or network.
>>
>> The first patch of the series is using the "pc" machine without any
>> "xen-platform" device, so we can't compare startup performance based on
>> that.
>>
>> > Specifically, startup time is longer and neither the grub vga drivers
>> > nor the windows vga drivers in early startup perform as well when the
>> > xen platform device is added via the qemu command line instead of being
>> > added immediately after the other emulated i440fx pci devices when the
>> > "xenfv" machine type is used.
>>
>> The "xen-platform" device is mostly an hint to a guest that they can use
>> pv-disk and pv-network devices. I don't think it would change anything
>> with regards to graphics.
>>
>> > For example, when using the "pc" machine, which adds the xen platform
>> > device using a command line option, the Linux guest could not display
>> > the grub boot menu at the native resolution of the monitor, but with the
>> > "xenfv" machine, the grub menu is displayed at the full 1920x1080
>> > native resolution of the monitor for testing. So improved startup
>> > performance is an advantage for the patch for qemu.
>>
>> I've just found out that when doing IGD passthrough, both machine
>> "xenfv" and "pc" are much more different than I though ... :-(
>> pc_xen_hvm_init_pci() in QEMU changes the pci-host device, which in
>> turns copy some informations from the real host bridge.
>> I guess this new host bridge help when the firmware setup the graphic
>> for grub.

Yes, it is needed - see below for the very simple patch to Qemu
upstream that fixes it for the "pc" machine!

> 
> I am surprised it works at all with the "pc" machine, that is, without the
> TYPE_IGD_PASSTHROUGH_I440FX_PCI_DEVICE that is used in the "xenfv"
> machine. This only seems to affect the legacy grub vga driver and the legacy
> Windows vga driver during early boot. Still, I much prefer keeping the "xenfv"
> machine for Intel IGD than this workaround of patching libxl to use the "pc"
> machine.
> 
>>
>> > I also call attention to the last point of the commit message of the
>> > second patch and the comments for reviewers section of the second patch.
>> > This approach, as opposed to fixing this in qemu upstream, makes
>> > maintaining the code in libxl__build_device_model_args_new more
>> > difficult and therefore increases the chances of problems caused by
>> > coding errors and typos for users of libxl. So that is another advantage
>> > of the patch for qemu.
>>
>> We would just needs to use a different approach in libxl when generating
>> the command line. We could probably avoid duplications.

I was thinking we could also either write a test to verify the correctness
of the second patch to ensure it generates the correct Qemu command line
or take the time to verify the second patch's accuracy before committing it.

>> I was hopping to
>> have patch series for libxl that would change the machine used to start
>> using "pc" instead of "xenfv" for all configurations, but based on the
>> point above (IGD specific change to "xenfv"), then I guess we can't
>> really do anything from libxl to fix IGD passthrough.
> 
> We could switch to the "pc" machine, but we would need to patch
> qemu also so the "pc" machine uses the special device the "xenfv"
> ma

Re: [XEN PATCH v2 0/3] Configure qemu upstream correctly by default for igd-passthru

[Chuck Zmudzinski]

On 1/25/2023 6:37 AM, Anthony PERARD wrote:
> On Tue, Jan 10, 2023 at 02:32:01AM -0500, Chuck Zmudzinski wrote:
> > I call attention to the commit message of the first patch which points
> > out that using the "pc" machine and adding the xen platform device on
> > the qemu upstream command line is not functionally equivalent to using
> > the "xenfv" machine which automatically adds the xen platform device
> > earlier in the guest creation process. As a result, there is a noticeable
> > reduction in the performance of the guest during startup with the "pc"
> > machne type even if the xen platform device is added via the qemu
> > command line options, although eventually both Linux and Windows guests
> > perform equally well once the guest operating system is fully loaded.
>
> There shouldn't be a difference between "xenfv" machine or using the
> "pc" machine while adding the "xen-platform" device, at least with
> regards to access to disk or network.
>
> The first patch of the series is using the "pc" machine without any
> "xen-platform" device, so we can't compare startup performance based on
> that.
>
> > Specifically, startup time is longer and neither the grub vga drivers
> > nor the windows vga drivers in early startup perform as well when the
> > xen platform device is added via the qemu command line instead of being
> > added immediately after the other emulated i440fx pci devices when the
> > "xenfv" machine type is used.
>
> The "xen-platform" device is mostly an hint to a guest that they can use
> pv-disk and pv-network devices. I don't think it would change anything
> with regards to graphics.
>
> > For example, when using the "pc" machine, which adds the xen platform
> > device using a command line option, the Linux guest could not display
> > the grub boot menu at the native resolution of the monitor, but with the
> > "xenfv" machine, the grub menu is displayed at the full 1920x1080
> > native resolution of the monitor for testing. So improved startup
> > performance is an advantage for the patch for qemu.
>
> I've just found out that when doing IGD passthrough, both machine
> "xenfv" and "pc" are much more different than I though ... :-(
> pc_xen_hvm_init_pci() in QEMU changes the pci-host device, which in
> turns copy some informations from the real host bridge.
> I guess this new host bridge help when the firmware setup the graphic
> for grub.

I am surprised it works at all with the "pc" machine, that is, without the
TYPE_IGD_PASSTHROUGH_I440FX_PCI_DEVICE that is used in the "xenfv"
machine. This only seems to affect the legacy grub vga driver and the legacy
Windows vga driver during early boot. Still, I much prefer keeping the "xenfv"
machine for Intel IGD than this workaround of patching libxl to use the "pc"
machine.

>
> > I also call attention to the last point of the commit message of the
> > second patch and the comments for reviewers section of the second patch.
> > This approach, as opposed to fixing this in qemu upstream, makes
> > maintaining the code in libxl__build_device_model_args_new more
> > difficult and therefore increases the chances of problems caused by
> > coding errors and typos for users of libxl. So that is another advantage
> > of the patch for qemu.
>
> We would just needs to use a different approach in libxl when generating
> the command line. We could probably avoid duplications. I was hopping to
> have patch series for libxl that would change the machine used to start
> using "pc" instead of "xenfv" for all configurations, but based on the
> point above (IGD specific change to "xenfv"), then I guess we can't
> really do anything from libxl to fix IGD passthrough.

We could switch to the "pc" machine, but we would need to patch
qemu also so the "pc" machine uses the special device the "xenfv"
machine uses (TYPE_IGD_PASSTHROUGH_I440FX_PCI_DEVICE).
So it is simpler to just use the other patch to qemu and not patch
libxl at all to fix this.

>
> > OTOH, fixing this in qemu causes newer qemu versions to behave
> > differently than previous versions of qemu, which the qemu community
> > does not like, although they seem OK with the other patch since it only
> > affects qemu "xenfv" machine types, but they do not want the patch to
> > affect toolstacks like libvirt that do not use qemu upstream's
> > autoconfiguration options as much as libxl does, and, of course, libvirt
> > can manage qemu "xenfv" machines so exising 

Re: [XEN PATCH v2 0/3] Configure qemu upstream correctly by default for igd-passthru

[Chuck Zmudzinski]

On 1/25/2023 6:37 AM, Anthony PERARD wrote:
> On Tue, Jan 10, 2023 at 02:32:01AM -0500, Chuck Zmudzinski wrote:
> > I call attention to the commit message of the first patch which points
> > out that using the "pc" machine and adding the xen platform device on
> > the qemu upstream command line is not functionally equivalent to using
> > the "xenfv" machine which automatically adds the xen platform device
> > earlier in the guest creation process. As a result, there is a noticeable
> > reduction in the performance of the guest during startup with the "pc"
> > machne type even if the xen platform device is added via the qemu
> > command line options, although eventually both Linux and Windows guests
> > perform equally well once the guest operating system is fully loaded.
>
> There shouldn't be a difference between "xenfv" machine or using the
> "pc" machine while adding the "xen-platform" device, at least with
> regards to access to disk or network.
>
> The first patch of the series is using the "pc" machine without any
> "xen-platform" device, so we can't compare startup performance based on
> that.
>
> > Specifically, startup time is longer and neither the grub vga drivers
> > nor the windows vga drivers in early startup perform as well when the
> > xen platform device is added via the qemu command line instead of being
> > added immediately after the other emulated i440fx pci devices when the
> > "xenfv" machine type is used.
>
> The "xen-platform" device is mostly an hint to a guest that they can use
> pv-disk and pv-network devices. I don't think it would change anything
> with regards to graphics.
>
> > For example, when using the "pc" machine, which adds the xen platform
> > device using a command line option, the Linux guest could not display
> > the grub boot menu at the native resolution of the monitor, but with the
> > "xenfv" machine, the grub menu is displayed at the full 1920x1080
> > native resolution of the monitor for testing. So improved startup
> > performance is an advantage for the patch for qemu.
>
> I've just found out that when doing IGD passthrough, both machine
> "xenfv" and "pc" are much more different than I though ... :-(
> pc_xen_hvm_init_pci() in QEMU changes the pci-host device, which in
> turns copy some informations from the real host bridge.
> I guess this new host bridge help when the firmware setup the graphic
> for grub.
>
> > I also call attention to the last point of the commit message of the
> > second patch and the comments for reviewers section of the second patch.
> > This approach, as opposed to fixing this in qemu upstream, makes
> > maintaining the code in libxl__build_device_model_args_new more
> > difficult and therefore increases the chances of problems caused by
> > coding errors and typos for users of libxl. So that is another advantage
> > of the patch for qemu.
>
> We would just needs to use a different approach in libxl when generating
> the command line. We could probably avoid duplications. I was hopping to
> have patch series for libxl that would change the machine used to start
> using "pc" instead of "xenfv" for all configurations, but based on the
> point above (IGD specific change to "xenfv"), then I guess we can't
> really do anything from libxl to fix IGD passthrough.
>
> > OTOH, fixing this in qemu causes newer qemu versions to behave
> > differently than previous versions of qemu, which the qemu community
> > does not like, although they seem OK with the other patch since it only
> > affects qemu "xenfv" machine types, but they do not want the patch to
> > affect toolstacks like libvirt that do not use qemu upstream's
> > autoconfiguration options as much as libxl does, and, of course, libvirt
> > can manage qemu "xenfv" machines so exising "xenfv" guests configured
> > manually by libvirt could be adversely affected by the patch to qemu,
> > but only if those same guests are also configured for igd-passthrough,
> > which is likely a very small number of possibly affected libvirt users
> > of qemu.
> > 
> > A year or two ago I tried to configure guests for pci passthrough on xen
> > using libvirt's tool to convert a libxl xl.cfg file to libvirt xml. It
> > could not convert an xl.cfg file with a configuration item
> > pci = [ "PCI_SPEC_STRING", "PCI_SPEC_STRING", ...] for pci passthrough.
> > So it is unlikely there are any users out there using libvirt to
> > config

Re: [PATCH v11] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

On 1/22/23 3:40 AM, Michael S. Tsirkin wrote:
> On Sat, Jan 21, 2023 at 07:57:02PM -0500, Chuck Zmudzinski wrote:
>> Intel specifies that the Intel IGD must occupy slot 2 on the PCI bus,
>> as noted in docs/igd-assign.txt in the Qemu source code.
>> 
>> Currently, when the xl toolstack is used to configure a Xen HVM guest with
>> Intel IGD passthrough to the guest with the Qemu upstream device model,
>> a Qemu emulated PCI device will occupy slot 2 and the Intel IGD will occupy
>> a different slot. This problem often prevents the guest from booting.
>> 
>> The only available workarounds are not good: Configure Xen HVM guests to
>> use the old and no longer maintained Qemu traditional device model
>> available from xenbits.xen.org which does reserve slot 2 for the Intel
>> IGD or use the "pc" machine type instead of the "xenfv" machine type and
>> add the xen platform device at slot 3 using a command line option
>> instead of patching qemu to fix the "xenfv" machine type directly. The
>> second workaround causes some degredation in startup performance such as
>> a longer boot time and reduced resolution of the grub menu that is
>> displayed on the monitor. This patch avoids that reduced startup
>> performance when using the Qemu upstream device model for Xen HVM guests
>> configured with the igd-passthru=on option.
>> 
>> To implement this feature in the Qemu upstream device model for Xen HVM
>> guests, introduce the following new functions, types, and macros:
>> 
>> * XEN_PT_DEVICE_CLASS declaration, based on the existing TYPE_XEN_PT_DEVICE
>> * XEN_PT_DEVICE_GET_CLASS macro helper function for XEN_PT_DEVICE_CLASS
>> * typedef XenPTQdevRealize function pointer
>> * XEN_PCI_IGD_SLOT_MASK, the value of slot_reserved_mask to reserve slot 2
>> * xen_igd_reserve_slot and xen_igd_clear_slot functions
>> 
>> Michael Tsirkin:
>> * Introduce XEN_PCI_IGD_DOMAIN, XEN_PCI_IGD_BUS, XEN_PCI_IGD_DEV, and
>>   XEN_PCI_IGD_FN - use them to compute the value of XEN_PCI_IGD_SLOT_MASK
>> 
>> The new xen_igd_reserve_slot function uses the existing slot_reserved_mask
>> member of PCIBus to reserve PCI slot 2 for Xen HVM guests configured using
>> the xl toolstack with the gfx_passthru option enabled, which sets the
>> igd-passthru=on option to Qemu for the Xen HVM machine type.
>> 
>> The new xen_igd_reserve_slot function also needs to be implemented in
>> hw/xen/xen_pt_stub.c to prevent FTBFS during the link stage for the case
>> when Qemu is configured with --enable-xen and --disable-xen-pci-passthrough,
>> in which case it does nothing.
>> 
>> The new xen_igd_clear_slot function overrides qdev->realize of the parent
>> PCI device class to enable the Intel IGD to occupy slot 2 on the PCI bus
>> since slot 2 was reserved by xen_igd_reserve_slot when the PCI bus was
>> created in hw/i386/pc_piix.c for the case when igd-passthru=on.
>> 
>> Move the call to xen_host_pci_device_get, and the associated error
>> handling, from xen_pt_realize to the new xen_igd_clear_slot function to
>> initialize the device class and vendor values which enables the checks for
>> the Intel IGD to succeed. The verification that the host device is an
>> Intel IGD to be passed through is done by checking the domain, bus, slot,
>> and function values as well as by checking that gfx_passthru is enabled,
>> the device class is VGA, and the device vendor in Intel.
>> 
>> Signed-off-by: Chuck Zmudzinski 
>> ---
>> Notes that might be helpful to reviewers of patched code in hw/xen:
>> 
>> The new functions and types are based on recommendations from Qemu docs:
>> https://qemu.readthedocs.io/en/latest/devel/qom.html
>> 
>> Notes that might be helpful to reviewers of patched code in hw/i386:
>> 
>> The small patch to hw/i386/pc_piix.c is protected by CONFIG_XEN so it does
>> not affect builds that do not have CONFIG_XEN defined.
>> 
>> xen_igd_gfx_pt_enabled() in the patched hw/i386/pc_piix.c file is an
>> existing function that is only true when Qemu is built with
>> xen-pci-passthrough enabled and the administrator has configured the Xen
>> HVM guest with Qemu's igd-passthru=on option.
>> 
>> v2: Remove From:  tag at top of commit message
>> 
>> v3: Changed the test for the Intel IGD in xen_igd_clear_slot:
>> 
>> if (is_igd_vga_passthrough(>real_device) &&
>> (s->real_device.vendor_id == PCI_VENDOR_ID_INTEL)) {
>> 
>> is changed to
>> 
>> if (xen_igd_gfx_pt_enabled() && (s->hostad

Re: [PATCH v6] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

On 1/21/23 1:06 PM, Chuck Zmudzinski wrote:
> On 1/6/2023 9:03 AM, Anthony PERARD wrote:
>> On Sun, Jan 01, 2023 at 06:52:03PM -0500, Chuck Zmudzinski wrote:
>> > Intel specifies that the Intel IGD must occupy slot 2 on the PCI bus,
>> > as noted in docs/igd-assign.txt in the Qemu source code.
>> > 
>> > Currently, when the xl toolstack is used to configure a Xen HVM guest with
>> > Intel IGD passthrough to the guest with the Qemu upstream device model,
>> > a Qemu emulated PCI device will occupy slot 2 and the Intel IGD will occupy
>> > a different slot. This problem often prevents the guest from booting.
>> > 
>> > The only available workaround is not good: Configure Xen HVM guests to use
>> > the old and no longer maintained Qemu traditional device model available
>> > from xenbits.xen.org which does reserve slot 2 for the Intel IGD.
>> > 
>> > To implement this feature in the Qemu upstream device model for Xen HVM
>> > guests, introduce the following new functions, types, and macros:
>> > 
>> > * XEN_PT_DEVICE_CLASS declaration, based on the existing TYPE_XEN_PT_DEVICE
>> > * XEN_PT_DEVICE_GET_CLASS macro helper function for XEN_PT_DEVICE_CLASS
>> > * typedef XenPTQdevRealize function pointer
>> > * XEN_PCI_IGD_SLOT_MASK, the value of slot_reserved_mask to reserve slot 2
>> > * xen_igd_reserve_slot and xen_igd_clear_slot functions
>> > 
>> > The new xen_igd_reserve_slot function uses the existing slot_reserved_mask
>> > member of PCIBus to reserve PCI slot 2 for Xen HVM guests configured using
>> > the xl toolstack with the gfx_passthru option enabled, which sets the
>> > igd-passthru=on option to Qemu for the Xen HVM machine type.
>> > 
>> > The new xen_igd_reserve_slot function also needs to be implemented in
>> > hw/xen/xen_pt_stub.c to prevent FTBFS during the link stage for the case
>> > when Qemu is configured with --enable-xen and 
>> > --disable-xen-pci-passthrough,
>> > in which case it does nothing.
>> > 
>> > The new xen_igd_clear_slot function overrides qdev->realize of the parent
>> > PCI device class to enable the Intel IGD to occupy slot 2 on the PCI bus
>> > since slot 2 was reserved by xen_igd_reserve_slot when the PCI bus was
>> > created in hw/i386/pc_piix.c for the case when igd-passthru=on.
>> > 
>> > Move the call to xen_host_pci_device_get, and the associated error
>> > handling, from xen_pt_realize to the new xen_igd_clear_slot function to
>> > initialize the device class and vendor values which enables the checks for
>> > the Intel IGD to succeed. The verification that the host device is an
>> > Intel IGD to be passed through is done by checking the domain, bus, slot,
>> > and function values as well as by checking that gfx_passthru is enabled,
>> > the device class is VGA, and the device vendor in Intel.
>> > 
>> > Signed-off-by: Chuck Zmudzinski 
>>
>>
>> This patch looks good enough. It only changes the "xenfv" machine so it
>> doesn't prevent a proper fix to be done in the toolstack libxl.
>>
>> The change in xen_pci_passthrough_class_init() to try to run some code
>> before pci_qdev_realize() could potentially break in the future due to
>> been uncommon but hopefully that will be ok.
>>
>> So if no work to fix libxl appear soon, I'm ok with this patch:
>>
>> Reviewed-by: Anthony PERARD 
>>
>> Thanks,
>>
> 
> Hi Anthony,
> 
> If you have been following this patch it is now at v10. Since there is
> another approach of patching libxl by using the "pc" machine instead of
> patching Qemu to fix the "xenfv" machine and there have been other
> changes, I did not include your Reviewed-by tag in the later versions.
> 
> I presume you are not interested in dealing with the technical debt
> of patching libxl as proposed by this patch to libxl:
> 
> https://lore.kernel.org/xen-devel/20230110073201.mdUvSjy1vKtxPriqMQuWAxIjQzf1eAqIlZgal1u3GBI@z/
> 
> because it would be more difficult to maintain and result in reduced
> startup performance with the Intel IGD than by patching Qemu and
> fixing the "xenfv" machine type with the Intel IGD directly.
> 
> So are you OK with v10 of this patch? If so, you can add your Reviewed-by
> again to v10. The v10 has several changes since v6 as requested by other
> reviewers (Michael, Stefano, Igor).
> 
> The v10 of the patch is here:
> 
> https://lore.kernel.org/qemu-devel/d473914c4d2dc38ae87dca4b898d75b44751c9cb.1674297794.git.brchu...@aol.com/
> 
> Thanks,
> 
> Chuck

Sorry to bother you again, Anthony, but no one noticed a style
mistake that has been present in the past few versions. The
v11 fixes that without making any other changes since v10, so
if you want to add your Reviewed-by to the most recent version,
here it is (v11) (you should also have it in your Inbox):

https://lore.kernel.org/qemu-devel/b1b4a21fe9a600b1322742dda55a40e9961daa57.1674346505.git.brchu...@aol.com/

Thanks,

Chuck

[PATCH v11] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

Intel specifies that the Intel IGD must occupy slot 2 on the PCI bus,
as noted in docs/igd-assign.txt in the Qemu source code.

Currently, when the xl toolstack is used to configure a Xen HVM guest with
Intel IGD passthrough to the guest with the Qemu upstream device model,
a Qemu emulated PCI device will occupy slot 2 and the Intel IGD will occupy
a different slot. This problem often prevents the guest from booting.

The only available workarounds are not good: Configure Xen HVM guests to
use the old and no longer maintained Qemu traditional device model
available from xenbits.xen.org which does reserve slot 2 for the Intel
IGD or use the "pc" machine type instead of the "xenfv" machine type and
add the xen platform device at slot 3 using a command line option
instead of patching qemu to fix the "xenfv" machine type directly. The
second workaround causes some degredation in startup performance such as
a longer boot time and reduced resolution of the grub menu that is
displayed on the monitor. This patch avoids that reduced startup
performance when using the Qemu upstream device model for Xen HVM guests
configured with the igd-passthru=on option.

To implement this feature in the Qemu upstream device model for Xen HVM
guests, introduce the following new functions, types, and macros:

* XEN_PT_DEVICE_CLASS declaration, based on the existing TYPE_XEN_PT_DEVICE
* XEN_PT_DEVICE_GET_CLASS macro helper function for XEN_PT_DEVICE_CLASS
* typedef XenPTQdevRealize function pointer
* XEN_PCI_IGD_SLOT_MASK, the value of slot_reserved_mask to reserve slot 2
* xen_igd_reserve_slot and xen_igd_clear_slot functions

Michael Tsirkin:
* Introduce XEN_PCI_IGD_DOMAIN, XEN_PCI_IGD_BUS, XEN_PCI_IGD_DEV, and
  XEN_PCI_IGD_FN - use them to compute the value of XEN_PCI_IGD_SLOT_MASK

The new xen_igd_reserve_slot function uses the existing slot_reserved_mask
member of PCIBus to reserve PCI slot 2 for Xen HVM guests configured using
the xl toolstack with the gfx_passthru option enabled, which sets the
igd-passthru=on option to Qemu for the Xen HVM machine type.

The new xen_igd_reserve_slot function also needs to be implemented in
hw/xen/xen_pt_stub.c to prevent FTBFS during the link stage for the case
when Qemu is configured with --enable-xen and --disable-xen-pci-passthrough,
in which case it does nothing.

The new xen_igd_clear_slot function overrides qdev->realize of the parent
PCI device class to enable the Intel IGD to occupy slot 2 on the PCI bus
since slot 2 was reserved by xen_igd_reserve_slot when the PCI bus was
created in hw/i386/pc_piix.c for the case when igd-passthru=on.

Move the call to xen_host_pci_device_get, and the associated error
handling, from xen_pt_realize to the new xen_igd_clear_slot function to
initialize the device class and vendor values which enables the checks for
the Intel IGD to succeed. The verification that the host device is an
Intel IGD to be passed through is done by checking the domain, bus, slot,
and function values as well as by checking that gfx_passthru is enabled,
the device class is VGA, and the device vendor in Intel.

Signed-off-by: Chuck Zmudzinski 
---
Notes that might be helpful to reviewers of patched code in hw/xen:

The new functions and types are based on recommendations from Qemu docs:
https://qemu.readthedocs.io/en/latest/devel/qom.html

Notes that might be helpful to reviewers of patched code in hw/i386:

The small patch to hw/i386/pc_piix.c is protected by CONFIG_XEN so it does
not affect builds that do not have CONFIG_XEN defined.

xen_igd_gfx_pt_enabled() in the patched hw/i386/pc_piix.c file is an
existing function that is only true when Qemu is built with
xen-pci-passthrough enabled and the administrator has configured the Xen
HVM guest with Qemu's igd-passthru=on option.

v2: Remove From:  tag at top of commit message

v3: Changed the test for the Intel IGD in xen_igd_clear_slot:

if (is_igd_vga_passthrough(>real_device) &&
(s->real_device.vendor_id == PCI_VENDOR_ID_INTEL)) {

is changed to

if (xen_igd_gfx_pt_enabled() && (s->hostaddr.slot == 2)
&& (s->hostaddr.function == 0)) {

I hoped that I could use the test in v2, since it matches the
other tests for the Intel IGD in Qemu and Xen, but those tests
do not work because the necessary data structures are not set with
their values yet. So instead use the test that the administrator
has enabled gfx_passthru and the device address on the host is
02.0. This test does detect the Intel IGD correctly.

v4: Use brchu...@aol.com instead of brchu...@netscape.net for the author's
email address to match the address used by the same author in commits
be9c61da and c0e86b76

Change variable for XEN_PT_DEVICE_CLASS: xptc changed to xpdc

v5: The patch of xen_pt.c was re-worked to allow a more consistent test
for the Intel IGD that uses the same criteria as in other places.
This involv

Re: [PATCH v6] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

On 1/6/2023 9:03 AM, Anthony PERARD wrote:
> On Sun, Jan 01, 2023 at 06:52:03PM -0500, Chuck Zmudzinski wrote:
> > Intel specifies that the Intel IGD must occupy slot 2 on the PCI bus,
> > as noted in docs/igd-assign.txt in the Qemu source code.
> > 
> > Currently, when the xl toolstack is used to configure a Xen HVM guest with
> > Intel IGD passthrough to the guest with the Qemu upstream device model,
> > a Qemu emulated PCI device will occupy slot 2 and the Intel IGD will occupy
> > a different slot. This problem often prevents the guest from booting.
> > 
> > The only available workaround is not good: Configure Xen HVM guests to use
> > the old and no longer maintained Qemu traditional device model available
> > from xenbits.xen.org which does reserve slot 2 for the Intel IGD.
> > 
> > To implement this feature in the Qemu upstream device model for Xen HVM
> > guests, introduce the following new functions, types, and macros:
> > 
> > * XEN_PT_DEVICE_CLASS declaration, based on the existing TYPE_XEN_PT_DEVICE
> > * XEN_PT_DEVICE_GET_CLASS macro helper function for XEN_PT_DEVICE_CLASS
> > * typedef XenPTQdevRealize function pointer
> > * XEN_PCI_IGD_SLOT_MASK, the value of slot_reserved_mask to reserve slot 2
> > * xen_igd_reserve_slot and xen_igd_clear_slot functions
> > 
> > The new xen_igd_reserve_slot function uses the existing slot_reserved_mask
> > member of PCIBus to reserve PCI slot 2 for Xen HVM guests configured using
> > the xl toolstack with the gfx_passthru option enabled, which sets the
> > igd-passthru=on option to Qemu for the Xen HVM machine type.
> > 
> > The new xen_igd_reserve_slot function also needs to be implemented in
> > hw/xen/xen_pt_stub.c to prevent FTBFS during the link stage for the case
> > when Qemu is configured with --enable-xen and --disable-xen-pci-passthrough,
> > in which case it does nothing.
> > 
> > The new xen_igd_clear_slot function overrides qdev->realize of the parent
> > PCI device class to enable the Intel IGD to occupy slot 2 on the PCI bus
> > since slot 2 was reserved by xen_igd_reserve_slot when the PCI bus was
> > created in hw/i386/pc_piix.c for the case when igd-passthru=on.
> > 
> > Move the call to xen_host_pci_device_get, and the associated error
> > handling, from xen_pt_realize to the new xen_igd_clear_slot function to
> > initialize the device class and vendor values which enables the checks for
> > the Intel IGD to succeed. The verification that the host device is an
> > Intel IGD to be passed through is done by checking the domain, bus, slot,
> > and function values as well as by checking that gfx_passthru is enabled,
> > the device class is VGA, and the device vendor in Intel.
> > 
> > Signed-off-by: Chuck Zmudzinski 
>
>
> This patch looks good enough. It only changes the "xenfv" machine so it
> doesn't prevent a proper fix to be done in the toolstack libxl.
>
> The change in xen_pci_passthrough_class_init() to try to run some code
> before pci_qdev_realize() could potentially break in the future due to
> been uncommon but hopefully that will be ok.
>
> So if no work to fix libxl appear soon, I'm ok with this patch:
>
> Reviewed-by: Anthony PERARD 
>
> Thanks,
>

Hi Anthony,

If you have been following this patch it is now at v10. Since there is
another approach of patching libxl by using the "pc" machine instead of
patching Qemu to fix the "xenfv" machine and there have been other
changes, I did not include your Reviewed-by tag in the later versions.

I presume you are not interested in dealing with the technical debt
of patching libxl as proposed by this patch to libxl:

https://lore.kernel.org/xen-devel/20230110073201.mdUvSjy1vKtxPriqMQuWAxIjQzf1eAqIlZgal1u3GBI@z/

because it would be more difficult to maintain and result in reduced
startup performance with the Intel IGD than by patching Qemu and
fixing the "xenfv" machine type with the Intel IGD directly.

So are you OK with v10 of this patch? If so, you can add your Reviewed-by
again to v10. The v10 has several changes since v6 as requested by other
reviewers (Michael, Stefano, Igor).

The v10 of the patch is here:

https://lore.kernel.org/qemu-devel/d473914c4d2dc38ae87dca4b898d75b44751c9cb.1674297794.git.brchu...@aol.com/

Thanks,

Chuck

Re: [PATCH v9] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

On 1/20/2023 4:34 PM, Stefano Stabellini wrote:
> On Sat, 14 Jan 2023, Chuck Zmudzinski wrote:
> > Intel specifies that the Intel IGD must occupy slot 2 on the PCI bus,
> > as noted in docs/igd-assign.txt in the Qemu source code.
> > 
> > Currently, when the xl toolstack is used to configure a Xen HVM guest with
> > Intel IGD passthrough to the guest with the Qemu upstream device model,
> > a Qemu emulated PCI device will occupy slot 2 and the Intel IGD will occupy
> > a different slot. This problem often prevents the guest from booting.
> > 
> > The only available workaround is not good: Configure Xen HVM guests to use
> > the old and no longer maintained Qemu traditional device model available
> > from xenbits.xen.org which does reserve slot 2 for the Intel IGD.
> > 
> > To implement this feature in the Qemu upstream device model for Xen HVM
> > guests, introduce the following new functions, types, and macros:
> > 
> > * XEN_PT_DEVICE_CLASS declaration, based on the existing TYPE_XEN_PT_DEVICE
> > * XEN_PT_DEVICE_GET_CLASS macro helper function for XEN_PT_DEVICE_CLASS
> > * typedef XenPTQdevRealize function pointer
> > * XEN_PCI_IGD_SLOT_MASK, the value of slot_reserved_mask to reserve slot 2
> > * xen_igd_reserve_slot and xen_igd_clear_slot functions
> > 
> > Michael Tsirkin:
> > * Introduce XEN_PCI_IGD_DOMAIN, XEN_PCI_IGD_BUS, XEN_PCI_IGD_DEV, and
> >   XEN_PCI_IGD_FN - use them to compute the value of XEN_PCI_IGD_SLOT_MASK
> > 
> > The new xen_igd_reserve_slot function uses the existing slot_reserved_mask
> > member of PCIBus to reserve PCI slot 2 for Xen HVM guests configured using
> > the xl toolstack with the gfx_passthru option enabled, which sets the
> > igd-passthru=on option to Qemu for the Xen HVM machine type.
> > 
> > The new xen_igd_reserve_slot function also needs to be implemented in
> > hw/xen/xen_pt_stub.c to prevent FTBFS during the link stage for the case
> > when Qemu is configured with --enable-xen and --disable-xen-pci-passthrough,
> > in which case it does nothing.
> > 
> > The new xen_igd_clear_slot function overrides qdev->realize of the parent
> > PCI device class to enable the Intel IGD to occupy slot 2 on the PCI bus
> > since slot 2 was reserved by xen_igd_reserve_slot when the PCI bus was
> > created in hw/i386/pc_piix.c for the case when igd-passthru=on.
> > 
> > Move the call to xen_host_pci_device_get, and the associated error
> > handling, from xen_pt_realize to the new xen_igd_clear_slot function to
> > initialize the device class and vendor values which enables the checks for
> > the Intel IGD to succeed. The verification that the host device is an
> > Intel IGD to be passed through is done by checking the domain, bus, slot,
> > and function values as well as by checking that gfx_passthru is enabled,
> > the device class is VGA, and the device vendor in Intel.
> > 
> > Signed-off-by: Chuck Zmudzinski 
>
> Hi Chuck,
>
> The approach looks OK in principle to me. I only have one question: for
> other PCI devices (not Intel IGD), where is xen_host_pci_device_get
> called now?
>
> It looks like that xen_igd_reserve_slot would return without setting
> slot_reserved_mask, hence xen_igd_clear_slot would also return without
> calling xen_host_pci_device_get. And xen_pt_realize doesn't call
> xen_host_pci_device_get any longer.
>
> Am I missing something?

Thanks for catching this. With v9 guest creation fails when the bit in
slot_reserved_mask that reserves slot 2 is not set.

It fails because v9 not only fails to call xen_host_pci_device_get when the
bit in slot_reserved_mask that reserves slot 2 is not set, it also does not
call xpdc->pci_qdev_realize either. So I uploaded v10 to fix that here:

https://lore.kernel.org/qemu-devel/d473914c4d2dc38ae87dca4b898d75b44751c9cb.1674297794.git.brchu...@aol.com/

Tests with v10 show it is now working for all cases.

Thanks,

Chuck

[PATCH v10] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

Intel specifies that the Intel IGD must occupy slot 2 on the PCI bus,
as noted in docs/igd-assign.txt in the Qemu source code.

Currently, when the xl toolstack is used to configure a Xen HVM guest with
Intel IGD passthrough to the guest with the Qemu upstream device model,
a Qemu emulated PCI device will occupy slot 2 and the Intel IGD will occupy
a different slot. This problem often prevents the guest from booting.

The only available workarounds are not good: Configure Xen HVM guests to
use the old and no longer maintained Qemu traditional device model
available from xenbits.xen.org which does reserve slot 2 for the Intel
IGD or use the "pc" machine type instead of the "xenfv" machine type and
add the xen platform device at slot 3 using a command line option
instead of patching qemu to fix the "xenfv" machine type directly. The
second workaround causes some degredation in startup performance such as
a longer boot time and reduced resolution of the grub menu that is
displayed on the monitor. This patch avoids that reduced startup
performance when using the Qemu upstream device model for Xen HVM guests
configured with the igd-passthru=on option.

To implement this feature in the Qemu upstream device model for Xen HVM
guests, introduce the following new functions, types, and macros:

* XEN_PT_DEVICE_CLASS declaration, based on the existing TYPE_XEN_PT_DEVICE
* XEN_PT_DEVICE_GET_CLASS macro helper function for XEN_PT_DEVICE_CLASS
* typedef XenPTQdevRealize function pointer
* XEN_PCI_IGD_SLOT_MASK, the value of slot_reserved_mask to reserve slot 2
* xen_igd_reserve_slot and xen_igd_clear_slot functions

Michael Tsirkin:
* Introduce XEN_PCI_IGD_DOMAIN, XEN_PCI_IGD_BUS, XEN_PCI_IGD_DEV, and
  XEN_PCI_IGD_FN - use them to compute the value of XEN_PCI_IGD_SLOT_MASK

The new xen_igd_reserve_slot function uses the existing slot_reserved_mask
member of PCIBus to reserve PCI slot 2 for Xen HVM guests configured using
the xl toolstack with the gfx_passthru option enabled, which sets the
igd-passthru=on option to Qemu for the Xen HVM machine type.

The new xen_igd_reserve_slot function also needs to be implemented in
hw/xen/xen_pt_stub.c to prevent FTBFS during the link stage for the case
when Qemu is configured with --enable-xen and --disable-xen-pci-passthrough,
in which case it does nothing.

The new xen_igd_clear_slot function overrides qdev->realize of the parent
PCI device class to enable the Intel IGD to occupy slot 2 on the PCI bus
since slot 2 was reserved by xen_igd_reserve_slot when the PCI bus was
created in hw/i386/pc_piix.c for the case when igd-passthru=on.

Move the call to xen_host_pci_device_get, and the associated error
handling, from xen_pt_realize to the new xen_igd_clear_slot function to
initialize the device class and vendor values which enables the checks for
the Intel IGD to succeed. The verification that the host device is an
Intel IGD to be passed through is done by checking the domain, bus, slot,
and function values as well as by checking that gfx_passthru is enabled,
the device class is VGA, and the device vendor in Intel.

Signed-off-by: Chuck Zmudzinski 
---
Notes that might be helpful to reviewers of patched code in hw/xen:

The new functions and types are based on recommendations from Qemu docs:
https://qemu.readthedocs.io/en/latest/devel/qom.html

Notes that might be helpful to reviewers of patched code in hw/i386:

The small patch to hw/i386/pc_piix.c is protected by CONFIG_XEN so it does
not affect builds that do not have CONFIG_XEN defined.

xen_igd_gfx_pt_enabled() in the patched hw/i386/pc_piix.c file is an
existing function that is only true when Qemu is built with
xen-pci-passthrough enabled and the administrator has configured the Xen
HVM guest with Qemu's igd-passthru=on option.

v2: Remove From:  tag at top of commit message

v3: Changed the test for the Intel IGD in xen_igd_clear_slot:

if (is_igd_vga_passthrough(>real_device) &&
(s->real_device.vendor_id == PCI_VENDOR_ID_INTEL)) {

is changed to

if (xen_igd_gfx_pt_enabled() && (s->hostaddr.slot == 2)
&& (s->hostaddr.function == 0)) {

I hoped that I could use the test in v2, since it matches the
other tests for the Intel IGD in Qemu and Xen, but those tests
do not work because the necessary data structures are not set with
their values yet. So instead use the test that the administrator
has enabled gfx_passthru and the device address on the host is
02.0. This test does detect the Intel IGD correctly.

v4: Use brchu...@aol.com instead of brchu...@netscape.net for the author's
email address to match the address used by the same author in commits
be9c61da and c0e86b76

Change variable for XEN_PT_DEVICE_CLASS: xptc changed to xpdc

v5: The patch of xen_pt.c was re-worked to allow a more consistent test
for the Intel IGD that uses the same criteria as in other places.
This involv

Re: [PATCH v9] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

On 1/20/23 4:34 PM, Stefano Stabellini wrote:
> On Sat, 14 Jan 2023, Chuck Zmudzinski wrote:
> > Intel specifies that the Intel IGD must occupy slot 2 on the PCI bus,
> > as noted in docs/igd-assign.txt in the Qemu source code.
> > 
> > Currently, when the xl toolstack is used to configure a Xen HVM guest with
> > Intel IGD passthrough to the guest with the Qemu upstream device model,
> > a Qemu emulated PCI device will occupy slot 2 and the Intel IGD will occupy
> > a different slot. This problem often prevents the guest from booting.
> > 
> > The only available workaround is not good: Configure Xen HVM guests to use
> > the old and no longer maintained Qemu traditional device model available
> > from xenbits.xen.org which does reserve slot 2 for the Intel IGD.
> > 
> > To implement this feature in the Qemu upstream device model for Xen HVM
> > guests, introduce the following new functions, types, and macros:
> > 
> > * XEN_PT_DEVICE_CLASS declaration, based on the existing TYPE_XEN_PT_DEVICE
> > * XEN_PT_DEVICE_GET_CLASS macro helper function for XEN_PT_DEVICE_CLASS
> > * typedef XenPTQdevRealize function pointer
> > * XEN_PCI_IGD_SLOT_MASK, the value of slot_reserved_mask to reserve slot 2
> > * xen_igd_reserve_slot and xen_igd_clear_slot functions
> > 
> > Michael Tsirkin:
> > * Introduce XEN_PCI_IGD_DOMAIN, XEN_PCI_IGD_BUS, XEN_PCI_IGD_DEV, and
> >   XEN_PCI_IGD_FN - use them to compute the value of XEN_PCI_IGD_SLOT_MASK
> > 
> > The new xen_igd_reserve_slot function uses the existing slot_reserved_mask
> > member of PCIBus to reserve PCI slot 2 for Xen HVM guests configured using
> > the xl toolstack with the gfx_passthru option enabled, which sets the
> > igd-passthru=on option to Qemu for the Xen HVM machine type.
> > 
> > The new xen_igd_reserve_slot function also needs to be implemented in
> > hw/xen/xen_pt_stub.c to prevent FTBFS during the link stage for the case
> > when Qemu is configured with --enable-xen and --disable-xen-pci-passthrough,
> > in which case it does nothing.
> > 
> > The new xen_igd_clear_slot function overrides qdev->realize of the parent
> > PCI device class to enable the Intel IGD to occupy slot 2 on the PCI bus
> > since slot 2 was reserved by xen_igd_reserve_slot when the PCI bus was
> > created in hw/i386/pc_piix.c for the case when igd-passthru=on.
> > 
> > Move the call to xen_host_pci_device_get, and the associated error
> > handling, from xen_pt_realize to the new xen_igd_clear_slot function to
> > initialize the device class and vendor values which enables the checks for
> > the Intel IGD to succeed. The verification that the host device is an
> > Intel IGD to be passed through is done by checking the domain, bus, slot,
> > and function values as well as by checking that gfx_passthru is enabled,
> > the device class is VGA, and the device vendor in Intel.
> > 
> > Signed-off-by: Chuck Zmudzinski 
>
> Hi Chuck,
>
> The approach looks OK in principle to me. I only have one question: for
> other PCI devices (not Intel IGD), where is xen_host_pci_device_get
> called now?

I think you are right that there might be a problem for the
devices being added after the Intel IGD. I think I only tested
the case when the Intel IGD is the last device being added.
I do expect if I add the Intel IGD first, then there will be
problems when the subsequent type XEN_PT devices are
added because xen_pt_realize will not like it if
xen_host_pci_device_get does not get called. I will check
this over the weekend and, if a change is needed, I will
post it in v10.

I also think there is the same problem when the bit in
slot_reserved_mask is never set, which happens when
the guest has pci devices passed through but without
configuring Qemu with the igd-passthru=on option for
the xenfv machine type. I will also test this over the
weekend.

> It looks like that xen_igd_reserve_slot would return without setting
> slot_reserved_mask, hence xen_igd_clear_slot would also return without
> calling xen_host_pci_device_get. And xen_pt_realize doesn't call
> xen_host_pci_device_get any longer.
>
> Am I missing something?

No, you are not missing anything. You are pointing to some
cases that I need to test that probably would not work.
I think the fix is to have this at the beginning of
xen_igd_clear_slot instead of what I have now:

    xen_host_pci_device_get(>real_device,
    s->hostaddr.domain, s->hostaddr.bus,
    s->hostaddr.slot, s->hostaddr.function,
    errp);
    if (*errp) {
    error_append_hint(errp, "Failed to \"open\" the real pci device");
    r

Re: [PATCH v8] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

On 1/20/2023 11:01 AM, Igor Mammedov wrote:
> On Tue, 17 Jan 2023 09:50:23 -0500
> Chuck Zmudzinski  wrote:
>
> > On 1/17/2023 5:35 AM, Igor Mammedov wrote:
> > > On Mon, 16 Jan 2023 13:00:53 -0500
> > > Chuck Zmudzinski  wrote:
> > >  
> > > > On 1/16/23 10:33, Igor Mammedov wrote:  
> > > > > On Fri, 13 Jan 2023 16:31:26 -0500
> > > > > Chuck Zmudzinski  wrote:
> > > > > 
> > > > >> On 1/13/23 4:33 AM, Igor Mammedov wrote:
> > > > >> > On Thu, 12 Jan 2023 23:14:26 -0500
> > > > >> > Chuck Zmudzinski  wrote:
> > > > >> >   
> > > > >> >> On 1/12/23 6:03 PM, Michael S. Tsirkin wrote:  
> > > > >> >> > On Thu, Jan 12, 2023 at 10:55:25PM +, Bernhard Beschow 
> > > > >> >> > wrote:
> > > > >> >> >> I think the change Michael suggests is very minimalistic: Move 
> > > > >> >> >> the if
> > > > >> >> >> condition around xen_igd_reserve_slot() into the function 
> > > > >> >> >> itself and
> > > > >> >> >> always call it there unconditionally -- basically turning 
> > > > >> >> >> three lines
> > > > >> >> >> into one. Since xen_igd_reserve_slot() seems very problem 
> > > > >> >> >> specific,
> > > > >> >> >> Michael further suggests to rename it to something more 
> > > > >> >> >> general. All
> > > > >> >> >> in all no big changes required.
> > > > >> >> > 
> > > > >> >> > yes, exactly.
> > > > >> >> > 
> > > > >> >> 
> > > > >> >> OK, got it. I can do that along with the other suggestions.  
> > > > >> > 
> > > > >> > have you considered instead of reservation, putting a slot check 
> > > > >> > in device model
> > > > >> > and if it's intel igd being passed through, fail at realize time  
> > > > >> > if it can't take
> > > > >> > required slot (with a error directing user to fix command line)?   
> > > > >> >
> > > > >> 
> > > > >> Yes, but the core pci code currently already fails at realize time
> > > > >> with a useful error message if the user tries to use slot 2 for the
> > > > >> igd, because of the xen platform device which has slot 2. The user
> > > > >> can fix this without patching qemu, but having the user fix it on
> > > > >> the command line is not the best way to solve the problem, primarily
> > > > >> because the user would need to hotplug the xen platform device via a
> > > > >> command line option instead of having the xen platform device added 
> > > > >> by
> > > > >> pc_xen_hvm_init functions almost immediately after creating the pci
> > > > >> bus, and that delay in adding the xen platform device degrades
> > > > >> startup performance of the guest.
> > > > >> 
> > > > >> > That could be less complicated than dealing with slot reservations 
> > > > >> > at the cost of
> > > > >> > being less convenient.  
> > > > >> 
> > > > >> And also a cost of reduced startup performance
> > > > > 
> > > > > Could you clarify how it affects performance (and how much).
> > > > > (as I see, setup done at board_init time is roughly the same
> > > > > as with '-device foo' CLI options, modulo time needed to parse
> > > > > options which should be negligible. and both ways are done before
> > > > > guest runs)
> > > > 
> > > > I preface my answer by saying there is a v9, but you don't
> > > > need to look at that. I will answer all your questions here.
> > > > 
> > > > I am going by what I observe on the main HDMI display with the
> > > > different approaches. With the approach of not patching Qemu
> > > > to fix this, which requires adding the Xen platform device a
> > > > little later, the length of time it takes to fully load the
> > > > guest is increased. I also n

Re: [PATCH v8] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

On 1/20/2023 11:57 AM, Stefano Stabellini wrote:
> On Tue, 17 Jan 2023, Chuck Zmudzinski wrote:
> > On 1/17/2023 6:04 AM, Igor Mammedov wrote:
> > > On Mon, 16 Jan 2023 13:00:53 -0500
> > > Chuck Zmudzinski  wrote:
> > >
> > > > On 1/16/23 10:33, Igor Mammedov wrote:
> > > > > On Fri, 13 Jan 2023 16:31:26 -0500
> > > > > Chuck Zmudzinski  wrote:
> > > > >   
> > > > >> On 1/13/23 4:33 AM, Igor Mammedov wrote:  
> > > > >> > On Thu, 12 Jan 2023 23:14:26 -0500
> > > > >> > Chuck Zmudzinski  wrote:
> > > > >> > 
> > > > >> >> On 1/12/23 6:03 PM, Michael S. Tsirkin wrote:
> > > > >> >> > On Thu, Jan 12, 2023 at 10:55:25PM +, Bernhard Beschow 
> > > > >> >> > wrote:  
> > > > >> >> >> I think the change Michael suggests is very minimalistic: Move 
> > > > >> >> >> the if
> > > > >> >> >> condition around xen_igd_reserve_slot() into the function 
> > > > >> >> >> itself and
> > > > >> >> >> always call it there unconditionally -- basically turning 
> > > > >> >> >> three lines
> > > > >> >> >> into one. Since xen_igd_reserve_slot() seems very problem 
> > > > >> >> >> specific,
> > > > >> >> >> Michael further suggests to rename it to something more 
> > > > >> >> >> general. All
> > > > >> >> >> in all no big changes required.  
> > > > >> >> > 
> > > > >> >> > yes, exactly.
> > > > >> >> >   
> > > > >> >> 
> > > > >> >> OK, got it. I can do that along with the other suggestions.
> > > > >> > 
> > > > >> > have you considered instead of reservation, putting a slot check 
> > > > >> > in device model
> > > > >> > and if it's intel igd being passed through, fail at realize time  
> > > > >> > if it can't take
> > > > >> > required slot (with a error directing user to fix command line)?   
> > > > >> >  
> > > > >> 
> > > > >> Yes, but the core pci code currently already fails at realize time
> > > > >> with a useful error message if the user tries to use slot 2 for the
> > > > >> igd, because of the xen platform device which has slot 2. The user
> > > > >> can fix this without patching qemu, but having the user fix it on
> > > > >> the command line is not the best way to solve the problem, primarily
> > > > >> because the user would need to hotplug the xen platform device via a
> > > > >> command line option instead of having the xen platform device added 
> > > > >> by
> > > > >> pc_xen_hvm_init functions almost immediately after creating the pci
> > > > >> bus, and that delay in adding the xen platform device degrades
> > > > >> startup performance of the guest.
> > > > >>   
> > > > >> > That could be less complicated than dealing with slot reservations 
> > > > >> > at the cost of
> > > > >> > being less convenient.
> > > > >> 
> > > > >> And also a cost of reduced startup performance  
> > > > > 
> > > > > Could you clarify how it affects performance (and how much).
> > > > > (as I see, setup done at board_init time is roughly the same
> > > > > as with '-device foo' CLI options, modulo time needed to parse
> > > > > options which should be negligible. and both ways are done before
> > > > > guest runs)  
> > > > 
> > > > I preface my answer by saying there is a v9, but you don't
> > > > need to look at that. I will answer all your questions here.
> > > > 
> > > > I am going by what I observe on the main HDMI display with the
> > > > different approaches. With the approach of not patching Qemu
> > > > to fix this, which requires adding the Xen platform device a
> > > > little later, the length of time it takes to fully load the
> > > > guest is increased. I also noticed with Linux guests that use
> > >

Re: [PATCH v8] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

On 1/17/2023 11:27 PM, Alex Williamson wrote:
> On Tue, 17 Jan 2023 19:15:57 -0500
> Chuck Zmudzinski  wrote:
>
> > On 1/17/2023 6:04 AM, Igor Mammedov wrote:
> > > On Mon, 16 Jan 2023 13:00:53 -0500
> > > Chuck Zmudzinski  wrote:
> > >  
> > > > On 1/16/23 10:33, Igor Mammedov wrote:  
> > > > > On Fri, 13 Jan 2023 16:31:26 -0500
> > > > > Chuck Zmudzinski  wrote:
> > > > > 
> > > > >> On 1/13/23 4:33 AM, Igor Mammedov wrote:
> > > > >> > On Thu, 12 Jan 2023 23:14:26 -0500
> > > > >> > Chuck Zmudzinski  wrote:
> > > > >> >   
> > > > >> >> On 1/12/23 6:03 PM, Michael S. Tsirkin wrote:  
> > > > >> >> > On Thu, Jan 12, 2023 at 10:55:25PM +, Bernhard Beschow 
> > > > >> >> > wrote:
> > > > >> >> >> I think the change Michael suggests is very minimalistic: Move 
> > > > >> >> >> the if
> > > > >> >> >> condition around xen_igd_reserve_slot() into the function 
> > > > >> >> >> itself and
> > > > >> >> >> always call it there unconditionally -- basically turning 
> > > > >> >> >> three lines
> > > > >> >> >> into one. Since xen_igd_reserve_slot() seems very problem 
> > > > >> >> >> specific,
> > > > >> >> >> Michael further suggests to rename it to something more 
> > > > >> >> >> general. All
> > > > >> >> >> in all no big changes required.
> > > > >> >> > 
> > > > >> >> > yes, exactly.
> > > > >> >> > 
> > > > >> >> 
> > > > >> >> OK, got it. I can do that along with the other suggestions.  
> > > > >> > 
> > > > >> > have you considered instead of reservation, putting a slot check 
> > > > >> > in device model
> > > > >> > and if it's intel igd being passed through, fail at realize time  
> > > > >> > if it can't take
> > > > >> > required slot (with a error directing user to fix command line)?   
> > > > >> >
> > > > >> 
> > > > >> Yes, but the core pci code currently already fails at realize time
> > > > >> with a useful error message if the user tries to use slot 2 for the
> > > > >> igd, because of the xen platform device which has slot 2. The user
> > > > >> can fix this without patching qemu, but having the user fix it on
> > > > >> the command line is not the best way to solve the problem, primarily
> > > > >> because the user would need to hotplug the xen platform device via a
> > > > >> command line option instead of having the xen platform device added 
> > > > >> by
> > > > >> pc_xen_hvm_init functions almost immediately after creating the pci
> > > > >> bus, and that delay in adding the xen platform device degrades
> > > > >> startup performance of the guest.
> > > > >> 
> > > > >> > That could be less complicated than dealing with slot reservations 
> > > > >> > at the cost of
> > > > >> > being less convenient.  
> > > > >> 
> > > > >> And also a cost of reduced startup performance
> > > > > 
> > > > > Could you clarify how it affects performance (and how much).
> > > > > (as I see, setup done at board_init time is roughly the same
> > > > > as with '-device foo' CLI options, modulo time needed to parse
> > > > > options which should be negligible. and both ways are done before
> > > > > guest runs)
> > > > 
> > > > I preface my answer by saying there is a v9, but you don't
> > > > need to look at that. I will answer all your questions here.
> > > > 
> > > > I am going by what I observe on the main HDMI display with the
> > > > different approaches. With the approach of not patching Qemu
> > > > to fix this, which requires adding the Xen platform device a
> > > > little later, the length of time it takes to fully load the
> > > > guest is increased

Re: [PATCH v8] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

On 1/17/2023 6:04 AM, Igor Mammedov wrote:
> On Mon, 16 Jan 2023 13:00:53 -0500
> Chuck Zmudzinski  wrote:
>
> > On 1/16/23 10:33, Igor Mammedov wrote:
> > > On Fri, 13 Jan 2023 16:31:26 -0500
> > > Chuck Zmudzinski  wrote:
> > >   
> > >> On 1/13/23 4:33 AM, Igor Mammedov wrote:  
> > >> > On Thu, 12 Jan 2023 23:14:26 -0500
> > >> > Chuck Zmudzinski  wrote:
> > >> > 
> > >> >> On 1/12/23 6:03 PM, Michael S. Tsirkin wrote:
> > >> >> > On Thu, Jan 12, 2023 at 10:55:25PM +, Bernhard Beschow wrote:   
> > >> >> >
> > >> >> >> I think the change Michael suggests is very minimalistic: Move the 
> > >> >> >> if
> > >> >> >> condition around xen_igd_reserve_slot() into the function itself 
> > >> >> >> and
> > >> >> >> always call it there unconditionally -- basically turning three 
> > >> >> >> lines
> > >> >> >> into one. Since xen_igd_reserve_slot() seems very problem specific,
> > >> >> >> Michael further suggests to rename it to something more general. 
> > >> >> >> All
> > >> >> >> in all no big changes required.  
> > >> >> > 
> > >> >> > yes, exactly.
> > >> >> >   
> > >> >> 
> > >> >> OK, got it. I can do that along with the other suggestions.
> > >> > 
> > >> > have you considered instead of reservation, putting a slot check in 
> > >> > device model
> > >> > and if it's intel igd being passed through, fail at realize time  if 
> > >> > it can't take
> > >> > required slot (with a error directing user to fix command line)?
> > >> 
> > >> Yes, but the core pci code currently already fails at realize time
> > >> with a useful error message if the user tries to use slot 2 for the
> > >> igd, because of the xen platform device which has slot 2. The user
> > >> can fix this without patching qemu, but having the user fix it on
> > >> the command line is not the best way to solve the problem, primarily
> > >> because the user would need to hotplug the xen platform device via a
> > >> command line option instead of having the xen platform device added by
> > >> pc_xen_hvm_init functions almost immediately after creating the pci
> > >> bus, and that delay in adding the xen platform device degrades
> > >> startup performance of the guest.
> > >>   
> > >> > That could be less complicated than dealing with slot reservations at 
> > >> > the cost of
> > >> > being less convenient.
> > >> 
> > >> And also a cost of reduced startup performance  
> > > 
> > > Could you clarify how it affects performance (and how much).
> > > (as I see, setup done at board_init time is roughly the same
> > > as with '-device foo' CLI options, modulo time needed to parse
> > > options which should be negligible. and both ways are done before
> > > guest runs)  
> > 
> > I preface my answer by saying there is a v9, but you don't
> > need to look at that. I will answer all your questions here.
> > 
> > I am going by what I observe on the main HDMI display with the
> > different approaches. With the approach of not patching Qemu
> > to fix this, which requires adding the Xen platform device a
> > little later, the length of time it takes to fully load the
> > guest is increased. I also noticed with Linux guests that use
> > the grub bootoader, the grub vga driver cannot display the
> > grub boot menu at the native resolution of the display, which
> > in the tested case is 1920x1080, when the Xen platform device
> > is added via a command line option instead of by the
> > pc_xen_hvm_init_pci fucntion in pc_piix.c, but with this patch
> > to Qemu, the grub menu is displayed at the full, 1920x1080
> > native resolution of the display. Once the guest fully loads,
> > there is no noticeable difference in performance. It is mainly
> > a degradation in startup performance, not performance once
> > the guest OS is fully loaded.
>
> Looking at igd-assign.txt, it recommends to add IGD using '-device' CLI
> option, and actually drop at least graphics defaults explicitly.
> So it is expected to work fine even when IGD is constructed with
> '-devic

Re: [PATCH v8] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

On 1/17/2023 5:35 AM, Igor Mammedov wrote:
> On Mon, 16 Jan 2023 13:00:53 -0500
> Chuck Zmudzinski  wrote:
>
> > On 1/16/23 10:33, Igor Mammedov wrote:
> > > On Fri, 13 Jan 2023 16:31:26 -0500
> > > Chuck Zmudzinski  wrote:
> > >   
> > >> On 1/13/23 4:33 AM, Igor Mammedov wrote:  
> > >> > On Thu, 12 Jan 2023 23:14:26 -0500
> > >> > Chuck Zmudzinski  wrote:
> > >> > 
> > >> >> On 1/12/23 6:03 PM, Michael S. Tsirkin wrote:
> > >> >> > On Thu, Jan 12, 2023 at 10:55:25PM +, Bernhard Beschow wrote:   
> > >> >> >
> > >> >> >> I think the change Michael suggests is very minimalistic: Move the 
> > >> >> >> if
> > >> >> >> condition around xen_igd_reserve_slot() into the function itself 
> > >> >> >> and
> > >> >> >> always call it there unconditionally -- basically turning three 
> > >> >> >> lines
> > >> >> >> into one. Since xen_igd_reserve_slot() seems very problem specific,
> > >> >> >> Michael further suggests to rename it to something more general. 
> > >> >> >> All
> > >> >> >> in all no big changes required.  
> > >> >> > 
> > >> >> > yes, exactly.
> > >> >> >   
> > >> >> 
> > >> >> OK, got it. I can do that along with the other suggestions.
> > >> > 
> > >> > have you considered instead of reservation, putting a slot check in 
> > >> > device model
> > >> > and if it's intel igd being passed through, fail at realize time  if 
> > >> > it can't take
> > >> > required slot (with a error directing user to fix command line)?
> > >> 
> > >> Yes, but the core pci code currently already fails at realize time
> > >> with a useful error message if the user tries to use slot 2 for the
> > >> igd, because of the xen platform device which has slot 2. The user
> > >> can fix this without patching qemu, but having the user fix it on
> > >> the command line is not the best way to solve the problem, primarily
> > >> because the user would need to hotplug the xen platform device via a
> > >> command line option instead of having the xen platform device added by
> > >> pc_xen_hvm_init functions almost immediately after creating the pci
> > >> bus, and that delay in adding the xen platform device degrades
> > >> startup performance of the guest.
> > >>   
> > >> > That could be less complicated than dealing with slot reservations at 
> > >> > the cost of
> > >> > being less convenient.
> > >> 
> > >> And also a cost of reduced startup performance  
> > > 
> > > Could you clarify how it affects performance (and how much).
> > > (as I see, setup done at board_init time is roughly the same
> > > as with '-device foo' CLI options, modulo time needed to parse
> > > options which should be negligible. and both ways are done before
> > > guest runs)  
> > 
> > I preface my answer by saying there is a v9, but you don't
> > need to look at that. I will answer all your questions here.
> > 
> > I am going by what I observe on the main HDMI display with the
> > different approaches. With the approach of not patching Qemu
> > to fix this, which requires adding the Xen platform device a
> > little later, the length of time it takes to fully load the
> > guest is increased. I also noticed with Linux guests that use
> > the grub bootoader, the grub vga driver cannot display the
> > grub boot menu at the native resolution of the display, which
> > in the tested case is 1920x1080, when the Xen platform device
> > is added via a command line option instead of by the
> > pc_xen_hvm_init_pci fucntion in pc_piix.c, but with this patch
> > to Qemu, the grub menu is displayed at the full, 1920x1080
> > native resolution of the display. Once the guest fully loads,
> > there is no noticeable difference in performance. It is mainly
> > a degradation in startup performance, not performance once
> > the guest OS is fully loaded.
> above hints on presence of bug[s] in igd-passthru implementation,
> and this patch effectively hides problem instead of trying to figure
> out what's wrong and fixing it.
>

I agree that the with the current Qemu there is a bug in the igd-passthru
implementation. My proposed patch fixes it. So why not support the
proposed patch with a Reviewed-by tag?

Re: [PATCH v8] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

On 1/17/2023 5:35 AM, Igor Mammedov wrote:
> On Mon, 16 Jan 2023 13:00:53 -0500
> Chuck Zmudzinski  wrote:
>
> > On 1/16/23 10:33, Igor Mammedov wrote:
> > > On Fri, 13 Jan 2023 16:31:26 -0500
> > > Chuck Zmudzinski  wrote:
> > >   
> > >> On 1/13/23 4:33 AM, Igor Mammedov wrote:  
> > >> > On Thu, 12 Jan 2023 23:14:26 -0500
> > >> > Chuck Zmudzinski  wrote:
> > >> > 
> > >> >> On 1/12/23 6:03 PM, Michael S. Tsirkin wrote:
> > >> >> > On Thu, Jan 12, 2023 at 10:55:25PM +, Bernhard Beschow wrote:   
> > >> >> >
> > >> >> >> I think the change Michael suggests is very minimalistic: Move the 
> > >> >> >> if
> > >> >> >> condition around xen_igd_reserve_slot() into the function itself 
> > >> >> >> and
> > >> >> >> always call it there unconditionally -- basically turning three 
> > >> >> >> lines
> > >> >> >> into one. Since xen_igd_reserve_slot() seems very problem specific,
> > >> >> >> Michael further suggests to rename it to something more general. 
> > >> >> >> All
> > >> >> >> in all no big changes required.  
> > >> >> > 
> > >> >> > yes, exactly.
> > >> >> >   
> > >> >> 
> > >> >> OK, got it. I can do that along with the other suggestions.
> > >> > 
> > >> > have you considered instead of reservation, putting a slot check in 
> > >> > device model
> > >> > and if it's intel igd being passed through, fail at realize time  if 
> > >> > it can't take
> > >> > required slot (with a error directing user to fix command line)?
> > >> 
> > >> Yes, but the core pci code currently already fails at realize time
> > >> with a useful error message if the user tries to use slot 2 for the
> > >> igd, because of the xen platform device which has slot 2. The user
> > >> can fix this without patching qemu, but having the user fix it on
> > >> the command line is not the best way to solve the problem, primarily
> > >> because the user would need to hotplug the xen platform device via a
> > >> command line option instead of having the xen platform device added by
> > >> pc_xen_hvm_init functions almost immediately after creating the pci
> > >> bus, and that delay in adding the xen platform device degrades
> > >> startup performance of the guest.
> > >>   
> > >> > That could be less complicated than dealing with slot reservations at 
> > >> > the cost of
> > >> > being less convenient.
> > >> 
> > >> And also a cost of reduced startup performance  
> > > 
> > > Could you clarify how it affects performance (and how much).
> > > (as I see, setup done at board_init time is roughly the same
> > > as with '-device foo' CLI options, modulo time needed to parse
> > > options which should be negligible. and both ways are done before
> > > guest runs)  
> > 
> > I preface my answer by saying there is a v9, but you don't
> > need to look at that. I will answer all your questions here.
> > 
> > I am going by what I observe on the main HDMI display with the
> > different approaches. With the approach of not patching Qemu
> > to fix this, which requires adding the Xen platform device a
> > little later, the length of time it takes to fully load the
> > guest is increased. I also noticed with Linux guests that use
> > the grub bootoader, the grub vga driver cannot display the
> > grub boot menu at the native resolution of the display, which
> > in the tested case is 1920x1080, when the Xen platform device
> > is added via a command line option instead of by the
> > pc_xen_hvm_init_pci fucntion in pc_piix.c, but with this patch
> > to Qemu, the grub menu is displayed at the full, 1920x1080
> > native resolution of the display. Once the guest fully loads,
> > there is no noticeable difference in performance. It is mainly
> > a degradation in startup performance, not performance once
> > the guest OS is fully loaded.
> above hints on presence of bug[s] in igd-passthru implementation,
> and this patch effectively hides problem instead of trying to figure
> out what's wrong and fixing it.
>

Why did you delete the rest of my answers to your other observations and
not respond to them?

Re: [PATCH v8] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

On 1/16/23 10:33, Igor Mammedov wrote:
> On Fri, 13 Jan 2023 16:31:26 -0500
> Chuck Zmudzinski  wrote:
> 
>> On 1/13/23 4:33 AM, Igor Mammedov wrote:
>> > On Thu, 12 Jan 2023 23:14:26 -0500
>> > Chuck Zmudzinski  wrote:
>> >   
>> >> On 1/12/23 6:03 PM, Michael S. Tsirkin wrote:  
>> >> > On Thu, Jan 12, 2023 at 10:55:25PM +, Bernhard Beschow wrote:
>> >> >> I think the change Michael suggests is very minimalistic: Move the if
>> >> >> condition around xen_igd_reserve_slot() into the function itself and
>> >> >> always call it there unconditionally -- basically turning three lines
>> >> >> into one. Since xen_igd_reserve_slot() seems very problem specific,
>> >> >> Michael further suggests to rename it to something more general. All
>> >> >> in all no big changes required.
>> >> > 
>> >> > yes, exactly.
>> >> > 
>> >> 
>> >> OK, got it. I can do that along with the other suggestions.  
>> > 
>> > have you considered instead of reservation, putting a slot check in device 
>> > model
>> > and if it's intel igd being passed through, fail at realize time  if it 
>> > can't take
>> > required slot (with a error directing user to fix command line)?  
>> 
>> Yes, but the core pci code currently already fails at realize time
>> with a useful error message if the user tries to use slot 2 for the
>> igd, because of the xen platform device which has slot 2. The user
>> can fix this without patching qemu, but having the user fix it on
>> the command line is not the best way to solve the problem, primarily
>> because the user would need to hotplug the xen platform device via a
>> command line option instead of having the xen platform device added by
>> pc_xen_hvm_init functions almost immediately after creating the pci
>> bus, and that delay in adding the xen platform device degrades
>> startup performance of the guest.
>> 
>> > That could be less complicated than dealing with slot reservations at the 
>> > cost of
>> > being less convenient.  
>> 
>> And also a cost of reduced startup performance
> 
> Could you clarify how it affects performance (and how much).
> (as I see, setup done at board_init time is roughly the same
> as with '-device foo' CLI options, modulo time needed to parse
> options which should be negligible. and both ways are done before
> guest runs)

I preface my answer by saying there is a v9, but you don't
need to look at that. I will answer all your questions here.

I am going by what I observe on the main HDMI display with the
different approaches. With the approach of not patching Qemu
to fix this, which requires adding the Xen platform device a
little later, the length of time it takes to fully load the
guest is increased. I also noticed with Linux guests that use
the grub bootoader, the grub vga driver cannot display the
grub boot menu at the native resolution of the display, which
in the tested case is 1920x1080, when the Xen platform device
is added via a command line option instead of by the
pc_xen_hvm_init_pci fucntion in pc_piix.c, but with this patch
to Qemu, the grub menu is displayed at the full, 1920x1080
native resolution of the display. Once the guest fully loads,
there is no noticeable difference in performance. It is mainly
a degradation in startup performance, not performance once
the guest OS is fully loaded.

> 
>> However, the performance hit can be prevented by assigning slot
>> 3 instead of slot 2 for the xen platform device if igd passthrough
>> is configured on the command line instead of doing slot reservation,
>> but there would still be less convenience and, for libxl users, an
>> inability to easily configure the command line so that the igd can
>> still have slot 2 without a hacky and error-prone patch to libxl to
>> deal with this problem.
> libvirt manages to get it right on management side without quirks on
> QEMU side.

I think the reason libvirt/kvm gets it right is simply because the
code implementing the libvirt/kvm approach got more attention and testing
than the code that implements the libxl/Xen approach. This patch
really represents what should have been done when support for the
igd-passthru=on option for xenfv machines was added seven years ago,
but the code was apparently added without much testing and is stale now
and needs this fix which is entirely implemented in either files maintained
by Xen maintainers or, in the case of the small patch to pc_piix.c,
entirely within a section guarded by #ifdef CONFIG_XEN. Not much
maintenance burden for hw/i386 maintainers.

[PATCH v9] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

Intel specifies that the Intel IGD must occupy slot 2 on the PCI bus,
as noted in docs/igd-assign.txt in the Qemu source code.

Currently, when the xl toolstack is used to configure a Xen HVM guest with
Intel IGD passthrough to the guest with the Qemu upstream device model,
a Qemu emulated PCI device will occupy slot 2 and the Intel IGD will occupy
a different slot. This problem often prevents the guest from booting.

The only available workaround is not good: Configure Xen HVM guests to use
the old and no longer maintained Qemu traditional device model available
from xenbits.xen.org which does reserve slot 2 for the Intel IGD.

To implement this feature in the Qemu upstream device model for Xen HVM
guests, introduce the following new functions, types, and macros:

* XEN_PT_DEVICE_CLASS declaration, based on the existing TYPE_XEN_PT_DEVICE
* XEN_PT_DEVICE_GET_CLASS macro helper function for XEN_PT_DEVICE_CLASS
* typedef XenPTQdevRealize function pointer
* XEN_PCI_IGD_SLOT_MASK, the value of slot_reserved_mask to reserve slot 2
* xen_igd_reserve_slot and xen_igd_clear_slot functions

Michael Tsirkin:
* Introduce XEN_PCI_IGD_DOMAIN, XEN_PCI_IGD_BUS, XEN_PCI_IGD_DEV, and
  XEN_PCI_IGD_FN - use them to compute the value of XEN_PCI_IGD_SLOT_MASK

The new xen_igd_reserve_slot function uses the existing slot_reserved_mask
member of PCIBus to reserve PCI slot 2 for Xen HVM guests configured using
the xl toolstack with the gfx_passthru option enabled, which sets the
igd-passthru=on option to Qemu for the Xen HVM machine type.

The new xen_igd_reserve_slot function also needs to be implemented in
hw/xen/xen_pt_stub.c to prevent FTBFS during the link stage for the case
when Qemu is configured with --enable-xen and --disable-xen-pci-passthrough,
in which case it does nothing.

The new xen_igd_clear_slot function overrides qdev->realize of the parent
PCI device class to enable the Intel IGD to occupy slot 2 on the PCI bus
since slot 2 was reserved by xen_igd_reserve_slot when the PCI bus was
created in hw/i386/pc_piix.c for the case when igd-passthru=on.

Move the call to xen_host_pci_device_get, and the associated error
handling, from xen_pt_realize to the new xen_igd_clear_slot function to
initialize the device class and vendor values which enables the checks for
the Intel IGD to succeed. The verification that the host device is an
Intel IGD to be passed through is done by checking the domain, bus, slot,
and function values as well as by checking that gfx_passthru is enabled,
the device class is VGA, and the device vendor in Intel.

Signed-off-by: Chuck Zmudzinski 
---
Notes that might be helpful to reviewers of patched code in hw/xen:

The new functions and types are based on recommendations from Qemu docs:
https://qemu.readthedocs.io/en/latest/devel/qom.html

Notes that might be helpful to reviewers of patched code in hw/i386:

The small patch to hw/i386/pc_piix.c is protected by CONFIG_XEN so it does
not affect builds that do not have CONFIG_XEN defined.

xen_igd_gfx_pt_enabled() in the patched hw/i386/pc_piix.c file is an
existing function that is only true when Qemu is built with
xen-pci-passthrough enabled and the administrator has configured the Xen
HVM guest with Qemu's igd-passthru=on option.

v2: Remove From:  tag at top of commit message

v3: Changed the test for the Intel IGD in xen_igd_clear_slot:

if (is_igd_vga_passthrough(>real_device) &&
(s->real_device.vendor_id == PCI_VENDOR_ID_INTEL)) {

is changed to

if (xen_igd_gfx_pt_enabled() && (s->hostaddr.slot == 2)
&& (s->hostaddr.function == 0)) {

I hoped that I could use the test in v2, since it matches the
other tests for the Intel IGD in Qemu and Xen, but those tests
do not work because the necessary data structures are not set with
their values yet. So instead use the test that the administrator
has enabled gfx_passthru and the device address on the host is
02.0. This test does detect the Intel IGD correctly.

v4: Use brchu...@aol.com instead of brchu...@netscape.net for the author's
email address to match the address used by the same author in commits
be9c61da and c0e86b76

Change variable for XEN_PT_DEVICE_CLASS: xptc changed to xpdc

v5: The patch of xen_pt.c was re-worked to allow a more consistent test
for the Intel IGD that uses the same criteria as in other places.
This involved moving the call to xen_host_pci_device_get from
xen_pt_realize to xen_igd_clear_slot and updating the checks for the
Intel IGD in xen_igd_clear_slot:

if (xen_igd_gfx_pt_enabled() && (s->hostaddr.slot == 2)
&& (s->hostaddr.function == 0)) {

is changed to

if (is_igd_vga_passthrough(>real_device) &&
s->real_device.domain == 0 && s->real_device.bus == 0 &&
s->real_device.dev == 2 && s->real_device.func == 0 &&
s->

Re: [PATCH v8] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

On 1/10/23 3:16 AM, Michael S. Tsirkin wrote:
> On Tue, Jan 10, 2023 at 02:08:34AM -0500, Chuck Zmudzinski wrote:
>> Intel specifies that the Intel IGD must occupy slot 2 on the PCI bus,
>> as noted in docs/igd-assign.txt in the Qemu source code.
>> 
>> Currently, when the xl toolstack is used to configure a Xen HVM guest with
>> Intel IGD passthrough to the guest with the Qemu upstream device model,
>> a Qemu emulated PCI device will occupy slot 2 and the Intel IGD will occupy
>> a different slot. This problem often prevents the guest from booting.
>> 
>> The only available workaround is not good: Configure Xen HVM guests to use
>> the old and no longer maintained Qemu traditional device model available
>> from xenbits.xen.org which does reserve slot 2 for the Intel IGD.
>> 
>> To implement this feature in the Qemu upstream device model for Xen HVM
>> guests, introduce the following new functions, types, and macros:
>> 
>> * XEN_PT_DEVICE_CLASS declaration, based on the existing TYPE_XEN_PT_DEVICE
>> * XEN_PT_DEVICE_GET_CLASS macro helper function for XEN_PT_DEVICE_CLASS
>> * typedef XenPTQdevRealize function pointer
>> * XEN_PCI_IGD_SLOT_MASK, the value of slot_reserved_mask to reserve slot 2
>> * xen_igd_reserve_slot and xen_igd_clear_slot functions
>> 
>> The new xen_igd_reserve_slot function uses the existing slot_reserved_mask
>> member of PCIBus to reserve PCI slot 2 for Xen HVM guests configured using
>> the xl toolstack with the gfx_passthru option enabled, which sets the
>> igd-passthru=on option to Qemu for the Xen HVM machine type.
>> 
>> The new xen_igd_reserve_slot function also needs to be implemented in
>> hw/xen/xen_pt_stub.c to prevent FTBFS during the link stage for the case
>> when Qemu is configured with --enable-xen and --disable-xen-pci-passthrough,
>> in which case it does nothing.
>> 
>> The new xen_igd_clear_slot function overrides qdev->realize of the parent
>> PCI device class to enable the Intel IGD to occupy slot 2 on the PCI bus
>> since slot 2 was reserved by xen_igd_reserve_slot when the PCI bus was
>> created in hw/i386/pc_piix.c for the case when igd-passthru=on.
>> 
>> Move the call to xen_host_pci_device_get, and the associated error
>> handling, from xen_pt_realize to the new xen_igd_clear_slot function to
>> initialize the device class and vendor values which enables the checks for
>> the Intel IGD to succeed. The verification that the host device is an
>> Intel IGD to be passed through is done by checking the domain, bus, slot,
>> and function values as well as by checking that gfx_passthru is enabled,
>> the device class is VGA, and the device vendor in Intel.
>> 
>> Signed-off-by: Chuck Zmudzinski 
>> ---
>> Notes that might be helpful to reviewers of patched code in hw/xen:
>> 
>> The new functions and types are based on recommendations from Qemu docs:
>> https://qemu.readthedocs.io/en/latest/devel/qom.html
>> 
>> Notes that might be helpful to reviewers of patched code in hw/i386:
>> 
>> The small patch to hw/i386/pc_piix.c is protected by CONFIG_XEN so it does
>> not affect builds that do not have CONFIG_XEN defined.
>> 
>> xen_igd_gfx_pt_enabled() in the patched hw/i386/pc_piix.c file is an
>> existing function that is only true when Qemu is built with
>> xen-pci-passthrough enabled and the administrator has configured the Xen
>> HVM guest with Qemu's igd-passthru=on option.
>> 
>> v2: Remove From:  tag at top of commit message
>> 
>> v3: Changed the test for the Intel IGD in xen_igd_clear_slot:
>> 
>> if (is_igd_vga_passthrough(>real_device) &&
>> (s->real_device.vendor_id == PCI_VENDOR_ID_INTEL)) {
>> 
>> is changed to
>> 
>> if (xen_igd_gfx_pt_enabled() && (s->hostaddr.slot == 2)
>> && (s->hostaddr.function == 0)) {
>> 
>> I hoped that I could use the test in v2, since it matches the
>> other tests for the Intel IGD in Qemu and Xen, but those tests
>> do not work because the necessary data structures are not set with
>> their values yet. So instead use the test that the administrator
>> has enabled gfx_passthru and the device address on the host is
>> 02.0. This test does detect the Intel IGD correctly.
>> 
>> v4: Use brchu...@aol.com instead of brchu...@netscape.net for the author's
>> email address to match the address used by the same author in commits
>> be9c61da and c0e86b76
>> 
>> Change variable for XEN_PT_DEVICE_CLASS: xptc changed t

Re: [PATCH v8] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

On 1/13/23 4:33 AM, Igor Mammedov wrote:
> On Thu, 12 Jan 2023 23:14:26 -0500
> Chuck Zmudzinski  wrote:
> 
>> On 1/12/23 6:03 PM, Michael S. Tsirkin wrote:
>> > On Thu, Jan 12, 2023 at 10:55:25PM +, Bernhard Beschow wrote:  
>> >> I think the change Michael suggests is very minimalistic: Move the if
>> >> condition around xen_igd_reserve_slot() into the function itself and
>> >> always call it there unconditionally -- basically turning three lines
>> >> into one. Since xen_igd_reserve_slot() seems very problem specific,
>> >> Michael further suggests to rename it to something more general. All
>> >> in all no big changes required.  
>> > 
>> > yes, exactly.
>> >   
>> 
>> OK, got it. I can do that along with the other suggestions.
> 
> have you considered instead of reservation, putting a slot check in device 
> model
> and if it's intel igd being passed through, fail at realize time  if it can't 
> take
> required slot (with a error directing user to fix command line)?

Yes, but the core pci code currently already fails at realize time
with a useful error message if the user tries to use slot 2 for the
igd, because of the xen platform device which has slot 2. The user
can fix this without patching qemu, but having the user fix it on
the command line is not the best way to solve the problem, primarily
because the user would need to hotplug the xen platform device via a
command line option instead of having the xen platform device added by
pc_xen_hvm_init functions almost immediately after creating the pci
bus, and that delay in adding the xen platform device degrades
startup performance of the guest.

> That could be less complicated than dealing with slot reservations at the 
> cost of
> being less convenient.

And also a cost of reduced startup performance.

However, the performance hit can be prevented by assigning slot
3 instead of slot 2 for the xen platform device if igd passthrough
is configured on the command line instead of doing slot reservation,
but there would still be less convenience and, for libxl users, an
inability to easily configure the command line so that the igd can
still have slot 2 without a hacky and error-prone patch to libxl to
deal with this problem.

I did post a patch on xen-devel to fix this using libxl, but so far
it has not yet been reviewed and I mentioned in that patch that the
approach of patching qemu so qemu reserves slot 2 for the igd is less
prone to coding errors and is easier to maintain than the patch that
would be required to implement the fix in libxl.

Re: [PATCH v8] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

On 1/12/23 6:03 PM, Michael S. Tsirkin wrote:
> On Thu, Jan 12, 2023 at 10:55:25PM +, Bernhard Beschow wrote:
>> I think the change Michael suggests is very minimalistic: Move the if
>> condition around xen_igd_reserve_slot() into the function itself and
>> always call it there unconditionally -- basically turning three lines
>> into one. Since xen_igd_reserve_slot() seems very problem specific,
>> Michael further suggests to rename it to something more general. All
>> in all no big changes required.
> 
> yes, exactly.
> 

OK, got it. I can do that along with the other suggestions.

Thanks.

Re: [PATCH v8] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

On 1/12/23 2:18 PM, Bernhard Beschow wrote:
> 
> 
> Am 11. Januar 2023 15:40:24 UTC schrieb Chuck Zmudzinski :
>>On 1/10/23 3:16 AM, Michael S. Tsirkin wrote:
>>> On Tue, Jan 10, 2023 at 02:08:34AM -0500, Chuck Zmudzinski wrote:
>>>> Intel specifies that the Intel IGD must occupy slot 2 on the PCI bus,
>>>> as noted in docs/igd-assign.txt in the Qemu source code.
>>>> 
>>>> Currently, when the xl toolstack is used to configure a Xen HVM guest with
>>>> Intel IGD passthrough to the guest with the Qemu upstream device model,
>>>> a Qemu emulated PCI device will occupy slot 2 and the Intel IGD will occupy
>>>> a different slot. This problem often prevents the guest from booting.
>>>> 
>>>> The only available workaround is not good: Configure Xen HVM guests to use
>>>> the old and no longer maintained Qemu traditional device model available
>>>> from xenbits.xen.org which does reserve slot 2 for the Intel IGD.
>>>> 
>>>> To implement this feature in the Qemu upstream device model for Xen HVM
>>>> guests, introduce the following new functions, types, and macros:
>>>> 
>>>> * XEN_PT_DEVICE_CLASS declaration, based on the existing TYPE_XEN_PT_DEVICE
>>>> * XEN_PT_DEVICE_GET_CLASS macro helper function for XEN_PT_DEVICE_CLASS
>>>> * typedef XenPTQdevRealize function pointer
>>>> * XEN_PCI_IGD_SLOT_MASK, the value of slot_reserved_mask to reserve slot 2
>>>> * xen_igd_reserve_slot and xen_igd_clear_slot functions
>>>> 
>>>> The new xen_igd_reserve_slot function uses the existing slot_reserved_mask
>>>> member of PCIBus to reserve PCI slot 2 for Xen HVM guests configured using
>>>> the xl toolstack with the gfx_passthru option enabled, which sets the
>>>> igd-passthru=on option to Qemu for the Xen HVM machine type.
>>>> 
>>>> The new xen_igd_reserve_slot function also needs to be implemented in
>>>> hw/xen/xen_pt_stub.c to prevent FTBFS during the link stage for the case
>>>> when Qemu is configured with --enable-xen and 
>>>> --disable-xen-pci-passthrough,
>>>> in which case it does nothing.
>>>> 
>>>> The new xen_igd_clear_slot function overrides qdev->realize of the parent
>>>> PCI device class to enable the Intel IGD to occupy slot 2 on the PCI bus
>>>> since slot 2 was reserved by xen_igd_reserve_slot when the PCI bus was
>>>> created in hw/i386/pc_piix.c for the case when igd-passthru=on.
>>>> 
>>>> Move the call to xen_host_pci_device_get, and the associated error
>>>> handling, from xen_pt_realize to the new xen_igd_clear_slot function to
>>>> initialize the device class and vendor values which enables the checks for
>>>> the Intel IGD to succeed. The verification that the host device is an
>>>> Intel IGD to be passed through is done by checking the domain, bus, slot,
>>>> and function values as well as by checking that gfx_passthru is enabled,
>>>> the device class is VGA, and the device vendor in Intel.
>>>> 
>>>> Signed-off-by: Chuck Zmudzinski 
>>>> ---
>>>> Notes that might be helpful to reviewers of patched code in hw/xen:
>>>> 
>>>> The new functions and types are based on recommendations from Qemu docs:
>>>> https://qemu.readthedocs.io/en/latest/devel/qom.html
>>>> 
>>>> Notes that might be helpful to reviewers of patched code in hw/i386:
>>>> 
>>>> The small patch to hw/i386/pc_piix.c is protected by CONFIG_XEN so it does
>>>> not affect builds that do not have CONFIG_XEN defined.
>>>> 
>>>> xen_igd_gfx_pt_enabled() in the patched hw/i386/pc_piix.c file is an
>>>> existing function that is only true when Qemu is built with
>>>> xen-pci-passthrough enabled and the administrator has configured the Xen
>>>> HVM guest with Qemu's igd-passthru=on option.
>>>> 
>>>> v2: Remove From:  tag at top of commit message
>>>> 
>>>> v3: Changed the test for the Intel IGD in xen_igd_clear_slot:
>>>> 
>>>> if (is_igd_vga_passthrough(>real_device) &&
>>>> (s->real_device.vendor_id == PCI_VENDOR_ID_INTEL)) {
>>>> 
>>>> is changed to
>>>> 
>>>> if (xen_igd_gfx_pt_enabled() && (s->hostaddr.slot == 2)
>>>> && (s->hostaddr.function == 0)) {
>>>> 
>&

Re: [PATCH v8] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

On 1/10/23 3:16 AM, Michael S. Tsirkin wrote:
> On Tue, Jan 10, 2023 at 02:08:34AM -0500, Chuck Zmudzinski wrote:
>> Intel specifies that the Intel IGD must occupy slot 2 on the PCI bus,
>> as noted in docs/igd-assign.txt in the Qemu source code.
>> 
>> Currently, when the xl toolstack is used to configure a Xen HVM guest with
>> Intel IGD passthrough to the guest with the Qemu upstream device model,
>> a Qemu emulated PCI device will occupy slot 2 and the Intel IGD will occupy
>> a different slot. This problem often prevents the guest from booting.
>> 
>> The only available workaround is not good: Configure Xen HVM guests to use
>> the old and no longer maintained Qemu traditional device model available
>> from xenbits.xen.org which does reserve slot 2 for the Intel IGD.
>> 
>> To implement this feature in the Qemu upstream device model for Xen HVM
>> guests, introduce the following new functions, types, and macros:
>> 
>> * XEN_PT_DEVICE_CLASS declaration, based on the existing TYPE_XEN_PT_DEVICE
>> * XEN_PT_DEVICE_GET_CLASS macro helper function for XEN_PT_DEVICE_CLASS
>> * typedef XenPTQdevRealize function pointer
>> * XEN_PCI_IGD_SLOT_MASK, the value of slot_reserved_mask to reserve slot 2
>> * xen_igd_reserve_slot and xen_igd_clear_slot functions
>> 
>> The new xen_igd_reserve_slot function uses the existing slot_reserved_mask
>> member of PCIBus to reserve PCI slot 2 for Xen HVM guests configured using
>> the xl toolstack with the gfx_passthru option enabled, which sets the
>> igd-passthru=on option to Qemu for the Xen HVM machine type.
>> 
>> The new xen_igd_reserve_slot function also needs to be implemented in
>> hw/xen/xen_pt_stub.c to prevent FTBFS during the link stage for the case
>> when Qemu is configured with --enable-xen and --disable-xen-pci-passthrough,
>> in which case it does nothing.
>> 
>> The new xen_igd_clear_slot function overrides qdev->realize of the parent
>> PCI device class to enable the Intel IGD to occupy slot 2 on the PCI bus
>> since slot 2 was reserved by xen_igd_reserve_slot when the PCI bus was
>> created in hw/i386/pc_piix.c for the case when igd-passthru=on.
>> 
>> Move the call to xen_host_pci_device_get, and the associated error
>> handling, from xen_pt_realize to the new xen_igd_clear_slot function to
>> initialize the device class and vendor values which enables the checks for
>> the Intel IGD to succeed. The verification that the host device is an
>> Intel IGD to be passed through is done by checking the domain, bus, slot,
>> and function values as well as by checking that gfx_passthru is enabled,
>> the device class is VGA, and the device vendor in Intel.
>> 
>> Signed-off-by: Chuck Zmudzinski 
>> ---
>> Notes that might be helpful to reviewers of patched code in hw/xen:
>> 
>> The new functions and types are based on recommendations from Qemu docs:
>> https://qemu.readthedocs.io/en/latest/devel/qom.html
>> 
>> Notes that might be helpful to reviewers of patched code in hw/i386:
>> 
>> The small patch to hw/i386/pc_piix.c is protected by CONFIG_XEN so it does
>> not affect builds that do not have CONFIG_XEN defined.
>> 
>> xen_igd_gfx_pt_enabled() in the patched hw/i386/pc_piix.c file is an
>> existing function that is only true when Qemu is built with
>> xen-pci-passthrough enabled and the administrator has configured the Xen
>> HVM guest with Qemu's igd-passthru=on option.
>> 
>> v2: Remove From:  tag at top of commit message
>> 
>> v3: Changed the test for the Intel IGD in xen_igd_clear_slot:
>> 
>> if (is_igd_vga_passthrough(>real_device) &&
>> (s->real_device.vendor_id == PCI_VENDOR_ID_INTEL)) {
>> 
>> is changed to
>> 
>> if (xen_igd_gfx_pt_enabled() && (s->hostaddr.slot == 2)
>> && (s->hostaddr.function == 0)) {
>> 
>> I hoped that I could use the test in v2, since it matches the
>> other tests for the Intel IGD in Qemu and Xen, but those tests
>> do not work because the necessary data structures are not set with
>> their values yet. So instead use the test that the administrator
>> has enabled gfx_passthru and the device address on the host is
>> 02.0. This test does detect the Intel IGD correctly.
>> 
>> v4: Use brchu...@aol.com instead of brchu...@netscape.net for the author's
>> email address to match the address used by the same author in commits
>> be9c61da and c0e86b76
>> 
>> Change variable for XEN_PT_DEVICE_CLASS: xptc changed t

Re: [PATCH v8] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

On 1/10/2023 3:16 AM, Michael S. Tsirkin wrote:
> On Tue, Jan 10, 2023 at 02:08:34AM -0500, Chuck Zmudzinski wrote:
> > Intel specifies that the Intel IGD must occupy slot 2 on the PCI bus,
> > as noted in docs/igd-assign.txt in the Qemu source code.
> > 
> > Currently, when the xl toolstack is used to configure a Xen HVM guest with
> > Intel IGD passthrough to the guest with the Qemu upstream device model,
> > a Qemu emulated PCI device will occupy slot 2 and the Intel IGD will occupy
> > a different slot. This problem often prevents the guest from booting.
> > 
> > The only available workaround is not good: Configure Xen HVM guests to use
> > the old and no longer maintained Qemu traditional device model available
> > from xenbits.xen.org which does reserve slot 2 for the Intel IGD.
> > 
> > To implement this feature in the Qemu upstream device model for Xen HVM
> > guests, introduce the following new functions, types, and macros:
> > 
> > * XEN_PT_DEVICE_CLASS declaration, based on the existing TYPE_XEN_PT_DEVICE
> > * XEN_PT_DEVICE_GET_CLASS macro helper function for XEN_PT_DEVICE_CLASS
> > * typedef XenPTQdevRealize function pointer
> > * XEN_PCI_IGD_SLOT_MASK, the value of slot_reserved_mask to reserve slot 2
> > * xen_igd_reserve_slot and xen_igd_clear_slot functions
> > 
> > The new xen_igd_reserve_slot function uses the existing slot_reserved_mask
> > member of PCIBus to reserve PCI slot 2 for Xen HVM guests configured using
> > the xl toolstack with the gfx_passthru option enabled, which sets the
> > igd-passthru=on option to Qemu for the Xen HVM machine type.
> > 
> > The new xen_igd_reserve_slot function also needs to be implemented in
> > hw/xen/xen_pt_stub.c to prevent FTBFS during the link stage for the case
> > when Qemu is configured with --enable-xen and --disable-xen-pci-passthrough,
> > in which case it does nothing.
> > 
> > The new xen_igd_clear_slot function overrides qdev->realize of the parent
> > PCI device class to enable the Intel IGD to occupy slot 2 on the PCI bus
> > since slot 2 was reserved by xen_igd_reserve_slot when the PCI bus was
> > created in hw/i386/pc_piix.c for the case when igd-passthru=on.
> > 
> > Move the call to xen_host_pci_device_get, and the associated error
> > handling, from xen_pt_realize to the new xen_igd_clear_slot function to
> > initialize the device class and vendor values which enables the checks for
> > the Intel IGD to succeed. The verification that the host device is an
> > Intel IGD to be passed through is done by checking the domain, bus, slot,
> > and function values as well as by checking that gfx_passthru is enabled,
> > the device class is VGA, and the device vendor in Intel.
> > 
> > Signed-off-by: Chuck Zmudzinski 
> > ---
> > Notes that might be helpful to reviewers of patched code in hw/xen:
> > 
> > The new functions and types are based on recommendations from Qemu docs:
> > https://qemu.readthedocs.io/en/latest/devel/qom.html
> > 
> > Notes that might be helpful to reviewers of patched code in hw/i386:
> > 
> > The small patch to hw/i386/pc_piix.c is protected by CONFIG_XEN so it does
> > not affect builds that do not have CONFIG_XEN defined.
> > 
> > xen_igd_gfx_pt_enabled() in the patched hw/i386/pc_piix.c file is an
> > existing function that is only true when Qemu is built with
> > xen-pci-passthrough enabled and the administrator has configured the Xen
> > HVM guest with Qemu's igd-passthru=on option.
> > 
> > v2: Remove From:  tag at top of commit message
> > 
> > v3: Changed the test for the Intel IGD in xen_igd_clear_slot:
> > 
> > if (is_igd_vga_passthrough(>real_device) &&
> > (s->real_device.vendor_id == PCI_VENDOR_ID_INTEL)) {
> > 
> > is changed to
> > 
> > if (xen_igd_gfx_pt_enabled() && (s->hostaddr.slot == 2)
> > && (s->hostaddr.function == 0)) {
> > 
> > I hoped that I could use the test in v2, since it matches the
> > other tests for the Intel IGD in Qemu and Xen, but those tests
> > do not work because the necessary data structures are not set with
> > their values yet. So instead use the test that the administrator
> > has enabled gfx_passthru and the device address on the host is
> > 02.0. This test does detect the Intel IGD correctly.
> > 
> > v4: Use brchu...@aol.com instead of brchu...@netscape.net for the author's
> > email address to match the address used by the same author in commits
> > be9c61da and c0e86

[PATCH v8] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

Intel specifies that the Intel IGD must occupy slot 2 on the PCI bus,
as noted in docs/igd-assign.txt in the Qemu source code.

Currently, when the xl toolstack is used to configure a Xen HVM guest with
Intel IGD passthrough to the guest with the Qemu upstream device model,
a Qemu emulated PCI device will occupy slot 2 and the Intel IGD will occupy
a different slot. This problem often prevents the guest from booting.

The only available workaround is not good: Configure Xen HVM guests to use
the old and no longer maintained Qemu traditional device model available
from xenbits.xen.org which does reserve slot 2 for the Intel IGD.

To implement this feature in the Qemu upstream device model for Xen HVM
guests, introduce the following new functions, types, and macros:

* XEN_PT_DEVICE_CLASS declaration, based on the existing TYPE_XEN_PT_DEVICE
* XEN_PT_DEVICE_GET_CLASS macro helper function for XEN_PT_DEVICE_CLASS
* typedef XenPTQdevRealize function pointer
* XEN_PCI_IGD_SLOT_MASK, the value of slot_reserved_mask to reserve slot 2
* xen_igd_reserve_slot and xen_igd_clear_slot functions

The new xen_igd_reserve_slot function uses the existing slot_reserved_mask
member of PCIBus to reserve PCI slot 2 for Xen HVM guests configured using
the xl toolstack with the gfx_passthru option enabled, which sets the
igd-passthru=on option to Qemu for the Xen HVM machine type.

The new xen_igd_reserve_slot function also needs to be implemented in
hw/xen/xen_pt_stub.c to prevent FTBFS during the link stage for the case
when Qemu is configured with --enable-xen and --disable-xen-pci-passthrough,
in which case it does nothing.

The new xen_igd_clear_slot function overrides qdev->realize of the parent
PCI device class to enable the Intel IGD to occupy slot 2 on the PCI bus
since slot 2 was reserved by xen_igd_reserve_slot when the PCI bus was
created in hw/i386/pc_piix.c for the case when igd-passthru=on.

Move the call to xen_host_pci_device_get, and the associated error
handling, from xen_pt_realize to the new xen_igd_clear_slot function to
initialize the device class and vendor values which enables the checks for
the Intel IGD to succeed. The verification that the host device is an
Intel IGD to be passed through is done by checking the domain, bus, slot,
and function values as well as by checking that gfx_passthru is enabled,
the device class is VGA, and the device vendor in Intel.

Signed-off-by: Chuck Zmudzinski 
---
Notes that might be helpful to reviewers of patched code in hw/xen:

The new functions and types are based on recommendations from Qemu docs:
https://qemu.readthedocs.io/en/latest/devel/qom.html

Notes that might be helpful to reviewers of patched code in hw/i386:

The small patch to hw/i386/pc_piix.c is protected by CONFIG_XEN so it does
not affect builds that do not have CONFIG_XEN defined.

xen_igd_gfx_pt_enabled() in the patched hw/i386/pc_piix.c file is an
existing function that is only true when Qemu is built with
xen-pci-passthrough enabled and the administrator has configured the Xen
HVM guest with Qemu's igd-passthru=on option.

v2: Remove From:  tag at top of commit message

v3: Changed the test for the Intel IGD in xen_igd_clear_slot:

if (is_igd_vga_passthrough(>real_device) &&
(s->real_device.vendor_id == PCI_VENDOR_ID_INTEL)) {

is changed to

if (xen_igd_gfx_pt_enabled() && (s->hostaddr.slot == 2)
&& (s->hostaddr.function == 0)) {

I hoped that I could use the test in v2, since it matches the
other tests for the Intel IGD in Qemu and Xen, but those tests
do not work because the necessary data structures are not set with
their values yet. So instead use the test that the administrator
has enabled gfx_passthru and the device address on the host is
02.0. This test does detect the Intel IGD correctly.

v4: Use brchu...@aol.com instead of brchu...@netscape.net for the author's
email address to match the address used by the same author in commits
be9c61da and c0e86b76

Change variable for XEN_PT_DEVICE_CLASS: xptc changed to xpdc

v5: The patch of xen_pt.c was re-worked to allow a more consistent test
for the Intel IGD that uses the same criteria as in other places.
This involved moving the call to xen_host_pci_device_get from
xen_pt_realize to xen_igd_clear_slot and updating the checks for the
Intel IGD in xen_igd_clear_slot:

if (xen_igd_gfx_pt_enabled() && (s->hostaddr.slot == 2)
&& (s->hostaddr.function == 0)) {

is changed to

if (is_igd_vga_passthrough(>real_device) &&
s->real_device.domain == 0 && s->real_device.bus == 0 &&
s->real_device.dev == 2 && s->real_device.func == 0 &&
s->real_device.vendor_id == PCI_VENDOR_ID_INTEL) {

Added an explanation for the move of xen_host_pci_device_get from
xen_pt_realize to xen_igd_cl

Re: [PATCH v7] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

On 1/10/2023 12:27 AM, Michael S. Tsirkin wrote:
> On Mon, Jan 09, 2023 at 07:05:35PM -0500, Chuck Zmudzinski wrote:
> > On 1/9/23 6:33 PM, Michael S. Tsirkin wrote:
> > > On Mon, Jan 09, 2023 at 04:55:42PM -0500, Chuck Zmudzinski wrote:
> > >> Intel specifies that the Intel IGD must occupy slot 2 on the PCI bus,
> > >> as noted in docs/igd-assign.txt in the Qemu source code.
> > >> 
> > >> Currently, when the xl toolstack is used to configure a Xen HVM guest 
> > >> with
> > >> Intel IGD passthrough to the guest with the Qemu upstream device model,
> > >> a Qemu emulated PCI device will occupy slot 2 and the Intel IGD will 
> > >> occupy
> > >> a different slot. This problem often prevents the guest from booting.
> > >> 
> > >> The only available workaround is not good: Configure Xen HVM guests to 
> > >> use
> > >> the old and no longer maintained Qemu traditional device model available
> > >> from xenbits.xen.org which does reserve slot 2 for the Intel IGD.
> > >> 
> > >> To implement this feature in the Qemu upstream device model for Xen HVM
> > >> guests, introduce the following new functions, types, and macros:
> > >> 
> > >> * XEN_PT_DEVICE_CLASS declaration, based on the existing 
> > >> TYPE_XEN_PT_DEVICE
> > >> * XEN_PT_DEVICE_GET_CLASS macro helper function for XEN_PT_DEVICE_CLASS
> > >> * typedef XenPTQdevRealize function pointer
> > >> * XEN_PCI_IGD_SLOT_MASK, the value of slot_reserved_mask to reserve slot 
> > >> 2
> > >> * xen_igd_reserve_slot and xen_igd_clear_slot functions
> > >> 
> > >> The new xen_igd_reserve_slot function uses the existing 
> > >> slot_reserved_mask
> > >> member of PCIBus to reserve PCI slot 2 for Xen HVM guests configured 
> > >> using
> > >> the xl toolstack with the gfx_passthru option enabled, which sets the
> > >> igd-passthru=on option to Qemu for the Xen HVM machine type.
> > >> 
> > >> The new xen_igd_reserve_slot function also needs to be implemented in
> > >> hw/xen/xen_pt_stub.c to prevent FTBFS during the link stage for the case
> > >> when Qemu is configured with --enable-xen and 
> > >> --disable-xen-pci-passthrough,
> > >> in which case it does nothing.
> > >> 
> > >> The new xen_igd_clear_slot function overrides qdev->realize of the parent
> > >> PCI device class to enable the Intel IGD to occupy slot 2 on the PCI bus
> > >> since slot 2 was reserved by xen_igd_reserve_slot when the PCI bus was
> > >> created in hw/i386/pc_piix.c for the case when igd-passthru=on.
> > >> 
> > >> Move the call to xen_host_pci_device_get, and the associated error
> > >> handling, from xen_pt_realize to the new xen_igd_clear_slot function to
> > >> initialize the device class and vendor values which enables the checks 
> > >> for
> > >> the Intel IGD to succeed. The verification that the host device is an
> > >> Intel IGD to be passed through is done by checking the domain, bus, slot,
> > >> and function values as well as by checking that gfx_passthru is enabled,
> > >> the device class is VGA, and the device vendor in Intel.
> > >> 
> > >> Signed-off-by: Chuck Zmudzinski 
> > >> ---
> > >> Notes that might be helpful to reviewers of patched code in hw/xen:
> > >> 
> > >> The new functions and types are based on recommendations from Qemu docs:
> > >> https://qemu.readthedocs.io/en/latest/devel/qom.html
> > >> 
> > >> Notes that might be helpful to reviewers of patched code in hw/i386:
> > >> 
> > >> The small patch to hw/i386/pc_piix.c is protected by CONFIG_XEN so it 
> > >> does
> > >> not affect builds that do not have CONFIG_XEN defined.
> > > 
> > > I'm not sure how you can claim that.
> > 
> > I mean the small patch to pc_piix.c in this patch sits
> > between an "#ifdef CONFIG_XEN" and the corresponding
> > "#endif" so the preprocessor will exclude it when CONFIG_XEN
> > is not defined. In other words, my patch is part of the
> > xen-specific code in pc_piix.c. Or am I missing something?
> > 
> > 
> > > 
> > > ...
> > > 
> > >> diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c
> > >> index b48047f50c..34a9736b5e 100644
> > >>

Re: [PATCH v7] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

On 1/9/2023 7:05 PM, Chuck Zmudzinski wrote:
> On 1/9/23 6:33 PM, Michael S. Tsirkin wrote:
> > On Mon, Jan 09, 2023 at 04:55:42PM -0500, Chuck Zmudzinski wrote:
> >> Intel specifies that the Intel IGD must occupy slot 2 on the PCI bus,
> >> as noted in docs/igd-assign.txt in the Qemu source code.
> >> 
> >> Currently, when the xl toolstack is used to configure a Xen HVM guest with
> >> Intel IGD passthrough to the guest with the Qemu upstream device model,
> >> a Qemu emulated PCI device will occupy slot 2 and the Intel IGD will occupy
> >> a different slot. This problem often prevents the guest from booting.
> >> 
> >> The only available workaround is not good: Configure Xen HVM guests to use
> >> the old and no longer maintained Qemu traditional device model available
> >> from xenbits.xen.org which does reserve slot 2 for the Intel IGD.
> >> 
> >> To implement this feature in the Qemu upstream device model for Xen HVM
> >> guests, introduce the following new functions, types, and macros:
> >> 
> >> * XEN_PT_DEVICE_CLASS declaration, based on the existing TYPE_XEN_PT_DEVICE
> >> * XEN_PT_DEVICE_GET_CLASS macro helper function for XEN_PT_DEVICE_CLASS
> >> * typedef XenPTQdevRealize function pointer
> >> * XEN_PCI_IGD_SLOT_MASK, the value of slot_reserved_mask to reserve slot 2
> >> * xen_igd_reserve_slot and xen_igd_clear_slot functions
> >> 
> >> The new xen_igd_reserve_slot function uses the existing slot_reserved_mask
> >> member of PCIBus to reserve PCI slot 2 for Xen HVM guests configured using
> >> the xl toolstack with the gfx_passthru option enabled, which sets the
> >> igd-passthru=on option to Qemu for the Xen HVM machine type.
> >> 
> >> The new xen_igd_reserve_slot function also needs to be implemented in
> >> hw/xen/xen_pt_stub.c to prevent FTBFS during the link stage for the case
> >> when Qemu is configured with --enable-xen and 
> >> --disable-xen-pci-passthrough,
> >> in which case it does nothing.
> >> 
> >> The new xen_igd_clear_slot function overrides qdev->realize of the parent
> >> PCI device class to enable the Intel IGD to occupy slot 2 on the PCI bus
> >> since slot 2 was reserved by xen_igd_reserve_slot when the PCI bus was
> >> created in hw/i386/pc_piix.c for the case when igd-passthru=on.
> >> 
> >> Move the call to xen_host_pci_device_get, and the associated error
> >> handling, from xen_pt_realize to the new xen_igd_clear_slot function to
> >> initialize the device class and vendor values which enables the checks for
> >> the Intel IGD to succeed. The verification that the host device is an
> >> Intel IGD to be passed through is done by checking the domain, bus, slot,
> >> and function values as well as by checking that gfx_passthru is enabled,
> >> the device class is VGA, and the device vendor in Intel.
> >> 
> >> Signed-off-by: Chuck Zmudzinski 
> >> ---
> >> Notes that might be helpful to reviewers of patched code in hw/xen:
> >> 
> >> The new functions and types are based on recommendations from Qemu docs:
> >> https://qemu.readthedocs.io/en/latest/devel/qom.html
> >> 
> >> Notes that might be helpful to reviewers of patched code in hw/i386:
> >> 
> >> The small patch to hw/i386/pc_piix.c is protected by CONFIG_XEN so it does
> >> not affect builds that do not have CONFIG_XEN defined.
> > 
> > I'm not sure how you can claim that.
>
> I mean the small patch to pc_piix.c in this patch sits
> between an "#ifdef CONFIG_XEN" and the corresponding
> "#endif" so the preprocessor will exclude it when CONFIG_XEN
> is not defined. In other words, my patch is part of the
> xen-specific code in pc_piix.c. Or am I missing something?
>
>
> > 
> > ...
> > 
> >> diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c
> >> index b48047f50c..34a9736b5e 100644
> >> --- a/hw/i386/pc_piix.c
> >> +++ b/hw/i386/pc_piix.c
> >> @@ -32,6 +32,7 @@
> >>  #include "hw/i386/pc.h"
> >>  #include "hw/i386/apic.h"
> >>  #include "hw/pci-host/i440fx.h"
> >> +#include "hw/rtc/mc146818rtc.h"
> >>  #include "hw/southbridge/piix.h"
> >>  #include "hw/display/ramfb.h"
> >>  #include "hw/firmware/smbios.h"
> >> @@ -40,16 +41,16 @@
> >>  #include "hw/usb.h"
> >>  #include "net/net.h"
> >>  #include &q

Re: [PATCH v7] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

On 1/9/23 6:35 PM, Michael S. Tsirkin wrote:
> On Mon, Jan 09, 2023 at 06:28:44PM -0500, Chuck Zmudzinski wrote:
>> On 1/9/23 5:33 PM, Michael S. Tsirkin wrote:
>> > On Mon, Jan 09, 2023 at 04:55:42PM -0500, Chuck Zmudzinski wrote:
>> >> Intel specifies that the Intel IGD must occupy slot 2 on the PCI bus,
>> >> as noted in docs/igd-assign.txt in the Qemu source code.
>> >> 
>> >> Currently, when the xl toolstack is used to configure a Xen HVM guest with
>> >> Intel IGD passthrough to the guest with the Qemu upstream device model,
>> >> a Qemu emulated PCI device will occupy slot 2 and the Intel IGD will 
>> >> occupy
>> >> a different slot. This problem often prevents the guest from booting.
>> >> 
>> >> The only available workaround is not good: Configure Xen HVM guests to use
>> >> the old and no longer maintained Qemu traditional device model available
>> >> from xenbits.xen.org which does reserve slot 2 for the Intel IGD.
>> >> 
>> >> To implement this feature in the Qemu upstream device model for Xen HVM
>> >> guests, introduce the following new functions, types, and macros:
>> >> 
>> >> * XEN_PT_DEVICE_CLASS declaration, based on the existing 
>> >> TYPE_XEN_PT_DEVICE
>> >> * XEN_PT_DEVICE_GET_CLASS macro helper function for XEN_PT_DEVICE_CLASS
>> >> * typedef XenPTQdevRealize function pointer
>> >> * XEN_PCI_IGD_SLOT_MASK, the value of slot_reserved_mask to reserve slot 2
>> >> * xen_igd_reserve_slot and xen_igd_clear_slot functions
>> >> 
>> >> The new xen_igd_reserve_slot function uses the existing slot_reserved_mask
>> >> member of PCIBus to reserve PCI slot 2 for Xen HVM guests configured using
>> >> the xl toolstack with the gfx_passthru option enabled, which sets the
>> >> igd-passthru=on option to Qemu for the Xen HVM machine type.
>> > 
>> > I don't like how slot_reserved_mask is set initially then cleared on
>> > device realize.
>> > To me this looks like a fragile hack. I suggest one of the following
>> > 1. adding a new mask
>> > "slot-manual-mask" or some such blocking auto-allocation of a given
>> > slot without blocking its use if address is specified on command line.
>> > 2. adding a special property that overrides slot_reserved_mask
>> > for a given device.
>> > 
>> > both need changes in pci core but look like something generally
>> > useful.
>> 
>> I was hoping to not need to touch pci core but I understand it would be
>> better for this patch to not affect machines that are manually configured
>> on the command line.
>> 
>> However, keep in mind that this patch will only actually reserve the slot
>> initially for xen hvm machines (machine type "xenfv") that also are 
>> configured
>> with the qemu igd-passthru=on option which, AFAIK, only applies to machines
>> witn accel=xen. It will not affect kvm users at all. So I don't think this 
>> patch
>> will break many machines out there that manually specify the pci slots. The
>> only machines it could affect are machines configured for igd-passthru on 
>> xen.
>> This patch also does *not* reserve the slot initially for "xenfv" machines 
>> that
>> are not configured with igd passthrough which I am sure is the vast majority
>> of all the xen virtual machines out in the wild.
> 
> I'm just saying that adding a capability that is generally useful
> as opposed to xen specific means less technical debt.
> 

I agree with that.

Re: [PATCH v7] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

On 1/9/23 6:33 PM, Michael S. Tsirkin wrote:
> On Mon, Jan 09, 2023 at 04:55:42PM -0500, Chuck Zmudzinski wrote:
>> Intel specifies that the Intel IGD must occupy slot 2 on the PCI bus,
>> as noted in docs/igd-assign.txt in the Qemu source code.
>> 
>> Currently, when the xl toolstack is used to configure a Xen HVM guest with
>> Intel IGD passthrough to the guest with the Qemu upstream device model,
>> a Qemu emulated PCI device will occupy slot 2 and the Intel IGD will occupy
>> a different slot. This problem often prevents the guest from booting.
>> 
>> The only available workaround is not good: Configure Xen HVM guests to use
>> the old and no longer maintained Qemu traditional device model available
>> from xenbits.xen.org which does reserve slot 2 for the Intel IGD.
>> 
>> To implement this feature in the Qemu upstream device model for Xen HVM
>> guests, introduce the following new functions, types, and macros:
>> 
>> * XEN_PT_DEVICE_CLASS declaration, based on the existing TYPE_XEN_PT_DEVICE
>> * XEN_PT_DEVICE_GET_CLASS macro helper function for XEN_PT_DEVICE_CLASS
>> * typedef XenPTQdevRealize function pointer
>> * XEN_PCI_IGD_SLOT_MASK, the value of slot_reserved_mask to reserve slot 2
>> * xen_igd_reserve_slot and xen_igd_clear_slot functions
>> 
>> The new xen_igd_reserve_slot function uses the existing slot_reserved_mask
>> member of PCIBus to reserve PCI slot 2 for Xen HVM guests configured using
>> the xl toolstack with the gfx_passthru option enabled, which sets the
>> igd-passthru=on option to Qemu for the Xen HVM machine type.
>> 
>> The new xen_igd_reserve_slot function also needs to be implemented in
>> hw/xen/xen_pt_stub.c to prevent FTBFS during the link stage for the case
>> when Qemu is configured with --enable-xen and --disable-xen-pci-passthrough,
>> in which case it does nothing.
>> 
>> The new xen_igd_clear_slot function overrides qdev->realize of the parent
>> PCI device class to enable the Intel IGD to occupy slot 2 on the PCI bus
>> since slot 2 was reserved by xen_igd_reserve_slot when the PCI bus was
>> created in hw/i386/pc_piix.c for the case when igd-passthru=on.
>> 
>> Move the call to xen_host_pci_device_get, and the associated error
>> handling, from xen_pt_realize to the new xen_igd_clear_slot function to
>> initialize the device class and vendor values which enables the checks for
>> the Intel IGD to succeed. The verification that the host device is an
>> Intel IGD to be passed through is done by checking the domain, bus, slot,
>> and function values as well as by checking that gfx_passthru is enabled,
>> the device class is VGA, and the device vendor in Intel.
>> 
>> Signed-off-by: Chuck Zmudzinski 
>> ---
>> Notes that might be helpful to reviewers of patched code in hw/xen:
>> 
>> The new functions and types are based on recommendations from Qemu docs:
>> https://qemu.readthedocs.io/en/latest/devel/qom.html
>> 
>> Notes that might be helpful to reviewers of patched code in hw/i386:
>> 
>> The small patch to hw/i386/pc_piix.c is protected by CONFIG_XEN so it does
>> not affect builds that do not have CONFIG_XEN defined.
> 
> I'm not sure how you can claim that.

I mean the small patch to pc_piix.c in this patch sits
between an "#ifdef CONFIG_XEN" and the corresponding
"#endif" so the preprocessor will exclude it when CONFIG_XEN
is not defined. In other words, my patch is part of the
xen-specific code in pc_piix.c. Or am I missing something?


> 
> ...
> 
>> diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c
>> index b48047f50c..34a9736b5e 100644
>> --- a/hw/i386/pc_piix.c
>> +++ b/hw/i386/pc_piix.c
>> @@ -32,6 +32,7 @@
>>  #include "hw/i386/pc.h"
>>  #include "hw/i386/apic.h"
>>  #include "hw/pci-host/i440fx.h"
>> +#include "hw/rtc/mc146818rtc.h"
>>  #include "hw/southbridge/piix.h"
>>  #include "hw/display/ramfb.h"
>>  #include "hw/firmware/smbios.h"
>> @@ -40,16 +41,16 @@
>>  #include "hw/usb.h"
>>  #include "net/net.h"
>>  #include "hw/ide/pci.h"
>> -#include "hw/ide/piix.h"
>>  #include "hw/irq.h"
>>  #include "sysemu/kvm.h"
>>  #include "hw/kvm/clock.h"
>>  #include "hw/sysbus.h"
>> +#include "hw/i2c/i2c.h"
>>  #include "hw/i2c/smbus_eeprom.h"
>>  #include "hw/xen/xen-x86.h"
>> +#include "hw/xen/xen.h"
>>  #include "exec/memory.h"
>>

Re: [PATCH v7] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

On 1/9/23 5:33 PM, Michael S. Tsirkin wrote:
> On Mon, Jan 09, 2023 at 04:55:42PM -0500, Chuck Zmudzinski wrote:
>> Intel specifies that the Intel IGD must occupy slot 2 on the PCI bus,
>> as noted in docs/igd-assign.txt in the Qemu source code.
>> 
>> Currently, when the xl toolstack is used to configure a Xen HVM guest with
>> Intel IGD passthrough to the guest with the Qemu upstream device model,
>> a Qemu emulated PCI device will occupy slot 2 and the Intel IGD will occupy
>> a different slot. This problem often prevents the guest from booting.
>> 
>> The only available workaround is not good: Configure Xen HVM guests to use
>> the old and no longer maintained Qemu traditional device model available
>> from xenbits.xen.org which does reserve slot 2 for the Intel IGD.
>> 
>> To implement this feature in the Qemu upstream device model for Xen HVM
>> guests, introduce the following new functions, types, and macros:
>> 
>> * XEN_PT_DEVICE_CLASS declaration, based on the existing TYPE_XEN_PT_DEVICE
>> * XEN_PT_DEVICE_GET_CLASS macro helper function for XEN_PT_DEVICE_CLASS
>> * typedef XenPTQdevRealize function pointer
>> * XEN_PCI_IGD_SLOT_MASK, the value of slot_reserved_mask to reserve slot 2
>> * xen_igd_reserve_slot and xen_igd_clear_slot functions
>> 
>> The new xen_igd_reserve_slot function uses the existing slot_reserved_mask
>> member of PCIBus to reserve PCI slot 2 for Xen HVM guests configured using
>> the xl toolstack with the gfx_passthru option enabled, which sets the
>> igd-passthru=on option to Qemu for the Xen HVM machine type.
> 
> I don't like how slot_reserved_mask is set initially then cleared on
> device realize.
> To me this looks like a fragile hack. I suggest one of the following
> 1. adding a new mask
> "slot-manual-mask" or some such blocking auto-allocation of a given
> slot without blocking its use if address is specified on command line.
> 2. adding a special property that overrides slot_reserved_mask
> for a given device.
> 
> both need changes in pci core but look like something generally
> useful.

I was hoping to not need to touch pci core but I understand it would be
better for this patch to not affect machines that are manually configured
on the command line.

However, keep in mind that this patch will only actually reserve the slot
initially for xen hvm machines (machine type "xenfv") that also are configured
with the qemu igd-passthru=on option which, AFAIK, only applies to machines
witn accel=xen. It will not affect kvm users at all. So I don't think this patch
will break many machines out there that manually specify the pci slots. The
only machines it could affect are machines configured for igd-passthru on xen.
This patch also does *not* reserve the slot initially for "xenfv" machines that
are not configured with igd passthrough which I am sure is the vast majority
of all the xen virtual machines out in the wild.

> 
> 
>> 
>> The new xen_igd_reserve_slot function also needs to be implemented in
>> hw/xen/xen_pt_stub.c to prevent FTBFS during the link stage for the case
>> when Qemu is configured with --enable-xen and --disable-xen-pci-passthrough,
>> in which case it does nothing.
>> 
>> The new xen_igd_clear_slot function overrides qdev->realize of the parent
>> PCI device class to enable the Intel IGD to occupy slot 2 on the PCI bus
>> since slot 2 was reserved by xen_igd_reserve_slot when the PCI bus was
>> created in hw/i386/pc_piix.c for the case when igd-passthru=on.
>> 
>> Move the call to xen_host_pci_device_get, and the associated error
>> handling, from xen_pt_realize to the new xen_igd_clear_slot function to
>> initialize the device class and vendor values which enables the checks for
>> the Intel IGD to succeed. The verification that the host device is an
>> Intel IGD to be passed through is done by checking the domain, bus, slot,
>> and function values as well as by checking that gfx_passthru is enabled,
>> the device class is VGA, and the device vendor in Intel.
>> 
>> Signed-off-by: Chuck Zmudzinski 
>> ---
>> Notes that might be helpful to reviewers of patched code in hw/xen:
>> 
>> The new functions and types are based on recommendations from Qemu docs:
>> https://qemu.readthedocs.io/en/latest/devel/qom.html
>> 
>> Notes that might be helpful to reviewers of patched code in hw/i386:
>> 
>> The small patch to hw/i386/pc_piix.c is protected by CONFIG_XEN so it does
>> not affect builds that do not have CONFIG_XEN defined.
>> 
>> xen_igd_gfx_pt_enabled() in the patched hw/i386/pc_piix.c file is an
>> existing function that

[XEN PATCH 3/3] libxl/dm: Assign slot 2 by default for Intel IGD passthrough

[Chuck Zmudzinski]

It is possible for the administrator to manually specify the virtual
slot addresses of passed through pci devices on the guest's pci bus
using the @VSLOT parameter in xl.cfg. With this patch, libxl will by
default assign the Intel IGD to slot 2 when gfx_passthru is configured
for the Intel IGD so it will no longer be necessary to use the @VSLOT
setting to configure the IGD correctly. Also, with this patch, libxl
will not override explicit @VSLOT settings by the administrator so
in that case the patch will have no effect on guest behavior.

The default behavior of letting qemu manage the slot addresses of passed
through pci devices when gfx_passthru is disabled and the administrator
does not set @VSLOT for passed through pci devices is also preserved.

Signed-off-by: Chuck Zmudzinski 
---
 tools/libs/light/libxl_dm.c | 22 ++
 1 file changed, 22 insertions(+)

diff --git a/tools/libs/light/libxl_dm.c b/tools/libs/light/libxl_dm.c
index 2720b5d4d0..b51ebae643 100644
--- a/tools/libs/light/libxl_dm.c
+++ b/tools/libs/light/libxl_dm.c
@@ -1207,6 +1207,7 @@ static int libxl__build_device_model_args_new(libxl__gc 
*gc,
 int rc;
 int next_slot;
 bool configure_pci_for_igd = false;
+const int igd_slot = 2;
 /*
  * next_slot is only used when we need to configure the pci
  * slots for the Intel IGD. Slot 2 will be for the Intel IGD.
@@ -2173,6 +2174,27 @@ static int libxl__build_device_model_args_new(libxl__gc 
*gc,
 flexarray_append(dm_envs, NULL);
 if (envs)
 *envs = (char **) flexarray_contents(dm_envs);
+if (configure_pci_for_igd) {
+libxl_device_pci *pci = NULL;
+for (i = 0; i < guest_config->num_pcidevs; i++) {
+pci = _config->pcidevs[i];
+if (!pci->vdevfn) {
+/*
+ * Find the Intel IGD and configure it for slot 2.
+ * Configure any other devices for slot next_slot.
+ * Since the guest is configured for IGD passthrough,
+ * assume the device on the host at slot 2 is the IGD.
+ */
+if (pci->domain == 0 && pci->bus == 0 &&
+pci->dev == igd_slot && pci->func == 0) {
+pci->vdevfn = PCI_DEVFN(igd_slot, 0);
+} else {
+pci->vdevfn = PCI_DEVFN(next_slot, 0);
+next_slot++;
+}
+}
+}
+}
 return 0;
 }
 
-- 
2.39.0

[XEN PATCH 2/3] libxl/dm: Manage pci slot assignment for Intel IGD passthrough

[Chuck Zmudzinski]

By default, except for the ich9-usb-uhci device which libxl assigns to
slot 29 (0xid), libxl defers to qemu upstream's automatic slot assignment
process, which is simply to assign each emulated device to the next
available slot on the pci bus. With this default behavior, libxl and
qemu will not configure the Intel IGD correctly. Specifically, the Intel
IGD must be assigned to slot 2, but the current default behavior is to
assign one of the emulated devices to slot 2.

With this patch libxl uses qemu command line options to specify the slot
address of each pci device in the guest when gfx_passthru is enabled
for the Intel IGD. It uses the same simple algorithm of assigning the
next available slot, except that it starts with slot 3 instead of slot 2
for the emulated devices. This process of slot assignment aims to
simulate the behavior of existing machines as much as possible.

The default behavior (when igd gfx_passthru is disabled) of letting qemu
manage the slot addresses of emulated pci devices is preserved. The
patch also preserves the special treatment for the ich9 usb2 controller
(ich9-usb-ehci1) that libxl currently assigns to slot.function 29.7 and
the associated ich9-usb-uhciN devices to slot 29 (0x1d).

For future maintainance of this code, it is important that pci devices
that are managed by the libxl__build_device_model_args_new function
follow the logic of this patch and use the new local counter next_slot
to assign the slot address instead of letting upstream qemu assign the
slot if the guest is configured for Intel IGD passthrough, and preserve
the current behavior of letting qemu assign the slot address otherwise.

Signed-off-by: Chuck Zmudzinski 
---
The diff of this patch is easier to review if it is generated using
the -b (aka --ignore-space-change) option of diff/git-diff to filter
out some of the changes that are only due to white space.

This patch is difficult to verify for correctness without testing all
the devices that could be added by the libxl__build_device_model_args_new
function. There are 12 places where the addr=%x option needed to be
added to the arguments of the "-device" qemu option that adds an
emulated pci device which corresponds to at least 12 different devices
that could be affected by this patch if the patch contains mistakes
that the compiler did not notice. One mistake the compiler would not
notice is a missing next_slot++; statement that would result in qemu
trying to assign a device to a slot that is already assigned, which
is an error in qemu. I did enough tests to find some mistakes in the
patch which of course I fixed before submitting the patch. So I cannot
guarantee that there are not any other mistakes in the patch because
I don't have the resources to do the necessary testing of the many
possible configurations that could be affected by this patch.

 tools/libs/light/libxl_dm.c | 168 ++--
 1 file changed, 141 insertions(+), 27 deletions(-)

diff --git a/tools/libs/light/libxl_dm.c b/tools/libs/light/libxl_dm.c
index 2048815611..2720b5d4d0 100644
--- a/tools/libs/light/libxl_dm.c
+++ b/tools/libs/light/libxl_dm.c
@@ -1205,6 +1205,20 @@ static int libxl__build_device_model_args_new(libxl__gc 
*gc,
 const char *path, *chardev;
 bool is_stubdom = libxl_defbool_val(b_info->device_model_stubdomain);
 int rc;
+int next_slot;
+bool configure_pci_for_igd = false;
+/*
+ * next_slot is only used when we need to configure the pci
+ * slots for the Intel IGD. Slot 2 will be for the Intel IGD.
+ */
+next_slot = 3;
+if (libxl_defbool_val(b_info->u.hvm.gfx_passthru)) {
+enum libxl_gfx_passthru_kind gfx_passthru_kind =
+libxl__detect_gfx_passthru_kind(gc, guest_config);
+if (gfx_passthru_kind == LIBXL_GFX_PASSTHRU_KIND_IGD) {
+configure_pci_for_igd = true;
+}
+}
 
 dm_args = flexarray_make(gc, 16, 1);
 dm_envs = flexarray_make(gc, 16, 1);
@@ -1372,6 +1386,20 @@ static int libxl__build_device_model_args_new(libxl__gc 
*gc,
 
 if (b_info->cmdline)
 flexarray_vappend(dm_args, "-append", b_info->cmdline, NULL);
+/*
+ * When the Intel IGD is configured for primary graphics
+ * passthrough, we need to manually add the xen platform
+ * device because the QEMU machine type is "pc". Add it first to
+ * simulate the behavior of the "xenfv" QEMU machine type which
+ * always adds the xen platform device first. But in this case it
+ * will be at slot 3 because we are reserving slot 2 for the IGD.
+ */
+if (configure_pci_for_igd &&
+libxl_defbool_val(b_info->u.hvm.xen_platform_pci)) {
+flexarray_append_pair(dm_args, "-device",
+GCSPRINTF("xen-platform,addr=%x", next_slot));
+next_slot++;
+

[XEN PATCH 1/3] libxl/dm: Use "pc" machine type for Intel IGD passthrough

[Chuck Zmudzinski]

The default qemu upstream "xenfv" machine type that is used when an HVM
guest is configured for Intel IGD passthrough assigns slot 2 to the
xen platform pci device. It is a requirement that slot 2 be assigned to
the Intel IGD when it is passed through as the primary graphics adapter.
Using the "pc" machine type instead of the "xenfv" machine type in that
case makes it possible for qemu upstream to assign slot 2 to the IGD.

Using the qemu "pc" machine and adding the xen platform device on the
qemu command line instead of using the qemu "xenfv" machine which
automatically adds the xen platform device earlier in the guest creation
process does come with some degredation of startup performance: startup
is slower and some vga drivers in use during early boot are unable to
display the screen at the native resolution of the monitor, but once
the guest operating system (Windows or Linux) is fully loaded, there
is no noticeable difference in the performance of the guest when using
the "pc" machine type instead of the "xenfv" machine type.

With this patch, libxl continues to use default "xenfv" machine type
with the default settings of xen_platform_pci enabled and igd
gfx_passthru disabled. The patch only affects machines configured with
gfx_passthru enabled.

Signed-off-by: Chuck Zmudzinski 
---
Reviewers might find this patch easier to review by looking at the
resulting code in the patched file instead of looking at the diff because
it is hard to follow the logical flow of the resulting code in the diff
because the patch moves the check for igd gfx_passthru before the check
for disabling the xen platform device. That change was made because it
results in a more simplified logical flow in the resulting code.

 tools/libs/light/libxl_dm.c | 37 -
 1 file changed, 20 insertions(+), 17 deletions(-)

diff --git a/tools/libs/light/libxl_dm.c b/tools/libs/light/libxl_dm.c
index fc264a3a13..2048815611 100644
--- a/tools/libs/light/libxl_dm.c
+++ b/tools/libs/light/libxl_dm.c
@@ -1809,7 +1809,26 @@ static int libxl__build_device_model_args_new(libxl__gc 
*gc,
 flexarray_append(dm_args, b_info->extra_pv[i]);
 break;
 case LIBXL_DOMAIN_TYPE_HVM:
-if (!libxl_defbool_val(b_info->u.hvm.xen_platform_pci)) {
+if (libxl_defbool_val(b_info->u.hvm.gfx_passthru)) {
+enum libxl_gfx_passthru_kind gfx_passthru_kind =
+libxl__detect_gfx_passthru_kind(gc, guest_config);
+switch (gfx_passthru_kind) {
+case LIBXL_GFX_PASSTHRU_KIND_IGD:
+/*
+ * Using the machine "pc" because with the default machine 
"xenfv"
+ * the xen-platform device will be assigned to slot 2, but with
+ * GFX_PASSTHRU_KIND_IGD, slot 2 needs to be reserved for the 
Intel IGD.
+ */
+machinearg = libxl__strdup(gc, 
"pc,accel=xen,suppress-vmdesc=on,igd-passthru=on");
+break;
+case LIBXL_GFX_PASSTHRU_KIND_DEFAULT:
+LOGD(ERROR, guest_domid, "unable to detect required 
gfx_passthru_kind");
+return ERROR_FAIL;
+default:
+LOGD(ERROR, guest_domid, "invalid value for 
gfx_passthru_kind");
+return ERROR_INVAL;
+}
+} else if (!libxl_defbool_val(b_info->u.hvm.xen_platform_pci)) {
 /* Switching here to the machine "pc" which does not add
  * the xen-platform device instead of the default "xenfv" machine.
  */
@@ -1831,22 +1850,6 @@ static int libxl__build_device_model_args_new(libxl__gc 
*gc,
 }
 }
 
-if (libxl_defbool_val(b_info->u.hvm.gfx_passthru)) {
-enum libxl_gfx_passthru_kind gfx_passthru_kind =
-libxl__detect_gfx_passthru_kind(gc, guest_config);
-switch (gfx_passthru_kind) {
-case LIBXL_GFX_PASSTHRU_KIND_IGD:
-machinearg = GCSPRINTF("%s,igd-passthru=on", machinearg);
-break;
-case LIBXL_GFX_PASSTHRU_KIND_DEFAULT:
-LOGD(ERROR, guest_domid, "unable to detect required 
gfx_passthru_kind");
-return ERROR_FAIL;
-default:
-LOGD(ERROR, guest_domid, "invalid value for 
gfx_passthru_kind");
-return ERROR_INVAL;
-}
-}
-
 flexarray_append(dm_args, machinearg);
 for (i = 0; b_info->extra_hvm && b_info->extra_hvm[i] != NULL; i++)
 flexarray_append(dm_args, b_info->extra_hvm[i]);
-- 
2.39.0

[XEN PATCH 0/3] Configure qemu upstream correctly by default for igd-passthru

[Chuck Zmudzinski]

ate properly
when passed through as the primary graphics adapter. The qemu
traditional device model takes care of this by reserving slot 2 for the
Intel IGD, but the upstream qemu device model currently does not reserve
slot 2 for the Intel IGD.

This patch series modifies libxl so the upstream qemu device model will
also, with default settings, assign slot 2 for the Intel IGD.

There are three reasons why it is difficult to configure the guest
so the Intel IGD is assigned to slot 2 in the guest using libxl and the
upstream device model, so the patch series is logically organized in
three separate patches; each patch resolves one of the three reasons
that cause problems:

The description of what each of the three libxl patches do:

1. With the default "xenfv" machine type, qemu upstream is hard-coded
   to assign the xen platform device to slot 2. The first patch fixes
   that by using the "pc" machine instead when gfx_passthru type is igd
   and, if xen_platform_pci is set in the guest config, libxl now assigns
   the xen platform device to slot 3, making it possible to assign the
   IGD to slot 2. The patch only affects guests with the gfx_passthru
   option enabled. The default behavior (xen_platform_pci is enabled
   but gfx_passthru option is disabled) of using the "xenfv" machine
   type is preserved. Another way to describe what the patch does is
   to say that it adds a second exception to the default choice of the
   "xenfv" machine type, with the first exception being that the "pc"
   machine type is also used instead of "xenfv" if the xen platform pci
   device is disabled in the guest xl.cfg file.

2. Currently, with libxl and qemu upstream, most emulated pci devices
   are by default automatically assigned a pci slot, and the emulated
   ones are assigned before the passed through ones, which means that
   even if libxl is patched so the xen platform device will not be
   assigned to slot 2, any other emulated device will be assigned slot 2
   unless libxl explicitly assigns the slot address of each emulated pci
   device in such a way that the IGD will be assigned slot 2. The second
   patch fixes this by hard coding the slot assignment for the emulated
   devices instead of deferring to qemu upstream's auto-assignment which
   does not do what is necessary to configure the Intel IGD correctly.
   With the second patch applied, it is possible to configure the Intel
   IGD correctly by using the @VSLOT parameter in xl.cfg to specify the
   slot address of each passed through pci device in the guest. The
   second patch is also designed to not change the default behavior of
   letting qemu autoconfigure the pci slot addresses when igd
   gfx_pasthru is disabled in xl.cfg.  

3. For convenience, the third patch automatically assigns slot 2 to the
   Intel IGD when the gfx_passthru type is igd so with the third patch
   appled it is not necessary to set the @VSLOT parameter to configure
   the Intel IGD correctly.

Testing:

I tested a system with Intel IGD passthrough and two other pci devices
passed through, with and without the xen platform device. I also did
tests on guests without any pci passthrough configured. In all cases
tested, libxl behaved as expected. For example, the device model
arguments are only changed if gfx_passthru is set for the IGD, libxl
respected administrator settings such as @VSLOT and xen_platform_pci
with the patch series applied, and not adding the xen platform device to
the guest caused reduced performance because in that case the guest
could not take advantage of the improvements offered by the Xen PV
drivers in the guest. I tested the following emulated devices on my
setup: xen-platform, e1000, and VGA. I also verified the device that is
added by the "hdtype = 'ahci'" xl.cfg option is configured correctly
with the patch applied. I did not test all 12 devices that could be
affected by patch 2 of the series. These include the intel-hda high
definition audio device, a virtio-serial, device, etc. Once can look
at the second patch for the full list of qemu emulated devices whose
behavior is affected by the second patch of the series when the guest
is configured for igd gfx_passthru. These devices are also subject
to mistakes in the patch not discovered by the compiler, as mentioned
in the comments for reviewers section of the second patch. 

[1] 
https://lore.kernel.org/qemu-devel/8349506149de6d81b0762f17623552c248439e93.1673297742.git.brchu...@aol.com/

Chuck Zmudzinski (3):
  libxl/dm: Use "pc" machine type for Intel IGD passthrough
  libxl/dm: Manage pci slot assignment for Intel IGD passthrough
  libxl/dm: Assign slot 2 by default for Intel IGD passthrough

 tools/libs/light/libxl_dm.c | 227 +---
 1 file changed, 183 insertions(+), 44 deletions(-)

-- 
2.39.0

[PATCH v7] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

Intel specifies that the Intel IGD must occupy slot 2 on the PCI bus,
as noted in docs/igd-assign.txt in the Qemu source code.

Currently, when the xl toolstack is used to configure a Xen HVM guest with
Intel IGD passthrough to the guest with the Qemu upstream device model,
a Qemu emulated PCI device will occupy slot 2 and the Intel IGD will occupy
a different slot. This problem often prevents the guest from booting.

The only available workaround is not good: Configure Xen HVM guests to use
the old and no longer maintained Qemu traditional device model available
from xenbits.xen.org which does reserve slot 2 for the Intel IGD.

To implement this feature in the Qemu upstream device model for Xen HVM
guests, introduce the following new functions, types, and macros:

* XEN_PT_DEVICE_CLASS declaration, based on the existing TYPE_XEN_PT_DEVICE
* XEN_PT_DEVICE_GET_CLASS macro helper function for XEN_PT_DEVICE_CLASS
* typedef XenPTQdevRealize function pointer
* XEN_PCI_IGD_SLOT_MASK, the value of slot_reserved_mask to reserve slot 2
* xen_igd_reserve_slot and xen_igd_clear_slot functions

The new xen_igd_reserve_slot function uses the existing slot_reserved_mask
member of PCIBus to reserve PCI slot 2 for Xen HVM guests configured using
the xl toolstack with the gfx_passthru option enabled, which sets the
igd-passthru=on option to Qemu for the Xen HVM machine type.

The new xen_igd_reserve_slot function also needs to be implemented in
hw/xen/xen_pt_stub.c to prevent FTBFS during the link stage for the case
when Qemu is configured with --enable-xen and --disable-xen-pci-passthrough,
in which case it does nothing.

The new xen_igd_clear_slot function overrides qdev->realize of the parent
PCI device class to enable the Intel IGD to occupy slot 2 on the PCI bus
since slot 2 was reserved by xen_igd_reserve_slot when the PCI bus was
created in hw/i386/pc_piix.c for the case when igd-passthru=on.

Move the call to xen_host_pci_device_get, and the associated error
handling, from xen_pt_realize to the new xen_igd_clear_slot function to
initialize the device class and vendor values which enables the checks for
the Intel IGD to succeed. The verification that the host device is an
Intel IGD to be passed through is done by checking the domain, bus, slot,
and function values as well as by checking that gfx_passthru is enabled,
the device class is VGA, and the device vendor in Intel.

Signed-off-by: Chuck Zmudzinski 
---
Notes that might be helpful to reviewers of patched code in hw/xen:

The new functions and types are based on recommendations from Qemu docs:
https://qemu.readthedocs.io/en/latest/devel/qom.html

Notes that might be helpful to reviewers of patched code in hw/i386:

The small patch to hw/i386/pc_piix.c is protected by CONFIG_XEN so it does
not affect builds that do not have CONFIG_XEN defined.

xen_igd_gfx_pt_enabled() in the patched hw/i386/pc_piix.c file is an
existing function that is only true when Qemu is built with
xen-pci-passthrough enabled and the administrator has configured the Xen
HVM guest with Qemu's igd-passthru=on option.

v2: Remove From:  tag at top of commit message

v3: Changed the test for the Intel IGD in xen_igd_clear_slot:

if (is_igd_vga_passthrough(>real_device) &&
(s->real_device.vendor_id == PCI_VENDOR_ID_INTEL)) {

is changed to

if (xen_igd_gfx_pt_enabled() && (s->hostaddr.slot == 2)
&& (s->hostaddr.function == 0)) {

I hoped that I could use the test in v2, since it matches the
other tests for the Intel IGD in Qemu and Xen, but those tests
do not work because the necessary data structures are not set with
their values yet. So instead use the test that the administrator
has enabled gfx_passthru and the device address on the host is
02.0. This test does detect the Intel IGD correctly.

v4: Use brchu...@aol.com instead of brchu...@netscape.net for the author's
email address to match the address used by the same author in commits
be9c61da and c0e86b76

Change variable for XEN_PT_DEVICE_CLASS: xptc changed to xpdc

v5: The patch of xen_pt.c was re-worked to allow a more consistent test
for the Intel IGD that uses the same criteria as in other places.
This involved moving the call to xen_host_pci_device_get from
xen_pt_realize to xen_igd_clear_slot and updating the checks for the
Intel IGD in xen_igd_clear_slot:

if (xen_igd_gfx_pt_enabled() && (s->hostaddr.slot == 2)
&& (s->hostaddr.function == 0)) {

is changed to

if (is_igd_vga_passthrough(>real_device) &&
s->real_device.domain == 0 && s->real_device.bus == 0 &&
s->real_device.dev == 2 && s->real_device.func == 0 &&
s->real_device.vendor_id == PCI_VENDOR_ID_INTEL) {

Added an explanation for the move of xen_host_pci_device_get from
xen_pt_realize to xen_igd_cl

Re: [PATCH v6] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

On 1/6/2023 10:02 AM, Chuck Zmudzinski wrote:
> On 1/6/23 9:31 AM, Chuck Zmudzinski wrote:
> > On 1/6/23 9:10 AM, Chuck Zmudzinski wrote:
> >> On 1/6/23 9:03 AM, Anthony PERARD wrote:
> >>> On Sun, Jan 01, 2023 at 06:52:03PM -0500, Chuck Zmudzinski wrote:
> >>>> ...
> >>>> 
> >>>> Signed-off-by: Chuck Zmudzinski 
> >>> 
> >>> 
> >>> This patch looks good enough. It only changes the "xenfv" machine so it
> >>> doesn't prevent a proper fix to be done in the toolstack libxl.
> >>> 
> >>> The change in xen_pci_passthrough_class_init() to try to run some code
> >>> before pci_qdev_realize() could potentially break in the future due to
> >>> been uncommon but hopefully that will be ok.
> >>> 
> >>> So if no work to fix libxl appear soon, I'm ok with this patch:

I have a patch that fixes it in libxl. It still needs a few tweaks before it is
ready for submission, but I plan to do that soon, perhaps later today or
tomorrow at the latest.

> > 
> > Well, I can tell you and others who use qemu are more comfortable
> > fixing this in libxl, so hold off for a week or so. I should have
> > a patch to fix this in libxl written and tested by then. If for
> > some reason that does not work out, then we can fix it in qemu.
>
> One last thought: the only donwnside to fixing this in libxl is that
> other toolstacks that configure qemu to use the xenfv machine will not
> benefit from the fix in qemu that would simplify configuring the
> guest correctly for the igd. Other toolstacks would still need to
> override the default behavior of adding the xen platform device at
> slot 2. I think no matter what, we should at least patch qemu to have
> the xen-platform device use slot 3 instead of being automatically assigned
> to slot 2 when igd-passthru=on. The rest of the fix could then be
> implemented in libxl so that other pci devices such as emulated network
> devices, other passed through pci devices, etc., do not take slot 2 when
> gfx_passthru in xl.cfg is set.

I decided to write the patch to libxl to fix this presuming no
changes to qemu. I think dealing with the "qemu behaves
differently starting from version 8 problem" is more trouble
that it's worth, so I am OK with implementing the fix completely
in libxl, which means libxl will now use the "pc" machine type
when igd-passthru=on and xen_platform_pci is true, but my patch
to libxl will still use the "xenfv" machine when xen_platform_pci
is true and igd-passthru is disabled.

Cheers,

Chuck

Re: [PATCH v2 6/6] hw/isa/piix: Resolve redundant TYPE_PIIX3_XEN_DEVICE

[Chuck Zmudzinski]

On 1/7/23 6:05 AM, Bernhard Beschow wrote:
> Am 7. Januar 2023 01:08:46 UTC schrieb Chuck Zmudzinski :
> >On 1/6/23 6:04 PM, Chuck Zmudzinski wrote:
> >> On 1/6/23 2:08 PM, Chuck Zmudzinski wrote:
> >>> On 1/6/23 7:25 AM, Philippe Mathieu-Daudé wrote:
> >>>> On 6/1/23 12:57, Bernhard Beschow wrote:
> >>>>> 
> >>>>> 
> >>>>> Am 4. Januar 2023 15:35:33 UTC schrieb "Philippe Mathieu-Daudé" 
> >>>>> :
> >>>>>> +Markus/Thomas
> >>>>>>
> >>>>>> On 4/1/23 15:44, Bernhard Beschow wrote:
> >>>>>>> During the last patches, TYPE_PIIX3_XEN_DEVICE turned into a clone of
> >>>>>>> TYPE_PIIX3_DEVICE. Remove this redundancy.
> >>>>>>>
> >>>>>>> Signed-off-by: Bernhard Beschow 
> >>>>>>> ---
> >>>>>>>hw/i386/pc_piix.c |  4 +---
> >>>>>>>hw/isa/piix.c | 20 
> >>>>>>>include/hw/southbridge/piix.h |  1 -
> >>>>>>>3 files changed, 1 insertion(+), 24 deletions(-)
> >>>> 
> >>>> 
> >>>>>>>-static void piix3_xen_class_init(ObjectClass *klass, void *data)
> >>>>>>> -{
> >>>>>>> -DeviceClass *dc = DEVICE_CLASS(klass);
> >>>>>>> -PCIDeviceClass *k = PCI_DEVICE_CLASS(klass);
> >>>>>>> -
> >>>>>>> -k->realize = piix3_realize;
> >>>>>>> -/* 82371SB PIIX3 PCI-to-ISA bridge (Step A1) */
> >>>>>>> -k->device_id = PCI_DEVICE_ID_INTEL_82371SB_0;
> >>>>>>> -dc->vmsd = _piix3;
> >>>>>>
> >>>>>> IIUC, since this device is user-creatable, we can't simply remove it
> >>>>>> without going thru the deprecation process.
> >>>>> 
> >>>>> AFAICS this device is actually not user-creatable since 
> >>>>> dc->user_creatable is set to false once in the base class. I think it 
> >>>>> is safe to remove the Xen class unless there are ABI issues.
> >>>> Great news!
> >>> 
> >>> I don't know if this means the device is user-creatable:
> >>> 
> >>> chuckz@bullseye:~$ qemu-system-i386 -device piix3-ide-xen,help
> >>> piix3-ide-xen options:
> >>>   addr=   - Slot and optional function number, example: 
> >>> 06.0 or 06 (default: -1)
> >>>   failover_pair_id=
> >>>   multifunction=   - on/off (default: false)
> >>>   rombar=-  (default: 1)
> >>>   romfile=
> >>>   x-pcie-extcap-init= - on/off (default: true)
> >>>   x-pcie-lnksta-dllla= - on/off (default: true)
> >>> 
> >>> Today I am running qemu-5.2 on Debian 11, so this output is for
> >>> qemu 5.2, and that version of qemu has a piix3-ide-xen device.
> >>> Is that this same device that is being removed? If so, it seems to
> >>> me that at least as of qemu 5.2, the device was user-creatable.
> >>> 
> >>> Chuck
> >> 
> >> Good news! It looks the device was removed as user-creatable since version 
> >> 5.2:
> >> 
> >> chuckz@debian:~$ qemu-system-i386-7.50 -device help | grep piix
> >> name "piix3-usb-uhci", bus PCI
> >> name "piix4-usb-uhci", bus PCI
> >> name "piix3-ide", bus PCI
> >> name "piix4-ide", bus PCI
> >> chuckz@debian:~$ qemu-system-i386-7.50-bernhard-v2 -device help | grep piix
> >> name "piix3-usb-uhci", bus PCI
> >> name "piix4-usb-uhci", bus PCI
> >> name "piix3-ide", bus PCI
> >> name "piix4-ide", bus PCI
> >> chuckz@debian:~$
> >> 
> >> The piix3-ide-xen device is not present either with or without Bernhard's 
> >> patches
> >> for current qemu 7.50, the development version for qemu 8.0
> >> 
> >> Cheers,
> >> 
> >> Chuck
> >
> >
> >I traced where the pciix3-ide-xen device was removed:
> >
> >It was 7851b21a81 (hw/ide/piix: Remove redundant "piix3-ide-xen" device 
> >class)
> >
> >https://gitlab.com/qemu-project/qemu/-/commit/7851b21a8192750adecbcf6e8780a20de5891ad6
> >
> >about six months ago. That was between 7.0 and 7.1. So the device being 
> >removed
> >here is definitely not user-creatable, but it appears that this piix3-ide-xen
> >device that was removed between 7.0 and 7.1 was user-creatable. Does that one
> >need to go through the deprecation process? Or, since no one has complained
> >it is gone, maybe we don't need to worry about it?
>
> Good point! Looks like it fell through the cracks...
>
> There are voices who claim that this device and its non-Xen counterpart 
> should have never been user-creatable in the firtst place:
> https://patchwork.kernel.org/project/qemu-devel/patch/20190718091740.6834-1-phi...@redhat.com/

Of course, only the xen variant was removed, so only users of the
xen variant were affected by the removal of the device. Any affected
users probably just substituted the non-xen variant for the xen variant
in their machines and didn't experience any problems.

Kind regards,

Chuck

Re: [PATCH v2 6/6] hw/isa/piix: Resolve redundant TYPE_PIIX3_XEN_DEVICE

[Chuck Zmudzinski]

On 1/6/23 6:04 PM, Chuck Zmudzinski wrote:
> On 1/6/23 2:08 PM, Chuck Zmudzinski wrote:
>> On 1/6/23 7:25 AM, Philippe Mathieu-Daudé wrote:
>>> On 6/1/23 12:57, Bernhard Beschow wrote:
>>>> 
>>>> 
>>>> Am 4. Januar 2023 15:35:33 UTC schrieb "Philippe Mathieu-Daudé" 
>>>> :
>>>>> +Markus/Thomas
>>>>>
>>>>> On 4/1/23 15:44, Bernhard Beschow wrote:
>>>>>> During the last patches, TYPE_PIIX3_XEN_DEVICE turned into a clone of
>>>>>> TYPE_PIIX3_DEVICE. Remove this redundancy.
>>>>>>
>>>>>> Signed-off-by: Bernhard Beschow 
>>>>>> ---
>>>>>>hw/i386/pc_piix.c |  4 +---
>>>>>>hw/isa/piix.c | 20 
>>>>>>include/hw/southbridge/piix.h |  1 -
>>>>>>3 files changed, 1 insertion(+), 24 deletions(-)
>>> 
>>> 
>>>>>>-static void piix3_xen_class_init(ObjectClass *klass, void *data)
>>>>>> -{
>>>>>> -DeviceClass *dc = DEVICE_CLASS(klass);
>>>>>> -PCIDeviceClass *k = PCI_DEVICE_CLASS(klass);
>>>>>> -
>>>>>> -k->realize = piix3_realize;
>>>>>> -/* 82371SB PIIX3 PCI-to-ISA bridge (Step A1) */
>>>>>> -k->device_id = PCI_DEVICE_ID_INTEL_82371SB_0;
>>>>>> -dc->vmsd = _piix3;
>>>>>
>>>>> IIUC, since this device is user-creatable, we can't simply remove it
>>>>> without going thru the deprecation process.
>>>> 
>>>> AFAICS this device is actually not user-creatable since dc->user_creatable 
>>>> is set to false once in the base class. I think it is safe to remove the 
>>>> Xen class unless there are ABI issues.
>>> Great news!
>> 
>> I don't know if this means the device is user-creatable:
>> 
>> chuckz@bullseye:~$ qemu-system-i386 -device piix3-ide-xen,help
>> piix3-ide-xen options:
>>   addr=   - Slot and optional function number, example: 06.0 
>> or 06 (default: -1)
>>   failover_pair_id=
>>   multifunction=   - on/off (default: false)
>>   rombar=-  (default: 1)
>>   romfile=
>>   x-pcie-extcap-init= - on/off (default: true)
>>   x-pcie-lnksta-dllla= - on/off (default: true)
>> 
>> Today I am running qemu-5.2 on Debian 11, so this output is for
>> qemu 5.2, and that version of qemu has a piix3-ide-xen device.
>> Is that this same device that is being removed? If so, it seems to
>> me that at least as of qemu 5.2, the device was user-creatable.
>> 
>> Chuck
> 
> Good news! It looks the device was removed as user-creatable since version 
> 5.2:
> 
> chuckz@debian:~$ qemu-system-i386-7.50 -device help | grep piix
> name "piix3-usb-uhci", bus PCI
> name "piix4-usb-uhci", bus PCI
> name "piix3-ide", bus PCI
> name "piix4-ide", bus PCI
> chuckz@debian:~$ qemu-system-i386-7.50-bernhard-v2 -device help | grep piix
> name "piix3-usb-uhci", bus PCI
> name "piix4-usb-uhci", bus PCI
> name "piix3-ide", bus PCI
> name "piix4-ide", bus PCI
> chuckz@debian:~$
> 
> The piix3-ide-xen device is not present either with or without Bernhard's 
> patches
> for current qemu 7.50, the development version for qemu 8.0
> 
> Cheers,
> 
> Chuck


I traced where the pciix3-ide-xen device was removed:

It was 7851b21a81 (hw/ide/piix: Remove redundant "piix3-ide-xen" device class)

https://gitlab.com/qemu-project/qemu/-/commit/7851b21a8192750adecbcf6e8780a20de5891ad6

about six months ago. That was between 7.0 and 7.1. So the device being removed
here is definitely not user-creatable, but it appears that this piix3-ide-xen
device that was removed between 7.0 and 7.1 was user-creatable. Does that one
need to go through the deprecation process? Or, since no one has complained
it is gone, maybe we don't need to worry about it?

Cheers,

Chuck

Re: [PATCH v2 6/6] hw/isa/piix: Resolve redundant TYPE_PIIX3_XEN_DEVICE

[Chuck Zmudzinski]

On 1/6/23 2:08 PM, Chuck Zmudzinski wrote:
> On 1/6/23 7:25 AM, Philippe Mathieu-Daudé wrote:
>> On 6/1/23 12:57, Bernhard Beschow wrote:
>>> 
>>> 
>>> Am 4. Januar 2023 15:35:33 UTC schrieb "Philippe Mathieu-Daudé" 
>>> :
>>>> +Markus/Thomas
>>>>
>>>> On 4/1/23 15:44, Bernhard Beschow wrote:
>>>>> During the last patches, TYPE_PIIX3_XEN_DEVICE turned into a clone of
>>>>> TYPE_PIIX3_DEVICE. Remove this redundancy.
>>>>>
>>>>> Signed-off-by: Bernhard Beschow 
>>>>> ---
>>>>>hw/i386/pc_piix.c |  4 +---
>>>>>hw/isa/piix.c | 20 
>>>>>include/hw/southbridge/piix.h |  1 -
>>>>>3 files changed, 1 insertion(+), 24 deletions(-)
>> 
>> 
>>>>>-static void piix3_xen_class_init(ObjectClass *klass, void *data)
>>>>> -{
>>>>> -DeviceClass *dc = DEVICE_CLASS(klass);
>>>>> -PCIDeviceClass *k = PCI_DEVICE_CLASS(klass);
>>>>> -
>>>>> -k->realize = piix3_realize;
>>>>> -/* 82371SB PIIX3 PCI-to-ISA bridge (Step A1) */
>>>>> -k->device_id = PCI_DEVICE_ID_INTEL_82371SB_0;
>>>>> -dc->vmsd = _piix3;
>>>>
>>>> IIUC, since this device is user-creatable, we can't simply remove it
>>>> without going thru the deprecation process.
>>> 
>>> AFAICS this device is actually not user-creatable since dc->user_creatable 
>>> is set to false once in the base class. I think it is safe to remove the 
>>> Xen class unless there are ABI issues.
>> Great news!
> 
> I don't know if this means the device is user-creatable:
> 
> chuckz@bullseye:~$ qemu-system-i386 -device piix3-ide-xen,help
> piix3-ide-xen options:
>   addr=   - Slot and optional function number, example: 06.0 
> or 06 (default: -1)
>   failover_pair_id=
>   multifunction=   - on/off (default: false)
>   rombar=-  (default: 1)
>   romfile=
>   x-pcie-extcap-init= - on/off (default: true)
>   x-pcie-lnksta-dllla= - on/off (default: true)
> 
> Today I am running qemu-5.2 on Debian 11, so this output is for
> qemu 5.2, and that version of qemu has a piix3-ide-xen device.
> Is that this same device that is being removed? If so, it seems to
> me that at least as of qemu 5.2, the device was user-creatable.
> 
> Chuck

Good news! It looks the device was removed as user-creatable since version 5.2:

chuckz@debian:~$ qemu-system-i386-7.50 -device help | grep piix
name "piix3-usb-uhci", bus PCI
name "piix4-usb-uhci", bus PCI
name "piix3-ide", bus PCI
name "piix4-ide", bus PCI
chuckz@debian:~$ qemu-system-i386-7.50-bernhard-v2 -device help | grep piix
name "piix3-usb-uhci", bus PCI
name "piix4-usb-uhci", bus PCI
name "piix3-ide", bus PCI
name "piix4-ide", bus PCI
chuckz@debian:~$

The piix3-ide-xen device is not present either with or without Bernhard's 
patches
for current qemu 7.50, the development version for qemu 8.0

Cheers,

Chuck

Re: [PATCH v2 6/6] hw/isa/piix: Resolve redundant TYPE_PIIX3_XEN_DEVICE

[Chuck Zmudzinski]

On 1/6/23 7:25 AM, Philippe Mathieu-Daudé wrote:
> On 6/1/23 12:57, Bernhard Beschow wrote:
>> 
>> 
>> Am 4. Januar 2023 15:35:33 UTC schrieb "Philippe Mathieu-Daudé" 
>> :
>>> +Markus/Thomas
>>>
>>> On 4/1/23 15:44, Bernhard Beschow wrote:
 During the last patches, TYPE_PIIX3_XEN_DEVICE turned into a clone of
 TYPE_PIIX3_DEVICE. Remove this redundancy.

 Signed-off-by: Bernhard Beschow 
 ---
hw/i386/pc_piix.c |  4 +---
hw/isa/piix.c | 20 
include/hw/southbridge/piix.h |  1 -
3 files changed, 1 insertion(+), 24 deletions(-)
> 
> 
-static void piix3_xen_class_init(ObjectClass *klass, void *data)
 -{
 -DeviceClass *dc = DEVICE_CLASS(klass);
 -PCIDeviceClass *k = PCI_DEVICE_CLASS(klass);
 -
 -k->realize = piix3_realize;
 -/* 82371SB PIIX3 PCI-to-ISA bridge (Step A1) */
 -k->device_id = PCI_DEVICE_ID_INTEL_82371SB_0;
 -dc->vmsd = _piix3;
>>>
>>> IIUC, since this device is user-creatable, we can't simply remove it
>>> without going thru the deprecation process.
>> 
>> AFAICS this device is actually not user-creatable since dc->user_creatable 
>> is set to false once in the base class. I think it is safe to remove the Xen 
>> class unless there are ABI issues.
> Great news!

I don't know if this means the device is user-creatable:

chuckz@bullseye:~$ qemu-system-i386 -device piix3-ide-xen,help
piix3-ide-xen options:
  addr=   - Slot and optional function number, example: 06.0 or 
06 (default: -1)
  failover_pair_id=
  multifunction=   - on/off (default: false)
  rombar=-  (default: 1)
  romfile=
  x-pcie-extcap-init= - on/off (default: true)
  x-pcie-lnksta-dllla= - on/off (default: true)

Today I am running qemu-5.2 on Debian 11, so this output is for
qemu 5.2, and that version of qemu has a piix3-ide-xen device.
Is that this same device that is being removed? If so, it seems to
me that at least as of qemu 5.2, the device was user-creatable.

Chuck

Re: [PATCH v2 3/6] hw/isa/piix: Wire up Xen PCI IRQ handling outside of PIIX3

[Chuck Zmudzinski]

On 1/6/23 12:46 PM, Chuck Zmudzinski wrote:
> On 1/6/23 12:35 PM, David Woodhouse wrote:
>> On Wed, 2023-01-04 at 15:44 +0100, Bernhard Beschow wrote:
>>> +    if (xen_enabled()) {
>> 
>> Could this perhaps be if (xen_mode != XEN_DISABLED) once we merge the
>> Xen-on-KVM series?
> 
> I am not an expert and just on here as a user/tester, but I think it
> would depend on whether or not the Xen-on-KVM mode uses Xen PCI IRQ
> handling or Linux/KVM PCI IRQ handling.
> 
> Chuck

I could try Bernhard's patches in a Xen-on-KVM configuration if that might help.
I would need help configuring the Xen-on-KVM mode to do it, though, because
I have never tried Xen-on-KVM before.

The test I could do would be to:

1) Test my Xen guest that uses the current PIIX3-xen device in the
Xen-on-KVM environment and get that working.

2) Apply Bernhard's patch series and see what, if anything, needs to
be done to make Bernhard's patch work in Xen-on-KVM. I have already
verified that Bernhard's patches work well with Xen on bare metal, so I
don't think we need to answer this question before going forward with
Bernhard's patches. This can be patched later to support Xen-on-KVM if
necessary as part of or in addition to the Xen-on-KVM series.

Chuck

Re: [PATCH v2 3/6] hw/isa/piix: Wire up Xen PCI IRQ handling outside of PIIX3

[Chuck Zmudzinski]

On 1/6/23 12:35 PM, David Woodhouse wrote:
> On Wed, 2023-01-04 at 15:44 +0100, Bernhard Beschow wrote:
>> +    if (xen_enabled()) {
> 
> Could this perhaps be if (xen_mode != XEN_DISABLED) once we merge the
> Xen-on-KVM series?

I am not an expert and just on here as a user/tester, but I think it
would depend on whether or not the Xen-on-KVM mode uses Xen PCI IRQ
handling or Linux/KVM PCI IRQ handling.

Chuck

Re: [PATCH v6] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

On 1/6/23 9:31 AM, Chuck Zmudzinski wrote:
> On 1/6/23 9:10 AM, Chuck Zmudzinski wrote:
>> On 1/6/23 9:03 AM, Anthony PERARD wrote:
>>> On Sun, Jan 01, 2023 at 06:52:03PM -0500, Chuck Zmudzinski wrote:
>>>> Intel specifies that the Intel IGD must occupy slot 2 on the PCI bus,
>>>> as noted in docs/igd-assign.txt in the Qemu source code.
>>>> 
>>>> Currently, when the xl toolstack is used to configure a Xen HVM guest with
>>>> Intel IGD passthrough to the guest with the Qemu upstream device model,
>>>> a Qemu emulated PCI device will occupy slot 2 and the Intel IGD will occupy
>>>> a different slot. This problem often prevents the guest from booting.
>>>> 
>>>> The only available workaround is not good: Configure Xen HVM guests to use
>>>> the old and no longer maintained Qemu traditional device model available
>>>> from xenbits.xen.org which does reserve slot 2 for the Intel IGD.
>>>> 
>>>> To implement this feature in the Qemu upstream device model for Xen HVM
>>>> guests, introduce the following new functions, types, and macros:
>>>> 
>>>> * XEN_PT_DEVICE_CLASS declaration, based on the existing TYPE_XEN_PT_DEVICE
>>>> * XEN_PT_DEVICE_GET_CLASS macro helper function for XEN_PT_DEVICE_CLASS
>>>> * typedef XenPTQdevRealize function pointer
>>>> * XEN_PCI_IGD_SLOT_MASK, the value of slot_reserved_mask to reserve slot 2
>>>> * xen_igd_reserve_slot and xen_igd_clear_slot functions
>>>> 
>>>> The new xen_igd_reserve_slot function uses the existing slot_reserved_mask
>>>> member of PCIBus to reserve PCI slot 2 for Xen HVM guests configured using
>>>> the xl toolstack with the gfx_passthru option enabled, which sets the
>>>> igd-passthru=on option to Qemu for the Xen HVM machine type.
>>>> 
>>>> The new xen_igd_reserve_slot function also needs to be implemented in
>>>> hw/xen/xen_pt_stub.c to prevent FTBFS during the link stage for the case
>>>> when Qemu is configured with --enable-xen and 
>>>> --disable-xen-pci-passthrough,
>>>> in which case it does nothing.
>>>> 
>>>> The new xen_igd_clear_slot function overrides qdev->realize of the parent
>>>> PCI device class to enable the Intel IGD to occupy slot 2 on the PCI bus
>>>> since slot 2 was reserved by xen_igd_reserve_slot when the PCI bus was
>>>> created in hw/i386/pc_piix.c for the case when igd-passthru=on.
>>>> 
>>>> Move the call to xen_host_pci_device_get, and the associated error
>>>> handling, from xen_pt_realize to the new xen_igd_clear_slot function to
>>>> initialize the device class and vendor values which enables the checks for
>>>> the Intel IGD to succeed. The verification that the host device is an
>>>> Intel IGD to be passed through is done by checking the domain, bus, slot,
>>>> and function values as well as by checking that gfx_passthru is enabled,
>>>> the device class is VGA, and the device vendor in Intel.
>>>> 
>>>> Signed-off-by: Chuck Zmudzinski 
>>> 
>>> 
>>> This patch looks good enough. It only changes the "xenfv" machine so it
>>> doesn't prevent a proper fix to be done in the toolstack libxl.
>>> 
>>> The change in xen_pci_passthrough_class_init() to try to run some code
>>> before pci_qdev_realize() could potentially break in the future due to
>>> been uncommon but hopefully that will be ok.
>>> 
>>> So if no work to fix libxl appear soon, I'm ok with this patch:
> 
> Well, I can tell you and others who use qemu are more comfortable
> fixing this in libxl, so hold off for a week or so. I should have
> a patch to fix this in libxl written and tested by then. If for
> some reason that does not work out, then we can fix it in qemu.

One last thought: the only donwnside to fixing this in libxl is that
other toolstacks that configure qemu to use the xenfv machine will not
benefit from the fix in qemu that would simplify configuring the
guest correctly for the igd. Other toolstacks would still need to
override the default behavior of adding the xen platform device at
slot 2. I think no matter what, we should at least patch qemu to have
the xen-platform device use slot 3 instead of being automatically assigned
to slot 2 when igd-passthru=on. The rest of the fix could then be
implemented in libxl so that other pci devices such as emulated network
devices, other passed through pci devices, etc., do not take slot 2 when
gfx_passthru in xl.cfg is set.

So, unless I hear any objection, my plan is to patch qemu to use slot
3 for the xen platform device when igd-passthru is on, and implement the
rest of the fix in libxl. I should have it ready within a week.

Thanks for your help,

Chuck

Re: [PATCH v6] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

On 1/6/23 9:10 AM, Chuck Zmudzinski wrote:
> On 1/6/23 9:03 AM, Anthony PERARD wrote:
>> On Sun, Jan 01, 2023 at 06:52:03PM -0500, Chuck Zmudzinski wrote:
>>> Intel specifies that the Intel IGD must occupy slot 2 on the PCI bus,
>>> as noted in docs/igd-assign.txt in the Qemu source code.
>>> 
>>> Currently, when the xl toolstack is used to configure a Xen HVM guest with
>>> Intel IGD passthrough to the guest with the Qemu upstream device model,
>>> a Qemu emulated PCI device will occupy slot 2 and the Intel IGD will occupy
>>> a different slot. This problem often prevents the guest from booting.
>>> 
>>> The only available workaround is not good: Configure Xen HVM guests to use
>>> the old and no longer maintained Qemu traditional device model available
>>> from xenbits.xen.org which does reserve slot 2 for the Intel IGD.
>>> 
>>> To implement this feature in the Qemu upstream device model for Xen HVM
>>> guests, introduce the following new functions, types, and macros:
>>> 
>>> * XEN_PT_DEVICE_CLASS declaration, based on the existing TYPE_XEN_PT_DEVICE
>>> * XEN_PT_DEVICE_GET_CLASS macro helper function for XEN_PT_DEVICE_CLASS
>>> * typedef XenPTQdevRealize function pointer
>>> * XEN_PCI_IGD_SLOT_MASK, the value of slot_reserved_mask to reserve slot 2
>>> * xen_igd_reserve_slot and xen_igd_clear_slot functions
>>> 
>>> The new xen_igd_reserve_slot function uses the existing slot_reserved_mask
>>> member of PCIBus to reserve PCI slot 2 for Xen HVM guests configured using
>>> the xl toolstack with the gfx_passthru option enabled, which sets the
>>> igd-passthru=on option to Qemu for the Xen HVM machine type.
>>> 
>>> The new xen_igd_reserve_slot function also needs to be implemented in
>>> hw/xen/xen_pt_stub.c to prevent FTBFS during the link stage for the case
>>> when Qemu is configured with --enable-xen and --disable-xen-pci-passthrough,
>>> in which case it does nothing.
>>> 
>>> The new xen_igd_clear_slot function overrides qdev->realize of the parent
>>> PCI device class to enable the Intel IGD to occupy slot 2 on the PCI bus
>>> since slot 2 was reserved by xen_igd_reserve_slot when the PCI bus was
>>> created in hw/i386/pc_piix.c for the case when igd-passthru=on.
>>> 
>>> Move the call to xen_host_pci_device_get, and the associated error
>>> handling, from xen_pt_realize to the new xen_igd_clear_slot function to
>>> initialize the device class and vendor values which enables the checks for
>>> the Intel IGD to succeed. The verification that the host device is an
>>> Intel IGD to be passed through is done by checking the domain, bus, slot,
>>> and function values as well as by checking that gfx_passthru is enabled,
>>> the device class is VGA, and the device vendor in Intel.
>>> 
>>> Signed-off-by: Chuck Zmudzinski 
>> 
>> 
>> This patch looks good enough. It only changes the "xenfv" machine so it
>> doesn't prevent a proper fix to be done in the toolstack libxl.
>> 
>> The change in xen_pci_passthrough_class_init() to try to run some code
>> before pci_qdev_realize() could potentially break in the future due to
>> been uncommon but hopefully that will be ok.
>> 
>> So if no work to fix libxl appear soon, I'm ok with this patch:

Well, I can tell you and others who use qemu are more comfortable
fixing this in libxl, so hold off for a week or so. I should have
a patch to fix this in libxl written and tested by then. If for
some reason that does not work out, then we can fix it in qemu.

Cheers,

Chuck

>> 
>> Reviewed-by: Anthony PERARD 
>> 
>> Thanks,
>> 

Re: [PATCH v6] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

On 1/6/23 9:03 AM, Anthony PERARD wrote:
> On Sun, Jan 01, 2023 at 06:52:03PM -0500, Chuck Zmudzinski wrote:
>> Intel specifies that the Intel IGD must occupy slot 2 on the PCI bus,
>> as noted in docs/igd-assign.txt in the Qemu source code.
>> 
>> Currently, when the xl toolstack is used to configure a Xen HVM guest with
>> Intel IGD passthrough to the guest with the Qemu upstream device model,
>> a Qemu emulated PCI device will occupy slot 2 and the Intel IGD will occupy
>> a different slot. This problem often prevents the guest from booting.
>> 
>> The only available workaround is not good: Configure Xen HVM guests to use
>> the old and no longer maintained Qemu traditional device model available
>> from xenbits.xen.org which does reserve slot 2 for the Intel IGD.
>> 
>> To implement this feature in the Qemu upstream device model for Xen HVM
>> guests, introduce the following new functions, types, and macros:
>> 
>> * XEN_PT_DEVICE_CLASS declaration, based on the existing TYPE_XEN_PT_DEVICE
>> * XEN_PT_DEVICE_GET_CLASS macro helper function for XEN_PT_DEVICE_CLASS
>> * typedef XenPTQdevRealize function pointer
>> * XEN_PCI_IGD_SLOT_MASK, the value of slot_reserved_mask to reserve slot 2
>> * xen_igd_reserve_slot and xen_igd_clear_slot functions
>> 
>> The new xen_igd_reserve_slot function uses the existing slot_reserved_mask
>> member of PCIBus to reserve PCI slot 2 for Xen HVM guests configured using
>> the xl toolstack with the gfx_passthru option enabled, which sets the
>> igd-passthru=on option to Qemu for the Xen HVM machine type.
>> 
>> The new xen_igd_reserve_slot function also needs to be implemented in
>> hw/xen/xen_pt_stub.c to prevent FTBFS during the link stage for the case
>> when Qemu is configured with --enable-xen and --disable-xen-pci-passthrough,
>> in which case it does nothing.
>> 
>> The new xen_igd_clear_slot function overrides qdev->realize of the parent
>> PCI device class to enable the Intel IGD to occupy slot 2 on the PCI bus
>> since slot 2 was reserved by xen_igd_reserve_slot when the PCI bus was
>> created in hw/i386/pc_piix.c for the case when igd-passthru=on.
>> 
>> Move the call to xen_host_pci_device_get, and the associated error
>> handling, from xen_pt_realize to the new xen_igd_clear_slot function to
>> initialize the device class and vendor values which enables the checks for
>> the Intel IGD to succeed. The verification that the host device is an
>> Intel IGD to be passed through is done by checking the domain, bus, slot,
>> and function values as well as by checking that gfx_passthru is enabled,
>> the device class is VGA, and the device vendor in Intel.
>> 
>> Signed-off-by: Chuck Zmudzinski 
> 
> 
> This patch looks good enough. It only changes the "xenfv" machine so it
> doesn't prevent a proper fix to be done in the toolstack libxl.
> 
> The change in xen_pci_passthrough_class_init() to try to run some code
> before pci_qdev_realize() could potentially break in the future due to
> been uncommon but hopefully that will be ok.
> 
> So if no work to fix libxl appear soon, I'm ok with this patch:
> 
> Reviewed-by: Anthony PERARD 
> 
> Thanks,
> 

Well, our messages almost collided! I just proposed a v7 that adds
a check to prevent the extra processing for cases when machine is
not xenfv and the slot does not need to be cleared because it was
never reserved. The proposed v7 would not change the behavior of the
patch at all but it would avoid some unnecessary processing. Do you
want me to submit that v7?

Chuck

Re: [PATCH v6] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

On 1/6/23 5:52 AM, Anthony PERARD wrote:
> On Tue, Jan 03, 2023 at 05:58:01PM -0500, Chuck Zmudzinski wrote:
>> Hello Anthony and Paul,
> 
> Hi Chuck,
> 
>> I am requesting your feedback to Alex Williamson's suggestion that this
>> problem with assigning the correct slot address to the igd on xen should
>> be fixed in libxl instead of in qemu.
>> 
>> It seems to me that the xen folks and the kvm folks have two different
>> philosophies regarding how a tool stack should be designed. kvm/libvirt
>> provides much greater flexibility in configuring the guest which puts
>> the burden on the administrator to set all the options correctly for
>> a given feature set, while xen/xenlight does not provide so much
>> flexibility and tries to automatically configure the guest based on
>> a high-level feature option such as the igd-passthrough=on option that
>> is available for xen guests using qemu but not for kvm guests using
>> qemu.
>> 
>> What do you think? Should libxl be patched instead of fixing the problem
>> with this patch to qemu, which is contrary to Alex's suggestion?
> 
> I do think that libxl should be able to deal with having to put a
> graphic card on slot 2. QEMU already provides every API necessary for a
> toolstack to be able to start a Xen guest with all the PCI card in the
> right slot. But it would just be a bit more complicated to implement in
> libxl.
> 
> At the moment, libxl makes use of the QEMU machine 'xenfv', libxl should
> instead start to use the 'pc' machine and add the "xen-platform" pci
> device. (libxl already uses 'pc' when the "xen-platform" pci card isn't
> needed.) Also probably add the other pci devices to specific slot to be
> able to add the passthrough graphic card at the right slot.
> 
> Next is to deal with migration when using the 'pc' machine, as it's just
> an alias to a specific version of the machine. We need to use the same
> machine on the receiving end, that is start with e.g. "pc-i440fx-7.1" if
> 'pc' was an alias for it at guest creation.
> 
> 
> I wonder if we can already avoid to patch the 'xenfv' machine with some
> xl config:
> # avoid 'xenfv' machine and use 'pc' instead
> xen_platform_pci=0
> # add xen-platform pci device back
> device_model_args_hvm = [
> "-device", "xen-platform,addr=3",
> ]
> But there's probably another device which is going to be auto-assigned
> to slot 2.
> 
> 
> If you feel like dealing with the technical dept in libxl, that is to
> stop using 'xenfv' and use 'pc' instead, then go for it, I can help with
> that. Otherwise, if the patch to QEMU only changes the behavior of the
> 'xenfv' machine then I think I would be ok with it.
> 
> I'll do a review of that QEMU patch in another email.
> 
> Cheers,
> 

Hello Anthony,

Thanks for responding!

The first part of my v6 of the patch only affects the xenfv
machine. Guests created with the pc machine type will not call
the new function that reserves slot 2 for the igd because that
function is only called when the machine type is xenfv (or xenfv-4.2).
But the new functions I added to configure the TYPE_XEN_PT_DEVICE
when igd-passthru=on will be called to check if the device is an
Intel igd and clear the slot if it is, but this will not have any
effect on the behavior in this case because the slot was never
reserved. Still, this would add some unnecessary processing in the
case of machines other than xenfv, which is undesirable.

So I can add a check for the machine type to a v7 of the patch
that will skip the new functions that clear the reserved slot if
slot 2 is not reserved and therefore does not need to be cleared.

Would that be OK?

Kind regards,

Chuck

Re: [PATCH v6] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

On 1/4/23 3:47 PM, Chuck Zmudzinski wrote:
> On 1/3/23 10:14 AM, Alex Williamson wrote:
> 
>> 
>> It's necessary to configure the assigned IGD at slot 2 to make it
>> functional, yes, but I don't really understand this notion of
>> "reserving" slot 2.  If something occupies address 00:02.0 in the
>> config, it's the user's or management tool's responsibility to move it
>> to make this configuration functional.  Why does QEMU need to play a
>> part in reserving this bus address.  IGD devices are not generally
>> hot-pluggable either, so it doesn't seem we need to reserve an address
>> in case an IGD device is added dynamically later.
> 
> The capability to reserve a bus address for a quirky device need not
> be limited to the case of hotplugged or dynamically added devices. The
> igd is a quirky device, and its presence in an emulated system like
> qemu requires special handling. The slot_reserved_mask member of PCIBus
> is also well-suited to the case of quirky device like Intel the igd that
> needs to be at slot 2. Just because it is not dynamically added later
> does not change the fact that it needs special handling at its initial
> configuration when the guest is being created.
> 
>>  
> 
> Here's the problem that answers Michael's question why this patch is
> specific to xen:
> 
> ---snip---
> #ifdef CONFIG_XEN
> 
> ...
> 
> static void pc_xen_hvm_init(MachineState *machine)
> {
> PCMachineState *pcms = PC_MACHINE(machine);
> 
> if (!xen_enabled()) {
> error_report("xenfv machine requires the xen accelerator");
> exit(1);
> }
> 
> pc_xen_hvm_init_pci(machine);
> pci_create_simple(pcms->bus, -1, "xen-platform");
> }
> #endif
> ---snip---
> 
> This code is from hw/i386/pc_piix.c. Note the call to
> pci_create_simple to create the xen platform pci device,
> which has -1 as the second argument. That -1 tells
> pci_create_simple to autoconfigure the pci bdf address.
> 
> It is *hard-coded* that way. That means no toolstack or
> management tool can change it. And what is hard-coded here
> is that the xen platform device will occupy slot 2, preventing
> the Intel igd or any other device from occupying slot 2.
> 

Actually, today I discovered it is possible to workaround
this issue with libxl that appears to hard-code the xen
platform device to slot 2.

The code referenced here that effectively hard codes the xen
platform device to slot 2 is only executed with the
'-machine xenfv' qemu option, which is the default that
libxl uses for hvm guests, but this behavior can be changed
by setting xen_platform_pci = '0' in the xl guest configuration
file, in which case libxl configures qemu with the
'-machine pc,accel=xen' option instead, and then one can add
the xen platform device at a different slot by adding, for
example,

device_model_args = ['-device','xen-platform,addr=03']

to the xl guest configuration file which adds the device at
slot 3 instead of slot 2.

This approach is an ugly workaround which has other side effects,
such as by having -machine as pc,accel=xen instead of xenfv, the
machine type is not exactly the same because the xenfv machine
type is based on a much older version of qemu's i440fx emulation
(qemu version 3.1 IIRC), so with this workaround there could be
some unintended side effects, although I just tested this
workaround and it does seem to work, but I have not been using
this approach to the problem for very long.

Also, this approach requires setting type=vif in the vif network
setting of the xl guest configuration to suppress the creation of
the qemu emulated network device that would grab slot 2 if it is
created by the ordinary libxl network configuration settings and,
for guests that do not have the xen pv network drivers installed,
this also would require manually building the qemu command line
using device_model_args to allow adding an emulated qemu network
device at a different slot and probably also manually configuring
the correct networking scripts outside of the normal xen networking
scripts, because the ordinary xl guest configuration options do not
have a setting for assigning an emulated qemu network device to a
specific slot, and the device would otherwise grab slot 2.

So,it is possible to configure the guest to have the Intel igd at slot
2 without patching either libxl or qemu, but it is a real PITA with
some possible unintended side effects, and that is why I think patching
qemu to reserve slot 2 is a much better solution to the problem.

Kind regards,

Chuck

Re: [PATCH v2 6/6] hw/isa/piix: Resolve redundant TYPE_PIIX3_XEN_DEVICE

[Chuck Zmudzinski]

On 1/4/2023 5:18 PM, Philippe Mathieu-Daudé wrote:
> On 4/1/23 20:29, Chuck Zmudzinski wrote:
> > On 1/4/23 1:48 PM, Philippe Mathieu-Daudé wrote:
>
> >> Here TYPE_PIIX3_DEVICE means for "PCI function part of the PIIX
> >> south bridge chipset, which expose a PCI-to-ISA bridge". A better
> >> name could be TYPE_PIIX3_ISA_PCI_DEVICE. Unfortunately this
> >> device is named "PIIX3" with no indication of ISA bridge.
> > 
> > 
> > Thanks, you are right, I see the PIIX3 device still exists after
> > this patch set is applied.
> > 
> > chuckz@debian:~/sources-sid/qemu/qemu-7.50+dfsg/hw/i386$ grep -r PIIX3 *
> > pc_piix.c:pci_dev = pci_new_multifunction(-1, true, 
> > TYPE_PIIX3_DEVICE);
> > 
> > I also understand there is the PCI-to-ISA bridge at 00:01.0 on the PCI bus:
> > 
> > chuckz@debian:~$ lspci
> > 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
>
> All these entries ('PCI functions') ...:
>
> > 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
> > 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton 
> > II]
> > 00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton 
> > II] (rev 01)
> > 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
>
> ... are part of the same *device*: the PIIX south bridge.
>
> This device is enumerated as #1 on the PCI bus #0.
> It currently exposes 4 functions: ISA/IDE/USB/ACPI.
>
> > 00:02.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 
> > 01)
> > 00:03.0 VGA compatible controller: Device 1234: (rev 02)
> > 
> > I also see with this patch, there is a bridge that is a PIIX4 ACPI at 
> > 00:01.3.
> > I get the exact same output from lspci without the patch series, so that 
> > gives
> > me confidence it is working as designed.
>
> Historically the PIIX3 and PIIX4 QEMU models have been written by
> different people with different goals.
>
> - PIIX3 comes from x86 machines and is important for KVM/Xen
>accelerators
> - PIIX4 was developed by hobbyist for MIPS machines
>
> PIIX4 added the ACPI function which was proven helpful for x86 machines.
>
> OS such Linux don't consider the PIIX south bridge as a whole chipset,
> and enumerate each PCI function individually. So it was possible to add
> the PIIX4 ACPI function to a PIIX3... A config that doesn't exist with
> real hardware :/
> While QEMU aims at modeling real HW, this config is still very useful
> for KVM/Xen. So this Frankenstein config is accepted / maintained.
>
> Bernhard is doing an incredible work merging the PIIX3/PIIX4 differences
> into a more maintainable model :)
>
> Regards,
>
> Phil.

Thanks for the nice explanation of the history. I understand the PIIX3 is 
associated
with the i440fx machine type for i386 - it goes all the way back to 1995 I think
with the original 32-bit Pentium processor and Windows 95. So it is a worthwhile
effort to work on updating this to something newer, and of course kvm can use
the newer Q35 chipset which goes back to 2009 or so, I think, but xen/x86 
languishes
on the i440fx for now.

Best regards,

Chuck

Re: [PATCH v2 6/6] hw/isa/piix: Resolve redundant TYPE_PIIX3_XEN_DEVICE

[Chuck Zmudzinski]

On 1/4/2023 5:35 PM, Philippe Mathieu-Daudé wrote:
> On 4/1/23 17:42, Chuck Zmudzinski wrote:
> > On 1/4/23 9:44 AM, Bernhard Beschow wrote:
> >> During the last patches, TYPE_PIIX3_XEN_DEVICE turned into a clone of
> >> TYPE_PIIX3_DEVICE. Remove this redundancy.
> >>
> >> Signed-off-by: Bernhard Beschow 
> >> ---
> >>   hw/i386/pc_piix.c |  4 +---
> >>   hw/isa/piix.c | 20 
> >>   include/hw/southbridge/piix.h |  1 -
> >>   3 files changed, 1 insertion(+), 24 deletions(-)
> >>
> >> diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c
> >> index 5738d9cdca..6b8de3d59d 100644
> >> --- a/hw/i386/pc_piix.c
> >> +++ b/hw/i386/pc_piix.c
> >> @@ -235,8 +235,6 @@ static void pc_init1(MachineState *machine,
> >>   if (pcmc->pci_enabled) {
> >>   DeviceState *dev;
> >>   PCIDevice *pci_dev;
> >> -const char *type = xen_enabled() ? TYPE_PIIX3_XEN_DEVICE
> >> - : TYPE_PIIX3_DEVICE;
> >>   int i;
> >>   
> >>   pci_bus = i440fx_init(pci_type,
> >> @@ -250,7 +248,7 @@ static void pc_init1(MachineState *machine,
> >>  : pci_slot_get_pirq);
> >>   pcms->bus = pci_bus;
> >>   
> >> -pci_dev = pci_new_multifunction(-1, true, type);
> >> +pci_dev = pci_new_multifunction(-1, true, TYPE_PIIX3_DEVICE);
> >>   object_property_set_bool(OBJECT(pci_dev), "has-usb",
> >>machine_usb(machine), _abort);
> >>   object_property_set_bool(OBJECT(pci_dev), "has-acpi",
> >> diff --git a/hw/isa/piix.c b/hw/isa/piix.c
> >> index 98e9b12661..e4587352c9 100644
> >> --- a/hw/isa/piix.c
> >> +++ b/hw/isa/piix.c
> >> @@ -33,7 +33,6 @@
> >>   #include "hw/qdev-properties.h"
> >>   #include "hw/ide/piix.h"
> >>   #include "hw/isa/isa.h"
> >> -#include "hw/xen/xen.h"
> >>   #include "sysemu/runstate.h"
> >>   #include "migration/vmstate.h"
> >>   #include "hw/acpi/acpi_aml_interface.h"
> >> @@ -465,24 +464,6 @@ static const TypeInfo piix3_info = {
> >>   .class_init= piix3_class_init,
> >>   };
> >>   
> >> -static void piix3_xen_class_init(ObjectClass *klass, void *data)
> >> -{
> >> -DeviceClass *dc = DEVICE_CLASS(klass);
> >> -PCIDeviceClass *k = PCI_DEVICE_CLASS(klass);
> >> -
> >> -k->realize = piix3_realize;
> >> -/* 82371SB PIIX3 PCI-to-ISA bridge (Step A1) */
> >> -k->device_id = PCI_DEVICE_ID_INTEL_82371SB_0;
> >> -dc->vmsd = _piix3;
> >> -}
> >> -
> >> -static const TypeInfo piix3_xen_info = {
> >> -.name  = TYPE_PIIX3_XEN_DEVICE,
> >> -.parent= TYPE_PIIX_PCI_DEVICE,
> >> -.instance_init = piix3_init,
> >> -.class_init= piix3_xen_class_init,
> >> -};
> >> -
> >>   static void piix4_realize(PCIDevice *dev, Error **errp)
> >>   {
> >>   ERRP_GUARD();
> >> @@ -534,7 +515,6 @@ static void piix3_register_types(void)
> >>   {
> >>   type_register_static(_pci_type_info);
> >>   type_register_static(_info);
> >> -type_register_static(_xen_info);
> >>   type_register_static(_info);
> >>   }
> >>   
> >> diff --git a/include/hw/southbridge/piix.h b/include/hw/southbridge/piix.h
> >> index 65ad8569da..b1fc94a742 100644
> >> --- a/include/hw/southbridge/piix.h
> >> +++ b/include/hw/southbridge/piix.h
> >> @@ -77,7 +77,6 @@ struct PIIXState {
> >>   OBJECT_DECLARE_SIMPLE_TYPE(PIIXState, PIIX_PCI_DEVICE)
> >>   
> >>   #define TYPE_PIIX3_DEVICE "PIIX3"
> >> -#define TYPE_PIIX3_XEN_DEVICE "PIIX3-xen"
> >>   #define TYPE_PIIX4_PCI_DEVICE "piix4-isa"
> >>   
> >>   #endif
> > 
> > 
> > This fixes the regression with the emulated usb tablet device that I 
> > reported in v1 here:
> > 
> > https://lore.kernel.org/qemu-devel/aed4f2c1-83f7-163a-fb44-f28437666...@aol.com/
> > 
> > I tested this patch again with all the prerequisites and now with v2 there 
> > are no regressions.
> > 
> > Tested-by: Chuck Zmudzinski 
>
> (IIUC Chuck meant to send this tag to the cover letter)
>

Is it customary to tag the cover letter instead? I thought it appropriate
to tag the last commit in the series because it best represents the actual
commit on which tests were carried out. Also, the cover letter does not
actually represent a real commit that can be tested, except maybe the
last commit in the series. I did read the document on submitting patches,
but I don't remember if it specifies how to tag a series of patches with
the Tested-by tag.

Kind regards,

Chuck

Re: [PATCH v2 6/6] hw/isa/piix: Resolve redundant TYPE_PIIX3_XEN_DEVICE

[Chuck Zmudzinski]

On 1/4/23 5:35 PM, Philippe Mathieu-Daudé wrote:
> On 4/1/23 17:42, Chuck Zmudzinski wrote:
>> On 1/4/23 9:44 AM, Bernhard Beschow wrote:
>>> During the last patches, TYPE_PIIX3_XEN_DEVICE turned into a clone of
>>> TYPE_PIIX3_DEVICE. Remove this redundancy.
>>>
>>> Signed-off-by: Bernhard Beschow 
>>> ---
>>>   hw/i386/pc_piix.c |  4 +---
>>>   hw/isa/piix.c | 20 
>>>   include/hw/southbridge/piix.h |  1 -
>>>   3 files changed, 1 insertion(+), 24 deletions(-)
>>>
>>> diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c
>>> index 5738d9cdca..6b8de3d59d 100644
>>> --- a/hw/i386/pc_piix.c
>>> +++ b/hw/i386/pc_piix.c
>>> @@ -235,8 +235,6 @@ static void pc_init1(MachineState *machine,
>>>   if (pcmc->pci_enabled) {
>>>   DeviceState *dev;
>>>   PCIDevice *pci_dev;
>>> -const char *type = xen_enabled() ? TYPE_PIIX3_XEN_DEVICE
>>> - : TYPE_PIIX3_DEVICE;
>>>   int i;
>>>   
>>>   pci_bus = i440fx_init(pci_type,
>>> @@ -250,7 +248,7 @@ static void pc_init1(MachineState *machine,
>>>  : pci_slot_get_pirq);
>>>   pcms->bus = pci_bus;
>>>   
>>> -pci_dev = pci_new_multifunction(-1, true, type);
>>> +pci_dev = pci_new_multifunction(-1, true, TYPE_PIIX3_DEVICE);
>>>   object_property_set_bool(OBJECT(pci_dev), "has-usb",
>>>machine_usb(machine), _abort);
>>>   object_property_set_bool(OBJECT(pci_dev), "has-acpi",
>>> diff --git a/hw/isa/piix.c b/hw/isa/piix.c
>>> index 98e9b12661..e4587352c9 100644
>>> --- a/hw/isa/piix.c
>>> +++ b/hw/isa/piix.c
>>> @@ -33,7 +33,6 @@
>>>   #include "hw/qdev-properties.h"
>>>   #include "hw/ide/piix.h"
>>>   #include "hw/isa/isa.h"
>>> -#include "hw/xen/xen.h"
>>>   #include "sysemu/runstate.h"
>>>   #include "migration/vmstate.h"
>>>   #include "hw/acpi/acpi_aml_interface.h"
>>> @@ -465,24 +464,6 @@ static const TypeInfo piix3_info = {
>>>   .class_init= piix3_class_init,
>>>   };
>>>   
>>> -static void piix3_xen_class_init(ObjectClass *klass, void *data)
>>> -{
>>> -DeviceClass *dc = DEVICE_CLASS(klass);
>>> -PCIDeviceClass *k = PCI_DEVICE_CLASS(klass);
>>> -
>>> -k->realize = piix3_realize;
>>> -/* 82371SB PIIX3 PCI-to-ISA bridge (Step A1) */
>>> -k->device_id = PCI_DEVICE_ID_INTEL_82371SB_0;
>>> -dc->vmsd = _piix3;
>>> -}
>>> -
>>> -static const TypeInfo piix3_xen_info = {
>>> -.name  = TYPE_PIIX3_XEN_DEVICE,
>>> -.parent= TYPE_PIIX_PCI_DEVICE,
>>> -.instance_init = piix3_init,
>>> -.class_init= piix3_xen_class_init,
>>> -};
>>> -
>>>   static void piix4_realize(PCIDevice *dev, Error **errp)
>>>   {
>>>   ERRP_GUARD();
>>> @@ -534,7 +515,6 @@ static void piix3_register_types(void)
>>>   {
>>>   type_register_static(_pci_type_info);
>>>   type_register_static(_info);
>>> -type_register_static(_xen_info);
>>>   type_register_static(_info);
>>>   }
>>>   
>>> diff --git a/include/hw/southbridge/piix.h b/include/hw/southbridge/piix.h
>>> index 65ad8569da..b1fc94a742 100644
>>> --- a/include/hw/southbridge/piix.h
>>> +++ b/include/hw/southbridge/piix.h
>>> @@ -77,7 +77,6 @@ struct PIIXState {
>>>   OBJECT_DECLARE_SIMPLE_TYPE(PIIXState, PIIX_PCI_DEVICE)
>>>   
>>>   #define TYPE_PIIX3_DEVICE "PIIX3"
>>> -#define TYPE_PIIX3_XEN_DEVICE "PIIX3-xen"
>>>   #define TYPE_PIIX4_PCI_DEVICE "piix4-isa"
>>>   
>>>   #endif
>> 
>> 
>> This fixes the regression with the emulated usb tablet device that I 
>> reported in v1 here:
>> 
>> https://lore.kernel.org/qemu-devel/aed4f2c1-83f7-163a-fb44-f28437666...@aol.com/
>> 
>> I tested this patch again with all the prerequisites and now with v2 there 
>> are no regressions.
>> 
>> Tested-by: Chuck Zmudzinski 
> 
> (IIUC Chuck meant to send this tag to the cover letter)
> 

Yes, I tested the whole patch series, not just this individual patch.
I tagged this one because it is the last patch in the series.

Re: [PATCH v2 6/6] hw/isa/piix: Resolve redundant TYPE_PIIX3_XEN_DEVICE

[Chuck Zmudzinski]

On 1/4/23 3:31 PM, Bernhard Beschow wrote:
> 
> 
> Am 4. Januar 2023 19:29:35 UTC schrieb Chuck Zmudzinski :
>>On 1/4/23 1:48 PM, Philippe Mathieu-Daudé wrote:
>>> On 4/1/23 18:54, Chuck Zmudzinski wrote:
>>>> On 1/4/23 10:35 AM, Philippe Mathieu-Daudé wrote:
>>>>> +Markus/Thomas
>>>>>
>>>>> On 4/1/23 15:44, Bernhard Beschow wrote:
>>>>>> During the last patches, TYPE_PIIX3_XEN_DEVICE turned into a clone of
>>>>>> TYPE_PIIX3_DEVICE. Remove this redundancy.
>>>>>>
>>>>>> Signed-off-by: Bernhard Beschow 
>>>>>> ---
>>>>>>hw/i386/pc_piix.c |  4 +---
>>>>>>hw/isa/piix.c | 20 
>>>>>>include/hw/southbridge/piix.h |  1 -
>>>>>>3 files changed, 1 insertion(+), 24 deletions(-)
>>>>>>
>>>>>> diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c
>>>>>> index 5738d9cdca..6b8de3d59d 100644
>>>>>> --- a/hw/i386/pc_piix.c
>>>>>> +++ b/hw/i386/pc_piix.c
>>>>>> @@ -235,8 +235,6 @@ static void pc_init1(MachineState *machine,
>>>>>>if (pcmc->pci_enabled) {
>>>>>>DeviceState *dev;
>>>>>>PCIDevice *pci_dev;
>>>>>> -const char *type = xen_enabled() ? TYPE_PIIX3_XEN_DEVICE
>>>>>> - : TYPE_PIIX3_DEVICE;
>>>>>>int i;
>>>>>>
>>>>>>pci_bus = i440fx_init(pci_type,
>>>>>> @@ -250,7 +248,7 @@ static void pc_init1(MachineState *machine,
>>>>>>   : pci_slot_get_pirq);
>>>>>>pcms->bus = pci_bus;
>>>>>>
>>>>>> -pci_dev = pci_new_multifunction(-1, true, type);
>>>>>> +pci_dev = pci_new_multifunction(-1, true, TYPE_PIIX3_DEVICE);
>>>>>>object_property_set_bool(OBJECT(pci_dev), "has-usb",
>>>>>> machine_usb(machine), _abort);
>>>>>>object_property_set_bool(OBJECT(pci_dev), "has-acpi",
>>>>>> diff --git a/hw/isa/piix.c b/hw/isa/piix.c
>>>>>> index 98e9b12661..e4587352c9 100644
>>>>>> --- a/hw/isa/piix.c
>>>>>> +++ b/hw/isa/piix.c
>>>>>> @@ -33,7 +33,6 @@
>>>>>>#include "hw/qdev-properties.h"
>>>>>>#include "hw/ide/piix.h"
>>>>>>#include "hw/isa/isa.h"
>>>>>> -#include "hw/xen/xen.h"
>>>>>>#include "sysemu/runstate.h"
>>>>>>#include "migration/vmstate.h"
>>>>>>#include "hw/acpi/acpi_aml_interface.h"
>>>>>> @@ -465,24 +464,6 @@ static const TypeInfo piix3_info = {
>>>>>>.class_init= piix3_class_init,
>>>>>>};
>>>>>>
>>>>>> -static void piix3_xen_class_init(ObjectClass *klass, void *data)
>>>>>> -{
>>>>>> -DeviceClass *dc = DEVICE_CLASS(klass);
>>>>>> -PCIDeviceClass *k = PCI_DEVICE_CLASS(klass);
>>>>>> -
>>>>>> -k->realize = piix3_realize;
>>>>>> -/* 82371SB PIIX3 PCI-to-ISA bridge (Step A1) */
>>>>>> -k->device_id = PCI_DEVICE_ID_INTEL_82371SB_0;
>>>>>> -dc->vmsd = _piix3;
>>>>>
>>>>> IIUC, since this device is user-creatable, we can't simply remove it
>>>>> without going thru the deprecation process. Alternatively we could
>>>>> add a type alias:
>>>>>
>>>>> -- >8 --
>>>>> diff --git a/softmmu/qdev-monitor.c b/softmmu/qdev-monitor.c
>>>>> index 4b0ef65780..d94f7ea369 100644
>>>>> --- a/softmmu/qdev-monitor.c
>>>>> +++ b/softmmu/qdev-monitor.c
>>>>> @@ -64,6 +64,7 @@ typedef struct QDevAlias
>>>>>  QEMU_ARCH_LOONGARCH)
>>>>>#define QEMU_ARCH_VIRTIO_CCW (QEMU_ARCH_S390X)
>>>>>#define QEMU_ARCH_VIRTIO_MMIO (QEMU_ARCH_M68K)
>>>>> +#define QEMU_ARCH_XEN (QEMU_ARCH_ARM | QEMU_ARCH_I386)
>>>>>
>>>>>/*

Re: [PATCH v2 6/6] hw/isa/piix: Resolve redundant TYPE_PIIX3_XEN_DEVICE

[Chuck Zmudzinski]

On 1/4/23 3:44 PM, Bernhard Beschow wrote:
> 
> 
> Am 4. Januar 2023 17:54:16 UTC schrieb Chuck Zmudzinski :
>>On 1/4/23 10:35 AM, Philippe Mathieu-Daudé wrote:
>>> +Markus/Thomas
>>> 
>>> On 4/1/23 15:44, Bernhard Beschow wrote:
>>>> During the last patches, TYPE_PIIX3_XEN_DEVICE turned into a clone of
>>>> TYPE_PIIX3_DEVICE. Remove this redundancy.
>>>> 
>>>> Signed-off-by: Bernhard Beschow 
>>>> ---
>>>>   hw/i386/pc_piix.c |  4 +---
>>>>   hw/isa/piix.c | 20 
>>>>   include/hw/southbridge/piix.h |  1 -
>>>>   3 files changed, 1 insertion(+), 24 deletions(-)
>>>> 
>>>> diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c
>>>> index 5738d9cdca..6b8de3d59d 100644
>>>> --- a/hw/i386/pc_piix.c
>>>> +++ b/hw/i386/pc_piix.c
>>>> @@ -235,8 +235,6 @@ static void pc_init1(MachineState *machine,
>>>>   if (pcmc->pci_enabled) {
>>>>   DeviceState *dev;
>>>>   PCIDevice *pci_dev;
>>>> -const char *type = xen_enabled() ? TYPE_PIIX3_XEN_DEVICE
>>>> - : TYPE_PIIX3_DEVICE;
>>>>   int i;
>>>>   
>>>>   pci_bus = i440fx_init(pci_type,
>>>> @@ -250,7 +248,7 @@ static void pc_init1(MachineState *machine,
>>>>  : pci_slot_get_pirq);
>>>>   pcms->bus = pci_bus;
>>>>   
>>>> -pci_dev = pci_new_multifunction(-1, true, type);
>>>> +pci_dev = pci_new_multifunction(-1, true, TYPE_PIIX3_DEVICE);
>>>>   object_property_set_bool(OBJECT(pci_dev), "has-usb",
>>>>machine_usb(machine), _abort);
>>>>   object_property_set_bool(OBJECT(pci_dev), "has-acpi",
>>>> diff --git a/hw/isa/piix.c b/hw/isa/piix.c
>>>> index 98e9b12661..e4587352c9 100644
>>>> --- a/hw/isa/piix.c
>>>> +++ b/hw/isa/piix.c
>>>> @@ -33,7 +33,6 @@
>>>>   #include "hw/qdev-properties.h"
>>>>   #include "hw/ide/piix.h"
>>>>   #include "hw/isa/isa.h"
>>>> -#include "hw/xen/xen.h"
>>>>   #include "sysemu/runstate.h"
>>>>   #include "migration/vmstate.h"
>>>>   #include "hw/acpi/acpi_aml_interface.h"
>>>> @@ -465,24 +464,6 @@ static const TypeInfo piix3_info = {
>>>>   .class_init= piix3_class_init,
>>>>   };
>>>>   
>>>> -static void piix3_xen_class_init(ObjectClass *klass, void *data)
>>>> -{
>>>> -DeviceClass *dc = DEVICE_CLASS(klass);
>>>> -PCIDeviceClass *k = PCI_DEVICE_CLASS(klass);
>>>> -
>>>> -k->realize = piix3_realize;
>>>> -/* 82371SB PIIX3 PCI-to-ISA bridge (Step A1) */
>>>> -k->device_id = PCI_DEVICE_ID_INTEL_82371SB_0;
>>>> -dc->vmsd = _piix3;
>>> 
>>> IIUC, since this device is user-creatable, we can't simply remove it
>>> without going thru the deprecation process. Alternatively we could
>>> add a type alias:
>>> 
>>> -- >8 --
>>> diff --git a/softmmu/qdev-monitor.c b/softmmu/qdev-monitor.c
>>> index 4b0ef65780..d94f7ea369 100644
>>> --- a/softmmu/qdev-monitor.c
>>> +++ b/softmmu/qdev-monitor.c
>>> @@ -64,6 +64,7 @@ typedef struct QDevAlias
>>> QEMU_ARCH_LOONGARCH)
>>>   #define QEMU_ARCH_VIRTIO_CCW (QEMU_ARCH_S390X)
>>>   #define QEMU_ARCH_VIRTIO_MMIO (QEMU_ARCH_M68K)
>>> +#define QEMU_ARCH_XEN (QEMU_ARCH_ARM | QEMU_ARCH_I386)
>>> 
>>>   /* Please keep this table sorted by typename. */
>>>   static const QDevAlias qdev_alias_table[] = {
>>> @@ -111,6 +112,7 @@ static const QDevAlias qdev_alias_table[] = {
>>>   { "virtio-tablet-device", "virtio-tablet", QEMU_ARCH_VIRTIO_MMIO },
>>>   { "virtio-tablet-ccw", "virtio-tablet", QEMU_ARCH_VIRTIO_CCW },
>>>   { "virtio-tablet-pci", "virtio-tablet", QEMU_ARCH_VIRTIO_PCI },
>>> +{ "PIIX3", "PIIX3-xen", QEMU_ARCH_XEN },
>>
>>Hi Bernhard,
>>
>>Can you comment if this should be:
>>
>>+{ "PIIX", "PIIX3-xen", QEMU_ARCH_XEN },
>>
>>instead? IIUC, the patch series also removed PIIX3 and PIIX4 and
>>replaced them with PIIX. Or am I not understanding correctly?
> 
> PIIX3 is correct. The PIIX consolidation is just about sharing code between 
> the PIIX3 and PIIX4 south bridges and should not cause any user or guest 
> observable differences.

I realize that now. I see the PIIX3 device still exists after applying the 
patch set.
Thanks,

Chuck

Re: [PATCH v6] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

On 1/3/23 10:14 AM, Alex Williamson wrote:

> 
> It's necessary to configure the assigned IGD at slot 2 to make it
> functional, yes, but I don't really understand this notion of
> "reserving" slot 2.  If something occupies address 00:02.0 in the
> config, it's the user's or management tool's responsibility to move it
> to make this configuration functional.  Why does QEMU need to play a
> part in reserving this bus address.  IGD devices are not generally
> hot-pluggable either, so it doesn't seem we need to reserve an address
> in case an IGD device is added dynamically later.

The capability to reserve a bus address for a quirky device need not
be limited to the case of hotplugged or dynamically added devices. The
igd is a quirky device, and its presence in an emulated system like
qemu requires special handling. The slot_reserved_mask member of PCIBus
is also well-suited to the case of quirky device like Intel the igd that
needs to be at slot 2. Just because it is not dynamically added later
does not change the fact that it needs special handling at its initial
configuration when the guest is being created.

>  

Here's the problem that answers Michael's question why this patch is
specific to xen:

---snip---
#ifdef CONFIG_XEN

...

static void pc_xen_hvm_init(MachineState *machine)
{
PCMachineState *pcms = PC_MACHINE(machine);

if (!xen_enabled()) {
error_report("xenfv machine requires the xen accelerator");
exit(1);
}

pc_xen_hvm_init_pci(machine);
pci_create_simple(pcms->bus, -1, "xen-platform");
}
#endif
---snip---

This code is from hw/i386/pc_piix.c. Note the call to
pci_create_simple to create the xen platform pci device,
which has -1 as the second argument. That -1 tells
pci_create_simple to autoconfigure the pci bdf address.

It is *hard-coded* that way. That means no toolstack or
management tool can change it. And what is hard-coded here
is that the xen platform device will occupy slot 2, preventing
the Intel igd or any other device from occupying slot 2.

So, even if xen developers wanted to create a version of the
libxl that is flexible enough to allow the xen platform device
to be at a different slot, they could not without patching
qemu to at least change that -1 to an initialization variable
that can be read from a qemu command line option that libxl
could configure.

So, why not just accept this patch as the best way to deal
with a xen-specific problem and fix it in a way that uses
the xen/libxl philosophy of autoconfiguring things as much as
possible except in cases of quirky devices like the Intel igd
in which case the existing slot_reserved_mask member of PCIBus
is very useful to accommodate the quirky igd device?

IMHO, trying to impose the kvm/libvirt philosophy of having
a very configurable toolstack on the xen/xenlight philosophy
of autoconfiguring things that can be autoconfigured and
using higher-level configuration options like igd-passthrough=on
to tweak how autoconfiguration is done in a way that is compatible
with quirky devices like the Intel igd is like trying to put
a square peg into a round hole. Actually, qemu with its qom is
able to accommodate both approaches to the design of a toolstack,
and each vendor or project that depends on qemu should be free to
use the approach it prefers.

Just my two cents, FWIW.

Kind regards,

Chuck

Re: [PATCH v2 6/6] hw/isa/piix: Resolve redundant TYPE_PIIX3_XEN_DEVICE

[Chuck Zmudzinski]

On 1/4/23 1:48 PM, Philippe Mathieu-Daudé wrote:
> On 4/1/23 18:54, Chuck Zmudzinski wrote:
>> On 1/4/23 10:35 AM, Philippe Mathieu-Daudé wrote:
>>> +Markus/Thomas
>>>
>>> On 4/1/23 15:44, Bernhard Beschow wrote:
>>>> During the last patches, TYPE_PIIX3_XEN_DEVICE turned into a clone of
>>>> TYPE_PIIX3_DEVICE. Remove this redundancy.
>>>>
>>>> Signed-off-by: Bernhard Beschow 
>>>> ---
>>>>hw/i386/pc_piix.c |  4 +---
>>>>hw/isa/piix.c | 20 
>>>>include/hw/southbridge/piix.h |  1 -
>>>>3 files changed, 1 insertion(+), 24 deletions(-)
>>>>
>>>> diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c
>>>> index 5738d9cdca..6b8de3d59d 100644
>>>> --- a/hw/i386/pc_piix.c
>>>> +++ b/hw/i386/pc_piix.c
>>>> @@ -235,8 +235,6 @@ static void pc_init1(MachineState *machine,
>>>>if (pcmc->pci_enabled) {
>>>>DeviceState *dev;
>>>>PCIDevice *pci_dev;
>>>> -const char *type = xen_enabled() ? TYPE_PIIX3_XEN_DEVICE
>>>> - : TYPE_PIIX3_DEVICE;
>>>>int i;
>>>>
>>>>pci_bus = i440fx_init(pci_type,
>>>> @@ -250,7 +248,7 @@ static void pc_init1(MachineState *machine,
>>>>   : pci_slot_get_pirq);
>>>>pcms->bus = pci_bus;
>>>>
>>>> -pci_dev = pci_new_multifunction(-1, true, type);
>>>> +pci_dev = pci_new_multifunction(-1, true, TYPE_PIIX3_DEVICE);
>>>>object_property_set_bool(OBJECT(pci_dev), "has-usb",
>>>> machine_usb(machine), _abort);
>>>>object_property_set_bool(OBJECT(pci_dev), "has-acpi",
>>>> diff --git a/hw/isa/piix.c b/hw/isa/piix.c
>>>> index 98e9b12661..e4587352c9 100644
>>>> --- a/hw/isa/piix.c
>>>> +++ b/hw/isa/piix.c
>>>> @@ -33,7 +33,6 @@
>>>>#include "hw/qdev-properties.h"
>>>>#include "hw/ide/piix.h"
>>>>#include "hw/isa/isa.h"
>>>> -#include "hw/xen/xen.h"
>>>>#include "sysemu/runstate.h"
>>>>#include "migration/vmstate.h"
>>>>#include "hw/acpi/acpi_aml_interface.h"
>>>> @@ -465,24 +464,6 @@ static const TypeInfo piix3_info = {
>>>>.class_init= piix3_class_init,
>>>>};
>>>>
>>>> -static void piix3_xen_class_init(ObjectClass *klass, void *data)
>>>> -{
>>>> -DeviceClass *dc = DEVICE_CLASS(klass);
>>>> -PCIDeviceClass *k = PCI_DEVICE_CLASS(klass);
>>>> -
>>>> -k->realize = piix3_realize;
>>>> -/* 82371SB PIIX3 PCI-to-ISA bridge (Step A1) */
>>>> -k->device_id = PCI_DEVICE_ID_INTEL_82371SB_0;
>>>> -dc->vmsd = _piix3;
>>>
>>> IIUC, since this device is user-creatable, we can't simply remove it
>>> without going thru the deprecation process. Alternatively we could
>>> add a type alias:
>>>
>>> -- >8 --
>>> diff --git a/softmmu/qdev-monitor.c b/softmmu/qdev-monitor.c
>>> index 4b0ef65780..d94f7ea369 100644
>>> --- a/softmmu/qdev-monitor.c
>>> +++ b/softmmu/qdev-monitor.c
>>> @@ -64,6 +64,7 @@ typedef struct QDevAlias
>>>  QEMU_ARCH_LOONGARCH)
>>>#define QEMU_ARCH_VIRTIO_CCW (QEMU_ARCH_S390X)
>>>#define QEMU_ARCH_VIRTIO_MMIO (QEMU_ARCH_M68K)
>>> +#define QEMU_ARCH_XEN (QEMU_ARCH_ARM | QEMU_ARCH_I386)
>>>
>>>/* Please keep this table sorted by typename. */
>>>static const QDevAlias qdev_alias_table[] = {
>>> @@ -111,6 +112,7 @@ static const QDevAlias qdev_alias_table[] = {
>>>{ "virtio-tablet-device", "virtio-tablet", QEMU_ARCH_VIRTIO_MMIO },
>>>{ "virtio-tablet-ccw", "virtio-tablet", QEMU_ARCH_VIRTIO_CCW },
>>>{ "virtio-tablet-pci", "virtio-tablet", QEMU_ARCH_VIRTIO_PCI },
>>> +{ "PIIX3", "PIIX3-xen", QEMU_ARCH_XEN },
>> 
>> Hi Bernhard,
>> 
>> Can you comment if this should be:
>> 
>> +{ "PIIX", "PIIX3-xen", QEMU_ARCH_XEN },
>&g

Re: [PATCH v2 6/6] hw/isa/piix: Resolve redundant TYPE_PIIX3_XEN_DEVICE

[Chuck Zmudzinski]

On 1/4/23 10:35 AM, Philippe Mathieu-Daudé wrote:
> +Markus/Thomas
> 
> On 4/1/23 15:44, Bernhard Beschow wrote:
>> During the last patches, TYPE_PIIX3_XEN_DEVICE turned into a clone of
>> TYPE_PIIX3_DEVICE. Remove this redundancy.
>> 
>> Signed-off-by: Bernhard Beschow 
>> ---
>>   hw/i386/pc_piix.c |  4 +---
>>   hw/isa/piix.c | 20 
>>   include/hw/southbridge/piix.h |  1 -
>>   3 files changed, 1 insertion(+), 24 deletions(-)
>> 
>> diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c
>> index 5738d9cdca..6b8de3d59d 100644
>> --- a/hw/i386/pc_piix.c
>> +++ b/hw/i386/pc_piix.c
>> @@ -235,8 +235,6 @@ static void pc_init1(MachineState *machine,
>>   if (pcmc->pci_enabled) {
>>   DeviceState *dev;
>>   PCIDevice *pci_dev;
>> -const char *type = xen_enabled() ? TYPE_PIIX3_XEN_DEVICE
>> - : TYPE_PIIX3_DEVICE;
>>   int i;
>>   
>>   pci_bus = i440fx_init(pci_type,
>> @@ -250,7 +248,7 @@ static void pc_init1(MachineState *machine,
>>  : pci_slot_get_pirq);
>>   pcms->bus = pci_bus;
>>   
>> -pci_dev = pci_new_multifunction(-1, true, type);
>> +pci_dev = pci_new_multifunction(-1, true, TYPE_PIIX3_DEVICE);
>>   object_property_set_bool(OBJECT(pci_dev), "has-usb",
>>machine_usb(machine), _abort);
>>   object_property_set_bool(OBJECT(pci_dev), "has-acpi",
>> diff --git a/hw/isa/piix.c b/hw/isa/piix.c
>> index 98e9b12661..e4587352c9 100644
>> --- a/hw/isa/piix.c
>> +++ b/hw/isa/piix.c
>> @@ -33,7 +33,6 @@
>>   #include "hw/qdev-properties.h"
>>   #include "hw/ide/piix.h"
>>   #include "hw/isa/isa.h"
>> -#include "hw/xen/xen.h"
>>   #include "sysemu/runstate.h"
>>   #include "migration/vmstate.h"
>>   #include "hw/acpi/acpi_aml_interface.h"
>> @@ -465,24 +464,6 @@ static const TypeInfo piix3_info = {
>>   .class_init= piix3_class_init,
>>   };
>>   
>> -static void piix3_xen_class_init(ObjectClass *klass, void *data)
>> -{
>> -DeviceClass *dc = DEVICE_CLASS(klass);
>> -PCIDeviceClass *k = PCI_DEVICE_CLASS(klass);
>> -
>> -k->realize = piix3_realize;
>> -/* 82371SB PIIX3 PCI-to-ISA bridge (Step A1) */
>> -k->device_id = PCI_DEVICE_ID_INTEL_82371SB_0;
>> -dc->vmsd = _piix3;
> 
> IIUC, since this device is user-creatable, we can't simply remove it
> without going thru the deprecation process. Alternatively we could
> add a type alias:
> 
> -- >8 --
> diff --git a/softmmu/qdev-monitor.c b/softmmu/qdev-monitor.c
> index 4b0ef65780..d94f7ea369 100644
> --- a/softmmu/qdev-monitor.c
> +++ b/softmmu/qdev-monitor.c
> @@ -64,6 +64,7 @@ typedef struct QDevAlias
> QEMU_ARCH_LOONGARCH)
>   #define QEMU_ARCH_VIRTIO_CCW (QEMU_ARCH_S390X)
>   #define QEMU_ARCH_VIRTIO_MMIO (QEMU_ARCH_M68K)
> +#define QEMU_ARCH_XEN (QEMU_ARCH_ARM | QEMU_ARCH_I386)
> 
>   /* Please keep this table sorted by typename. */
>   static const QDevAlias qdev_alias_table[] = {
> @@ -111,6 +112,7 @@ static const QDevAlias qdev_alias_table[] = {
>   { "virtio-tablet-device", "virtio-tablet", QEMU_ARCH_VIRTIO_MMIO },
>   { "virtio-tablet-ccw", "virtio-tablet", QEMU_ARCH_VIRTIO_CCW },
>   { "virtio-tablet-pci", "virtio-tablet", QEMU_ARCH_VIRTIO_PCI },
> +{ "PIIX3", "PIIX3-xen", QEMU_ARCH_XEN },

Hi Bernhard,

Can you comment if this should be:

+{ "PIIX", "PIIX3-xen", QEMU_ARCH_XEN },

instead? IIUC, the patch series also removed PIIX3 and PIIX4 and
replaced them with PIIX. Or am I not understanding correctly?

Best regards,

Chuck


>   { }
>   };
> ---
> 
> But I'm not sure due to this comment from commit ee46d8a503
> (2011-12-22 15:24:20 -0600):
> 
> 47) /*
> 48)  * Aliases were a bad idea from the start.  Let's keep them
> 49)  * from spreading further.
> 50)  */
> 
> Maybe using qdev_alias_table[] during device deprecation is
> acceptable?
> 
>> -}
>> -
>> -static const TypeInfo piix3_xen_info = {
>> -.name  = TYPE_PIIX3_XEN_DEVICE,
>> -.parent= TYPE_PIIX_PCI_DEVICE,
>> -.instance_init = piix3_init,
>> -.class_init= piix3_xen_class_init,
>> -};
>> -
>>   static void piix4_realize(PCIDevice *dev, Error **errp)
>>   {
>>   ERRP_GUARD();
>> @@ -534,7 +515,6 @@ static void piix3_register_types(void)
>>   {
>>   type_register_static(_pci_type_info);
>>   type_register_static(_info);
>> -type_register_static(_xen_info);
>>   type_register_static(_info);
>>   }
>>   
>> diff --git a/include/hw/southbridge/piix.h b/include/hw/southbridge/piix.h
>> index 65ad8569da..b1fc94a742 100644
>> --- a/include/hw/southbridge/piix.h
>> +++ b/include/hw/southbridge/piix.h
>> @@ -77,7 +77,6 @@ struct PIIXState {
>>   OBJECT_DECLARE_SIMPLE_TYPE(PIIXState, PIIX_PCI_DEVICE)
>>   
>>   #define TYPE_PIIX3_DEVICE "PIIX3"
>> -#define TYPE_PIIX3_XEN_DEVICE "PIIX3-xen"
>>   #define TYPE_PIIX4_PCI_DEVICE 

Re: [PATCH 0/6] Resolve TYPE_PIIX3_XEN_DEVICE

[Chuck Zmudzinski]

On 1/4/23 11:12 AM, Bernhard Beschow wrote:
> 
> 
> Am 4. Januar 2023 13:11:16 UTC schrieb Chuck Zmudzinski :
>>On 1/4/2023 7:13 AM, Bernhard Beschow wrote:
>>> Am 4. Januar 2023 08:18:59 UTC schrieb Chuck Zmudzinski :
>>> >On 1/3/2023 8:38 AM, Bernhard Beschow wrote:
>>> >>
>>> >>
>>> >> On Tue, Jan 3, 2023 at 2:17 PM Philippe Mathieu-Daudé 
>>> >>  wrote:
>>> >>
>>> >> Hi Chuck,
>>> >>
>>> >> On 3/1/23 04:15, Chuck Zmudzinski wrote:
>>> >> > On 1/2/23 4:34 PM, Bernhard Beschow wrote:
>>> >> >> This series first renders TYPE_PIIX3_XEN_DEVICE redundant and 
>>> >> finally removes
>>> >> >> it. The motivation is to 1/ decouple PIIX from Xen and 2/ to make 
>>> >> Xen in the PC
>>> >> >> machine agnostic to the precise southbridge being used. 2/ will 
>>> >> become
>>> >> >> particularily interesting once PIIX4 becomes usable in the PC 
>>> >> machine, avoiding
>>> >> >> the "Frankenstein" use of PIIX4_ACPI in PIIX3.
>>> >> >>
>>> >> >> Testing done:
>>> >> >> None, because I don't know how to conduct this properly :(
>>> >> >>
>>> >> >> Based-on: <20221221170003.2929-1-shen...@gmail.com>
>>> >> >>            "[PATCH v4 00/30] Consolidate PIIX south bridges"
>>> >>
>>> >> This series is based on a previous series:
>>> >> 
>>> >> https://lore.kernel.org/qemu-devel/20221221170003.2929-1-shen...@gmail.com/
>>> >> (which itself also is).
>>> >>
>>> >> >> Bernhard Beschow (6):
>>> >> >>    include/hw/xen/xen: Make xen_piix3_set_irq() generic and 
>>> >> rename it
>>> >> >>    hw/isa/piix: Reuse piix3_realize() in piix3_xen_realize()
>>> >> >>    hw/isa/piix: Wire up Xen PCI IRQ handling outside of PIIX3
>>> >> >>    hw/isa/piix: Avoid Xen-specific variant of piix_write_config()
>>> >> >>    hw/isa/piix: Resolve redundant k->config_write assignments
>>> >> >>    hw/isa/piix: Resolve redundant TYPE_PIIX3_XEN_DEVICE
>>> >> >>
>>> >> >>   hw/i386/pc_piix.c             | 34 --
>>> >> >>   hw/i386/xen/xen-hvm.c         |  9 +++--
>>> >> >>   hw/isa/piix.c                 | 66 
>>> >> +--
>>> >> >
>>> >> > This file does not exist on the Qemu master branch.
>>> >> > But hw/isa/piix3.c and hw/isa/piix4.c do exist.
>>> >> >
>>> >> > I tried renaming it from piix.c to piix3.c in the patch, but
>>> >> > the patch set still does not apply cleanly on my tree.
>>> >> >
>>> >> > Is this patch set re-based against something other than
>>> >> > the current master Qemu branch?
>>> >> >
>>> >> > I have a system that is suitable for testing this patch set, but
>>> >> > I need guidance on how to apply it to the Qemu source tree.
>>> >>
>>> >> You can ask Bernhard to publish a branch with the full work,
>>> >>
>>> >>
>>> >> Hi Chuck,
>>> >>
>>> >> ... or just visit 
>>> >> https://patchew.org/QEMU/20230102213504.14646-1-shen...@gmail.com/ . 
>>> >> There you'll find a git tag with a complete history and all instructions!
>>> >>
>>> >> Thanks for giving my series a test ride!
>>> >>
>>> >> Best regards,
>>> >> Bernhard
>>> >>
>>> >> or apply each series locally. I use the b4 tool for that:
>>> >> https://b4.docs.kernel.org/en/latest/installing.html
>>> >>
>>> >> i.e.:
>>> >>
>>> >> $ git checkout -b shentey_work
>>> >> $ b4 am 20221120150550.63059-1-shen...@gmail.com
>>> >> $ git am
>>> >> 
>>> >> ./v2_20221120_shentey_decouple_intx_to_lnkx_routing_from_south_bridges.mbx
>>&

Re: [PATCH v2 6/6] hw/isa/piix: Resolve redundant TYPE_PIIX3_XEN_DEVICE

[Chuck Zmudzinski]

On 1/4/23 9:44 AM, Bernhard Beschow wrote:
> During the last patches, TYPE_PIIX3_XEN_DEVICE turned into a clone of
> TYPE_PIIX3_DEVICE. Remove this redundancy.
> 
> Signed-off-by: Bernhard Beschow 
> ---
>  hw/i386/pc_piix.c |  4 +---
>  hw/isa/piix.c | 20 
>  include/hw/southbridge/piix.h |  1 -
>  3 files changed, 1 insertion(+), 24 deletions(-)
> 
> diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c
> index 5738d9cdca..6b8de3d59d 100644
> --- a/hw/i386/pc_piix.c
> +++ b/hw/i386/pc_piix.c
> @@ -235,8 +235,6 @@ static void pc_init1(MachineState *machine,
>  if (pcmc->pci_enabled) {
>  DeviceState *dev;
>  PCIDevice *pci_dev;
> -const char *type = xen_enabled() ? TYPE_PIIX3_XEN_DEVICE
> - : TYPE_PIIX3_DEVICE;
>  int i;
>  
>  pci_bus = i440fx_init(pci_type,
> @@ -250,7 +248,7 @@ static void pc_init1(MachineState *machine,
> : pci_slot_get_pirq);
>  pcms->bus = pci_bus;
>  
> -pci_dev = pci_new_multifunction(-1, true, type);
> +pci_dev = pci_new_multifunction(-1, true, TYPE_PIIX3_DEVICE);
>  object_property_set_bool(OBJECT(pci_dev), "has-usb",
>   machine_usb(machine), _abort);
>  object_property_set_bool(OBJECT(pci_dev), "has-acpi",
> diff --git a/hw/isa/piix.c b/hw/isa/piix.c
> index 98e9b12661..e4587352c9 100644
> --- a/hw/isa/piix.c
> +++ b/hw/isa/piix.c
> @@ -33,7 +33,6 @@
>  #include "hw/qdev-properties.h"
>  #include "hw/ide/piix.h"
>  #include "hw/isa/isa.h"
> -#include "hw/xen/xen.h"
>  #include "sysemu/runstate.h"
>  #include "migration/vmstate.h"
>  #include "hw/acpi/acpi_aml_interface.h"
> @@ -465,24 +464,6 @@ static const TypeInfo piix3_info = {
>  .class_init= piix3_class_init,
>  };
>  
> -static void piix3_xen_class_init(ObjectClass *klass, void *data)
> -{
> -DeviceClass *dc = DEVICE_CLASS(klass);
> -PCIDeviceClass *k = PCI_DEVICE_CLASS(klass);
> -
> -k->realize = piix3_realize;
> -/* 82371SB PIIX3 PCI-to-ISA bridge (Step A1) */
> -k->device_id = PCI_DEVICE_ID_INTEL_82371SB_0;
> -dc->vmsd = _piix3;
> -}
> -
> -static const TypeInfo piix3_xen_info = {
> -.name  = TYPE_PIIX3_XEN_DEVICE,
> -.parent= TYPE_PIIX_PCI_DEVICE,
> -.instance_init = piix3_init,
> -.class_init= piix3_xen_class_init,
> -};
> -
>  static void piix4_realize(PCIDevice *dev, Error **errp)
>  {
>  ERRP_GUARD();
> @@ -534,7 +515,6 @@ static void piix3_register_types(void)
>  {
>  type_register_static(_pci_type_info);
>  type_register_static(_info);
> -type_register_static(_xen_info);
>  type_register_static(_info);
>  }
>  
> diff --git a/include/hw/southbridge/piix.h b/include/hw/southbridge/piix.h
> index 65ad8569da..b1fc94a742 100644
> --- a/include/hw/southbridge/piix.h
> +++ b/include/hw/southbridge/piix.h
> @@ -77,7 +77,6 @@ struct PIIXState {
>  OBJECT_DECLARE_SIMPLE_TYPE(PIIXState, PIIX_PCI_DEVICE)
>  
>  #define TYPE_PIIX3_DEVICE "PIIX3"
> -#define TYPE_PIIX3_XEN_DEVICE "PIIX3-xen"
>  #define TYPE_PIIX4_PCI_DEVICE "piix4-isa"
>  
>  #endif


This fixes the regression with the emulated usb tablet device that I reported 
in v1 here:

https://lore.kernel.org/qemu-devel/aed4f2c1-83f7-163a-fb44-f28437666...@aol.com/

I tested this patch again with all the prerequisites and now with v2 there are 
no regressions.

Tested-by: Chuck Zmudzinski 

Re: [PATCH 0/6] Resolve TYPE_PIIX3_XEN_DEVICE

[Chuck Zmudzinski]

On 1/4/2023 7:13 AM, Bernhard Beschow wrote:
> Am 4. Januar 2023 08:18:59 UTC schrieb Chuck Zmudzinski :
> >On 1/3/2023 8:38 AM, Bernhard Beschow wrote:
> >>
> >>
> >> On Tue, Jan 3, 2023 at 2:17 PM Philippe Mathieu-Daudé  
> >> wrote:
> >>
> >> Hi Chuck,
> >>
> >> On 3/1/23 04:15, Chuck Zmudzinski wrote:
> >> > On 1/2/23 4:34 PM, Bernhard Beschow wrote:
> >> >> This series first renders TYPE_PIIX3_XEN_DEVICE redundant and 
> >> finally removes
> >> >> it. The motivation is to 1/ decouple PIIX from Xen and 2/ to make 
> >> Xen in the PC
> >> >> machine agnostic to the precise southbridge being used. 2/ will 
> >> become
> >> >> particularily interesting once PIIX4 becomes usable in the PC 
> >> machine, avoiding
> >> >> the "Frankenstein" use of PIIX4_ACPI in PIIX3.
> >> >>
> >> >> Testing done:
> >> >> None, because I don't know how to conduct this properly :(
> >> >>
> >> >> Based-on: <20221221170003.2929-1-shen...@gmail.com>
> >> >>            "[PATCH v4 00/30] Consolidate PIIX south bridges"
> >>
> >> This series is based on a previous series:
> >> 
> >> https://lore.kernel.org/qemu-devel/20221221170003.2929-1-shen...@gmail.com/
> >> (which itself also is).
> >>
> >> >> Bernhard Beschow (6):
> >> >>    include/hw/xen/xen: Make xen_piix3_set_irq() generic and rename 
> >> it
> >> >>    hw/isa/piix: Reuse piix3_realize() in piix3_xen_realize()
> >> >>    hw/isa/piix: Wire up Xen PCI IRQ handling outside of PIIX3
> >> >>    hw/isa/piix: Avoid Xen-specific variant of piix_write_config()
> >> >>    hw/isa/piix: Resolve redundant k->config_write assignments
> >> >>    hw/isa/piix: Resolve redundant TYPE_PIIX3_XEN_DEVICE
> >> >>
> >> >>   hw/i386/pc_piix.c             | 34 --
> >> >>   hw/i386/xen/xen-hvm.c         |  9 +++--
> >> >>   hw/isa/piix.c                 | 66 
> >> +--
> >> >
> >> > This file does not exist on the Qemu master branch.
> >> > But hw/isa/piix3.c and hw/isa/piix4.c do exist.
> >> >
> >> > I tried renaming it from piix.c to piix3.c in the patch, but
> >> > the patch set still does not apply cleanly on my tree.
> >> >
> >> > Is this patch set re-based against something other than
> >> > the current master Qemu branch?
> >> >
> >> > I have a system that is suitable for testing this patch set, but
> >> > I need guidance on how to apply it to the Qemu source tree.
> >>
> >> You can ask Bernhard to publish a branch with the full work,
> >>
> >>
> >> Hi Chuck,
> >>
> >> ... or just visit 
> >> https://patchew.org/QEMU/20230102213504.14646-1-shen...@gmail.com/ . There 
> >> you'll find a git tag with a complete history and all instructions!
> >>
> >> Thanks for giving my series a test ride!
> >>
> >> Best regards,
> >> Bernhard
> >>
> >> or apply each series locally. I use the b4 tool for that:
> >> https://b4.docs.kernel.org/en/latest/installing.html
> >>
> >> i.e.:
> >>
> >> $ git checkout -b shentey_work
> >> $ b4 am 20221120150550.63059-1-shen...@gmail.com
> >> $ git am
> >> 
> >> ./v2_20221120_shentey_decouple_intx_to_lnkx_routing_from_south_bridges.mbx
> >> $ b4 am 20221221170003.2929-1-shen...@gmail.com
> >> $ git am
> >> 
> >> ./v4_20221221_shentey_this_series_consolidates_the_implementations_of_the_piix3_and_piix4_south.mbx
> >> $ b4 am 20230102213504.14646-1-shen...@gmail.com
> >> $ git am ./20230102_shentey_resolve_type_piix3_xen_device.mbx
> >>
> >> Now the branch 'shentey_work' contains all the patches and you can 
> >> test.
> >>
> >> Regards,
> >>
> >> Phil.
> >>
> >
> >Hi Phil and Bernard,
> >
> >I tried applying these 3 patch series on top of the current qemu
> >master branch.
> >
> >Unfortunately, I saw a regression, so I can'

Re: [PATCH 0/6] Resolve TYPE_PIIX3_XEN_DEVICE

[Chuck Zmudzinski]

On 1/3/2023 8:38 AM, Bernhard Beschow wrote:
>
>
> On Tue, Jan 3, 2023 at 2:17 PM Philippe Mathieu-Daudé  
> wrote:
>
> Hi Chuck,
>
>     On 3/1/23 04:15, Chuck Zmudzinski wrote:
> > On 1/2/23 4:34 PM, Bernhard Beschow wrote:
> >> This series first renders TYPE_PIIX3_XEN_DEVICE redundant and finally 
> removes
> >> it. The motivation is to 1/ decouple PIIX from Xen and 2/ to make Xen 
> in the PC
> >> machine agnostic to the precise southbridge being used. 2/ will become
> >> particularily interesting once PIIX4 becomes usable in the PC machine, 
> avoiding
> >> the "Frankenstein" use of PIIX4_ACPI in PIIX3.
> >>
> >> Testing done:
> >> None, because I don't know how to conduct this properly :(
> >>
> >> Based-on: <20221221170003.2929-1-shen...@gmail.com>
> >>            "[PATCH v4 00/30] Consolidate PIIX south bridges"
>
> This series is based on a previous series:
> 
> https://lore.kernel.org/qemu-devel/20221221170003.2929-1-shen...@gmail.com/
> (which itself also is).
>
> >> Bernhard Beschow (6):
> >>    include/hw/xen/xen: Make xen_piix3_set_irq() generic and rename it
> >>    hw/isa/piix: Reuse piix3_realize() in piix3_xen_realize()
> >>    hw/isa/piix: Wire up Xen PCI IRQ handling outside of PIIX3
> >>    hw/isa/piix: Avoid Xen-specific variant of piix_write_config()
> >>    hw/isa/piix: Resolve redundant k->config_write assignments
> >>    hw/isa/piix: Resolve redundant TYPE_PIIX3_XEN_DEVICE
> >>
> >>   hw/i386/pc_piix.c             | 34 --
> >>   hw/i386/xen/xen-hvm.c         |  9 +++--
> >>   hw/isa/piix.c                 | 66 
> +--
> >
> > This file does not exist on the Qemu master branch.
> > But hw/isa/piix3.c and hw/isa/piix4.c do exist.
> >
> > I tried renaming it from piix.c to piix3.c in the patch, but
> > the patch set still does not apply cleanly on my tree.
> >
> > Is this patch set re-based against something other than
> > the current master Qemu branch?
> >
> > I have a system that is suitable for testing this patch set, but
> > I need guidance on how to apply it to the Qemu source tree.
>
> You can ask Bernhard to publish a branch with the full work,
>
>
> Hi Chuck,
>
> ... or just visit 
> https://patchew.org/QEMU/20230102213504.14646-1-shen...@gmail.com/ . There 
> you'll find a git tag with a complete history and all instructions!
>
> Thanks for giving my series a test ride!
>
> Best regards,
> Bernhard
>
> or apply each series locally. I use the b4 tool for that:
> https://b4.docs.kernel.org/en/latest/installing.html
>
> i.e.:
>
> $ git checkout -b shentey_work
> $ b4 am 20221120150550.63059-1-shen...@gmail.com
> $ git am
> ./v2_20221120_shentey_decouple_intx_to_lnkx_routing_from_south_bridges.mbx
> $ b4 am 20221221170003.2929-1-shen...@gmail.com
> $ git am
> 
> ./v4_20221221_shentey_this_series_consolidates_the_implementations_of_the_piix3_and_piix4_south.mbx
> $ b4 am 20230102213504.14646-1-shen...@gmail.com
> $ git am ./20230102_shentey_resolve_type_piix3_xen_device.mbx
>
> Now the branch 'shentey_work' contains all the patches and you can test.
>
> Regards,
>
> Phil.
>

Hi Phil and Bernard,

I tried applying these 3 patch series on top of the current qemu
master branch.

Unfortunately, I saw a regression, so I can't give a tested-by tag yet.

Here are the details of the testing I did so far:

Xen only needs one target, the i386-softmmu target which creates
the qemu-system-i386 binary that Xen uses for its device model.
That target compiled and linked with no problems with these 3
patch series applied on top of qemu master. I didn't try building
any other targets.

My tests used the xenfv machine type with the xen platform
pci device, which is ordinarily called a xen hvm guest with xen
paravirtualized network and block device drivers. It is based on the
i440fx machine type and so emulates piix3. I tested the xen
hvm guests with two different configurations as described below.

I tested both Linux and Windows guests, with mixed results. With the
current Qemu master (commit 222059a0fccf4 without the 3 patch
series applied), all tested guest configurations work normally for both
Linux and Windows guests.

With these 3 patch series applied on top of the qemu master branch,
which tries to consolidate piix3 and piix4 and resolve the xen piix3
device tha

Re: [PATCH v6] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski]

On 1/3/2023 4:50 PM, Chuck Zmudzinski wrote:
> On 1/3/2023 10:14 AM, Alex Williamson wrote:
> > On Mon, 2 Jan 2023 18:10:24 -0500
> > Chuck Zmudzinski  wrote:
> >
> > > On 1/2/23 12:46 PM, Michael S. Tsirkin wrote:
> > > > On Sun, Jan 01, 2023 at 06:52:03PM -0500, Chuck Zmudzinski wrote:  
> > > > > Intel specifies that the Intel IGD must occupy slot 2 on the PCI bus,
> > > > > as noted in docs/igd-assign.txt in the Qemu source code.
> > > > > ... 
> > > > > Signed-off-by: Chuck Zmudzinski   
> > > >
> > > > I'm not sure why is the issue xen specific. Can you explain?
> > > > Doesn't it affect kvm too?  
> > > 
> > > Recall from docs/igd-assign.txt that there are two modes for
> > > igd passthrough: legacy and upt, and the igd needs to be
> > > at slot 2 only when using legacy mode which gives one
> > > single guest exclusive access to the Intel igd.
> > > 
> > > It's only xen specific insofar as xen does not have support
> > > for the upt mode so xen must use legacy mode which
> > > requires the igd to be at slot 2. I am not an expert with
> >
> > UPT mode never fully materialized for direct assignment, the folks at
> > Intel championing this scenario left.
>
> Thanks for clarifying that for me.
>
> >
> > > kvm, but if I understand correctly, with kvm one can use
> > > the upt mode with the Intel i915 kvmgt kernel module
> > > and in that case the guest will see a virtual Intel gpu
> > > that can be at any arbitrary slot when using kvmgt, and
> > > also, in that case, more than one guest can access the
> > > igd through the kvmgt kernel module.
> >
> > This is true, IIRC an Intel vGPU does not need to be in slot 2.
> >
> > > Again, I am not an expert and do not have as much
> > > experience with kvm, but if I understand correctly it is
> > > possible to use the legacy mode with kvm and I think you
> > > are correct that if one uses kvm in legacy mode and without
> > > using the Intel i915 kvmgt kernel module, then it would be
> > > necessary to reserve slot 2 for the igd on kvm.
> >
> > It's necessary to configure the assigned IGD at slot 2 to make it
> > functional, yes, but I don't really understand this notion of
> > "reserving" slot 2.  If something occupies address 00:02.0 in the
> > config, it's the user's or management tool's responsibility to move it
> > to make this configuration functional.  Why does QEMU need to play a
> > part in reserving this bus address.  IGD devices are not generally
> > hot-pluggable either, so it doesn't seem we need to reserve an address
> > in case an IGD device is added dynamically later.
>
> As I said in earlier message in this thread, the xenlight toolstack (libxl) 
> that is
> provided as the default toolstack for building xen guests with pci passthrough
> is not the most flexible management tool, and that is why, in the case of xen,
> it is simpler to reserve slot 2 while qemu assigns the slot addresses of the
> qemu emulated pci devices so that the igd can use slot 2. IIRC, In 
> hw/pci/pci.c,
> once the slot value is assigned, it is constant and cannot be changed later on
> by a management tool.
>
> >  
> > > Your question makes me curious, and I have not been able
> > > to determine if anyone has tried igd passthrough using
> > > legacy mode on kvm with recent versions of linux and qemu.
> >
> > Yes, it works.
> >
> > > I will try reproducing the problem on kvm in legacy mode with
> > > current versions of linux and qemu and report my findings.
> > > With kvm, there might be enough flexibility to specify the
> > > slot number for every pci device in the guest. Such a
> >
> > I think this is always the recommendation, libvirt will do this by
> > default in order to make sure the configuration is reproducible.  This
> > is what we generally rely on for kvm/vfio IGD assignment to place the
> > GPU at the correct address.
> >
> > > capability is not available using the xenlight toolstack
> > > for managing xen guests, so I have been using this patch
> > > to ensure that the Intel igd is at slot 2 with xen guests
> > > created by the xenlight toolstack.
> >
> > Seems like a deficiency in xenlight.  I'm not sure why QEMU should take
> > on this burden to support support tool stacks that lack such basic
> > features.
>
> So you would prefer to patch xenlight (libxl) to make it flexible enough to 
> proper