Re: [Qemu-devel] [PATCH] vfio/pci-quirks.c: Make stolen memory size adjustable for igd VFIO

[Alex Williamson]

On Tue, 17 Apr 2018 02:37:21 +
"Zhang, Xiong Y"  wrote:

> > On Wed, 11 Apr 2018 13:24:10 +0800
> > Xiong Zhang  wrote:
> >   
> > > Currenly linux guest with kernel above 3.19 couldn't boot up on igd
> > > passthrough env. The root case is i915 driver use stolen memory, but
> > > qemu vfio doesn't support it.
> > >
> > > This patch set stolen memory size to zero default, so guest i915 won't
> > > use it. But this breaks old windows igd driver which will be unloaded
> > > once it see zero stolen memory size. Then this patch also use
> > > x-igd-gms option to adjust stolen memory size. New windows igd driver
> > > fixes this and could work with zero stolen memory size.
> > >
> > > Finally with this patch, Linux guest and windows guest with igd driver
> > > above 15.45.4860 could work successfully, windows guest with igd
> > > driver below 15.45.4860 should add x-igd-gms=* option.  
> > 
> > Sorry, I can't understand the above commit log in comparison to what this
> > patch is actually doing.
> >   
> > > Signed-off-by: Xiong Zhang 
> > > ---
> > >  hw/vfio/pci-quirks.c | 95
> > > 
> > >  1 file changed, 52 insertions(+), 43 deletions(-)
> > >
> > > diff --git a/hw/vfio/pci-quirks.c b/hw/vfio/pci-quirks.c index
> > > 60ad5fb..6e9ed7f 100644
> > > --- a/hw/vfio/pci-quirks.c
> > > +++ b/hw/vfio/pci-quirks.c
> > > @@ -1372,14 +1372,60 @@ static void  
> > vfio_probe_igd_bar4_quirk(VFIOPCIDevice *vdev, int nr)  
> > >  uint16_t cmd_orig, cmd;
> > >  Error *err = NULL;
> > >
> > > +/* This must be an Intel VGA device. */
> > > +if (!vfio_pci_is(vdev, PCI_VENDOR_ID_INTEL, PCI_ANY_ID) ||
> > > +!vfio_is_vga(vdev) || nr != 4) {
> > > +return;
> > > +}
> > > +
> > >  /*
> > > - * This must be an Intel VGA device at address 00:02.0 for us to even
> > > - * consider enabling legacy mode.  The vBIOS has dependencies on  
> > the  
> > > - * PCI bus address.
> > > + * IGD is not a standard, they like to change their specs often.  We
> > > + * only attempt to support back to SandBridge and we hope that  
> > newer  
> > > + * devices maintain compatibility with generation 8.
> > >   */
> > > -if (!vfio_pci_is(vdev, PCI_VENDOR_ID_INTEL, PCI_ANY_ID) ||
> > > -!vfio_is_vga(vdev) || nr != 4 ||
> > > ->pdev !=  
> > pci_find_device(pci_device_root_bus(>pdev),  
> > > +gen = igd_gen(vdev);
> > > +if (gen != 6 && gen != 8) {
> > > +error_report("IGD device %s is unsupported by IGD quirks, "
> > > + "try SandyBridge or newer",  
> > vdev->vbasedev.name);  
> > > +return;
> > > +}
> > > +
> > > +/*
> > > + * Regardless of running in UPT or legacy mode, the guest graphics
> > > + * driver may attempt to use stolen memory, however only legacy  
> > mode  
> > > + * has BIOS support for reserving stolen memory in the guest VM.
> > > + * Emulate the GMCH register in all cases and zero out the stolen
> > > + * memory size here.
> > > + */
> > > +gmch = vfio_pci_read_config(>pdev, IGD_GMCH, 4);
> > > +gmch &= ~((gen < 8 ? 0x1f : 0xff) << (gen < 8 ? 3 : 8));  
> > 
> > Aha!  You want to enable stolen memory in either UPT or legacy mode.  I
> > could not possibly have determined that from the commit log.  Please
> > rewrite the commit log to describe what is actually changing here.  
> [Zhang, Xiong Y] Here, I want to disable stolen memory in UPT and legacy mode.
> Sure, I will rewrite the commit log.
> >   
> > > +
> > > +/*
> > > + * Assume we have no GMS memory, but allow it to be overrided by  
> > device  
> > > + * option (experimental). Old windows igd driver must see non-zero  
> > stolen  
> > > + * memory size, otherwise it will be unloaded. New igd windows  
> > driver fix  
> > > + * this issue and could load with zero stolen memory size.
> > > + */
> > > +if (vdev->igd_gms) {
> > > +if (vdev->igd_gms <= 0x10) {
> > > +gms_mb = vdev->igd_gms * 32;
> > > +gmch |= vdev->igd_gms << (gen < 8 ? 3 : 8);
> > > +} else {
> > > +error_report("Unsupported IGD GMS value 0x%x",  
> > vdev->igd_gms);  
> > > +vdev->igd_gms = 0;
> > > +}
> > > +}  
> > 
> > You state in the previous code block that only legacy mode has BIOS support
> > for reserving stolen memory, so why should UPT mode be allowed to specify
> > stolen memory?  Where is it allocated?  
> [Zhang, Xiong Y] As the previous code disable stolen memory and set the size 
> of stolen memory to zero,

Only in Legacy Mode, GMCH is not virtualized in UPT Mode.

> this cause old windows igd driver couldn't be loaded as the old
> windows igd driver will be unloaded once it see zero size stolen
> memory but the old windows igd driver doesn't use stolen memory, so
> for old windows igd 

Re: [Qemu-devel] [PATCH] vfio/pci-quirks.c: Make stolen memory size adjustable for igd VFIO

[Zhang, Xiong Y]

> On Wed, 11 Apr 2018 13:24:10 +0800
> Xiong Zhang  wrote:
> 
> > Currenly linux guest with kernel above 3.19 couldn't boot up on igd
> > passthrough env. The root case is i915 driver use stolen memory, but
> > qemu vfio doesn't support it.
> >
> > This patch set stolen memory size to zero default, so guest i915 won't
> > use it. But this breaks old windows igd driver which will be unloaded
> > once it see zero stolen memory size. Then this patch also use
> > x-igd-gms option to adjust stolen memory size. New windows igd driver
> > fixes this and could work with zero stolen memory size.
> >
> > Finally with this patch, Linux guest and windows guest with igd driver
> > above 15.45.4860 could work successfully, windows guest with igd
> > driver below 15.45.4860 should add x-igd-gms=* option.
> 
> Sorry, I can't understand the above commit log in comparison to what this
> patch is actually doing.
> 
> > Signed-off-by: Xiong Zhang 
> > ---
> >  hw/vfio/pci-quirks.c | 95
> > 
> >  1 file changed, 52 insertions(+), 43 deletions(-)
> >
> > diff --git a/hw/vfio/pci-quirks.c b/hw/vfio/pci-quirks.c index
> > 60ad5fb..6e9ed7f 100644
> > --- a/hw/vfio/pci-quirks.c
> > +++ b/hw/vfio/pci-quirks.c
> > @@ -1372,14 +1372,60 @@ static void
> vfio_probe_igd_bar4_quirk(VFIOPCIDevice *vdev, int nr)
> >  uint16_t cmd_orig, cmd;
> >  Error *err = NULL;
> >
> > +/* This must be an Intel VGA device. */
> > +if (!vfio_pci_is(vdev, PCI_VENDOR_ID_INTEL, PCI_ANY_ID) ||
> > +!vfio_is_vga(vdev) || nr != 4) {
> > +return;
> > +}
> > +
> >  /*
> > - * This must be an Intel VGA device at address 00:02.0 for us to even
> > - * consider enabling legacy mode.  The vBIOS has dependencies on
> the
> > - * PCI bus address.
> > + * IGD is not a standard, they like to change their specs often.  We
> > + * only attempt to support back to SandBridge and we hope that
> newer
> > + * devices maintain compatibility with generation 8.
> >   */
> > -if (!vfio_pci_is(vdev, PCI_VENDOR_ID_INTEL, PCI_ANY_ID) ||
> > -!vfio_is_vga(vdev) || nr != 4 ||
> > ->pdev !=
> pci_find_device(pci_device_root_bus(>pdev),
> > +gen = igd_gen(vdev);
> > +if (gen != 6 && gen != 8) {
> > +error_report("IGD device %s is unsupported by IGD quirks, "
> > + "try SandyBridge or newer",
> vdev->vbasedev.name);
> > +return;
> > +}
> > +
> > +/*
> > + * Regardless of running in UPT or legacy mode, the guest graphics
> > + * driver may attempt to use stolen memory, however only legacy
> mode
> > + * has BIOS support for reserving stolen memory in the guest VM.
> > + * Emulate the GMCH register in all cases and zero out the stolen
> > + * memory size here.
> > + */
> > +gmch = vfio_pci_read_config(>pdev, IGD_GMCH, 4);
> > +gmch &= ~((gen < 8 ? 0x1f : 0xff) << (gen < 8 ? 3 : 8));
> 
> Aha!  You want to enable stolen memory in either UPT or legacy mode.  I
> could not possibly have determined that from the commit log.  Please
> rewrite the commit log to describe what is actually changing here.
[Zhang, Xiong Y] Here, I want to disable stolen memory in UPT and legacy mode.
Sure, I will rewrite the commit log.
> 
> > +
> > +/*
> > + * Assume we have no GMS memory, but allow it to be overrided by
> device
> > + * option (experimental). Old windows igd driver must see non-zero
> stolen
> > + * memory size, otherwise it will be unloaded. New igd windows
> driver fix
> > + * this issue and could load with zero stolen memory size.
> > + */
> > +if (vdev->igd_gms) {
> > +if (vdev->igd_gms <= 0x10) {
> > +gms_mb = vdev->igd_gms * 32;
> > +gmch |= vdev->igd_gms << (gen < 8 ? 3 : 8);
> > +} else {
> > +error_report("Unsupported IGD GMS value 0x%x",
> vdev->igd_gms);
> > +vdev->igd_gms = 0;
> > +}
> > +}
> 
> You state in the previous code block that only legacy mode has BIOS support
> for reserving stolen memory, so why should UPT mode be allowed to specify
> stolen memory?  Where is it allocated?
[Zhang, Xiong Y] As the previous code disable stolen memory and set the size of 
stolen memory to zero, this cause old windows igd driver couldn't be loaded as 
the old windows igd driver will be unloaded once it see zero size stolen memory 
but the old windows igd driver doesn't use stolen memory, so for old windows 
igd driver, we just need to set a none zero size stolen memory and don't need 
to allocate an emulated stolen memory . This is old windows igd driver issue 
and has been fixed in new windows igd driver.
So the above code is just for old windows igd driver. if user use an old 
windows igd driver, he could use x-igd-gms option to workaround old windows igd 
driver issue. Without this code, old windows igd driver couldn't 

Re: [Qemu-devel] [PATCH] vfio/pci-quirks.c: Make stolen memory size adjustable for igd VFIO

[Alex Williamson]

On Wed, 11 Apr 2018 13:24:10 +0800
Xiong Zhang  wrote:

> Currenly linux guest with kernel above 3.19 couldn't boot up on igd
> passthrough env. The root case is i915 driver use stolen memory, but
> qemu vfio doesn't support it.
> 
> This patch set stolen memory size to zero default, so guest i915 won't
> use it. But this breaks old windows igd driver which will be unloaded
> once it see zero stolen memory size. Then this patch also use x-igd-gms
> option to adjust stolen memory size. New windows igd driver fixes this and
> could work with zero stolen memory size.
> 
> Finally with this patch, Linux guest and windows guest with igd driver
> above 15.45.4860 could work successfully, windows guest with igd driver
> below 15.45.4860 should add x-igd-gms=* option.

Sorry, I can't understand the above commit log in comparison to what
this patch is actually doing.

> Signed-off-by: Xiong Zhang 
> ---
>  hw/vfio/pci-quirks.c | 95 
> 
>  1 file changed, 52 insertions(+), 43 deletions(-)
> 
> diff --git a/hw/vfio/pci-quirks.c b/hw/vfio/pci-quirks.c
> index 60ad5fb..6e9ed7f 100644
> --- a/hw/vfio/pci-quirks.c
> +++ b/hw/vfio/pci-quirks.c
> @@ -1372,14 +1372,60 @@ static void vfio_probe_igd_bar4_quirk(VFIOPCIDevice 
> *vdev, int nr)
>  uint16_t cmd_orig, cmd;
>  Error *err = NULL;
>  
> +/* This must be an Intel VGA device. */
> +if (!vfio_pci_is(vdev, PCI_VENDOR_ID_INTEL, PCI_ANY_ID) ||
> +!vfio_is_vga(vdev) || nr != 4) {
> +return;
> +}
> +
>  /*
> - * This must be an Intel VGA device at address 00:02.0 for us to even
> - * consider enabling legacy mode.  The vBIOS has dependencies on the
> - * PCI bus address.
> + * IGD is not a standard, they like to change their specs often.  We
> + * only attempt to support back to SandBridge and we hope that newer
> + * devices maintain compatibility with generation 8.
>   */
> -if (!vfio_pci_is(vdev, PCI_VENDOR_ID_INTEL, PCI_ANY_ID) ||
> -!vfio_is_vga(vdev) || nr != 4 ||
> ->pdev != pci_find_device(pci_device_root_bus(>pdev),
> +gen = igd_gen(vdev);
> +if (gen != 6 && gen != 8) {
> +error_report("IGD device %s is unsupported by IGD quirks, "
> + "try SandyBridge or newer", vdev->vbasedev.name);
> +return;
> +}
> +
> +/*
> + * Regardless of running in UPT or legacy mode, the guest graphics
> + * driver may attempt to use stolen memory, however only legacy mode
> + * has BIOS support for reserving stolen memory in the guest VM.
> + * Emulate the GMCH register in all cases and zero out the stolen
> + * memory size here.
> + */
> +gmch = vfio_pci_read_config(>pdev, IGD_GMCH, 4);
> +gmch &= ~((gen < 8 ? 0x1f : 0xff) << (gen < 8 ? 3 : 8));

Aha!  You want to enable stolen memory in either UPT or legacy mode.  I
could not possibly have determined that from the commit log.  Please
rewrite the commit log to describe what is actually changing here.

> +
> +/*
> + * Assume we have no GMS memory, but allow it to be overrided by device
> + * option (experimental). Old windows igd driver must see non-zero stolen
> + * memory size, otherwise it will be unloaded. New igd windows driver fix
> + * this issue and could load with zero stolen memory size.
> + */
> +if (vdev->igd_gms) {
> +if (vdev->igd_gms <= 0x10) {
> +gms_mb = vdev->igd_gms * 32;
> +gmch |= vdev->igd_gms << (gen < 8 ? 3 : 8);
> +} else {
> +error_report("Unsupported IGD GMS value 0x%x", vdev->igd_gms);
> +vdev->igd_gms = 0;
> +}
> +}

You state in the previous code block that only legacy mode has BIOS
support for reserving stolen memory, so why should UPT mode be allowed
to specify stolen memory?  Where is it allocated?

> +
> +/* GMCH is read-only, emulated */
> +pci_set_long(vdev->pdev.config + IGD_GMCH, gmch);
> +pci_set_long(vdev->pdev.wmask + IGD_GMCH, 0);
> +pci_set_long(vdev->emulated_config_bits + IGD_GMCH, ~0);
> +
> +/*
> + * This must be at address 00:02.0 for us to even onsider enabling
> + * legacy mode.  The vBIOS has dependencies on the PCI bus address.
> + */
> +if (>pdev != pci_find_device(pci_device_root_bus(>pdev),
> 0, PCI_DEVFN(0x2, 0))) {

So we allow the user to specify stolen memory size in UPT mode, we
emulate that in the GMCH register, but we do nothing with the BDSM
register which tells the guest driver the address of the stolen
memory.  What's the point?  I'm lost.  Thanks,

Alex

>  return;
>  }
> @@ -1399,18 +1445,6 @@ static void vfio_probe_igd_bar4_quirk(VFIOPCIDevice 
> *vdev, int nr)
>  }
>  
>  /*
> - * IGD is not a standard, they like to change their specs often.  We
> - * only attempt to support back to SandBridge and 

[Qemu-devel] [PATCH] vfio/pci-quirks.c: Make stolen memory size adjustable for igd VFIO

[Xiong Zhang]

Currenly linux guest with kernel above 3.19 couldn't boot up on igd
passthrough env. The root case is i915 driver use stolen memory, but
qemu vfio doesn't support it.

This patch set stolen memory size to zero default, so guest i915 won't
use it. But this breaks old windows igd driver which will be unloaded
once it see zero stolen memory size. Then this patch also use x-igd-gms
option to adjust stolen memory size. New windows igd driver fixes this and
could work with zero stolen memory size.

Finally with this patch, Linux guest and windows guest with igd driver
above 15.45.4860 could work successfully, windows guest with igd driver
below 15.45.4860 should add x-igd-gms=* option.

Signed-off-by: Xiong Zhang 
---
 hw/vfio/pci-quirks.c | 95 
 1 file changed, 52 insertions(+), 43 deletions(-)

diff --git a/hw/vfio/pci-quirks.c b/hw/vfio/pci-quirks.c
index 60ad5fb..6e9ed7f 100644
--- a/hw/vfio/pci-quirks.c
+++ b/hw/vfio/pci-quirks.c
@@ -1372,14 +1372,60 @@ static void vfio_probe_igd_bar4_quirk(VFIOPCIDevice 
*vdev, int nr)
 uint16_t cmd_orig, cmd;
 Error *err = NULL;
 
+/* This must be an Intel VGA device. */
+if (!vfio_pci_is(vdev, PCI_VENDOR_ID_INTEL, PCI_ANY_ID) ||
+!vfio_is_vga(vdev) || nr != 4) {
+return;
+}
+
 /*
- * This must be an Intel VGA device at address 00:02.0 for us to even
- * consider enabling legacy mode.  The vBIOS has dependencies on the
- * PCI bus address.
+ * IGD is not a standard, they like to change their specs often.  We
+ * only attempt to support back to SandBridge and we hope that newer
+ * devices maintain compatibility with generation 8.
  */
-if (!vfio_pci_is(vdev, PCI_VENDOR_ID_INTEL, PCI_ANY_ID) ||
-!vfio_is_vga(vdev) || nr != 4 ||
->pdev != pci_find_device(pci_device_root_bus(>pdev),
+gen = igd_gen(vdev);
+if (gen != 6 && gen != 8) {
+error_report("IGD device %s is unsupported by IGD quirks, "
+ "try SandyBridge or newer", vdev->vbasedev.name);
+return;
+}
+
+/*
+ * Regardless of running in UPT or legacy mode, the guest graphics
+ * driver may attempt to use stolen memory, however only legacy mode
+ * has BIOS support for reserving stolen memory in the guest VM.
+ * Emulate the GMCH register in all cases and zero out the stolen
+ * memory size here.
+ */
+gmch = vfio_pci_read_config(>pdev, IGD_GMCH, 4);
+gmch &= ~((gen < 8 ? 0x1f : 0xff) << (gen < 8 ? 3 : 8));
+
+/*
+ * Assume we have no GMS memory, but allow it to be overrided by device
+ * option (experimental). Old windows igd driver must see non-zero stolen
+ * memory size, otherwise it will be unloaded. New igd windows driver fix
+ * this issue and could load with zero stolen memory size.
+ */
+if (vdev->igd_gms) {
+if (vdev->igd_gms <= 0x10) {
+gms_mb = vdev->igd_gms * 32;
+gmch |= vdev->igd_gms << (gen < 8 ? 3 : 8);
+} else {
+error_report("Unsupported IGD GMS value 0x%x", vdev->igd_gms);
+vdev->igd_gms = 0;
+}
+}
+
+/* GMCH is read-only, emulated */
+pci_set_long(vdev->pdev.config + IGD_GMCH, gmch);
+pci_set_long(vdev->pdev.wmask + IGD_GMCH, 0);
+pci_set_long(vdev->emulated_config_bits + IGD_GMCH, ~0);
+
+/*
+ * This must be at address 00:02.0 for us to even onsider enabling
+ * legacy mode.  The vBIOS has dependencies on the PCI bus address.
+ */
+if (>pdev != pci_find_device(pci_device_root_bus(>pdev),
0, PCI_DEVFN(0x2, 0))) {
 return;
 }
@@ -1399,18 +1445,6 @@ static void vfio_probe_igd_bar4_quirk(VFIOPCIDevice 
*vdev, int nr)
 }
 
 /*
- * IGD is not a standard, they like to change their specs often.  We
- * only attempt to support back to SandBridge and we hope that newer
- * devices maintain compatibility with generation 8.
- */
-gen = igd_gen(vdev);
-if (gen != 6 && gen != 8) {
-error_report("IGD device %s is unsupported in legacy mode, "
- "try SandyBridge or newer", vdev->vbasedev.name);
-return;
-}
-
-/*
  * Most of what we're doing here is to enable the ROM to run, so if
  * there's no ROM, there's no point in setting up this quirk.
  * NB. We only seem to get BIOS ROMs, so a UEFI VM would need CSM support.
@@ -1465,8 +1499,6 @@ static void vfio_probe_igd_bar4_quirk(VFIOPCIDevice 
*vdev, int nr)
 goto out;
 }
 
-gmch = vfio_pci_read_config(>pdev, IGD_GMCH, 4);
-
 /*
  * If IGD VGA Disable is clear (expected) and VGA is not already enabled,
  * try to enable it.  Probably shouldn't be using legacy mode without VGA,
@@ -1532,24 +1564,6 @@ static void vfio_probe_igd_bar4_quirk(VFIOPCIDevice 
*vdev, int nr)
 }
 
 /*
- * Assume we have no GMS memory, but