Re: [SeaBIOS] [Qemu-devel] [RFC PATCH v3 3/3] fw/pci: Allocate IGD stolen memory

[Gerd Hoffmann]

  Hi,

> I don't mind the allocation of guest memory issue, the fact that guest
> memory is consumed by built-in devices is exactly what happens on bare
> metal.

And following in qemu what happens on bare metal usually works best
long-term.

> to pre-populate it, we could.  It at least puts QEMU in control, but
> relaying back the address via a device specific register is still a bit
> ugly.

It is the usual way.  We do the same for a few other addresses too
(mmconfig xbar, acpi registers).  Firmware programs the hardware, and
qemu picks up the addresses from there (and puts them into the generated
acpi tables for example).

Main advantage is that we don't need a paravirtual firmware <=> qemu
interface for each address.

Yes, each firmware (seabios/ovmf/...) needs to handle it then.  But if
we allocate the stolen memory from guest ram the firmware has to handle
it _anyway_ so it is marked reserved in e820 etc.

cheers,
  Gerd


___
SeaBIOS mailing list
SeaBIOS@seabios.org
http://www.seabios.org/mailman/listinfo/seabios

Re: [SeaBIOS] [Qemu-devel] [RFC PATCH v3 3/3] fw/pci: Allocate IGD stolen memory

[Alex Williamson]

On Mon, 15 Feb 2016 21:29:26 +0200
"Michael S. Tsirkin"  wrote:

> On Mon, Feb 15, 2016 at 12:20:23PM -0700, Alex Williamson wrote:
> > On Mon, 15 Feb 2016 10:54:51 +0100
> > Igor Mammedov  wrote:
> >   
> > > On Sat, 13 Feb 2016 18:03:31 -0700
> > > Alex Williamson  wrote:
> > >   
> > > > On Sat, 13 Feb 2016 19:20:32 -0500
> > > > "Kevin O'Connor"  wrote:
> > > > 
> > > > > On Sat, Feb 13, 2016 at 01:57:09PM -0700, Alex Williamson wrote:  
> > > > > > On Sat, 13 Feb 2016 15:05:09 -0500
> > > > > > "Kevin O'Connor"  wrote:
> > > > > > > On Sat, Feb 13, 2016 at 11:51:51AM -0700, Alex Williamson wrote:  
> > > > > > >   
> > > > > > > > On Sat, 13 Feb 2016 13:18:39 -0500
> > > > > > > > "Kevin O'Connor"  wrote:  
> > > > > > > > > On Sat, Feb 13, 2016 at 08:12:09AM -0700, Alex Williamson 
> > > > > > > > > wrote:  
> > > > > > > > > > On Fri, 12 Feb 2016 21:49:04 -0500
> > > > > > > > > > "Kevin O'Connor"  wrote:
> > > > > > > > > > > On Fri, Feb 12, 2016 at 05:23:18PM -0700, Alex Williamson 
> > > > > > > > > > > wrote:
> > > > > > > > > > > > Intel IGD makes use of memory allocated and marked 
> > > > > > > > > > > > reserved by the
> > > > > > > > > > > > BIOS as a stolen memory range.  For the most part, 
> > > > > > > > > > > > guest drivers don't
> > > > > > > > > > > > make use of this, but our achilles heel is the vBIOS.  
> > > > > > > > > > > > The vBIOS
> > > > > > > > > > > > programs the device to use the host stolen memory range 
> > > > > > > > > > > > and it's used
> > > > > > > > > > > > in the pre-boot environment.  Generally the guest won't 
> > > > > > > > > > > > have access to
> > > > > > > > > > > > the host stolen memory area, so these accesses either 
> > > > > > > > > > > > land in VM
> > > > > > > > > > > > memory or unassigned space and generate IOMMU faults.  
> > > > > > > > > > > > By allocating
> > > > > > > > > > > > this range in SeaBIOS and programming it into the 
> > > > > > > > > > > > device, QEMU (via
> > > > > > > > > > > > vfio) can make sure this guest allocated stolen memory 
> > > > > > > > > > > > range is used
> > > > > > > > > > > > instead.  
> > > > > > > > > > > 
> > > > > > > > > > > What does "vBIOS" mean in this context?  Is it the video 
> > > > > > > > > > > device option
> > > > > > > > > > > rom or something else?
> > > > > > > > > > 
> > > > > > > > > > vBIOS = video BIOS, you're correct, it's just the video 
> > > > > > > > > > device option
> > > > > > > > > > ROM.
> > > > > > > > > 
> > > > > > > > > Is the problem from when the host runs the video option rom, 
> > > > > > > > > or is the
> > > > > > > > > problem from the guest (via SeaBIOS) running the video option 
> > > > > > > > > rom?  If
> > > > > > > > > the guest is running the option rom, is it the first time the 
> > > > > > > > > option
> > > > > > > > > rom has been run for the machine (ie, was the option rom not 
> > > > > > > > > executed
> > > > > > > > > on the host when the host machine first booted)?
> > > > > > > > > 
> > > > > > > > > FWIW, many of the chromebooks use coreboot with Intel 
> > > > > > > > > graphics and a
> > > > > > > > > number of users have installed SeaBIOS (running natively) on 
> > > > > > > > > their
> > > > > > > > > machines.  Running the intel video option rom more than once 
> > > > > > > > > has been
> > > > > > > > > known to cause issues.  
> > > > > > > > 
> > > > > > > > The issue is in the VM and it occurs every time the option ROM 
> > > > > > > > is
> > > > > > > > executed.  Standard VGA text mode displays fine (ex. SeaBIOS 
> > > > > > > > version
> > > > > > > > string and boot menu), but any sort of extended graphics mode 
> > > > > > > > (ex. live
> > > > > > > > CD boot menu) tries to make use of the host memory area which
> > > > > > > > corresponds to the stolen memory area of the physical device.  
> > > > > > > > We're
> > > > > > > > not really sure how the ROM execution arrives at these 
> > > > > > > > addresses (not
> > > > > > > > from the device according to access traces), but we can see 
> > > > > > > > when the
> > > > > > > > ROM is writing these addresses to the device and modify they 
> > > > > > > > addresses
> > > > > > > > to point at a VM memory range which we've allocated.  That's 
> > > > > > > > what this
> > > > > > > > code attempts to do, allocate a buffer and tell QEMU about it 
> > > > > > > > via the
> > > > > > > > BDSM (Base Data of Stolen Memory) register.  
> > > > > > > 
> > > > > > > Forgive me if I'm not fully understanding this.  If I read what 
> > > > > > > you're
> > > > > > > saying then the sequence is something like:
> > > > > > > 
> > > > > > > 1 - the host system bios (or vgabios) programs the GTT/stolen 
> > > > > > > memory
> > > > > > 

Re: [SeaBIOS] [Qemu-devel] [RFC PATCH v3 3/3] fw/pci: Allocate IGD stolen memory

[Kevin O'Connor]

On Mon, Feb 15, 2016 at 09:29:26PM +0200, Michael S. Tsirkin wrote:
> I can build a generic interface to pass addresses
> allocated by bios back to QEMU. It looks like this would
> be useful for other purposes as well.  Interested?

If this is undertaken, I suggest extending fw_cfg to support "writable
files" instead of introducing a whole new interface.

-Kevin

___
SeaBIOS mailing list
SeaBIOS@seabios.org
http://www.seabios.org/mailman/listinfo/seabios

Re: [SeaBIOS] [Qemu-devel] [RFC PATCH v3 3/3] fw/pci: Allocate IGD stolen memory

[Michael S. Tsirkin]

On Mon, Feb 15, 2016 at 12:20:23PM -0700, Alex Williamson wrote:
> On Mon, 15 Feb 2016 10:54:51 +0100
> Igor Mammedov  wrote:
> 
> > On Sat, 13 Feb 2016 18:03:31 -0700
> > Alex Williamson  wrote:
> > 
> > > On Sat, 13 Feb 2016 19:20:32 -0500
> > > "Kevin O'Connor"  wrote:
> > >   
> > > > On Sat, Feb 13, 2016 at 01:57:09PM -0700, Alex Williamson wrote:
> > > > > On Sat, 13 Feb 2016 15:05:09 -0500
> > > > > "Kevin O'Connor"  wrote:  
> > > > > > On Sat, Feb 13, 2016 at 11:51:51AM -0700, Alex Williamson wrote:
> > > > > >   
> > > > > > > On Sat, 13 Feb 2016 13:18:39 -0500
> > > > > > > "Kevin O'Connor"  wrote:
> > > > > > > > On Sat, Feb 13, 2016 at 08:12:09AM -0700, Alex Williamson 
> > > > > > > > wrote:
> > > > > > > > > On Fri, 12 Feb 2016 21:49:04 -0500
> > > > > > > > > "Kevin O'Connor"  wrote:  
> > > > > > > > > > On Fri, Feb 12, 2016 at 05:23:18PM -0700, Alex Williamson 
> > > > > > > > > > wrote:  
> > > > > > > > > > > Intel IGD makes use of memory allocated and marked 
> > > > > > > > > > > reserved by the
> > > > > > > > > > > BIOS as a stolen memory range.  For the most part, guest 
> > > > > > > > > > > drivers don't
> > > > > > > > > > > make use of this, but our achilles heel is the vBIOS.  
> > > > > > > > > > > The vBIOS
> > > > > > > > > > > programs the device to use the host stolen memory range 
> > > > > > > > > > > and it's used
> > > > > > > > > > > in the pre-boot environment.  Generally the guest won't 
> > > > > > > > > > > have access to
> > > > > > > > > > > the host stolen memory area, so these accesses either 
> > > > > > > > > > > land in VM
> > > > > > > > > > > memory or unassigned space and generate IOMMU faults.  By 
> > > > > > > > > > > allocating
> > > > > > > > > > > this range in SeaBIOS and programming it into the device, 
> > > > > > > > > > > QEMU (via
> > > > > > > > > > > vfio) can make sure this guest allocated stolen memory 
> > > > > > > > > > > range is used
> > > > > > > > > > > instead.
> > > > > > > > > > 
> > > > > > > > > > What does "vBIOS" mean in this context?  Is it the video 
> > > > > > > > > > device option
> > > > > > > > > > rom or something else?  
> > > > > > > > > 
> > > > > > > > > vBIOS = video BIOS, you're correct, it's just the video 
> > > > > > > > > device option
> > > > > > > > > ROM.  
> > > > > > > > 
> > > > > > > > Is the problem from when the host runs the video option rom, or 
> > > > > > > > is the
> > > > > > > > problem from the guest (via SeaBIOS) running the video option 
> > > > > > > > rom?  If
> > > > > > > > the guest is running the option rom, is it the first time the 
> > > > > > > > option
> > > > > > > > rom has been run for the machine (ie, was the option rom not 
> > > > > > > > executed
> > > > > > > > on the host when the host machine first booted)?
> > > > > > > > 
> > > > > > > > FWIW, many of the chromebooks use coreboot with Intel graphics 
> > > > > > > > and a
> > > > > > > > number of users have installed SeaBIOS (running natively) on 
> > > > > > > > their
> > > > > > > > machines.  Running the intel video option rom more than once 
> > > > > > > > has been
> > > > > > > > known to cause issues.
> > > > > > > 
> > > > > > > The issue is in the VM and it occurs every time the option ROM is
> > > > > > > executed.  Standard VGA text mode displays fine (ex. SeaBIOS 
> > > > > > > version
> > > > > > > string and boot menu), but any sort of extended graphics mode 
> > > > > > > (ex. live
> > > > > > > CD boot menu) tries to make use of the host memory area which
> > > > > > > corresponds to the stolen memory area of the physical device.  
> > > > > > > We're
> > > > > > > not really sure how the ROM execution arrives at these addresses 
> > > > > > > (not
> > > > > > > from the device according to access traces), but we can see when 
> > > > > > > the
> > > > > > > ROM is writing these addresses to the device and modify they 
> > > > > > > addresses
> > > > > > > to point at a VM memory range which we've allocated.  That's what 
> > > > > > > this
> > > > > > > code attempts to do, allocate a buffer and tell QEMU about it via 
> > > > > > > the
> > > > > > > BDSM (Base Data of Stolen Memory) register.
> > > > > > 
> > > > > > Forgive me if I'm not fully understanding this.  If I read what 
> > > > > > you're
> > > > > > saying then the sequence is something like:
> > > > > > 
> > > > > > 1 - the host system bios (or vgabios) programs the GTT/stolen memory
> > > > > > base register at host system bootup time and reserves it in the
> > > > > > host e820 map.
> > > > > > 
> > > > > > 2 - upon running qemu, the guest reruns the vga bios option rom 
> > > > > > which
> > > > > > seems to work (ie, text mode works)
> > > > > > 
> > > > > > 3 - in the guest, upon running a bootloader 

Re: [SeaBIOS] [Qemu-devel] [RFC PATCH v3 3/3] fw/pci: Allocate IGD stolen memory

[Alex Williamson]

On Mon, 15 Feb 2016 10:54:51 +0100
Igor Mammedov  wrote:

> On Sat, 13 Feb 2016 18:03:31 -0700
> Alex Williamson  wrote:
> 
> > On Sat, 13 Feb 2016 19:20:32 -0500
> > "Kevin O'Connor"  wrote:
> >   
> > > On Sat, Feb 13, 2016 at 01:57:09PM -0700, Alex Williamson wrote:
> > > > On Sat, 13 Feb 2016 15:05:09 -0500
> > > > "Kevin O'Connor"  wrote:  
> > > > > On Sat, Feb 13, 2016 at 11:51:51AM -0700, Alex Williamson wrote:  
> > > > > > On Sat, 13 Feb 2016 13:18:39 -0500
> > > > > > "Kevin O'Connor"  wrote:
> > > > > > > On Sat, Feb 13, 2016 at 08:12:09AM -0700, Alex Williamson wrote:  
> > > > > > >   
> > > > > > > > On Fri, 12 Feb 2016 21:49:04 -0500
> > > > > > > > "Kevin O'Connor"  wrote:  
> > > > > > > > > On Fri, Feb 12, 2016 at 05:23:18PM -0700, Alex Williamson 
> > > > > > > > > wrote:  
> > > > > > > > > > Intel IGD makes use of memory allocated and marked reserved 
> > > > > > > > > > by the
> > > > > > > > > > BIOS as a stolen memory range.  For the most part, guest 
> > > > > > > > > > drivers don't
> > > > > > > > > > make use of this, but our achilles heel is the vBIOS.  The 
> > > > > > > > > > vBIOS
> > > > > > > > > > programs the device to use the host stolen memory range and 
> > > > > > > > > > it's used
> > > > > > > > > > in the pre-boot environment.  Generally the guest won't 
> > > > > > > > > > have access to
> > > > > > > > > > the host stolen memory area, so these accesses either land 
> > > > > > > > > > in VM
> > > > > > > > > > memory or unassigned space and generate IOMMU faults.  By 
> > > > > > > > > > allocating
> > > > > > > > > > this range in SeaBIOS and programming it into the device, 
> > > > > > > > > > QEMU (via
> > > > > > > > > > vfio) can make sure this guest allocated stolen memory 
> > > > > > > > > > range is used
> > > > > > > > > > instead.
> > > > > > > > > 
> > > > > > > > > What does "vBIOS" mean in this context?  Is it the video 
> > > > > > > > > device option
> > > > > > > > > rom or something else?  
> > > > > > > > 
> > > > > > > > vBIOS = video BIOS, you're correct, it's just the video device 
> > > > > > > > option
> > > > > > > > ROM.  
> > > > > > > 
> > > > > > > Is the problem from when the host runs the video option rom, or 
> > > > > > > is the
> > > > > > > problem from the guest (via SeaBIOS) running the video option 
> > > > > > > rom?  If
> > > > > > > the guest is running the option rom, is it the first time the 
> > > > > > > option
> > > > > > > rom has been run for the machine (ie, was the option rom not 
> > > > > > > executed
> > > > > > > on the host when the host machine first booted)?
> > > > > > > 
> > > > > > > FWIW, many of the chromebooks use coreboot with Intel graphics 
> > > > > > > and a
> > > > > > > number of users have installed SeaBIOS (running natively) on their
> > > > > > > machines.  Running the intel video option rom more than once has 
> > > > > > > been
> > > > > > > known to cause issues.
> > > > > > 
> > > > > > The issue is in the VM and it occurs every time the option ROM is
> > > > > > executed.  Standard VGA text mode displays fine (ex. SeaBIOS version
> > > > > > string and boot menu), but any sort of extended graphics mode (ex. 
> > > > > > live
> > > > > > CD boot menu) tries to make use of the host memory area which
> > > > > > corresponds to the stolen memory area of the physical device.  We're
> > > > > > not really sure how the ROM execution arrives at these addresses 
> > > > > > (not
> > > > > > from the device according to access traces), but we can see when the
> > > > > > ROM is writing these addresses to the device and modify they 
> > > > > > addresses
> > > > > > to point at a VM memory range which we've allocated.  That's what 
> > > > > > this
> > > > > > code attempts to do, allocate a buffer and tell QEMU about it via 
> > > > > > the
> > > > > > BDSM (Base Data of Stolen Memory) register.
> > > > > 
> > > > > Forgive me if I'm not fully understanding this.  If I read what you're
> > > > > saying then the sequence is something like:
> > > > > 
> > > > > 1 - the host system bios (or vgabios) programs the GTT/stolen memory
> > > > > base register at host system bootup time and reserves it in the
> > > > > host e820 map.
> > > > > 
> > > > > 2 - upon running qemu, the guest reruns the vga bios option rom which
> > > > > seems to work (ie, text mode works)
> > > > > 
> > > > > 3 - in the guest, upon running a bootloader that uses graphics mode,
> > > > > the bootloader calls the vgabios to switch to graphics mode, and
> > > > > the vgabios sends commands to the graphics hardware that somehow
> > > > > reference the host stolen memory  
> > > > 
> > > > What exactly happens here isn't clear to me, but this is a plausible
> > > > explanation.  What we see 

Re: [SeaBIOS] [Qemu-devel] [RFC PATCH v3 3/3] fw/pci: Allocate IGD stolen memory

[Igor Mammedov]

On Sat, 13 Feb 2016 18:03:31 -0700
Alex Williamson  wrote:

> On Sat, 13 Feb 2016 19:20:32 -0500
> "Kevin O'Connor"  wrote:
> 
> > On Sat, Feb 13, 2016 at 01:57:09PM -0700, Alex Williamson wrote:  
> > > On Sat, 13 Feb 2016 15:05:09 -0500
> > > "Kevin O'Connor"  wrote:
> > > > On Sat, Feb 13, 2016 at 11:51:51AM -0700, Alex Williamson wrote:
> > > > > On Sat, 13 Feb 2016 13:18:39 -0500
> > > > > "Kevin O'Connor"  wrote:  
> > > > > > On Sat, Feb 13, 2016 at 08:12:09AM -0700, Alex Williamson wrote:
> > > > > >   
> > > > > > > On Fri, 12 Feb 2016 21:49:04 -0500
> > > > > > > "Kevin O'Connor"  wrote:
> > > > > > > > On Fri, Feb 12, 2016 at 05:23:18PM -0700, Alex Williamson 
> > > > > > > > wrote:
> > > > > > > > > Intel IGD makes use of memory allocated and marked reserved 
> > > > > > > > > by the
> > > > > > > > > BIOS as a stolen memory range.  For the most part, guest 
> > > > > > > > > drivers don't
> > > > > > > > > make use of this, but our achilles heel is the vBIOS.  The 
> > > > > > > > > vBIOS
> > > > > > > > > programs the device to use the host stolen memory range and 
> > > > > > > > > it's used
> > > > > > > > > in the pre-boot environment.  Generally the guest won't have 
> > > > > > > > > access to
> > > > > > > > > the host stolen memory area, so these accesses either land in 
> > > > > > > > > VM
> > > > > > > > > memory or unassigned space and generate IOMMU faults.  By 
> > > > > > > > > allocating
> > > > > > > > > this range in SeaBIOS and programming it into the device, 
> > > > > > > > > QEMU (via
> > > > > > > > > vfio) can make sure this guest allocated stolen memory range 
> > > > > > > > > is used
> > > > > > > > > instead.  
> > > > > > > > 
> > > > > > > > What does "vBIOS" mean in this context?  Is it the video device 
> > > > > > > > option
> > > > > > > > rom or something else?
> > > > > > > 
> > > > > > > vBIOS = video BIOS, you're correct, it's just the video device 
> > > > > > > option
> > > > > > > ROM.
> > > > > > 
> > > > > > Is the problem from when the host runs the video option rom, or is 
> > > > > > the
> > > > > > problem from the guest (via SeaBIOS) running the video option rom?  
> > > > > > If
> > > > > > the guest is running the option rom, is it the first time the option
> > > > > > rom has been run for the machine (ie, was the option rom not 
> > > > > > executed
> > > > > > on the host when the host machine first booted)?
> > > > > > 
> > > > > > FWIW, many of the chromebooks use coreboot with Intel graphics and a
> > > > > > number of users have installed SeaBIOS (running natively) on their
> > > > > > machines.  Running the intel video option rom more than once has 
> > > > > > been
> > > > > > known to cause issues.  
> > > > > 
> > > > > The issue is in the VM and it occurs every time the option ROM is
> > > > > executed.  Standard VGA text mode displays fine (ex. SeaBIOS version
> > > > > string and boot menu), but any sort of extended graphics mode (ex. 
> > > > > live
> > > > > CD boot menu) tries to make use of the host memory area which
> > > > > corresponds to the stolen memory area of the physical device.  We're
> > > > > not really sure how the ROM execution arrives at these addresses (not
> > > > > from the device according to access traces), but we can see when the
> > > > > ROM is writing these addresses to the device and modify they addresses
> > > > > to point at a VM memory range which we've allocated.  That's what this
> > > > > code attempts to do, allocate a buffer and tell QEMU about it via the
> > > > > BDSM (Base Data of Stolen Memory) register.  
> > > > 
> > > > Forgive me if I'm not fully understanding this.  If I read what you're
> > > > saying then the sequence is something like:
> > > > 
> > > > 1 - the host system bios (or vgabios) programs the GTT/stolen memory
> > > > base register at host system bootup time and reserves it in the
> > > > host e820 map.
> > > > 
> > > > 2 - upon running qemu, the guest reruns the vga bios option rom which
> > > > seems to work (ie, text mode works)
> > > > 
> > > > 3 - in the guest, upon running a bootloader that uses graphics mode,
> > > > the bootloader calls the vgabios to switch to graphics mode, and
> > > > the vgabios sends commands to the graphics hardware that somehow
> > > > reference the host stolen memory
> > > 
> > > What exactly happens here isn't clear to me, but this is a plausible
> > > explanation.  What we see in tracing access to the hardware is that a
> > > bunch of addresses are written to the device that fall within the host
> > > e820 reserved area and then the device starts generating IOMMU faults
> > > trying to access those addresses.
> > > 
> > > > 4 - your patch causes QEMU to catch these commands with references to
> > > > the host stolen memory and replace