Re: [PATCH v2] piix: fix regression during unplug in Xen HVM domUs
Am Tue, 16 May 2023 13:38:42 -0400 schrieb John Snow : > I haven't touched IDE or block code in quite a long while now -- I > don't think I can help land this fix, but I won't get in anyone's way, > either. Maybe just re-submit the patches with an improved commit > message / cover letter that helps collect the info from the previous > thread, the core issue, etc. I poked at it some more in the past days. Paolo was right in 2019, this issue needs to be debugged more to really understand why fiddling with one PCI devices breaks another, apparently unrelated PCI device. Once I know more, I will suggest a new change. The old one is stale, and needs to be rebased anyway. Olaf pgpwzgFQSRXim.pgp Description: Digitale Signatur von OpenPGP
Re: [PATCH v2] piix: fix regression during unplug in Xen HVM domUs
On Fri, May 12, 2023 at 5:14 PM Stefano Stabellini wrote: > > On Wed, 10 May 2023, Olaf Hering wrote: > > Wed, 10 May 2023 00:58:27 +0200 Olaf Hering : > > > > > In my debugging (with v8.0.0) it turned out the three pci_set_word > > > causes the domU to hang. In fact, it is just the last one: > > > > > >pci_set_byte(pci_conf + 0x20, 0x01); /* BMIBA: 20-23h */ > > > > > > It changes the value from 0xc121 to 0x1. > > > > If I disable just "pci_set_word(pci_conf + PCI_COMMAND, 0x);" it works > > as well. > > It changes the value from 0x5 to 0. > > > > In general I feel it is wrong to fiddle with PCI from the host side. > > This is most likely not the intention of the Xen unplug protocol. > > I'm sure the guest does not expect such changes under the hood. > > It happens to work by luck with pvops kernels because their PCI discovery > > is done after the unplug. > > > > So, what do we do here to get this off the table? > > I don't have a concrete suggestion because I don't understand the root > cause of the issue. Looking back at Paolo's reply from 2021 > > https://marc.info/?l=xen-devel=161669099305992=2 > > I think he was right. We can either fix the root cause of the issue or > avoid calling qdev_reset_all on unplug. I am OK with either one. I haven't touched IDE or block code in quite a long while now -- I don't think I can help land this fix, but I won't get in anyone's way, either. Maybe just re-submit the patches with an improved commit message / cover letter that helps collect the info from the previous thread, the core issue, etc. --js
Re: [PATCH v2] piix: fix regression during unplug in Xen HVM domUs
On Wed, 10 May 2023, Olaf Hering wrote: > Wed, 10 May 2023 00:58:27 +0200 Olaf Hering : > > > In my debugging (with v8.0.0) it turned out the three pci_set_word > > causes the domU to hang. In fact, it is just the last one: > > > >pci_set_byte(pci_conf + 0x20, 0x01); /* BMIBA: 20-23h */ > > > > It changes the value from 0xc121 to 0x1. > > If I disable just "pci_set_word(pci_conf + PCI_COMMAND, 0x);" it works as > well. > It changes the value from 0x5 to 0. > > In general I feel it is wrong to fiddle with PCI from the host side. > This is most likely not the intention of the Xen unplug protocol. > I'm sure the guest does not expect such changes under the hood. > It happens to work by luck with pvops kernels because their PCI discovery > is done after the unplug. > > So, what do we do here to get this off the table? I don't have a concrete suggestion because I don't understand the root cause of the issue. Looking back at Paolo's reply from 2021 https://marc.info/?l=xen-devel=161669099305992=2 I think he was right. We can either fix the root cause of the issue or avoid calling qdev_reset_all on unplug. I am OK with either one.
Re: [PATCH v2] piix: fix regression during unplug in Xen HVM domUs
Wed, 10 May 2023 00:58:27 +0200 Olaf Hering : > In my debugging (with v8.0.0) it turned out the three pci_set_word > causes the domU to hang. In fact, it is just the last one: > >pci_set_byte(pci_conf + 0x20, 0x01); /* BMIBA: 20-23h */ > > It changes the value from 0xc121 to 0x1. If I disable just "pci_set_word(pci_conf + PCI_COMMAND, 0x);" it works as well. It changes the value from 0x5 to 0. In general I feel it is wrong to fiddle with PCI from the host side. This is most likely not the intention of the Xen unplug protocol. I'm sure the guest does not expect such changes under the hood. It happens to work by luck with pvops kernels because their PCI discovery is done after the unplug. So, what do we do here to get this off the table? Olaf pgpTyTksMtE1Y.pgp Description: Digitale Signatur von OpenPGP
Re: [PATCH v2] piix: fix regression during unplug in Xen HVM domUs
Resuming this old thread about an unfixed bug, which was introduced in qemu-4.2: qemu ends up in piix_ide_reset from pci_unplug_disks. This was not the case prior 4.2, the removed call to qemu_register_reset(piix3_reset, d) in ee358e919e385fdc79d59d0d47b4a81e349cd5c9 did apparently nothing. In my debugging (with v8.0.0) it turned out the three pci_set_word causes the domU to hang. In fact, it is just the last one: pci_set_byte(pci_conf + 0x20, 0x01); /* BMIBA: 20-23h */ It changes the value from 0xc121 to 0x1. The question is: what does this do in practice? Starting with recent qemu (like 7.2), the domU sometimes proceeds with these messages: [1.631161] uhci_hcd :00:01.2: host system error, PCI problems? [1.634965] uhci_hcd :00:01.2: host controller process error, something bad happened! [1.634965] uhci_hcd :00:01.2: host controller halted, very bad! [1.634965] uhci_hcd :00:01.2: HC died; cleaning up Loading basic drivers...[2.398048] Disabling IRQ #23 Is anyone familiar enough with PIIX3 and knows how these devices are interacting? 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01) 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03) 00:02.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) 00:03.0 VGA compatible controller: Cirrus Logic GD 5446 Thanks, Olaf On Thu, Mar 25, Paolo Bonzini wrote: > On 25/03/21 12:12, Olaf Hering wrote: > > Am Mon, 22 Mar 2021 18:09:17 -0400 > > schrieb John Snow : > > > > > My understanding is that XEN has some extra disks that it unplugs when > > > it later figures out it doesn't need them. How exactly this works is > > > something I've not looked into too closely. > > > > It has no extra disks, why would it? > > > > I assume each virtualization variant has some sort of unplug if it has to > > support guests that lack PV/virtio/enlightened/whatever drivers. > > No, it's Xen only and really should be legacy. Ideally one would just have > devices supported at all levels from firmware to kernel. > > > > So if these IDE devices have been "unplugged" already, we avoid > > > resetting them here. What about this reset causes the bug you describe > > > in the commit message? > > > > > > Does this reset now happen earlier/later as compared to what it did > > > prior to ee358e91 ? > > > > Prior this commit, piix_ide_reset was only called when the entire > > emulated machine was reset. Like: never. With this commit, > > piix_ide_reset will be called from pci_piix3_xen_ide_unplug. For some > > reason it confuses the emulated USB hardware. Why it does confused > > it, no idea. > > > I wonder what the purpose of the qdev_reset_all() call really is. It > > is 10 years old. It might be stale. > > piix_ide_reset is only calling ide_bus_reset, and from there ide_reset and > bmdma_reset. All of these functions do just two things: reset internal > registers and ensure pending I/O is completed or canceled. The latter is > indeed unnecessary; drain/flush/detach is already done before the call to > qdev_reset_all. > > But the fact that it breaks USB is weird. That's the part that needs to be > debugged, because changing IDE to unbreak USB needs an explanation even if > it's the right thing to do. > > If you don't want to debug it, removing the qdev_reset_all call might do the > job; you'll have to see what the Xen maintainers think of it. But if you > don't debug the USB issue now, it will come back later almost surely. > > Paolo signature.asc Description: Digital signature
Re: [PATCH v2] piix: fix regression during unplug in Xen HVM domUs
On 25/03/21 12:12, Olaf Hering wrote: Am Mon, 22 Mar 2021 18:09:17 -0400 schrieb John Snow : My understanding is that XEN has some extra disks that it unplugs when it later figures out it doesn't need them. How exactly this works is something I've not looked into too closely. It has no extra disks, why would it? I assume each virtualization variant has some sort of unplug if it has to support guests that lack PV/virtio/enlightened/whatever drivers. No, it's Xen only and really should be legacy. Ideally one would just have devices supported at all levels from firmware to kernel. So if these IDE devices have been "unplugged" already, we avoid resetting them here. What about this reset causes the bug you describe in the commit message? Does this reset now happen earlier/later as compared to what it did prior to ee358e91 ? Prior this commit, piix_ide_reset was only called when the entire emulated machine was reset. Like: never. With this commit, piix_ide_reset will be called from pci_piix3_xen_ide_unplug. For some reason it confuses the emulated USB hardware. Why it does confused it, no idea. I wonder what the purpose of the qdev_reset_all() call really is. It is 10 years old. It might be stale. piix_ide_reset is only calling ide_bus_reset, and from there ide_reset and bmdma_reset. All of these functions do just two things: reset internal registers and ensure pending I/O is completed or canceled. The latter is indeed unnecessary; drain/flush/detach is already done before the call to qdev_reset_all. But the fact that it breaks USB is weird. That's the part that needs to be debugged, because changing IDE to unbreak USB needs an explanation even if it's the right thing to do. If you don't want to debug it, removing the qdev_reset_all call might do the job; you'll have to see what the Xen maintainers think of it. But if you don't debug the USB issue now, it will come back later almost surely. Paolo
Re: [PATCH v2] piix: fix regression during unplug in Xen HVM domUs
Am Mon, 22 Mar 2021 18:09:17 -0400 schrieb John Snow : > My understanding is that XEN has some extra disks that it unplugs when > it later figures out it doesn't need them. How exactly this works is > something I've not looked into too closely. It has no extra disks, why would it? I assume each virtualization variant has some sort of unplug if it has to support guests that lack PV/virtio/enlightened/whatever drivers. In case of HVM, the configured block or network devices can be either accessed via emulated PCI or via the PV drivers. Since the BIOS, the bootloader and potentially the operating system kernel typically lack PV drivers, they will find the devices only via the PCI bus. In case they happen to have PV drivers in addition to PCI drivers, both drivers will find and offer the same resource via different paths. In case of a block device, ata_piix.ko will show it via "/dev/sda" and xen-blkfront.ko will show it via "/dev/xvda". This is obviously bad, at least in the read-write case. The pvops kernel triggers the unplug of the emulated PCI hardware early, prior any other PCI initialization. As a result the PCI drivers will not find their hardware anymore. In case of ata_piix, only the non-CDROM storage will be removed in qmeu, because there is no PV-CDROM driver. The PV support in old xenlinux based kernels is only available as modules. As a result the unplug will happen after PCI was initialized, but it must happen before any PCI device drivers are loaded. > So if these IDE devices have been "unplugged" already, we avoid > resetting them here. What about this reset causes the bug you describe > in the commit message? > > Does this reset now happen earlier/later as compared to what it did > prior to ee358e91 ? Prior this commit, piix_ide_reset was only called when the entire emulated machine was reset. Like: never. With this commit, piix_ide_reset will be called from pci_piix3_xen_ide_unplug. For some reason it confuses the emulated USB hardware. Why it does confused it, no idea. I wonder what the purpose of the qdev_reset_all() call really is. It is 10 years old. It might be stale. Olaf pgpiJF0BreQPC.pgp Description: Digitale Signatur von OpenPGP
Re: [PATCH v2] piix: fix regression during unplug in Xen HVM domUs
On 3/17/21 3:00 AM, Olaf Hering wrote:
Commit ee358e919e385fdc79d59d0d47b4a81e349cd5c9 causes a regression in
Xen HVM domUs which run xenlinux based kernels.
If the domU has an USB device assigned, for example with
"usbdevice=['tablet']" in domU.cfg, the late unplug of devices will
kill the emulated USB host. As a result the khubd thread hangs, and as
a result the entire boot process.
For some reason this does not affect pvops based kernels. This is
most likely caused by the fact that unplugging happens very early
during boot.
I'm not entirely sure of how the commit message relates to the patch,
actually. (Sorry, I am not well familiar with XEN.)
Signed-off-by: Olaf Hering
---
hw/ide/piix.c| 5 +
include/hw/ide/pci.h | 1 +
2 files changed, 6 insertions(+)
diff --git a/hw/ide/piix.c b/hw/ide/piix.c
index b9860e35a5..7f1998bf04 100644
--- a/hw/ide/piix.c
+++ b/hw/ide/piix.c
@@ -109,6 +109,9 @@ static void piix_ide_reset(DeviceState *dev)
uint8_t *pci_conf = pd->config;
int i;
+if (d->xen_unplug_done == true) {
+return;
+}
My understanding is that XEN has some extra disks that it unplugs when
it later figures out it doesn't need them. How exactly this works is
something I've not looked into too closely.
So if these IDE devices have been "unplugged" already, we avoid
resetting them here. What about this reset causes the bug you describe
in the commit message?
Does this reset now happen earlier/later as compared to what it did
prior to ee358e91 ?
for (i = 0; i < 2; i++) {
ide_bus_reset(>bus[i]);
}
@@ -151,6 +154,7 @@ static void pci_piix_ide_realize(PCIDevice *dev, Error
**errp)
PCIIDEState *d = PCI_IDE(dev);
uint8_t *pci_conf = dev->config;
+d->xen_unplug_done = false;
pci_conf[PCI_CLASS_PROG] = 0x80; // legacy ATA mode
bmdma_setup_bar(d);
@@ -170,6 +174,7 @@ int pci_piix3_xen_ide_unplug(DeviceState *dev, bool aux)
BlockBackend *blk;
pci_ide = PCI_IDE(dev);
+pci_ide->xen_unplug_done = true;
for (i = aux ? 1 : 0; i < 4; i++) {
idebus = _ide->bus[i / 2];
diff --git a/include/hw/ide/pci.h b/include/hw/ide/pci.h
index d8384e1c42..9e71cfec3b 100644
--- a/include/hw/ide/pci.h
+++ b/include/hw/ide/pci.h
@@ -50,6 +50,7 @@ struct PCIIDEState {
IDEBus bus[2];
BMDMAState bmdma[2];
uint32_t secondary; /* used only for cmd646 */
+bool xen_unplug_done;
I am hesitant to put a new XEN-specific boolean here, but don't know
enough about the problem to outright say "no".
This looks like a band-aid that's out of place, but I don't understand
the problem well enough yet to suggest a better place.
MemoryRegion bmdma_bar;
MemoryRegion cmd_bar[2];
MemoryRegion data_bar[2];
(If anyone else with more experience with XEN wants to take over the
review of this patch, let me know. I only really care about the IDE bits.)
[PATCH v2] piix: fix regression during unplug in Xen HVM domUs
Commit ee358e919e385fdc79d59d0d47b4a81e349cd5c9 causes a regression in
Xen HVM domUs which run xenlinux based kernels.
If the domU has an USB device assigned, for example with
"usbdevice=['tablet']" in domU.cfg, the late unplug of devices will
kill the emulated USB host. As a result the khubd thread hangs, and as
a result the entire boot process.
For some reason this does not affect pvops based kernels. This is
most likely caused by the fact that unplugging happens very early
during boot.
Signed-off-by: Olaf Hering
---
hw/ide/piix.c| 5 +
include/hw/ide/pci.h | 1 +
2 files changed, 6 insertions(+)
diff --git a/hw/ide/piix.c b/hw/ide/piix.c
index b9860e35a5..7f1998bf04 100644
--- a/hw/ide/piix.c
+++ b/hw/ide/piix.c
@@ -109,6 +109,9 @@ static void piix_ide_reset(DeviceState *dev)
uint8_t *pci_conf = pd->config;
int i;
+if (d->xen_unplug_done == true) {
+return;
+}
for (i = 0; i < 2; i++) {
ide_bus_reset(>bus[i]);
}
@@ -151,6 +154,7 @@ static void pci_piix_ide_realize(PCIDevice *dev, Error
**errp)
PCIIDEState *d = PCI_IDE(dev);
uint8_t *pci_conf = dev->config;
+d->xen_unplug_done = false;
pci_conf[PCI_CLASS_PROG] = 0x80; // legacy ATA mode
bmdma_setup_bar(d);
@@ -170,6 +174,7 @@ int pci_piix3_xen_ide_unplug(DeviceState *dev, bool aux)
BlockBackend *blk;
pci_ide = PCI_IDE(dev);
+pci_ide->xen_unplug_done = true;
for (i = aux ? 1 : 0; i < 4; i++) {
idebus = _ide->bus[i / 2];
diff --git a/include/hw/ide/pci.h b/include/hw/ide/pci.h
index d8384e1c42..9e71cfec3b 100644
--- a/include/hw/ide/pci.h
+++ b/include/hw/ide/pci.h
@@ -50,6 +50,7 @@ struct PCIIDEState {
IDEBus bus[2];
BMDMAState bmdma[2];
uint32_t secondary; /* used only for cmd646 */
+bool xen_unplug_done;
MemoryRegion bmdma_bar;
MemoryRegion cmd_bar[2];
MemoryRegion data_bar[2];
