Re: PCI passthrough resource remapping
On 03/31/2010 06:18 PM, Chris Wright wrote: Hrm, I'm not sure these would be related to the small BAR region patch. It looks more like a timing issue. small BAR == slow path == timing issue? Would be interesting to verify using perf with the 'kvm:kvm_mmio' software event, see how many happen per second. -- Do not meddle in the internals of kernels, for they are subtle and quick to panic. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
Chris Wright wrote: > * Alexander Graf (ag...@suse.de) wrote: > >> Kenni Lund wrote: >> >>> 2010/3/31 Kenni Lund : >>> >>> Booting the guest and tuning to a DVB-T channel _works_ !!! :-D Thanks a lot for your help...I have one more question, though: If I have two devices (like the ivtv tuner and the USB card) and they share an IRQ, if I then assign BOTH of them to the same guest, will it then work? Alexander, the patch works, I hope to see it in a stable release in the near future ;) >>> Unfortunately, I have a correction to this :( It _almost_ works. I had >>> some video/audio artifacts which I though was caused by bad reception, >>> but after switching the DVB-T tuner back and forth between the PCI USB >>> card and a laptop, it got clear to me, that this was a passthrough >>> issue. I get no errors in dmesg on the host or in the guest. >>> >>> I recorded a short videoclip which illustrates the issue: >>> https://docs.google.com/leaf?id=0B-_nZameGeN-NDk4YzQ3N2EtMmEzMi00NTU4LWFjMjgtNzkxMzcxYzg2MTM1&hl=en >>> >>> >> Hrm, I'm not sure these would be related to the small BAR region patch. >> It looks more like a timing issue. >> > > small BAR == slow path == timing issue? > Could be, yeah. The only chance I see to avoid that is to have in-kernel small BAR code that would issue the MMIO accesses without the userspace churn. One thing I could imagine would be < PAGE_SIZE memory slots. Then userspace just maps the 4K to the BAR, but passes only a part of it on as slot. Alex -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
* Alexander Graf (ag...@suse.de) wrote: > Kenni Lund wrote: > > 2010/3/31 Kenni Lund : > > > >> Booting the guest and tuning to a DVB-T channel _works_ !!! :-D Thanks > >> a lot for your help...I have one more question, though: If I have two > >> devices (like the ivtv tuner and the USB card) and they share an IRQ, > >> if I then assign BOTH of them to the same guest, will it then work? > >> > >> Alexander, the patch works, I hope to see it in a stable release in > >> the near future ;) > >> > > > > Unfortunately, I have a correction to this :( It _almost_ works. I had > > some video/audio artifacts which I though was caused by bad reception, > > but after switching the DVB-T tuner back and forth between the PCI USB > > card and a laptop, it got clear to me, that this was a passthrough > > issue. I get no errors in dmesg on the host or in the guest. > > > > I recorded a short videoclip which illustrates the issue: > > https://docs.google.com/leaf?id=0B-_nZameGeN-NDk4YzQ3N2EtMmEzMi00NTU4LWFjMjgtNzkxMzcxYzg2MTM1&hl=en > > > > Hrm, I'm not sure these would be related to the small BAR region patch. > It looks more like a timing issue. small BAR == slow path == timing issue? thanks, -chris -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
* Kenni Lund (ke...@kelu.dk) wrote: > 2010/3/31 Chris Wright : > > * Kenni Lund (ke...@kelu.dk) wrote: > >> 2010/3/31 Chris Wright : > >> >> So I suppose I'll need to get rid of this shared IRQ before I can > >> >> conclude anything on the patch in git. Hmm, is there some cleaver way > >> >> of fixing this in Linux, or do I have to fix it by changing BIOS IRQ > >> >> settings, disabling hardware and/or moving the hardware around in > >> >> various PCI slots? > >> > > >> > The way I typically work around this is simply unbinding the driver from > >> > the device in the host (and thus freeing the irq). > >> > >> Doh...anyway, I went all the way, found a USB->PS2 adaptor, disabled > >> the onboard USB controller and PATA controller in BIOS, and now got > >> "kvm_assigned_intx_device" for all 4 IRQs :) > > > > A little extreme...but hey, that works ;-) I'm still curious what's > > going on w/ your PCI card and the irq routing. Something is suspect. > > I'll not be able to do any further testing for the next two weeks, but > if you want me to perform some tests, test patches etc., just write > what you want me to do and I'll do it in two weeks. Thanks for your willingness to do some tests. I'll poke around a bit and see if I can come up w/ something useful for you to try out. thanks, -chris -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
2010/3/31 Alexander Graf : > Kenni Lund wrote: >> 2010/3/31 Kenni Lund : >> >>> Booting the guest and tuning to a DVB-T channel _works_ !!! :-D Thanks >>> a lot for your help...I have one more question, though: If I have two >>> devices (like the ivtv tuner and the USB card) and they share an IRQ, >>> if I then assign BOTH of them to the same guest, will it then work? >>> >>> Alexander, the patch works, I hope to see it in a stable release in >>> the near future ;) >>> >> >> Unfortunately, I have a correction to this :( It _almost_ works. I had >> some video/audio artifacts which I though was caused by bad reception, >> but after switching the DVB-T tuner back and forth between the PCI USB >> card and a laptop, it got clear to me, that this was a passthrough >> issue. I get no errors in dmesg on the host or in the guest. >> >> I recorded a short videoclip which illustrates the issue: >> https://docs.google.com/leaf?id=0B-_nZameGeN-NDk4YzQ3N2EtMmEzMi00NTU4LWFjMjgtNzkxMzcxYzg2MTM1&hl=en >> > > Hrm, I'm not sure these would be related to the small BAR region patch. > It looks more like a timing issue. For what it's worth, I quickly tried to passthrough one of the onboard USB controllers instead (Intel Corporation 82801JD/DO (ICH10 Family), which is also affected by the 4k limitation). It improves the artifact problems, bringing the artifacts down to one artifact every 5 seconds. I'll be back in ~1½ week, please tell me if you want me to test something when I get back. Thanks for your help :) Best Regards Kenni -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
Kenni Lund wrote: > 2010/3/31 Kenni Lund : > >> Booting the guest and tuning to a DVB-T channel _works_ !!! :-D Thanks >> a lot for your help...I have one more question, though: If I have two >> devices (like the ivtv tuner and the USB card) and they share an IRQ, >> if I then assign BOTH of them to the same guest, will it then work? >> >> Alexander, the patch works, I hope to see it in a stable release in >> the near future ;) >> > > Unfortunately, I have a correction to this :( It _almost_ works. I had > some video/audio artifacts which I though was caused by bad reception, > but after switching the DVB-T tuner back and forth between the PCI USB > card and a laptop, it got clear to me, that this was a passthrough > issue. I get no errors in dmesg on the host or in the guest. > > I recorded a short videoclip which illustrates the issue: > https://docs.google.com/leaf?id=0B-_nZameGeN-NDk4YzQ3N2EtMmEzMi00NTU4LWFjMjgtNzkxMzcxYzg2MTM1&hl=en > Hrm, I'm not sure these would be related to the small BAR region patch. It looks more like a timing issue. Alex -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
2010/3/31 Kenni Lund : > Booting the guest and tuning to a DVB-T channel _works_ !!! :-D Thanks > a lot for your help...I have one more question, though: If I have two > devices (like the ivtv tuner and the USB card) and they share an IRQ, > if I then assign BOTH of them to the same guest, will it then work? > > Alexander, the patch works, I hope to see it in a stable release in > the near future ;) Unfortunately, I have a correction to this :( It _almost_ works. I had some video/audio artifacts which I though was caused by bad reception, but after switching the DVB-T tuner back and forth between the PCI USB card and a laptop, it got clear to me, that this was a passthrough issue. I get no errors in dmesg on the host or in the guest. I recorded a short videoclip which illustrates the issue: https://docs.google.com/leaf?id=0B-_nZameGeN-NDk4YzQ3N2EtMmEzMi00NTU4LWFjMjgtNzkxMzcxYzg2MTM1&hl=en Best Regards Kenni -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
2010/3/31 Chris Wright : > * Kenni Lund (ke...@kelu.dk) wrote: >> 2010/3/31 Chris Wright : >> >> So I suppose I'll need to get rid of this shared IRQ before I can >> >> conclude anything on the patch in git. Hmm, is there some cleaver way >> >> of fixing this in Linux, or do I have to fix it by changing BIOS IRQ >> >> settings, disabling hardware and/or moving the hardware around in >> >> various PCI slots? >> > >> > The way I typically work around this is simply unbinding the driver from >> > the device in the host (and thus freeing the irq). >> >> Doh...anyway, I went all the way, found a USB->PS2 adaptor, disabled >> the onboard USB controller and PATA controller in BIOS, and now got >> "kvm_assigned_intx_device" for all 4 IRQs :) > > A little extreme...but hey, that works ;-) I'm still curious what's > going on w/ your PCI card and the irq routing. Something is suspect. I'll not be able to do any further testing for the next two weeks, but if you want me to perform some tests, test patches etc., just write what you want me to do and I'll do it in two weeks. Best Regards Kenni -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
* Kenni Lund (ke...@kelu.dk) wrote: > 2010/3/31 Chris Wright : > >> So I suppose I'll need to get rid of this shared IRQ before I can > >> conclude anything on the patch in git. Hmm, is there some cleaver way > >> of fixing this in Linux, or do I have to fix it by changing BIOS IRQ > >> settings, disabling hardware and/or moving the hardware around in > >> various PCI slots? > > > > The way I typically work around this is simply unbinding the driver from > > the device in the host (and thus freeing the irq). > > Doh...anyway, I went all the way, found a USB->PS2 adaptor, disabled > the onboard USB controller and PATA controller in BIOS, and now got > "kvm_assigned_intx_device" for all 4 IRQs :) A little extreme...but hey, that works ;-) I'm still curious what's going on w/ your PCI card and the irq routing. Something is suspect. > Booting the guest and tuning to a DVB-T channel _works_ !!! :-D Thanks > a lot for your help...I have one more question, though: If I have two > devices (like the ivtv tuner and the USB card) and they share an IRQ, > if I then assign BOTH of them to the same guest, will it then work? No, it won't. The first one will request an exclusive interrupt, and the second one will fail its request_irq. There's two options, one is to use a different driver for kvm (if the device is new enough to handle PCI 2.3) or use MSI/MSI-X based devices (that works best ;) Neither of those options are readily available in your case. thanks, -chris -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
* Kenni Lund (ke...@kelu.dk) wrote: > 2010/3/31 Alexander Graf : > > The easiest thing coming to mind is to unplug the ivtv card for now. It's > > really only to verify that the patch does something useful :-). > > This was not sufficient. Same issue with the ivtv card unplugged...if > I interpret the content of /proc/interrupt correctly, the USB PCI > cards uses 4 IRQs and out of these three IRQs are still shared with > other components. > > The card uses IRQ 16, 18, 19, 20: > pci-stub :02:01.0: PCI INT B -> GSI 18 (level, low) -> IRQ 18 > pci-stub :02:01.1: PCI INT C -> GSI 16 (level, low) -> IRQ 16 > pci-stub :02:01.2: PCI INT D -> GSI 20 (level, low) -> IRQ 20 > pci-stub :02:01.3: PCI INT A -> GSI 19 (level, low) -> IRQ 19 > > Only IRQ 20 gets "kvm_assigned_intx_device" after > unbinding+binding+qemu-kvm launch: What is your qemu command line? It's acting as if function .2 (one of the ochi controllers) is generating the interrupt rather than .3 (the ehci controller). And indeed, there is no handler to ack that interrupt (just the likely on-board uhci one). before: 19: 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb3, uhci_hcd:usb9 20: 0 0 0 0 IO-APIC-fasteoi ohci_hcd:usb12 after: 19: 75010 73438 76479 75074 IO-APIC-fasteoi uhci_hcd:usb9 20: 0 0 0 0 IO-APIC-fasteoi kvm_assigned_intx_device Can you unbind uhci_hcd:usb9 and then give both .2 and .3 to the guest? thanks, -chris -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
2010/3/31 Chris Wright : >> So I suppose I'll need to get rid of this shared IRQ before I can >> conclude anything on the patch in git. Hmm, is there some cleaver way >> of fixing this in Linux, or do I have to fix it by changing BIOS IRQ >> settings, disabling hardware and/or moving the hardware around in >> various PCI slots? > > The way I typically work around this is simply unbinding the driver from > the device in the host (and thus freeing the irq). Doh...anyway, I went all the way, found a USB->PS2 adaptor, disabled the onboard USB controller and PATA controller in BIOS, and now got "kvm_assigned_intx_device" for all 4 IRQs :) Booting the guest and tuning to a DVB-T channel _works_ !!! :-D Thanks a lot for your help...I have one more question, though: If I have two devices (like the ivtv tuner and the USB card) and they share an IRQ, if I then assign BOTH of them to the same guest, will it then work? Alexander, the patch works, I hope to see it in a stable release in the near future ;) Best Regards Kenni -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
* Kenni Lund (ke...@kelu.dk) wrote: > 2010/3/30 Chris Wright : > > * Kenni Lund (ke...@kelu.dk) wrote: > >> Client dmesg: http://pastebin.com/uNG4QK5j > >> Host dmesg: http://pastebin.com/jZu3WKZW > >> > >> I just verified it and I do get the call trace in the host (which > >> disables IRQ 19, used by the PCI USB card), exactly at the same second > > > > It looks like IRQ 19 is shared between the ehci controller and the > > ivtv tuner. What do you see in /proc/interrupts on the host (before > > you unbind and after you bind to pci stub)? > > Ahh, and even if the ivtv module is not loaded, I will still have a > shared IRQ, right? I didn't see ivtv in /proc/interrupts before, as I > unload the ivtv driver on boot in /etc/rc.local, before unbinding the > ivtv tuner and binding it to pci stub. (the ivtv tuner is normally > assigned to the same guest, but not now while testing the PCI USB > card). > > If I don't unload (and unbind/bind) the ivtv driver/tuner on boot in > /etc/local, I get the following in /proc/interrupts on a clean boot: > http://pastebin.com/SFQj58LC > > If I now unbind and bind the PCI USB card to pci stub, I get no > changes in /proc/interrupts. Sorry, I meant bind to pci_stub and launch guest. IOW, you should see kvm_assigned_{msi,msix,intx}_device (from your lspci, I'd expect intx). What's odd is a device is asserting an interrupt to a line w/ no handler acking. The IRQ 19, should have kvm handling the interrupt, in which case it'd always return IRQ_HANDLED. And for the case of shared interrupts, we won't let you start the guest with an assigned device that's sharing an interrupt. IOW, we do request_irq() w/out specifying IRQF_SHARED (meaning we want an exclusive irq). > So I suppose I'll need to get rid of this shared IRQ before I can > conclude anything on the patch in git. Hmm, is there some cleaver way > of fixing this in Linux, or do I have to fix it by changing BIOS IRQ > settings, disabling hardware and/or moving the hardware around in > various PCI slots? The way I typically work around this is simply unbinding the driver from the device in the host (and thus freeing the irq). thanks, -chris -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
2010/3/31 Alexander Graf : > > On 31.03.2010, at 00:27, Kenni Lund wrote: > >> 2010/3/30 Chris Wright : >>> * Kenni Lund (ke...@kelu.dk) wrote: Client dmesg: http://pastebin.com/uNG4QK5j Host dmesg: http://pastebin.com/jZu3WKZW I just verified it and I do get the call trace in the host (which disables IRQ 19, used by the PCI USB card), exactly at the same second >>> >>> It looks like IRQ 19 is shared between the ehci controller and the >>> ivtv tuner. What do you see in /proc/interrupts on the host (before >>> you unbind and after you bind to pci stub)? >> >> Ahh, and even if the ivtv module is not loaded, I will still have a >> shared IRQ, right? I didn't see ivtv in /proc/interrupts before, as I >> unload the ivtv driver on boot in /etc/rc.local, before unbinding the >> ivtv tuner and binding it to pci stub. (the ivtv tuner is normally >> assigned to the same guest, but not now while testing the PCI USB >> card). >> >> If I don't unload (and unbind/bind) the ivtv driver/tuner on boot in >> /etc/local, I get the following in /proc/interrupts on a clean boot: >> http://pastebin.com/SFQj58LC >> >> If I now unbind and bind the PCI USB card to pci stub, I get no >> changes in /proc/interrupts. >> >> So I suppose I'll need to get rid of this shared IRQ before I can >> conclude anything on the patch in git. Hmm, is there some cleaver way >> of fixing this in Linux, or do I have to fix it by changing BIOS IRQ >> settings, disabling hardware and/or moving the hardware around in >> various PCI slots? > > The easiest thing coming to mind is to unplug the ivtv card for now. It's > really only to verify that the patch does something useful :-). This was not sufficient. Same issue with the ivtv card unplugged...if I interpret the content of /proc/interrupt correctly, the USB PCI cards uses 4 IRQs and out of these three IRQs are still shared with other components. The card uses IRQ 16, 18, 19, 20: pci-stub :02:01.0: PCI INT B -> GSI 18 (level, low) -> IRQ 18 pci-stub :02:01.1: PCI INT C -> GSI 16 (level, low) -> IRQ 16 pci-stub :02:01.2: PCI INT D -> GSI 20 (level, low) -> IRQ 20 pci-stub :02:01.3: PCI INT A -> GSI 19 (level, low) -> IRQ 19 Only IRQ 20 gets "kvm_assigned_intx_device" after unbinding+binding+qemu-kvm launch: interrupts before: http://pastebin.com/DdJEv29z interrupts after: http://pastebin.com/zasWEZsL It seems like i still have shared IRQs with the onboard USB controller as well as the PATA controller. In the BIOS i have the option of setting the IRQ of "PCI Card 1" and "PCI Card 2". I tried changing these into IRQ 10 and 11 (which are free according to /proc/interrupts), but it changed absolutely nothing in /proc/interrupts after booting(?) :-( Any kind of hint which could lead me in hte right direction, would be highly appreciated...thanks. Best Regards Kenni -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
On 31.03.2010, at 00:27, Kenni Lund wrote: > 2010/3/30 Chris Wright : >> * Kenni Lund (ke...@kelu.dk) wrote: >>> Client dmesg: http://pastebin.com/uNG4QK5j >>> Host dmesg: http://pastebin.com/jZu3WKZW >>> >>> I just verified it and I do get the call trace in the host (which >>> disables IRQ 19, used by the PCI USB card), exactly at the same second >> >> It looks like IRQ 19 is shared between the ehci controller and the >> ivtv tuner. What do you see in /proc/interrupts on the host (before >> you unbind and after you bind to pci stub)? > > Ahh, and even if the ivtv module is not loaded, I will still have a > shared IRQ, right? I didn't see ivtv in /proc/interrupts before, as I > unload the ivtv driver on boot in /etc/rc.local, before unbinding the > ivtv tuner and binding it to pci stub. (the ivtv tuner is normally > assigned to the same guest, but not now while testing the PCI USB > card). > > If I don't unload (and unbind/bind) the ivtv driver/tuner on boot in > /etc/local, I get the following in /proc/interrupts on a clean boot: > http://pastebin.com/SFQj58LC > > If I now unbind and bind the PCI USB card to pci stub, I get no > changes in /proc/interrupts. > > So I suppose I'll need to get rid of this shared IRQ before I can > conclude anything on the patch in git. Hmm, is there some cleaver way > of fixing this in Linux, or do I have to fix it by changing BIOS IRQ > settings, disabling hardware and/or moving the hardware around in > various PCI slots? The easiest thing coming to mind is to unplug the ivtv card for now. It's really only to verify that the patch does something useful :-). Alex-- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
2010/3/30 Chris Wright : > * Kenni Lund (ke...@kelu.dk) wrote: >> Client dmesg: http://pastebin.com/uNG4QK5j >> Host dmesg: http://pastebin.com/jZu3WKZW >> >> I just verified it and I do get the call trace in the host (which >> disables IRQ 19, used by the PCI USB card), exactly at the same second > > It looks like IRQ 19 is shared between the ehci controller and the > ivtv tuner. What do you see in /proc/interrupts on the host (before > you unbind and after you bind to pci stub)? Ahh, and even if the ivtv module is not loaded, I will still have a shared IRQ, right? I didn't see ivtv in /proc/interrupts before, as I unload the ivtv driver on boot in /etc/rc.local, before unbinding the ivtv tuner and binding it to pci stub. (the ivtv tuner is normally assigned to the same guest, but not now while testing the PCI USB card). If I don't unload (and unbind/bind) the ivtv driver/tuner on boot in /etc/local, I get the following in /proc/interrupts on a clean boot: http://pastebin.com/SFQj58LC If I now unbind and bind the PCI USB card to pci stub, I get no changes in /proc/interrupts. So I suppose I'll need to get rid of this shared IRQ before I can conclude anything on the patch in git. Hmm, is there some cleaver way of fixing this in Linux, or do I have to fix it by changing BIOS IRQ settings, disabling hardware and/or moving the hardware around in various PCI slots? Best Regards Kenni -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
* Kenni Lund (ke...@kelu.dk) wrote: > Client dmesg: http://pastebin.com/uNG4QK5j > Host dmesg: http://pastebin.com/jZu3WKZW > > I just verified it and I do get the call trace in the host (which > disables IRQ 19, used by the PCI USB card), exactly at the same second It looks like IRQ 19 is shared between the ehci controller and the ivtv tuner. What do you see in /proc/interrupts on the host (before you unbind and after you bind to pci stub)? The host dmesg shows other possible devices sharing that interrupt: $ grep "IRQ 19" jZu3WKZW 482. ahci :00:1f.2: PCI INT B -> GSI 19 (level, low) -> IRQ 19 578. ehci_hcd :02:01.3: PCI INT A -> GSI 19 (level, low) -> IRQ 19 623. uhci_hcd :00:1d.1: PCI INT B -> GSI 19 (level, low) -> IRQ 19 795. ivtv :03:09.0: PCI INT A -> GSI 19 (level, low) -> IRQ 19 814. IRQ 19/ivtv1: IRQF_DISABLED is not guaranteed on shared IRQs 955. pci-stub :02:01.3: PCI INT A -> GSI 19 (level, low) -> IRQ 19 thanks, -chris -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
2010/3/30 Chris Wright : > * Alexander Graf (ag...@suse.de) wrote: >> On 30.03.2010, at 01:00, Kenni Lund wrote: >> >> > 2010/3/29 Alexander Graf : >> >> >> >> On 29.03.2010, at 19:23, Kenni Lund wrote: >> >> >> > 2010/1/9 Alexander Graf : >> >> >> >> On 09.01.2010, at 03:45, Ryan C. Underwood wrote: >> >> >> >>> >> >>> I have a multifunction PCI device that I'd like to pass through to >> >>> KVM. >> >>> In order to do that, I'm reading that the PCI memory region must be >> >>> 4K-page >> >>> aligned and the PCI memory resources itself must also be exact >> >>> multiples >> >>> of 4K pages. >> >>> >> >>> I have added the following on my kernel command line: >> >>> reassign_resources >> >>> reassigndev=08:09.0,08:09.1,08:09.2,08:09.3,08:09.4 >> >>> >> >>> But I don't know if it has any effect. The resources are still not >> >>> sized in 4K pages. Also, this seems to screw up the last device. >> >> >> >> I submitted a patch to qemu-kvm recently that got rid of that >> >> limitation. >> >> Please try out if the current git head works for you. >> >> >> >> Alex-- >> > >> > I just upgraded to kernel 2.6.32.10 with qemu-kvm 0.12.3 and I still >> > get the following error when trying to pass through a dedicated PCI >> > USB card: >> > >> > "Unable to assign device: PCI region 0 at address 0xe9403000 has size >> > 0x100, which is not a multiple of 4K >> > Error initializing device pci-assign" >> > >> > Didn't the above patch make it into qemu-kvm? I don't know why, but I >> > was under the impression that this was fixed when I upgraded to >> > qemu-kvm 0.12.3. >> > >> It's only in qemu-kvm.git. Maybe it should go into qemu-kvm-0.12.4 if >> there >> is one >> >>> >> >>> That would be highly appriciated...with the current USB support in >> >>> QEMU, PCI passthrough is the only way to get USB 2.0 support. I've >> >>> bought two dedicated PCI USB cards for this, but none of them works >> >>> due to the above limitation. >> >>> >> >>> Perhaps a developer can comment on this? Are there any plans on >> >>> including this patch in the stable releases in the near future? >> >> >> >> Please first try out to build the current git snapshot of qemu-kvm. If it >> >> works properly for you then I agree that we should take this into >> >> 0.12-stable. >> >> >> >> I wrote the support for a card that still didn't work even with this >> >> patch. So having someone say it makes things work for him is definitely a >> >> must :-). >> > >> > Sure, I have compiled the current git snapshot and performed some >> > tests...It's at least mostly working, so I'm a bit unsure if this is a >> > bug related to this or to something else. >> >> Chris, any idea on this? Looks like something's going wrong with function >> assignment. > > Hmm, one thing that sticks out to me is the debug port. Kenni, can you > post full dmesg on both host and guest, nothing is obviously broken (and > in fact the guest should never "see" the debug port). > Uploaded here: Client dmesg: http://pastebin.com/uNG4QK5j Host dmesg: http://pastebin.com/jZu3WKZW I just verified it and I do get the call trace in the host (which disables IRQ 19, used by the PCI USB card), exactly at the same second I ask the DVB-T tuner to view a channel in the guest. Thanks.. Best Regards Kenni Lund -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
* Alexander Graf (ag...@suse.de) wrote: > On 30.03.2010, at 01:00, Kenni Lund wrote: > > > 2010/3/29 Alexander Graf : > >> > >> On 29.03.2010, at 19:23, Kenni Lund wrote: > >> > > 2010/1/9 Alexander Graf : > >> > >> On 09.01.2010, at 03:45, Ryan C. Underwood wrote: > >> > >>> > >>> I have a multifunction PCI device that I'd like to pass through to > >>> KVM. > >>> In order to do that, I'm reading that the PCI memory region must be > >>> 4K-page > >>> aligned and the PCI memory resources itself must also be exact > >>> multiples > >>> of 4K pages. > >>> > >>> I have added the following on my kernel command line: > >>> reassign_resources reassigndev=08:09.0,08:09.1,08:09.2,08:09.3,08:09.4 > >>> > >>> But I don't know if it has any effect. The resources are still not > >>> sized in 4K pages. Also, this seems to screw up the last device. > >> > >> I submitted a patch to qemu-kvm recently that got rid of that > >> limitation. > >> Please try out if the current git head works for you. > >> > >> Alex-- > > > > I just upgraded to kernel 2.6.32.10 with qemu-kvm 0.12.3 and I still > > get the following error when trying to pass through a dedicated PCI > > USB card: > > > > "Unable to assign device: PCI region 0 at address 0xe9403000 has size > > 0x100, which is not a multiple of 4K > > Error initializing device pci-assign" > > > > Didn't the above patch make it into qemu-kvm? I don't know why, but I > > was under the impression that this was fixed when I upgraded to > > qemu-kvm 0.12.3. > > > It's only in qemu-kvm.git. Maybe it should go into qemu-kvm-0.12.4 if > there > is one > >>> > >>> That would be highly appriciated...with the current USB support in > >>> QEMU, PCI passthrough is the only way to get USB 2.0 support. I've > >>> bought two dedicated PCI USB cards for this, but none of them works > >>> due to the above limitation. > >>> > >>> Perhaps a developer can comment on this? Are there any plans on > >>> including this patch in the stable releases in the near future? > >> > >> Please first try out to build the current git snapshot of qemu-kvm. If it > >> works properly for you then I agree that we should take this into > >> 0.12-stable. > >> > >> I wrote the support for a card that still didn't work even with this > >> patch. So having someone say it makes things work for him is definitely a > >> must :-). > > > > Sure, I have compiled the current git snapshot and performed some > > tests...It's at least mostly working, so I'm a bit unsure if this is a > > bug related to this or to something else. > > Chris, any idea on this? Looks like something's going wrong with function > assignment. Hmm, one thing that sticks out to me is the debug port. Kenni, can you post full dmesg on both host and guest, nothing is obviously broken (and in fact the guest should never "see" the debug port). thanks, -chris -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
On 30.03.2010, at 01:00, Kenni Lund wrote: > 2010/3/29 Alexander Graf : >> >> On 29.03.2010, at 19:23, Kenni Lund wrote: >> > 2010/1/9 Alexander Graf : >> >> On 09.01.2010, at 03:45, Ryan C. Underwood wrote: >> >>> >>> I have a multifunction PCI device that I'd like to pass through to KVM. >>> In order to do that, I'm reading that the PCI memory region must be >>> 4K-page >>> aligned and the PCI memory resources itself must also be exact multiples >>> of 4K pages. >>> >>> I have added the following on my kernel command line: >>> reassign_resources reassigndev=08:09.0,08:09.1,08:09.2,08:09.3,08:09.4 >>> >>> But I don't know if it has any effect. The resources are still not >>> sized in 4K pages. Also, this seems to screw up the last device. >> >> I submitted a patch to qemu-kvm recently that got rid of that limitation. >> Please try out if the current git head works for you. >> >> Alex-- > > I just upgraded to kernel 2.6.32.10 with qemu-kvm 0.12.3 and I still > get the following error when trying to pass through a dedicated PCI > USB card: > > "Unable to assign device: PCI region 0 at address 0xe9403000 has size > 0x100, which is not a multiple of 4K > Error initializing device pci-assign" > > Didn't the above patch make it into qemu-kvm? I don't know why, but I > was under the impression that this was fixed when I upgraded to > qemu-kvm 0.12.3. > It's only in qemu-kvm.git. Maybe it should go into qemu-kvm-0.12.4 if there is one >>> >>> That would be highly appriciated...with the current USB support in >>> QEMU, PCI passthrough is the only way to get USB 2.0 support. I've >>> bought two dedicated PCI USB cards for this, but none of them works >>> due to the above limitation. >>> >>> Perhaps a developer can comment on this? Are there any plans on >>> including this patch in the stable releases in the near future? >> >> Please first try out to build the current git snapshot of qemu-kvm. If it >> works properly for you then I agree that we should take this into >> 0.12-stable. >> >> I wrote the support for a card that still didn't work even with this patch. >> So having someone say it makes things work for him is definitely a must :-). > > Sure, I have compiled the current git snapshot and performed some > tests...It's at least mostly working, so I'm a bit unsure if this is a > bug related to this or to something else. Chris, any idea on this? Looks like something's going wrong with function assignment. Alex-- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
2010/3/29 Alexander Graf : > > On 29.03.2010, at 19:23, Kenni Lund wrote: > 2010/1/9 Alexander Graf : > > On 09.01.2010, at 03:45, Ryan C. Underwood wrote: > >> >> I have a multifunction PCI device that I'd like to pass through to KVM. >> In order to do that, I'm reading that the PCI memory region must be >> 4K-page >> aligned and the PCI memory resources itself must also be exact multiples >> of 4K pages. >> >> I have added the following on my kernel command line: >> reassign_resources reassigndev=08:09.0,08:09.1,08:09.2,08:09.3,08:09.4 >> >> But I don't know if it has any effect. The resources are still not >> sized in 4K pages. Also, this seems to screw up the last device. > > I submitted a patch to qemu-kvm recently that got rid of that limitation. > Please try out if the current git head works for you. > > Alex-- I just upgraded to kernel 2.6.32.10 with qemu-kvm 0.12.3 and I still get the following error when trying to pass through a dedicated PCI USB card: "Unable to assign device: PCI region 0 at address 0xe9403000 has size 0x100, which is not a multiple of 4K Error initializing device pci-assign" Didn't the above patch make it into qemu-kvm? I don't know why, but I was under the impression that this was fixed when I upgraded to qemu-kvm 0.12.3. >>> It's only in qemu-kvm.git. Maybe it should go into qemu-kvm-0.12.4 if there >>> is one >> >> That would be highly appriciated...with the current USB support in >> QEMU, PCI passthrough is the only way to get USB 2.0 support. I've >> bought two dedicated PCI USB cards for this, but none of them works >> due to the above limitation. >> >> Perhaps a developer can comment on this? Are there any plans on >> including this patch in the stable releases in the near future? > > Please first try out to build the current git snapshot of qemu-kvm. If it > works properly for you then I agree that we should take this into 0.12-stable. > > I wrote the support for a card that still didn't work even with this patch. > So having someone say it makes things work for him is definitely a must :-). Sure, I have compiled the current git snapshot and performed some tests...It's at least mostly working, so I'm a bit unsure if this is a bug related to this or to something else. Here's my test results on trying to passthrough a PCI USB card (I've copy-pasted the text below into http://pastebin.com/8RJE36wG in case formatting is lost below): qemu-kvm complete command line: qemu-kvm -usbdevice tablet -net nic,macaddr=52:54:00:00:00:01,model=virtio -net tap,ifname=tap0 -vnc :1 -smp 2 -m 2048 -cdrom /data/server/Linux/mythbuntu-9.10-desktop-amd64.iso -drive file=/data/virtualization/01_Mythbuntu.img,if=virtio,boot=on -boot c -localtime -daemonize -pcidevice host=02:01.0 -pcidevice host=02:01.1 -pcidevice host=02:01.2 -pcidevice host=02:01.3 -monitor unix:/var/run/kvm/01.socket,server,nowait -k da uname -a on host Linux mediaserver 2.6.32-ARCH #1 SMP PREEMPT Fri Mar 26 02:03:53 CET 2010 x86_64 Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz GenuineIntel GNU/Linux Exact kernel version is 2.6.32.10. lspci -v on host, only for the USB PCI card: 02:01.0 USB Controller: ALi Corporation USB 1.1 Controller (rev 03) (prog-if 10 [OHCI]) Subsystem: ALi Corporation ASRock 939Dual-SATA2 Motherboard Flags: bus master, 66MHz, medium devsel, latency 32, IRQ 18 Memory at e940 (32-bit, non-prefetchable) [size=4K] Capabilities: [60] Power Management version 2 Kernel driver in use: pci-stub Kernel modules: ohci-hcd 02:01.1 USB Controller: ALi Corporation USB 1.1 Controller (rev 03) (prog-if 10 [OHCI]) Subsystem: ALi Corporation ASRock 939Dual-SATA2 Motherboard Flags: bus master, 66MHz, medium devsel, latency 32, IRQ 16 Memory at e9401000 (32-bit, non-prefetchable) [size=4K] Capabilities: [60] Power Management version 2 Kernel driver in use: pci-stub Kernel modules: ohci-hcd 02:01.2 USB Controller: ALi Corporation USB 1.1 Controller (rev 03) (prog-if 10 [OHCI]) Subsystem: ALi Corporation ASRock 939Dual-SATA2 Motherboard Flags: bus master, 66MHz, medium devsel, latency 32, IRQ 20 Memory at e9402000 (32-bit, non-prefetchable) [size=4K] Capabilities: [60] Power Management version 2 Kernel driver in use: pci-stub Kernel modules: ohci-hcd 02:01.3 USB Controller: ALi Corporation USB 2.0 Controller (rev 01) (prog-if 20 [EHCI]) Subsystem: Device 2020: Flags: bus master, 66MHz, medium devsel, latency 32, IRQ 19 Memory at e9403000 (32-bit, non-prefetchable) [size=256] Capabilities: [50] Power Management version 2 Capabilities: [58] Debug port: BAR=1 offset=0090 Kernel driver in use: pci-stub Kernel modules: ehci-hcd -
Re: PCI passthrough resource remapping
On 29.03.2010, at 19:23, Kenni Lund wrote: >>> 2010/1/9 Alexander Graf : On 09.01.2010, at 03:45, Ryan C. Underwood wrote: > > I have a multifunction PCI device that I'd like to pass through to KVM. > In order to do that, I'm reading that the PCI memory region must be > 4K-page > aligned and the PCI memory resources itself must also be exact multiples > of 4K pages. > > I have added the following on my kernel command line: > reassign_resources reassigndev=08:09.0,08:09.1,08:09.2,08:09.3,08:09.4 > > But I don't know if it has any effect. The resources are still not > sized in 4K pages. Also, this seems to screw up the last device. I submitted a patch to qemu-kvm recently that got rid of that limitation. Please try out if the current git head works for you. Alex-- >>> >>> I just upgraded to kernel 2.6.32.10 with qemu-kvm 0.12.3 and I still >>> get the following error when trying to pass through a dedicated PCI >>> USB card: >>> >>> "Unable to assign device: PCI region 0 at address 0xe9403000 has size >>> 0x100, which is not a multiple of 4K >>> Error initializing device pci-assign" >>> >>> Didn't the above patch make it into qemu-kvm? I don't know why, but I >>> was under the impression that this was fixed when I upgraded to >>> qemu-kvm 0.12.3. >>> >> It's only in qemu-kvm.git. Maybe it should go into qemu-kvm-0.12.4 if there >> is one > > That would be highly appriciated...with the current USB support in > QEMU, PCI passthrough is the only way to get USB 2.0 support. I've > bought two dedicated PCI USB cards for this, but none of them works > due to the above limitation. > > Perhaps a developer can comment on this? Are there any plans on > including this patch in the stable releases in the near future? Please first try out to build the current git snapshot of qemu-kvm. If it works properly for you then I agree that we should take this into 0.12-stable. I wrote the support for a card that still didn't work even with this patch. So having someone say it makes things work for him is definitely a must :-). Alex-- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
>> 2010/1/9 Alexander Graf : >>> >>> On 09.01.2010, at 03:45, Ryan C. Underwood wrote: >>> I have a multifunction PCI device that I'd like to pass through to KVM. In order to do that, I'm reading that the PCI memory region must be 4K-page aligned and the PCI memory resources itself must also be exact multiples of 4K pages. I have added the following on my kernel command line: reassign_resources reassigndev=08:09.0,08:09.1,08:09.2,08:09.3,08:09.4 But I don't know if it has any effect. The resources are still not sized in 4K pages. Also, this seems to screw up the last device. >>> >>> I submitted a patch to qemu-kvm recently that got rid of that limitation. >>> Please try out if the current git head works for you. >>> >>> Alex-- >> >> I just upgraded to kernel 2.6.32.10 with qemu-kvm 0.12.3 and I still >> get the following error when trying to pass through a dedicated PCI >> USB card: >> >> "Unable to assign device: PCI region 0 at address 0xe9403000 has size >> 0x100, which is not a multiple of 4K >> Error initializing device pci-assign" >> >> Didn't the above patch make it into qemu-kvm? I don't know why, but I >> was under the impression that this was fixed when I upgraded to >> qemu-kvm 0.12.3. >> > It's only in qemu-kvm.git. Maybe it should go into qemu-kvm-0.12.4 if there > is one That would be highly appriciated...with the current USB support in QEMU, PCI passthrough is the only way to get USB 2.0 support. I've bought two dedicated PCI USB cards for this, but none of them works due to the above limitation. Perhaps a developer can comment on this? Are there any plans on including this patch in the stable releases in the near future? Thanks :) Best Regards Kenni Lund -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
It's only in qemu-kvm.git. Maybe it should go into qemu-kvm-0.12.4 if there is one Sent from my iPhone On Mar 25, 2010, at 9:37 PM, Kenni Lund wrote: 2010/1/9 Alexander Graf : On 09.01.2010, at 03:45, Ryan C. Underwood wrote: I have a multifunction PCI device that I'd like to pass through to KVM. In order to do that, I'm reading that the PCI memory region must be 4K-page aligned and the PCI memory resources itself must also be exact multiples of 4K pages. I have added the following on my kernel command line: reassign_resources reassigndev=08:09.0,08:09.1,08:09.2,08:09.3,08:09.4 But I don't know if it has any effect. The resources are still not sized in 4K pages. Also, this seems to screw up the last device. I submitted a patch to qemu-kvm recently that got rid of that limitation. Please try out if the current git head works for you. Alex-- I just upgraded to kernel 2.6.32.10 with qemu-kvm 0.12.3 and I still get the following error when trying to pass through a dedicated PCI USB card: "Unable to assign device: PCI region 0 at address 0xe9403000 has size 0x100, which is not a multiple of 4K Error initializing device pci-assign" Didn't the above patch make it into qemu-kvm? I don't know why, but I was under the impression that this was fixed when I upgraded to qemu-kvm 0.12.3. Thanks Best Regards Kenni -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
2010/1/9 Alexander Graf : > > On 09.01.2010, at 03:45, Ryan C. Underwood wrote: > >> >> I have a multifunction PCI device that I'd like to pass through to KVM. >> In order to do that, I'm reading that the PCI memory region must be 4K-page >> aligned and the PCI memory resources itself must also be exact multiples >> of 4K pages. >> >> I have added the following on my kernel command line: >> reassign_resources reassigndev=08:09.0,08:09.1,08:09.2,08:09.3,08:09.4 >> >> But I don't know if it has any effect. The resources are still not >> sized in 4K pages. Also, this seems to screw up the last device. > > I submitted a patch to qemu-kvm recently that got rid of that limitation. > Please try out if the current git head works for you. > > Alex-- I just upgraded to kernel 2.6.32.10 with qemu-kvm 0.12.3 and I still get the following error when trying to pass through a dedicated PCI USB card: "Unable to assign device: PCI region 0 at address 0xe9403000 has size 0x100, which is not a multiple of 4K Error initializing device pci-assign" Didn't the above patch make it into qemu-kvm? I don't know why, but I was under the impression that this was fixed when I upgraded to qemu-kvm 0.12.3. Thanks Best Regards Kenni -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
On 01/14/2010 09:34 PM, Ryan C. Underwood wrote: On Thu, Jan 14, 2010 at 09:09:32PM +0200, Avi Kivity wrote: PCI cards can access system memory directly. If you assign a card to a guest, the guest will program the card to transfer data to system memory using guest addresses; since guest addresses don't correspond to host addresses, memory corruption will ensue. I see, so the only way to fix this would be either with a special guest driver for the device that does not perform DMA, or if that is impossible (due to no docs), to trap and rewrite any command writes to the device's MMIO region that reference a DMA write target buffer. Forgive my ignorance, but is it possible that the latter is already possible with qemu-kvm (somewhat like hardware memory breakpoints in Soft-ICE)? Yes, you can easily trap mmio writes to a device, and in fact kvm does this in some non-default scenarios. If qemu-kvm can be made to break and log on PCI memory accesses, I would then hack around the safety limitations, assuming that's all they are, and analyze the PCI writes one by one to find the cases where a physical address is passed to the card. Then I would perform the IOMMU translation myself in software whenever a physmem address shows up in the command stream. (Somewhat like the security validation of a 3D graphics card command stream in the DRM.) That's definitely doable, but you would need to know exactly how the device does dma. You would also need to lock all pages into memory (mlockall()) and how pages are mapped (/proc/$pid/pagemap?) -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
On 15.01.2010, at 14:11, Pasi Kärkkäinen wrote: > On Thu, Jan 14, 2010 at 12:31:32PM -0600, Ryan C. Underwood wrote: >> >> >> On Thu, Jan 14, 2010 at 05:54:51PM +0200, Avi Kivity wrote: >>> On 01/14/2010 05:47 PM, Michael S. Tsirkin wrote: > Michael, I think 'DisINTx-' means the device is not PCI 2.3 compliant? No it doesn't, just that interrupt disable bit is not set. >>> >>> Thanks. Ryan, while kvm doesn't support assigning a device with >>> shared interrupts now, in the future it will likely be possible to >>> share it. You'll still need an iommu. >> >> No IOMMU on this machine and this is all integrated hardware. >> >> This IOMMU requirement seems so strange. I used to pass through PCI >> devices ages ago when using the DOSEMU emulator. It emulated PCI BIOS >> functions and mapped the PCI config space and memory regions into the >> emulator process. The device interrupt was grabbed and handled in the >> emulator's kernel support, waking up the emulator when an interrupt came >> in. >> >> I don't really know anything about kvm internals, but I'd like to >> understand more about the particulars of the IOMMU requirement if you >> don't mind. >> > > Xen supports PCI passthrough to PV guests without IOMMU. This can create > security problems, since the guests get DMA access to physical hardware, > but that's usually OK in the situations where you want to use PCI > passthrough on your desktop or on your development box. That's why there way PV support for DMA in KVM too, but it turned out to be rather unmaintained and hard to detect if it actually works. Because if the guest then just didn't use the PV parts to remap its DMA regions, your PCI card ended up writing into random host memory regions. WIthout you knowing. Xen doesn't have that problem as badly as we do, because it can guarantee that a PV guest is PV aware. On KVM PV is an optional add-in. All guests start off being fully virtualized. So we voted for dropping PV DMA support in KVM and just went with the IOMMU only approach. In the long run that's a pretty straight-forward hardware requirement. And if you're using KVM you're used to hardware requirements already anyways ;-). Alex-- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
On Thu, Jan 14, 2010 at 12:31:32PM -0600, Ryan C. Underwood wrote: > > > On Thu, Jan 14, 2010 at 05:54:51PM +0200, Avi Kivity wrote: > > On 01/14/2010 05:47 PM, Michael S. Tsirkin wrote: > > > > > >>Michael, I think 'DisINTx-' means the device is not PCI 2.3 compliant? > > >No it doesn't, just that interrupt disable bit is not set. > > > > Thanks. Ryan, while kvm doesn't support assigning a device with > > shared interrupts now, in the future it will likely be possible to > > share it. You'll still need an iommu. > > No IOMMU on this machine and this is all integrated hardware. > > This IOMMU requirement seems so strange. I used to pass through PCI > devices ages ago when using the DOSEMU emulator. It emulated PCI BIOS > functions and mapped the PCI config space and memory regions into the > emulator process. The device interrupt was grabbed and handled in the > emulator's kernel support, waking up the emulator when an interrupt came > in. > > I don't really know anything about kvm internals, but I'd like to > understand more about the particulars of the IOMMU requirement if you > don't mind. > Xen supports PCI passthrough to PV guests without IOMMU. This can create security problems, since the guests get DMA access to physical hardware, but that's usually OK in the situations where you want to use PCI passthrough on your desktop or on your development box. -- Pasi -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
On Thu, Jan 14, 2010 at 09:09:32PM +0200, Avi Kivity wrote: > > PCI cards can access system memory directly. If you assign a card > to a guest, the guest will program the card to transfer data to > system memory using guest addresses; since guest addresses don't > correspond to host addresses, memory corruption will ensue. I see, so the only way to fix this would be either with a special guest driver for the device that does not perform DMA, or if that is impossible (due to no docs), to trap and rewrite any command writes to the device's MMIO region that reference a DMA write target buffer. Forgive my ignorance, but is it possible that the latter is already possible with qemu-kvm (somewhat like hardware memory breakpoints in Soft-ICE)? If qemu-kvm can be made to break and log on PCI memory accesses, I would then hack around the safety limitations, assuming that's all they are, and analyze the PCI writes one by one to find the cases where a physical address is passed to the card. Then I would perform the IOMMU translation myself in software whenever a physmem address shows up in the command stream. (Somewhat like the security validation of a 3D graphics card command stream in the DRM.) > Not sure about your experience with DOSEMU; maybe these devices were > not dma capable. Well, more likely the MS-DOS drivers were simply not using PCI DMA transfers, since there is very little point to add such complexity to the driver in a non-multitasking OS. Of course that makes perfect sense now! -- Ryan C. Underwood, -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
On 01/14/2010 08:31 PM, Ryan C. Underwood wrote: On Thu, Jan 14, 2010 at 05:54:51PM +0200, Avi Kivity wrote: On 01/14/2010 05:47 PM, Michael S. Tsirkin wrote: Michael, I think 'DisINTx-' means the device is not PCI 2.3 compliant? No it doesn't, just that interrupt disable bit is not set. Thanks. Ryan, while kvm doesn't support assigning a device with shared interrupts now, in the future it will likely be possible to share it. You'll still need an iommu. No IOMMU on this machine and this is all integrated hardware. This IOMMU requirement seems so strange. I used to pass through PCI devices ages ago when using the DOSEMU emulator. It emulated PCI BIOS functions and mapped the PCI config space and memory regions into the emulator process. The device interrupt was grabbed and handled in the emulator's kernel support, waking up the emulator when an interrupt came in. I don't really know anything about kvm internals, but I'd like to understand more about the particulars of the IOMMU requirement if you don't mind. PCI cards can access system memory directly. If you assign a card to a guest, the guest will program the card to transfer data to system memory using guest addresses; since guest addresses don't correspond to host addresses, memory corruption will ensue. Not sure about your experience with DOSEMU; maybe these devices were not dma capable. -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
On Thu, Jan 14, 2010 at 05:54:51PM +0200, Avi Kivity wrote: > On 01/14/2010 05:47 PM, Michael S. Tsirkin wrote: > > > >>Michael, I think 'DisINTx-' means the device is not PCI 2.3 compliant? > >No it doesn't, just that interrupt disable bit is not set. > > Thanks. Ryan, while kvm doesn't support assigning a device with > shared interrupts now, in the future it will likely be possible to > share it. You'll still need an iommu. No IOMMU on this machine and this is all integrated hardware. This IOMMU requirement seems so strange. I used to pass through PCI devices ages ago when using the DOSEMU emulator. It emulated PCI BIOS functions and mapped the PCI config space and memory regions into the emulator process. The device interrupt was grabbed and handled in the emulator's kernel support, waking up the emulator when an interrupt came in. I don't really know anything about kvm internals, but I'd like to understand more about the particulars of the IOMMU requirement if you don't mind. -- Ryan C. Underwood, -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
On 01/14/2010 05:47 PM, Michael S. Tsirkin wrote: Michael, I think 'DisINTx-' means the device is not PCI 2.3 compliant? No it doesn't, just that interrupt disable bit is not set. Thanks. Ryan, while kvm doesn't support assigning a device with shared interrupts now, in the future it will likely be possible to share it. You'll still need an iommu. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
On Thu, Jan 14, 2010 at 05:34:42PM +0200, Avi Kivity wrote: > On 01/14/2010 05:26 PM, Ryan C. Underwood wrote: >> On Thu, Jan 14, 2010 at 03:59:53PM +0200, Avi Kivity wrote: >> >>> > > >Also, just for further complication, the Ricoh chip does not support > >MSI and shares an IRQ on the system board with the USB host controller. > >I have rebound the USB host controller to pci-stub, but I'm not sure if > >that totally takes care of the IRQ-sharing-without-MSI issue. >>> > > Can you post lspci -vv output for that card? If it is pci 2.3 >>> > compliant we might be able to handle the sharing. >>> >> It claims to be PCI 3.0 compliant. But I don't see any mention MSI >> anywhere in the specification: >> http://www.aeneas.com.cn/PDF/Ricoh/2005/R5C832E1%5B1%5D.00.pdf >> >> Attached is lspci -vv, thanks. >> >> >> >> 08:09.0 FireWire (IEEE 1394): Ricoh Co Ltd R5C832 IEEE 1394 Controller (rev >> 05) (prog-if 10) >> Subsystem: Hewlett-Packard Company Device 30cd >> Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- >> Stepping- SERR- FastB2B- DisINTx- >> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- >> DEVSEL=medium>TAbort-SERR-> Latency: 64 (500ns min, 1000ns max) >> Interrupt: pin A routed to IRQ 16 >> Region 0: [virtual] Memory at f440 (32-bit, non-prefetchable) >> [size=2K] >> Capabilities: >> Kernel driver in use: pci-stub >> Kernel modules: firewire-ohci, ohci1394 >> >> > > Michael, I think 'DisINTx-' means the device is not PCI 2.3 compliant? No it doesn't, just that interrupt disable bit is not set. >> 08:09.4 System peripheral: Ricoh Co Ltd xD-Picture Card Controller (rev 12) >> Subsystem: Hewlett-Packard Company Device 30cd >> Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ >> Stepping- SERR+ FastB2B- DisINTx+ >> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- >> DEVSEL=medium>TAbort-SERR-> Latency: 248, Cache Line Size: 1020 bytes >> Interrupt: pin B routed to IRQ 10 >> Region 0: Memory at f4401400 (32-bit, non-prefetchable) [size=256] >> Capabilities: >> Kernel driver in use: pci-stub >> > > But another function has this feature? Another function has this bit set. > -- > error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
On 01/14/2010 05:26 PM, Ryan C. Underwood wrote: On Thu, Jan 14, 2010 at 03:59:53PM +0200, Avi Kivity wrote: > > >Also, just for further complication, the Ricoh chip does not support > >MSI and shares an IRQ on the system board with the USB host controller. > >I have rebound the USB host controller to pci-stub, but I'm not sure if > >that totally takes care of the IRQ-sharing-without-MSI issue. > > Can you post lspci -vv output for that card? If it is pci 2.3 > compliant we might be able to handle the sharing. It claims to be PCI 3.0 compliant. But I don't see any mention MSI anywhere in the specification: http://www.aeneas.com.cn/PDF/Ricoh/2005/R5C832E1%5B1%5D.00.pdf Attached is lspci -vv, thanks. 08:09.0 FireWire (IEEE 1394): Ricoh Co Ltd R5C832 IEEE 1394 Controller (rev 05) (prog-if 10) Subsystem: Hewlett-Packard Company Device 30cd Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium>TAbort-SERR- Kernel driver in use: pci-stub Kernel modules: firewire-ohci, ohci1394 Michael, I think 'DisINTx-' means the device is not PCI 2.3 compliant? 08:09.4 System peripheral: Ricoh Co Ltd xD-Picture Card Controller (rev 12) Subsystem: Hewlett-Packard Company Device 30cd Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium>TAbort-SERR- Kernel driver in use: pci-stub But another function has this feature? -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
On Thu, Jan 14, 2010 at 03:59:53PM +0200, Avi Kivity wrote: > > >Also, just for further complication, the Ricoh chip does not support > >MSI and shares an IRQ on the system board with the USB host controller. > >I have rebound the USB host controller to pci-stub, but I'm not sure if > >that totally takes care of the IRQ-sharing-without-MSI issue. > > Can you post lspci -vv output for that card? If it is pci 2.3 > compliant we might be able to handle the sharing. It claims to be PCI 3.0 compliant. But I don't see any mention MSI anywhere in the specification: http://www.aeneas.com.cn/PDF/Ricoh/2005/R5C832E1%5B1%5D.00.pdf Attached is lspci -vv, thanks. -- Ryan C. Underwood, 00:00.0 Host bridge: Intel Corporation Mobile PM965/GM965/GL960 Memory Controller Hub (rev 0c) Subsystem: Hewlett-Packard Company Device 30cd Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- SERR- Kernel driver in use: agpgart-intel Kernel modules: intel-agp 00:02.0 VGA compatible controller: Intel Corporation Mobile GM965/GL960 Integrated Graphics Controller (rev 0c) Subsystem: Hewlett-Packard Company Device 30cd Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- SERR- Kernel driver in use: i915 Kernel modules: i915 00:02.1 Display controller: Intel Corporation Mobile GM965/GL960 Integrated Graphics Controller (rev 0c) Subsystem: Hewlett-Packard Company Device 30cd Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- SERR- 00:1a.0 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #4 (rev 03) Subsystem: Hewlett-Packard Company Device 30cd Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- SERR- TAbort- SERR- TAbort- SERR- Kernel driver in use: ehci_hcd 00:1b.0 Audio device: Intel Corporation 82801H (ICH8 Family) HD Audio Controller (rev 03) Subsystem: Hewlett-Packard Company Device 30cd Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Kernel driver in use: HDA Intel Kernel modules: snd-hda-intel 00:1c.0 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 1 (rev 03) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- TAbort- Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: Kernel driver in use: pcieport-driver Kernel modules: shpchp 00:1c.1 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 2 (rev 03) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- TAbort- Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: Kernel driver in use: pcieport-driver Kernel modules: shpchp 00:1c.2 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 3 (rev 03) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- TAbort- Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: Kernel driver in use: pcieport-driver Kernel modules: shpchp 00:1c.3 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 4 (rev 03) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- TAbort- Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: Kernel driver in use: pcieport-driver Kernel modules: shpchp 00:1d.0 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #1 (rev 03) Subsystem: Hewlett-Packard Company Device 30cd Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- SERR- TAbort- SERR- TAbort- SERR- TAbort- SERR- Kernel driver in
Re: PCI passthrough resource remapping
On 01/11/2010 12:15 AM, Ryan C. Underwood wrote: I guess I'll run the things I've found by the list to see if I'm off track or not. There is this patch: http://marc.info/?l=xen-devel&m=124748015304566&w=4 which would seem to be related to what I'm doing, trying to pass through a multifunction device (a Ricoh firewire controller which has digital media card reader subfunctions). Then there is this patch: http://patchwork.kernel.org/patch/20195/ Which is related to PCI passthrough without VT-d, but I have no idea if this patch is necessary or sufficient. That effort has been discontinues. Also, just for further complication, the Ricoh chip does not support MSI and shares an IRQ on the system board with the USB host controller. I have rebound the USB host controller to pci-stub, but I'm not sure if that totally takes care of the IRQ-sharing-without-MSI issue. Can you post lspci -vv output for that card? If it is pci 2.3 compliant we might be able to handle the sharing. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
I guess I'll run the things I've found by the list to see if I'm off track or not. There is this patch: http://marc.info/?l=xen-devel&m=124748015304566&w=4 which would seem to be related to what I'm doing, trying to pass through a multifunction device (a Ricoh firewire controller which has digital media card reader subfunctions). Then there is this patch: http://patchwork.kernel.org/patch/20195/ Which is related to PCI passthrough without VT-d, but I have no idea if this patch is necessary or sufficient. Also, just for further complication, the Ricoh chip does not support MSI and shares an IRQ on the system board with the USB host controller. I have rebound the USB host controller to pci-stub, but I'm not sure if that totally takes care of the IRQ-sharing-without-MSI issue. -- Ryan C. Underwood, -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
> > I have a multifunction PCI device that I'd like to pass through to KVM. > > In order to do that, I'm reading that the PCI memory region must be 4K-page > > aligned and the PCI memory resources itself must also be exact multiples > > of 4K pages. > > > > I have added the following on my kernel command line: > > reassign_resources reassigndev=08:09.0,08:09.1,08:09.2,08:09.3,08:09.4 > > > > But I don't know if it has any effect. The resources are still not > > sized in 4K pages. Also, this seems to screw up the last device. > > I submitted a patch to qemu-kvm recently that got rid of that limitation. > Please try out if the current git head works for you. This works around that particular limitation, but now it looks like I have to have VT-D for this to work at all. (It seems so hard to find documentation about PCI passthrough, probably because things are changing all the time.) device: 08:09.0: driver="pci-assign" host="08:09.0" device: 08:09.1: driver="pci-assign" host="08:09.1" device: 08:09.2: driver="pci-assign" host="08:09.2" device: 08:09.3: driver="pci-assign" host="08:09.3" device: 08:09.4: driver="pci-assign" host="08:09.4" PCI region 0 at address 0xf440 has size 0x800, which is not a multiple of 4K. You might experience some performance hit due to that. No IOMMU found. Unable to assign device "08:09.0" Failed to deassign device "08:09.0" : Invalid argument Error initializing device pci-assign -- Ryan C. Underwood, -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough resource remapping
On 09.01.2010, at 03:45, Ryan C. Underwood wrote: > > I have a multifunction PCI device that I'd like to pass through to KVM. > In order to do that, I'm reading that the PCI memory region must be 4K-page > aligned and the PCI memory resources itself must also be exact multiples > of 4K pages. > > I have added the following on my kernel command line: > reassign_resources reassigndev=08:09.0,08:09.1,08:09.2,08:09.3,08:09.4 > > But I don't know if it has any effect. The resources are still not > sized in 4K pages. Also, this seems to screw up the last device. I submitted a patch to qemu-kvm recently that got rid of that limitation. Please try out if the current git head works for you. Alex-- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html