Re: [Xen-devel] Question about the XEN platform pci

[karim.allah.ah...@gmail.com]

On Wed, Apr 13, 2016 at 11:55 AM, Konrad Rzeszutek Wilk
<konrad.w...@oracle.com> wrote:
> On Wed, Apr 13, 2016 at 09:00:18AM +0200, karim.allah.ah...@gmail.com wrote:
>> On Tue, Apr 12, 2016 at 5:45 PM, Konrad Rzeszutek Wilk
>> <konrad.w...@oracle.com> wrote:
>> > On Tue, Apr 12, 2016 at 05:33:47PM +0200, karim.allah.ah...@gmail.com 
>> > wrote:
>> >> The INTx interrupt of this platform device can be used by Xen in HVM case 
>> >> to
>> >> notify the guest of pending events in the event channel. However that's 
>> >> usually
>> >> not used in favor of vector callbacks support in Xen where a vector is 
>> >> injected
>> >> directly to the vCPU bypassing LAPIC.
>> >>
>> >> (that said, the platform-pci driver in linux is actually broken when 
>> >> vector
>> >> callbacks are not used anyway)
>> >
>> > Oh? Is there an report/bug somewhere?
>> >
>>
>> I'm not sure if it's reported as a bug somewhere or not.
>>
>> I've always assumed that INTx is deperecated and vector callbacks are the one
>> that's supported that's why I never tried to fix it, in addition I never 
>> tried
>> to reproduce it I just looked at the code and it seemed a little bit off as
>> explained below.
>>
>> Mainly xenbus_init is called during postcore_initcall which will eventually 
>> try
>> to read a value from XenStore and will get stuck on read_reply at xenbus
>> forever since the platform driver is not probed yet and its INTx interrupt
>> handler is not registered yet which basically means that the guest can not be
>> notified at this moment of any pending event channels and none of the 
>> per-event
>> handlers will ever be invoked (including the XenStore one) and the reply will
>> never be picked up by the kernel.
>
> Argh, well that is not good. Also - thanks for reporting this.
>
> No simple ideas come to my mind on how this can be fixed - unless we
> move the xenbus_init and its driver past the platform-pci init?
>
> Or if platform-pci init code ends up being called earlier?
>

Let me test a few alternatives and send a patch to fix that first.

>>
>> The exact stack where things get stuck during xenbus_init:
>>
>> -xenbus_init
>>  -xs_init
>>   -xs_reset_watches
>>-xenbus_scanf
>> -xenbus_read
>>  -xs_single
>>   -xs_single
>>-xs_talkv
>>
>> > Thanks!
>> >>
>> >> I also think that the grant-table lives on this PCI device MMIO BAR (?!)
>> >
>> > The area may be usurped for grant-table as the OS won't touch that memory
>> > area (it after all belongs to the device).
>> >>
>> >> If you looked at hw/i386/xen/xen_platform.c in QEMU source , you will get 
>> >> a
>> >> general idea what this device is supposed todo (like logging to syslog 
>> >> stuff
>> >> for example).
>> >>
>> >> That said the platform device is really not fully utilized anyway in 
>> >> Linux.
>> >>
>> >> On Tue, Apr 12, 2016 at 4:09 PM, Konrad Rzeszutek Wilk
>> >> <konrad.w...@oracle.com> wrote:
>> >> > On Tue, Apr 12, 2016 at 02:19:48AM +, Wu, Bob wrote:
>> >> >>
>> >> >> Really thanks for your reply.
>> >> >
>> >> > Hey!
>> >> >
>> >> > CC-ing Xen-devel back on. Please do not drop it and please don't
>> >> > top-post.
>> >> >>
>> >> >> Can you explain a little more?
>> >> >
>> >> > I am not sure what you want me to explain. Perhaps if you
>> >> > read http://xenbits.xen.org/docs/unstable/misc/hvm-emulated-unplug.html
>> >> > it may become clearer?
>> >> >
>> >> >> Is the xen platform pci driver the only purpose for telling QEMU  that 
>> >> >> don’t emulate the IDE driver?
>> >> >
>> >> > And network.
>> >> >> I think it can be done by a simple way, but don't need use this huge 
>> >> >> platform driver.
>> >> >
>> >> > ?
>> >> >>
>> >> >> I guess this is for PCI pass through in XEN HVM mode, but don't sure.
>> >> >
>> >> > No.
>> >> >>
>> >> >> Thanks,
>> >> >> Bob
>> >> >>
>> >> >>
>> >> >> -

Re: [Xen-devel] Question about the XEN platform pci

[karim.allah.ah...@gmail.com]

On Wed, Apr 13, 2016 at 9:28 AM, Wu, Bob <bob...@emc.com> wrote:
>
> From: karim.allah.ah...@gmail.com [mailto:karim.allah.ah...@gmail.com]
> Sent: 2016年4月13日 15:00
> To: Konrad Rzeszutek Wilk
> Cc: Wu, Bob; xen-de...@lists.xenproject.org
> Subject: Re: [Xen-devel] Question about the XEN platform pci
>
> On Tue, Apr 12, 2016 at 5:45 PM, Konrad Rzeszutek Wilk 
> <konrad.w...@oracle.com> wrote:
>> On Tue, Apr 12, 2016 at 05:33:47PM +0200, karim.allah.ah...@gmail.com wrote:
>>> The INTx interrupt of this platform device can be used by Xen in HVM
>>> case to notify the guest of pending events in the event channel.
>>> However that's usually not used in favor of vector callbacks support
>>> in Xen where a vector is injected directly to the vCPU bypassing LAPIC.
>>>
>>> (that said, the platform-pci driver in linux is actually broken when
>>> vector callbacks are not used anyway)
>>
>> Oh? Is there an report/bug somewhere?
>>
>
> I'm not sure if it's reported as a bug somewhere or not.
>
> I've always assumed that INTx is deperecated and vector callbacks are the one 
> that's supported that's why I never tried to fix it, in addition I never 
> tried to reproduce it I just looked at the code and it seemed a little bit 
> off as explained below.
>
> Mainly xenbus_init is called during postcore_initcall which will eventually 
> try to read a value from XenStore and will get stuck on read_reply at xenbus 
> forever since the platform driver is not probed yet and its INTx interrupt 
> handler is not registered yet which basically means that the guest can not be 
> notified at this moment of any pending event channels and none of the 
> per-event handlers will ever be invoked (including the XenStore one) and the 
> reply will never be picked up by the kernel.
>
> The exact stack where things get stuck during xenbus_init:
>
> -xenbus_init
>  -xs_init
>   -xs_reset_watches
>-xenbus_scanf
> -xenbus_read
>  -xs_single
>   -xs_single
>-xs_talkv
>
>> Thanks!
>>>
>>> I also think that the grant-table lives on this PCI device MMIO BAR
>>> (?!)
>>
>> The area may be usurped for grant-table as the OS won't touch that
>> memory area (it after all belongs to the device).
>>>
>>> If you looked at hw/i386/xen/xen_platform.c in QEMU source , you will
>>> get a general idea what this device is supposed todo (like logging to
>>> syslog stuff for example).
>>>
>>> >>
>>> >> -Original Message-
>>> >> From: Konrad Rzeszutek Wilk [mailto:konrad.w...@oracle.com]
>>> >> Sent: 2016年4月11日 22:24
>>> >> To: Wu, Bob
>>> >> Cc: xen-devel@lists.xen.org
>>> >> Subject: Re: [Xen-devel] Question about the XEN platform pci
>>> >>
>>> >> On Fri, Apr 08, 2016 at 08:52:08AM +, Wu, Bob wrote:
>>> >> >
>>> >> > Sorry bother, I read the XEN source code recently, and found the
>>> >> > XEN platform PCI code under drivers/xen/platform-pci.c, and I can't 
>>> >> > fully under this driver's affect, can anybody explain a little for me?
>>> >> >
>>> >> > Is the platform PCI driver for PV-split-PCI-driver-model such as the 
>>> >> > pci-frontend/pci-backend? or for PCI pass-through model? Or for other 
>>> >> > purpose?
>>> >> > I saw the xenbus_pcifront_driver/ xenbus_xen_pcibk_driver are 
>>> >> > registered on XENBUS, so I guess the platform-PCI-driver is not for PV 
>>> >> > PCI driver.
>>> >> >
>>> >>
>>> >> It is for the QEMU driver. To tell QEMU to stop emulating the 
>>> >> IDE/network.
>>> >>
>>> >> > Really thank you for your replay.
>>> >> >
>>> >> > Thanks,
>>> >> > Bob
>>> >> >
>
> Hi, Karim,
>
> I still have one more question, can you share your understanding?
> The PCI platform driver just register a pci driver on pci bus, but when the 
> real device attaching occur?
> I mean who trigger the real device attach behavior? The QEMU or others? I 
> there any piece of code can indicate it?

This Xen PCI device is a normal PCI device that's emulated by QEMU
and the kernel will just find it while scanning the PCI bus during the
boot process so the driver will just get probed at some point during
boot = It's not hot-plugged.

(Information about PCI devices can be guessed from scanning the
 host bridge or through ACPI tables)

>
> Thanks,
> Bob



-- 
Karim Allah Ahmed.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Question about the XEN platform pci

[karim.allah.ah...@gmail.com]

On Tue, Apr 12, 2016 at 5:45 PM, Konrad Rzeszutek Wilk
<konrad.w...@oracle.com> wrote:
> On Tue, Apr 12, 2016 at 05:33:47PM +0200, karim.allah.ah...@gmail.com wrote:
>> The INTx interrupt of this platform device can be used by Xen in HVM case to
>> notify the guest of pending events in the event channel. However that's 
>> usually
>> not used in favor of vector callbacks support in Xen where a vector is 
>> injected
>> directly to the vCPU bypassing LAPIC.
>>
>> (that said, the platform-pci driver in linux is actually broken when vector
>> callbacks are not used anyway)
>
> Oh? Is there an report/bug somewhere?
>

I'm not sure if it's reported as a bug somewhere or not.

I've always assumed that INTx is deperecated and vector callbacks are the one
that's supported that's why I never tried to fix it, in addition I never tried
to reproduce it I just looked at the code and it seemed a little bit off as
explained below.

Mainly xenbus_init is called during postcore_initcall which will eventually try
to read a value from XenStore and will get stuck on read_reply at xenbus
forever since the platform driver is not probed yet and its INTx interrupt
handler is not registered yet which basically means that the guest can not be
notified at this moment of any pending event channels and none of the per-event
handlers will ever be invoked (including the XenStore one) and the reply will
never be picked up by the kernel.

The exact stack where things get stuck during xenbus_init:

-xenbus_init
 -xs_init
  -xs_reset_watches
   -xenbus_scanf
-xenbus_read
 -xs_single
  -xs_single
   -xs_talkv

> Thanks!
>>
>> I also think that the grant-table lives on this PCI device MMIO BAR (?!)
>
> The area may be usurped for grant-table as the OS won't touch that memory
> area (it after all belongs to the device).
>>
>> If you looked at hw/i386/xen/xen_platform.c in QEMU source , you will get a
>> general idea what this device is supposed todo (like logging to syslog stuff
>> for example).
>>
>> That said the platform device is really not fully utilized anyway in Linux.
>>
>> On Tue, Apr 12, 2016 at 4:09 PM, Konrad Rzeszutek Wilk
>> <konrad.w...@oracle.com> wrote:
>> > On Tue, Apr 12, 2016 at 02:19:48AM +, Wu, Bob wrote:
>> >>
>> >> Really thanks for your reply.
>> >
>> > Hey!
>> >
>> > CC-ing Xen-devel back on. Please do not drop it and please don't
>> > top-post.
>> >>
>> >> Can you explain a little more?
>> >
>> > I am not sure what you want me to explain. Perhaps if you
>> > read http://xenbits.xen.org/docs/unstable/misc/hvm-emulated-unplug.html
>> > it may become clearer?
>> >
>> >> Is the xen platform pci driver the only purpose for telling QEMU  that 
>> >> don’t emulate the IDE driver?
>> >
>> > And network.
>> >> I think it can be done by a simple way, but don't need use this huge 
>> >> platform driver.
>> >
>> > ?
>> >>
>> >> I guess this is for PCI pass through in XEN HVM mode, but don't sure.
>> >
>> > No.
>> >>
>> >> Thanks,
>> >> Bob
>> >>
>> >>
>> >> -Original Message-
>> >> From: Konrad Rzeszutek Wilk [mailto:konrad.w...@oracle.com]
>> >> Sent: 2016年4月11日 22:24
>> >> To: Wu, Bob
>> >> Cc: xen-devel@lists.xen.org
>> >> Subject: Re: [Xen-devel] Question about the XEN platform pci
>> >>
>> >> On Fri, Apr 08, 2016 at 08:52:08AM +, Wu, Bob wrote:
>> >> >
>> >> > Sorry bother, I read the XEN source code recently, and found the XEN
>> >> > platform PCI code under drivers/xen/platform-pci.c, and I can't fully 
>> >> > under this driver's affect, can anybody explain a little for me?
>> >> >
>> >> > Is the platform PCI driver for PV-split-PCI-driver-model such as the 
>> >> > pci-frontend/pci-backend? or for PCI pass-through model? Or for other 
>> >> > purpose?
>> >> > I saw the xenbus_pcifront_driver/ xenbus_xen_pcibk_driver are 
>> >> > registered on XENBUS, so I guess the platform-PCI-driver is not for PV 
>> >> > PCI driver.
>> >> >
>> >>
>> >> It is for the QEMU driver. To tell QEMU to stop emulating the IDE/network.
>> >>
>> >> > Really thank you for your replay.
>> >> >
>> >> > Thanks,
>> >> > Bob
>> >> >
>> >>
>> >> > ___
>> >> > Xen-devel mailing list
>> >> > Xen-devel@lists.xen.org
>> >> > http://lists.xen.org/xen-devel
>> >>
>> >
>> > ___
>> > Xen-devel mailing list
>> > Xen-devel@lists.xen.org
>> > http://lists.xen.org/xen-devel
>>
>>
>>
>> --
>> Karim Allah Ahmed.



-- 
Karim Allah Ahmed.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Question about the XEN platform pci

[karim.allah.ah...@gmail.com]

The INTx interrupt of this platform device can be used by Xen in HVM case to
notify the guest of pending events in the event channel. However that's usually
not used in favor of vector callbacks support in Xen where a vector is injected
directly to the vCPU bypassing LAPIC.

(that said, the platform-pci driver in linux is actually broken when vector
callbacks are not used anyway)

I also think that the grant-table lives on this PCI device MMIO BAR (?!)

If you looked at hw/i386/xen/xen_platform.c in QEMU source , you will get a
general idea what this device is supposed todo (like logging to syslog stuff
for example).

That said the platform device is really not fully utilized anyway in Linux.

On Tue, Apr 12, 2016 at 4:09 PM, Konrad Rzeszutek Wilk
 wrote:
> On Tue, Apr 12, 2016 at 02:19:48AM +, Wu, Bob wrote:
>>
>> Really thanks for your reply.
>
> Hey!
>
> CC-ing Xen-devel back on. Please do not drop it and please don't
> top-post.
>>
>> Can you explain a little more?
>
> I am not sure what you want me to explain. Perhaps if you
> read http://xenbits.xen.org/docs/unstable/misc/hvm-emulated-unplug.html
> it may become clearer?
>
>> Is the xen platform pci driver the only purpose for telling QEMU  that don’t 
>> emulate the IDE driver?
>
> And network.
>> I think it can be done by a simple way, but don't need use this huge 
>> platform driver.
>
> ?
>>
>> I guess this is for PCI pass through in XEN HVM mode, but don't sure.
>
> No.
>>
>> Thanks,
>> Bob
>>
>>
>> -Original Message-
>> From: Konrad Rzeszutek Wilk [mailto:konrad.w...@oracle.com]
>> Sent: 2016年4月11日 22:24
>> To: Wu, Bob
>> Cc: xen-devel@lists.xen.org
>> Subject: Re: [Xen-devel] Question about the XEN platform pci
>>
>> On Fri, Apr 08, 2016 at 08:52:08AM +, Wu, Bob wrote:
>> >
>> > Sorry bother, I read the XEN source code recently, and found the XEN
>> > platform PCI code under drivers/xen/platform-pci.c, and I can't fully 
>> > under this driver's affect, can anybody explain a little for me?
>> >
>> > Is the platform PCI driver for PV-split-PCI-driver-model such as the 
>> > pci-frontend/pci-backend? or for PCI pass-through model? Or for other 
>> > purpose?
>> > I saw the xenbus_pcifront_driver/ xenbus_xen_pcibk_driver are registered 
>> > on XENBUS, so I guess the platform-PCI-driver is not for PV PCI driver.
>> >
>>
>> It is for the QEMU driver. To tell QEMU to stop emulating the IDE/network.
>>
>> > Really thank you for your replay.
>> >
>> > Thanks,
>> > Bob
>> >
>>
>> > ___
>> > Xen-devel mailing list
>> > Xen-devel@lists.xen.org
>> > http://lists.xen.org/xen-devel
>>
>
> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel



-- 
Karim Allah Ahmed.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel