Re: [Xen-devel] [Draft F] Xen on ARM vITS Handling
On 16/06/15 16:10, Ian Campbell wrote: This could happen when the device is not quiescent. We had this issue on the vexpress at boot time when the network card was trying to send an interrupt before DOM0 is setup. I don't fully understand the issue you are trying to describe, but do you want to propose a change to the spec? I actually don't know how to modify it. So it's an open question. For SPI too, or just for LPI? Only LPI. How is it fixed for SPI? We queue the SPI but don't inject it (gic_raise_guest_irq) to the guest. The injection will be done when the guest enable the IRQ. We can't use it for LPIs as the mapping vLPI - pLPI may not yet exists. vgic_vcpu_inject_irq doesn't queue the interrupt if a VCPU is down. I think this is because the state of the VCPU wouldn't be correct. The process would be something like: - Creation of the domain = All vCPUs are down - Device is assigned to the guest = Enable physical LPIs * physical LPI is received * = Will be ignored and not EOIed (VCPU0 is down) = The LPI will never fired again during the guest life - Domain is started by the toolstack = VCPU0 is online Is it sufficient to queue interrupts even for VCPUs which are down? How does the lack of a vITT entry when this interrupt occurred affect this? Well, in this case we don't know on which vLPI we have to inject it. But as said above, I guess we don't care if we ensure that the device can't send an event (by ensuring that the device doesn't know the GITS_TRANSLATER address is). Yes, I think that works. I'm not sure where to spell that out though. We want to find a way to avoid the PCI device to send an event. I see multiple possibility to do it: 1) Mask the event MSI-X allow to mask a specific event. There is a similar feature for MSI but it's optional. 2) Compose the MSI message only when the vLPI is enabled. That require Xen/PCI-back to delay the write in the config space. None of this solution satisfy me. But I don't have much knowledge on how PCI-passthrough works on Xen. Regards, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [Draft F] Xen on ARM vITS Handling
On Tue, 2015-06-16 at 20:20 +0530, Vijay Kilari wrote:
On Thu, Jun 11, 2015 at 3:10 PM, Ian Campbell ian.campb...@citrix.com wrote:
Draft F follows. Also at:
http://xenbits.xen.org/people/ianc/vits/draftF.{pdf,html}
## Per-domain `struct pending_irq` for `vLPI`s
Internally Xen uses a `struct pending_irq` to track the status of any
pending virtual IRQ, including a virtual LPI.
Upon domain creation an array of such `struct pending_irq`'s will be
allocated to cover the range `8192..nr_lpis` (for the number of LPIs
which the guest is configured with) and a pointer this array will be
stored in the `struct domain`. The function `irq_to_pending` will be
modified to lookup interupts in the LPI range in this array.
.
nr_lpis can be large if more devices are assigned to domain.
As I was suggesting on #xenarm chat, is it ok to use RB-tree instead of array?
what should be value for nr_lpis?
It should be user configurable and default to the sum of the number of
events on all devices at start of day.
I think this removes the need for it to be an R-B tree, an array is
tolerable here.
Adding an R-B tree not only has a memory overhead, but it then needs
more complex management when inserting, searching, etc.
Ian.
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [Draft F] Xen on ARM vITS Handling
On Thu, Jun 11, 2015 at 3:10 PM, Ian Campbell ian.campb...@citrix.com wrote:
Draft F follows. Also at:
http://xenbits.xen.org/people/ianc/vits/draftF.{pdf,html}
## Per-domain `struct pending_irq` for `vLPI`s
Internally Xen uses a `struct pending_irq` to track the status of any
pending virtual IRQ, including a virtual LPI.
Upon domain creation an array of such `struct pending_irq`'s will be
allocated to cover the range `8192..nr_lpis` (for the number of LPIs
which the guest is configured with) and a pointer this array will be
stored in the `struct domain`. The function `irq_to_pending` will be
modified to lookup interupts in the LPI range in this array.
.
nr_lpis can be large if more devices are assigned to domain.
As I was suggesting on #xenarm chat, is it ok to use RB-tree instead of array?
what should be value for nr_lpis?
Regards
Vijay
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [Draft F] Xen on ARM vITS Handling
On Fri, 2015-06-12 at 13:55 -0400, Julien Grall wrote: On 12/06/2015 10:24, Ian Campbell wrote: On Fri, 2015-06-12 at 09:32 -0400, Julien Grall wrote: On 12/06/2015 09:16, Ian Campbell wrote: On Fri, 2015-06-12 at 09:09 -0400, Julien Grall wrote: Hi Ian, On 12/06/2015 04:52, Ian Campbell wrote: On Fri, 2015-06-12 at 14:07 +0530, Vijay Kilari wrote: So pLPIs must be routed at device assignment time because in the vLPI configuration table trap there is no mapping back to a single pLPI. I just remembered the exact reason that made use to differ SPI enabling. I can't parse this sentence, differ how? deferring sorry. When the device is assigned, the domain VCPUs are still down (even VCPU0). If we receive an interrupt before the VCPU0 is unpaused, the interrupt will be lost. Same if the interrupt is not yet configured (i.e before the vITS setup correctly the table) with your proposal. Is this any different to booting with the ITT not setup? I don't understand your question. During boot the ITT is not configured and a spurious event will go undelivered to an LPI then too, even on native. It's different. In the case of native, the event is not recorded by the ITS so it can fire up again later when the ITT is setup (for instance because the device has been reset). With your proposal, the interrupt will go in Active state (from the CPU POV see 4.8.3) in the GIC. If you don't EOI it, it will never fire again when the guest has setup the vITT. Our other option is to EIO it, which has its own issues (namely possible interrupt storms). For PCI passthrough it may be that we can mitigate that some what by resetting the device ourselves or something. Although, the device won't know where it has to write the event ID (i.e in GITS_TRANSLATER) because it should not have been configured. So it will get ignored, right? I think so, yes. This could happen when the device is not quiescent. We had this issue on the vexpress at boot time when the network card was trying to send an interrupt before DOM0 is setup. I don't fully understand the issue you are trying to describe, but do you want to propose a change to the spec? I actually don't know how to modify it. So it's an open question. For SPI too, or just for LPI? Only LPI. How is it fixed for SPI? vgic_vcpu_inject_irq doesn't queue the interrupt if a VCPU is down. I think this is because the state of the VCPU wouldn't be correct. The process would be something like: - Creation of the domain = All vCPUs are down - Device is assigned to the guest = Enable physical LPIs * physical LPI is received * = Will be ignored and not EOIed (VCPU0 is down) = The LPI will never fired again during the guest life - Domain is started by the toolstack = VCPU0 is online Is it sufficient to queue interrupts even for VCPUs which are down? How does the lack of a vITT entry when this interrupt occurred affect this? Well, in this case we don't know on which vLPI we have to inject it. But as said above, I guess we don't care if we ensure that the device can't send an event (by ensuring that the device doesn't know the GITS_TRANSLATER address is). Yes, I think that works. I'm not sure where to spell that out though. Ian. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [Draft F] Xen on ARM vITS Handling
On Thu, Jun 11, 2015 at 3:10 PM, Ian Campbell ian.campb...@citrix.com wrote:
Draft F follows. Also at:
http://xenbits.xen.org/people/ianc/vits/draftF.{pdf,html}
Here's a quick update based on feedback prior to meeting on #xenarm at
12:00AM BST / 7:00AM EDT / 4:30PM IST (which is ~1:20 from now)
Ian.
% Xen on ARM vITS Handling
% Ian Campbell ian.campb...@citrix.com
% Draft F
# Changelog
## Since Draft E
* Discussion of `struct pending_irq`
* Fix handling of enable/disable, requiring switching back to trapping
the virtual cfg table again. get_vlpi_cfg is no longer needed.
* Fix p2m_lookup to also use get_page_from_gfn.
## Since Draft D
* Fixed assumptions about vLPI-pLPI mapping, which is not
possible. This lead to changes to the model for enabling and
disabling pLPI and vLPI and the handling of the virtual LPI
configuration table, resolving _Unresolved Issue 1_.
* Made the pLPI and vLPI interrupt priorities explicit.
* Attempted to clarify the trust issues regarding in-guest data
structures.
* Mandate a particular cacheability for tables in guest memory.
## Since Draft C
* _Major_ rework, in an attempt to simplify everything into something
more likely to be achievable for 4.6.
* Made some simplifying assumptions.
* Reduced the scope of some support.
* Command emulation is now mostly trivial.
* Expanded detail on host setup, allowing other assumptions to be
made during emulation.
* Many other things lost in the noise of the above.
## Since Draft B
* Details of command translation (thanks to Julien and Vijay)
* Added background on LPI Translation and Pending tables
* Added background on Collections
* Settled on `N:N` scheme for vITS:pat's mapping.
* Rejigged section nesting a bit.
* Since we now thing translation should be cheap, settle on
translation at scheduling time.
* Lazy `INVALL` and `SYNC`
## Since Draft A
* Added discussion of when/where command translation occurs.
* Contention on scheduler lock, suggestion to use SOFTIRQ.
* Handling of domain shutdown.
* More detailed discussion of multiple vs single vits pros/cons.
# Introduction
ARM systems containing a GIC version 3 or later may contain one or
more ITS logical blocks. An ITS is used to route Message Signalled
interrupts from devices into an LPI injection on the processor.
The following summarises the ITS hardware design and serves as a set
of assumptions for the vITS software design. For full details of the
ITS see the GIC Architecture Specification.
## Locality-specific Peripheral Interrupts (`LPI`)
This is a new class of message signalled interrupts introduced in
GICv3. They occupy the interrupt ID space from `8192..(2^32)-1`.
The number of LPIs support by an ITS is exposed via
`GITS_TYPER.IDbits` (as number of bits - 1), it may be up to
2^32. _Note_: This field also contains the number of Event IDs
supported by the ITS.
### LPI Configuration Table
Each LPI has an associated configuration byte in the LPI Configuration
Table (managed via the GIC Redistributor and placed at
`GICR_PROPBASER` or `GICR_VPROPBASER`). This byte configures:
* The LPI's priority;
* Whether the LPI is enabled or disabled.
Software updates the Configuration Table directly but must then issue
an invalidate command (per-device `INV` ITS command, global `INVALL`
ITS command or write `GICR_INVLPIR`) for the affect to be guaranteed
to become visible (possibly requiring an ITS `SYNC` command to ensure
completion of the `INV` or `INVALL`). Note that it is valid for an
implementation to reread the configuration table at any time (IOW it
is _not_ guaranteed that a change to the LPI Configuration Table won't
be visible until an invalidate is issued).
### LPI Pending Table
Each LPI also has an associated bit in the LPI Pending Table (managed
by the GIC redistributor). This bit signals whether the LPI is pending
or not.
This region may contain out of date information and the mechanism to
synchronise is `IMPLEMENTATION DEFINED`.
## Interrupt Translation Service (`ITS`)
### Device Identifiers
Each device using the ITS is associated with a unique Device
Identifier.
The device IDs are properties of the implementation and are typically
described via system firmware, e.g. the ACPI IORT table or via device
tree.
The number of device ids in a system depends on the implementation and
can be discovered via `GITS_TYPER.Devbits`. This field allows an ITS
to have up to 2^32 devices.
### Events
Each device can generate Events (called `ID` in the spec) these
correspond to possible interrupt sources in the device (e.g. MSI
offset).
The maximum number of interrupt sources is device specific. It is
usually discovered either from firmware tables (e.g. DT or ACPI) or
from bus specific mechanisms (e.g. PCI config space).
The maximum number of events ids support by an ITS is exposed via
`GITS_TYPER.IDbits` (as number of bits - 1), it
Re: [Xen-devel] [Draft F] Xen on ARM vITS Handling
On Fri, 2015-06-12 at 14:07 +0530, Vijay Kilari wrote:
Please could you trim your quotes to only include the bit you are
referring to. Otherwise there is a high chance I will miss a one line
comment in the middle of the thousand lines of quoted matter.
The `GITS_BASER0` will be setup to request sufficient memory for a
device table consisting of entries of:
struct vdevice_table {
uint64_t vitt_ipa;
uint32_t vitt_size;
uint32_t padding;
};
How about adding valid bit to know if the entry is valid or not?
I suggest to use vitt_ipa == INVALID_PADDR to signal this rather than
using another bit.
## Handling of unrouted/spurious LPIs
Since there is no 1:1 link between a `vLPI` and `pLPI` enabling and
disabling of phyiscal LPIs cannot be driven from the state of an
associated vLPI.
Each `pLPI` is routed and enabled during device assignment, therefore
it is possible to receive a physical LPI which has yet to be routed
(via a `vITS`) to a `vLPI`.
Why do we need to enable LPIs during device assignment?
Can't we do it only on LPI configuration update, which is trapped in
Xen as mentioned in 7.8? ( ## Enabling and disabling LPIs)
Quoting the first sentence/paragraph of this section:
Since there is no 1:1 link between a `vLPI` and `pLPI` enabling
and disabling of phyiscal LPIs cannot be driven from the state
of an associated vLPI.
To expand on that: The vITT can map multiple (vDevice,vEvent) pairs to
the same LPI, and each of those (vDevice,vEvent) pairs is related to a
different (pDevice,pEvent) which in turn has a unique pLPI associated
with it. Thus a vLPI can be associated with more than one pLPI.
Enumerating all pLPIs associated with a given vLPI would be expensive (a
complete walk of the vITT).
In addition if it were possible to do so we would also need to manage
enabling/disabling the pLPI in several other places that in vPLI cfg
traps, specifically MAPC and MAPD at least.
So pLPIs must be routed at device assignment time because in the vLPI
configuration table trap there is no mapping back to a single pLPI.
Ian.
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [Draft F] Xen on ARM vITS Handling
On 12/06/2015 08:55, Ian Campbell wrote: On Thu, 2015-06-11 at 10:40 +0100, Ian Campbell wrote: ## Command Queue Virtualisation The command translation/emulation in this design has been arranged to be as cheap as possible (e.g. in many cases the actions are NOPs), avoiding previous concerns about the length of time which an emulated write to a `CWRITER` register may block the vcpu. The vits will simply track its reader and writer pointers. On write to `CWRITER` it will immediately and synchronously process all commands in the queue and update its state accordingly. It might be possible to implement a rudimentary form of preemption by periodically (as determined by `hypercall_preempt_check()`) returning to the guest without incrementing PC but with updated internal `CREADR` state, meaning it will reexecute the write to `CWRITER` and we can pickup where we left off for another iteration. This at least lets us schedule other vcpus etc and prevents a monopoly. In the presence of multiple VCPUs writing to GITS_CWRITER preemption actually gets pretty subtle. I suggest leaving it out for now. Would it be possible to do it with re-doing the write to the GITS_CWRITER? Regards, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [Draft F] Xen on ARM vITS Handling
On Fri, 2015-06-12 at 09:09 -0400, Julien Grall wrote: Hi Ian, On 12/06/2015 04:52, Ian Campbell wrote: On Fri, 2015-06-12 at 14:07 +0530, Vijay Kilari wrote: So pLPIs must be routed at device assignment time because in the vLPI configuration table trap there is no mapping back to a single pLPI. I just remembered the exact reason that made use to differ SPI enabling. I can't parse this sentence, differ how? When the device is assigned, the domain VCPUs are still down (even VCPU0). If we receive an interrupt before the VCPU0 is unpaused, the interrupt will be lost. Same if the interrupt is not yet configured (i.e before the vITS setup correctly the table) with your proposal. Is this any different to booting with the ITT not setup? (SPIs are a slightly different case because they don't need h/w routing) This could happen when the device is not quiescent. We had this issue on the vexpress at boot time when the network card was trying to send an interrupt before DOM0 is setup. I don't fully understand the issue you are trying to describe, but do you want to propose a change to the spec? Ian. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [Draft F] Xen on ARM vITS Handling
On Thu, 2015-06-11 at 10:40 +0100, Ian Campbell wrote: ## Command Queue Virtualisation The command translation/emulation in this design has been arranged to be as cheap as possible (e.g. in many cases the actions are NOPs), avoiding previous concerns about the length of time which an emulated write to a `CWRITER` register may block the vcpu. The vits will simply track its reader and writer pointers. On write to `CWRITER` it will immediately and synchronously process all commands in the queue and update its state accordingly. It might be possible to implement a rudimentary form of preemption by periodically (as determined by `hypercall_preempt_check()`) returning to the guest without incrementing PC but with updated internal `CREADR` state, meaning it will reexecute the write to `CWRITER` and we can pickup where we left off for another iteration. This at least lets us schedule other vcpus etc and prevents a monopoly. In the presence of multiple VCPUs writing to GITS_CWRITER preemption actually gets pretty subtle. I suggest leaving it out for now. Ian. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [Draft F] Xen on ARM vITS Handling
On Fri, 2015-06-12 at 09:32 -0400, Julien Grall wrote: On 12/06/2015 09:16, Ian Campbell wrote: On Fri, 2015-06-12 at 09:09 -0400, Julien Grall wrote: Hi Ian, On 12/06/2015 04:52, Ian Campbell wrote: On Fri, 2015-06-12 at 14:07 +0530, Vijay Kilari wrote: So pLPIs must be routed at device assignment time because in the vLPI configuration table trap there is no mapping back to a single pLPI. I just remembered the exact reason that made use to differ SPI enabling. I can't parse this sentence, differ how? deferring sorry. When the device is assigned, the domain VCPUs are still down (even VCPU0). If we receive an interrupt before the VCPU0 is unpaused, the interrupt will be lost. Same if the interrupt is not yet configured (i.e before the vITS setup correctly the table) with your proposal. Is this any different to booting with the ITT not setup? I don't understand your question. During boot the ITT is not configured and a spurious event will go undelivered to an LPI then too, even on native. (SPIs are a slightly different case because they don't need h/w routing) I think you mixed PPIs with SPIs. SPIs (shared private interrupt) requires h/w routing. I don't think they do, GICD_ICFGR (or the GICv3 equivalent) come up in a state where the interrupt will go _somewhere_, which differs from things injected via the ITS. This could happen when the device is not quiescent. We had this issue on the vexpress at boot time when the network card was trying to send an interrupt before DOM0 is setup. I don't fully understand the issue you are trying to describe, but do you want to propose a change to the spec? I actually don't know how to modify it. So it's an open question. For SPI too, or just for LPI? vgic_vcpu_inject_irq doesn't queue the interrupt if a VCPU is down. I think this is because the state of the VCPU wouldn't be correct. The process would be something like: - Creation of the domain = All vCPUs are down - Device is assigned to the guest = Enable physical LPIs * physical LPI is received * = Will be ignored and not EOIed (VCPU0 is down) = The LPI will never fired again during the guest life - Domain is started by the toolstack = VCPU0 is online Is it sufficient to queue interrupts even for VCPUs which are down? How does the lack of a vITT entry when this interrupt occurred affect this? Ian. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [Draft F] Xen on ARM vITS Handling
On 12/06/2015 09:16, Ian Campbell wrote: On Fri, 2015-06-12 at 09:09 -0400, Julien Grall wrote: Hi Ian, On 12/06/2015 04:52, Ian Campbell wrote: On Fri, 2015-06-12 at 14:07 +0530, Vijay Kilari wrote: So pLPIs must be routed at device assignment time because in the vLPI configuration table trap there is no mapping back to a single pLPI. I just remembered the exact reason that made use to differ SPI enabling. I can't parse this sentence, differ how? deferring sorry. When the device is assigned, the domain VCPUs are still down (even VCPU0). If we receive an interrupt before the VCPU0 is unpaused, the interrupt will be lost. Same if the interrupt is not yet configured (i.e before the vITS setup correctly the table) with your proposal. Is this any different to booting with the ITT not setup? I don't understand your question. (SPIs are a slightly different case because they don't need h/w routing) I think you mixed PPIs with SPIs. SPIs (shared private interrupt) requires h/w routing. This could happen when the device is not quiescent. We had this issue on the vexpress at boot time when the network card was trying to send an interrupt before DOM0 is setup. I don't fully understand the issue you are trying to describe, but do you want to propose a change to the spec? I actually don't know how to modify it. So it's an open question. vgic_vcpu_inject_irq doesn't queue the interrupt if a VCPU is down. I think this is because the state of the VCPU wouldn't be correct. The process would be something like: - Creation of the domain = All vCPUs are down - Device is assigned to the guest = Enable physical LPIs * physical LPI is received * = Will be ignored and not EOIed (VCPU0 is down) = The LPI will never fired again during the guest life - Domain is started by the toolstack = VCPU0 is online Regards, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [Draft F] Xen on ARM vITS Handling
On 12/06/2015 10:24, Ian Campbell wrote: On Fri, 2015-06-12 at 09:32 -0400, Julien Grall wrote: On 12/06/2015 09:16, Ian Campbell wrote: On Fri, 2015-06-12 at 09:09 -0400, Julien Grall wrote: Hi Ian, On 12/06/2015 04:52, Ian Campbell wrote: On Fri, 2015-06-12 at 14:07 +0530, Vijay Kilari wrote: So pLPIs must be routed at device assignment time because in the vLPI configuration table trap there is no mapping back to a single pLPI. I just remembered the exact reason that made use to differ SPI enabling. I can't parse this sentence, differ how? deferring sorry. When the device is assigned, the domain VCPUs are still down (even VCPU0). If we receive an interrupt before the VCPU0 is unpaused, the interrupt will be lost. Same if the interrupt is not yet configured (i.e before the vITS setup correctly the table) with your proposal. Is this any different to booting with the ITT not setup? I don't understand your question. During boot the ITT is not configured and a spurious event will go undelivered to an LPI then too, even on native. It's different. In the case of native, the event is not recorded by the ITS so it can fire up again later when the ITT is setup (for instance because the device has been reset). With your proposal, the interrupt will go in Active state (from the CPU POV see 4.8.3) in the GIC. If you don't EOI it, it will never fire again when the guest has setup the vITT. Although, the device won't know where it has to write the event ID (i.e in GITS_TRANSLATER) because it should not have been configured. So it will get ignored, right? (SPIs are a slightly different case because they don't need h/w routing) I think you mixed PPIs with SPIs. SPIs (shared private interrupt) requires h/w routing. I don't think they do, GICD_ICFGR (or the GICv3 equivalent) come up in a state where the interrupt will go _somewhere_, which differs from things injected via the ITS. Right, I was confused with the h/w routing meaning. This could happen when the device is not quiescent. We had this issue on the vexpress at boot time when the network card was trying to send an interrupt before DOM0 is setup. I don't fully understand the issue you are trying to describe, but do you want to propose a change to the spec? I actually don't know how to modify it. So it's an open question. For SPI too, or just for LPI? Only LPI. vgic_vcpu_inject_irq doesn't queue the interrupt if a VCPU is down. I think this is because the state of the VCPU wouldn't be correct. The process would be something like: - Creation of the domain = All vCPUs are down - Device is assigned to the guest = Enable physical LPIs * physical LPI is received * = Will be ignored and not EOIed (VCPU0 is down) = The LPI will never fired again during the guest life - Domain is started by the toolstack = VCPU0 is online Is it sufficient to queue interrupts even for VCPUs which are down? How does the lack of a vITT entry when this interrupt occurred affect this? Well, in this case we don't know on which vLPI we have to inject it. But as said above, I guess we don't care if we ensure that the device can't send an event (by ensuring that the device doesn't know the GITS_TRANSLATER address is). Regards, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [Draft F] Xen on ARM vITS Handling
On Fri, 2015-06-12 at 09:14 -0400, Julien Grall wrote: On 12/06/2015 08:55, Ian Campbell wrote: On Thu, 2015-06-11 at 10:40 +0100, Ian Campbell wrote: ## Command Queue Virtualisation The command translation/emulation in this design has been arranged to be as cheap as possible (e.g. in many cases the actions are NOPs), avoiding previous concerns about the length of time which an emulated write to a `CWRITER` register may block the vcpu. The vits will simply track its reader and writer pointers. On write to `CWRITER` it will immediately and synchronously process all commands in the queue and update its state accordingly. It might be possible to implement a rudimentary form of preemption by periodically (as determined by `hypercall_preempt_check()`) returning to the guest without incrementing PC but with updated internal `CREADR` state, meaning it will reexecute the write to `CWRITER` and we can pickup where we left off for another iteration. This at least lets us schedule other vcpus etc and prevents a monopoly. In the presence of multiple VCPUs writing to GITS_CWRITER preemption actually gets pretty subtle. I suggest leaving it out for now. Would it be possible to do it with re-doing the write to the GITS_CWRITER? Not easily. For example one subtle case is: * VCPUA writes W1 to GITS_CWRITER, processes a proportion and is preempted. * VCPUB writes W2 (W1) GITS_CWRITER W2 and processes a proportion such that W1 GITS_CREADR W2, is preempted. * VCPUA resumes, sees GITS_CREADR != W1, processes up to W2, and then keeps going processing junk until it wraps around to W1 again. The != is necessary because the command queue is a circular buffer, so GITS_CREADR GITS_CWRITER is not a reliable way to determine if there is anything on the ring. Another case I haven't decided if I'm concerned about yet is an OS which issues two writes to GITS_CWRITER in what it things is the proper order (maybe it's using barriers or atmomic variables or something to try and ensure this) but the trap for the first takes longer to get to the handler than the second (maybe it takes an interrupt on the way in, or more cache misses etc). This may or may not be an issue, I've not decided yet (mainly because I haven't decided if that OS behaviour is valid or not). Unlike with proper hypercall preemption we do not have the ability to record progress in the hypercall arguments, nor to indicate that a particular call is a continuation rather than a fresh call, both of which make this harder to deal with. It's possible something could be constructed with some per-vcpu data, I'm still thinking it through. Ian. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [Draft F] Xen on ARM vITS Handling
On 12/06/2015 09:32, Julien Grall wrote: On 12/06/2015 09:16, Ian Campbell wrote: On Fri, 2015-06-12 at 09:09 -0400, Julien Grall wrote: Hi Ian, On 12/06/2015 04:52, Ian Campbell wrote: On Fri, 2015-06-12 at 14:07 +0530, Vijay Kilari wrote: So pLPIs must be routed at device assignment time because in the vLPI configuration table trap there is no mapping back to a single pLPI. I just remembered the exact reason that made use to differ SPI enabling. I can't parse this sentence, differ how? deferring sorry. When the device is assigned, the domain VCPUs are still down (even VCPU0). If we receive an interrupt before the VCPU0 is unpaused, the interrupt will be lost. Same if the interrupt is not yet configured (i.e before the vITS setup correctly the table) with your proposal. Is this any different to booting with the ITT not setup? I don't understand your question. (SPIs are a slightly different case because they don't need h/w routing) I think you mixed PPIs with SPIs. SPIs (shared private interrupt) s/private/processor/ requires h/w routing. This could happen when the device is not quiescent. We had this issue on the vexpress at boot time when the network card was trying to send an interrupt before DOM0 is setup. I don't fully understand the issue you are trying to describe, but do you want to propose a change to the spec? I actually don't know how to modify it. So it's an open question. vgic_vcpu_inject_irq doesn't queue the interrupt if a VCPU is down. I think this is because the state of the VCPU wouldn't be correct. The process would be something like: - Creation of the domain = All vCPUs are down - Device is assigned to the guest = Enable physical LPIs * physical LPI is received * = Will be ignored and not EOIed (VCPU0 is down) = The LPI will never fired again during the guest life - Domain is started by the toolstack = VCPU0 is online Regards, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
[Xen-devel] [Draft F] Xen on ARM vITS Handling
Draft F follows. Also at:
http://xenbits.xen.org/people/ianc/vits/draftF.{pdf,html}
Here's a quick update based on feedback prior to meeting on #xenarm at
12:00AM BST / 7:00AM EDT / 4:30PM IST (which is ~1:20 from now)
Ian.
% Xen on ARM vITS Handling
% Ian Campbell ian.campb...@citrix.com
% Draft F
# Changelog
## Since Draft E
* Discussion of `struct pending_irq`
* Fix handling of enable/disable, requiring switching back to trapping
the virtual cfg table again. get_vlpi_cfg is no longer needed.
* Fix p2m_lookup to also use get_page_from_gfn.
## Since Draft D
* Fixed assumptions about vLPI-pLPI mapping, which is not
possible. This lead to changes to the model for enabling and
disabling pLPI and vLPI and the handling of the virtual LPI
configuration table, resolving _Unresolved Issue 1_.
* Made the pLPI and vLPI interrupt priorities explicit.
* Attempted to clarify the trust issues regarding in-guest data
structures.
* Mandate a particular cacheability for tables in guest memory.
## Since Draft C
* _Major_ rework, in an attempt to simplify everything into something
more likely to be achievable for 4.6.
* Made some simplifying assumptions.
* Reduced the scope of some support.
* Command emulation is now mostly trivial.
* Expanded detail on host setup, allowing other assumptions to be
made during emulation.
* Many other things lost in the noise of the above.
## Since Draft B
* Details of command translation (thanks to Julien and Vijay)
* Added background on LPI Translation and Pending tables
* Added background on Collections
* Settled on `N:N` scheme for vITS:pat's mapping.
* Rejigged section nesting a bit.
* Since we now thing translation should be cheap, settle on
translation at scheduling time.
* Lazy `INVALL` and `SYNC`
## Since Draft A
* Added discussion of when/where command translation occurs.
* Contention on scheduler lock, suggestion to use SOFTIRQ.
* Handling of domain shutdown.
* More detailed discussion of multiple vs single vits pros/cons.
# Introduction
ARM systems containing a GIC version 3 or later may contain one or
more ITS logical blocks. An ITS is used to route Message Signalled
interrupts from devices into an LPI injection on the processor.
The following summarises the ITS hardware design and serves as a set
of assumptions for the vITS software design. For full details of the
ITS see the GIC Architecture Specification.
## Locality-specific Peripheral Interrupts (`LPI`)
This is a new class of message signalled interrupts introduced in
GICv3. They occupy the interrupt ID space from `8192..(2^32)-1`.
The number of LPIs support by an ITS is exposed via
`GITS_TYPER.IDbits` (as number of bits - 1), it may be up to
2^32. _Note_: This field also contains the number of Event IDs
supported by the ITS.
### LPI Configuration Table
Each LPI has an associated configuration byte in the LPI Configuration
Table (managed via the GIC Redistributor and placed at
`GICR_PROPBASER` or `GICR_VPROPBASER`). This byte configures:
* The LPI's priority;
* Whether the LPI is enabled or disabled.
Software updates the Configuration Table directly but must then issue
an invalidate command (per-device `INV` ITS command, global `INVALL`
ITS command or write `GICR_INVLPIR`) for the affect to be guaranteed
to become visible (possibly requiring an ITS `SYNC` command to ensure
completion of the `INV` or `INVALL`). Note that it is valid for an
implementation to reread the configuration table at any time (IOW it
is _not_ guaranteed that a change to the LPI Configuration Table won't
be visible until an invalidate is issued).
### LPI Pending Table
Each LPI also has an associated bit in the LPI Pending Table (managed
by the GIC redistributor). This bit signals whether the LPI is pending
or not.
This region may contain out of date information and the mechanism to
synchronise is `IMPLEMENTATION DEFINED`.
## Interrupt Translation Service (`ITS`)
### Device Identifiers
Each device using the ITS is associated with a unique Device
Identifier.
The device IDs are properties of the implementation and are typically
described via system firmware, e.g. the ACPI IORT table or via device
tree.
The number of device ids in a system depends on the implementation and
can be discovered via `GITS_TYPER.Devbits`. This field allows an ITS
to have up to 2^32 devices.
### Events
Each device can generate Events (called `ID` in the spec) these
correspond to possible interrupt sources in the device (e.g. MSI
offset).
The maximum number of interrupt sources is device specific. It is
usually discovered either from firmware tables (e.g. DT or ACPI) or
from bus specific mechanisms (e.g. PCI config space).
The maximum number of events ids support by an ITS is exposed via
`GITS_TYPER.IDbits` (as number of bits - 1), it may be up to
2^32. _Note_: This field also contains the number of `LPIs` supported
by the ITS.
### Interrupt Collections
Each interrupt is a member of an Interrupt Collection.
Re: [Xen-devel] [Draft F] Xen on ARM vITS Handling
On Thu, 2015-06-11 at 10:40 +0100, Ian Campbell wrote: Here's a quick update based on feedback prior to meeting on #xenarm at 12:00AM BST / 7:00AM EDT / 4:30PM IST (which is ~1:20 from now) Here is the log. (12:02:38) ijc: VK: So, are you happy that the design doc is something which could be implemented? (12:03:26) VK: ijc: I have some doubts as conclusion is not done in some cases or I have missed to follow up (12:03:53) ijc: OK, shall we go through them then? (12:04:12) ijc: I'll be working from the Draft F I sent out an hour ago (12:04:24) VK: ijc: I have listed down queries and go through topic wise as per draft E (12:04:44) ijc: ok (12:05:59) VK: ijc: 2.2.8 - Xen will use completion INT mechanism and trigger softIRQ for scheduling. (12:05:59) VK:and one completion INT per domain is allocated for mapping completion INT to domain's (12:05:59) VK:vITS. OK? (12:06:41) ijc: VK: Why is that needed? AFAICT the pITS driver can either poll or use a single host wide completion interrupt (12:06:52) ijc: From the PoV of vits we don't care how the pits driver gets completions I think (12:07:01) ijc: Or at least this design does not require it (12:08:07) ijc: There is no softirq and nor ITS scheduling in draft E, so I don't htink it is neeed, od you understand differently? (12:08:58) VK: ijc: As there is no info about ITS scheduling in draft E, I want some clarification. (12:09:29) ijc: VK: Everything is done synchronously in the GITS_CWIRTE handler (12:09:48) ijc: THings have been arranged such that the commands are all cheap enough to do this (12:10:16) ijc: The section Command Queue Virtualisation covers this I think (12:10:29) ijc: 7.11 in draft E (12:10:41) VK: ijc: Ok. then it is same as what I have done RFC v2 patch (12:10:49) ijc: and 7.14 in draft F (12:11:21) ijc: VK: At least that aspect may be I think, I don't know though. (12:11:56) VK: ijc: OK (12:12:28) julieng: VK: At the difference that there is no physical command send to the ITS (12:13:00) julieng: neither allocation (12:13:39) ijc: julieng: Right, the intention was to do as much stuff at setup or assignment time such that the command processing was cheap (12:15:10) VK: ijc: But VCPU still polls right and for that you have proposed rudimentary form of preemption (12:15:55) VK: ijc: in 7.14 in draft F. I am not aware of this any guidance on this? (12:16:01) ijc: VK I think the polling and the preemption are unrelated. The write to GITS_WRITER is processed synchronously, so the vcpu cannot be polling then, I think? (12:16:26) ijc: The premption thing is an optional extension to consider to allow that synchronous processing to be split up e.g. to allow other vcpus to run (12:17:22) ijc: If other vcpus are reading GITS_READR then I suppose we would want them to see progress, i.e. by updating the internal CREADR stepwise rather than all at once at the end. (12:18:36) VK: ijc: but when VCPU posts command on CWRITER write, then VCPU polls for completion in pITS driver (12:19:04) julieng: VK: There is no command send to the physical ITS. (12:19:20) ijc: If a command generates a request to the generic code which results in a call to the pits driver then it is up to the pits driver how to deal with that and polling would be a valid response (12:19:27) ijc: s/response/way to implement that/ (12:19:52) ijc: VK: I've deliberately decupled the vits and pits here (via the abstraction of the generic code) so that from a vits PoV you are not required to worry about it (12:20:22) julieng: ijc: AFAICT, there no command requiring physical command anymore (12:20:37) ijc: julieng: that would be even better ;-) (12:20:44) ijc: and I think you are right (12:21:01) julieng: If not, this would be a concern as a guest would be able to block a pCPU for a while. (12:21:20) ijc: (I wouldn't be too worried about that in the end, but it is moot anyway) (12:22:43) ijc: VK: Does that resolve your concern? (12:23:54) VK: ijc: I am not getting it here. How vITS command does _not_ translate to physical ITS command (12:24:13) ijc: VK: Everything is setup at start of day, so there is nothing to do during vits command processing (12:24:36) ijc: Look through 7.15.2.* and you should see no calls to anything which interacts with the physical its (12:25:28) ijc: (NB: 7.15.2.7 and .8 should read Since LPI Configuration table updates are handled synchronously, there (12:25:28) ijc: is nothing to do here. in Draft F, I missed updating them (12:32:22) VK: ijc: when you say it set up start of the day. you mean the guest Device Table and ITT table is directly updated by Xen instead of sending physical ITS command? (12:32:59) ijc: VK: http://xenbits.xen.org/people/ianc/vits/draftF.html#device-discoveryregistration-and-configuration (12:33:11) ijc: and the following section 6 Device Assignment (12:33:28) ijc: All of the events are routed to pLPIs during setup (either xen boot or during device assignment) (12:37:36) VK: ijc: OK, on
