Re: [Xen-devel] [RFC] ARM: New (Xen) VGIC design document

[Christoffer Dall]

On Wed, Nov 1, 2017 at 10:54 PM, Stefano Stabellini
 wrote:

[...]

>
>> > The suggestion of using this model in Xen was made in the past already.
>> > I always objected for the reason that we don't actually know how many
>> > LRs the hardware provides, potentially very many, and it is expensive
>> > and needless to read/write them all every time on entry/exit.
>> >
>> > I would prefer to avoid that, but I'll be honest: I can be convinced
>> > that that model of handling LRs is so much simpler that it is worth it.
>> > I am more concerned about the future maintainance of a separate new
>> > driver developed elsewhere.
>>
>> I think this LR topic should have been covered in that other email.
>>
>> Beside being a strong supporter of the KISS principle in general, I
>> believe in case of the GIC emulation we should avoid (premature)
>> optimizations like the plague, as there are quite some corner cases in
>> any VGIC, and handling all of them explicitly with some hacks will not
>> fly (been there, done that).
>> So I can just support Christoffer's point: having an architecture
>> compliant VGIC emulation in an maintainable manner requires a
>> straight-forward and clear design. Everything else should be secondary,
>> and can be evaluated later, if there are good reasons (numbers!).
>
> The reason why I stated the above is that I ran the numbers back in the
> day and reading or writing LRs on an XGene was so slow that it made
> sense to avoid it as much as possible. But maybe things have changed if
> Christoffer also ran the numbers and managed to demonstrate the
> opposite.

Accessing LRs on GICv2 is terrible indeed, and we already optimize it
as much as it makes sense.  It's just that with the current KVM code
base reading/writing the same value almost never happens, so there's
no more room (in practice) for optimization.

Also, note with GICv3 the cost goes down a lot, and potentially also
for other integrations of GICv2.

-Christoffer

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC] ARM: New (Xen) VGIC design document

[Christoffer Dall]

On Wed, Nov 1, 2017 at 10:15 AM, Andre Przywara
<andre.przyw...@linaro.org> wrote:
> Hi,
>
> On 01/11/17 04:31, Christoffer Dall wrote:
>> On Wed, Nov 1, 2017 at 9:58 AM, Stefano Stabellini
>> <sstabell...@kernel.org> wrote:
>>
>> []
>
> Christoffer, many thanks for answering this!
> I think we have a lot of assumptions about the whole VGIC life cycle
> floating around, but it would indeed be good to get some numbers behind it.
> I would be all too happy to trace some workloads on Xen again and
> getting some metrics, though this sounds time consuming if done properly.
>
> Do you have any numbers on VGIC performance available somewhere?
>

I ran the full set of workloads and micro numbers that I've used for
my papers with/without the optimization, and couldn't find a
measurable difference.

-Christoffer

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC] ARM: New (Xen) VGIC design document

[Christoffer Dall]

On Wed, Nov 1, 2017 at 9:58 AM, Stefano Stabellini
 wrote:

[]

>
>> ### List register management
>>
>> A list register (LR) holds the state of a virtual interrupt, which will
>> be used by the GIC hardware to simulate an IRQ life cycle for a guest.
>> Each GIC hardware implementation can choose to implement a number of LRs,
>> having four of them seems to be a common value. This design here does not
>> try to manage the LRs very cleverly, instead on every guest exit every LR
>> in use will be synced to the emulated state, then cleared. Upon guest entry
>> the top priority virtual IRQs will be inserted into the LRs. If there are
>> more pending or active IRQs than list registers, the GIC management IRQ
>> will be configured to notify the hypervisor of a free LR (once the guest
>> has EOIed one IRQ). This will trigger a normal exit, which will go through
>> the normal cleanup/repopulate scheme, possibly now queuing the leftover
>> interrupt(s).
>> To facilitate quick guest exit and entry times, the VGIC maintains the list
>> of pending or active interrupts (ap\_list) sorted by their priority. Active
>> interrupts always go first on the list, since a guest and the hardware GIC
>> expect those to stay until they have been explicitly deactivated. Failure
>> in keeping active IRQs around will result in error conditions in the GIC.
>> The second sort criteria for the ap\_list is their priority, so higher
>> priority pending interrupt always go first into the LRs.
>
> The suggestion of using this model in Xen was made in the past already.
> I always objected for the reason that we don't actually know how many
> LRs the hardware provides, potentially very many, and it is expensive
> and needless to read/write them all every time on entry/exit.
>
> I would prefer to avoid that, but I'll be honest: I can be convinced
> that that model of handling LRs is so much simpler that it is worth it.
> I am more concerned about the future maintainance of a separate new
> driver developed elsewhere.

[Having just spent a fair amount of time optimizing KVM/ARM and
measuring GIC interaction, I'll comment on this and leave it up to
Andre to drive the rest of the discussion].

In KVM we currently only ever touch an LR when we absolutely have to.
For example, if there are no interrupts, we do not touch an LR.

When you do have an interrupt in flight, and have programmed one or
more LRs, you have to either read back that LR, or read one of the
status registers to figure out if the interrupt has become inactive
(and should potentially be injected again).  I measured both on KVM
for various workloads and it was faster to never read the status
registers, but simply read back the LRs that were in use when entering
the guest.

You can potentially micro-optimize slightly by remembering the exit
value of an LR (and not clearing it on guest exit), but you have to
pay the cost in terms of additional logic during VCPU migration and
when you enter a VM again, maintaining a mapping of the LR and the
virtual state, to avoid rewriting the same value to the LR again.  We
tried that in KVM and could not measure any benefit using either a
pinned or oversubscribed workload; I speculate that the number of
times you exit with unprocessed interrupts in the LRs is extremely
rare.

In terms of the number of LRs, I stil haven't seen an implementation
with anything else than 4 LRs.

-Christoffer

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC] ARM: New (Xen) VGIC design document

[Christoffer Dall]

Hi Andre,

On Wed, Oct 11, 2017 at 03:33:03PM +0100, Andre Przywara wrote:
> Hi,
> 
> (CC:ing some KVM/ARM folks involved in the VGIC)

Very nice writeup!

I added a bunch of comments, mostly for the writing and clarity, I hope
it helps.

> 
> starting with the addition of the ITS support we were seeing more and
> more issues with the current implementation of our ARM Generic Interrupt
> Controller (GIC) emulation, the VGIC.
> Among other approaches to fix those issues it was proposed to copy the
> VGIC emulation used in KVM. This one was suffering from very similar
> issues, and a clean design from scratch lead to a very robust and
> capable re-implementation. Interestingly this implementation is fairly
> self-contained, so it seems feasible to copy it. Hopefully we only need
> minor adjustments, possibly we can even copy it verbatim with some
> additional glue layer code.
> Stefano asked for getting a design overview, to assess the feasibility
> of copying the KVM code without reviewing tons of code in the first
> place.
> So to follow Xen rules for new features, this design document below is
> an attempt to describe the current KVM VGIC design - in a hypervisor
> agnostic session. It is a bit of a retro-fit design description, as it
> is not strictly forward-looking only, but actually describing the
> existing implemenation [1].
> 
> Please have a look and let me know:
> 1) if this document has the right scope
> 2) if this document has the right level of detail
> 3) if there are points missing from the document
> 3) if the design in general is a fit
> 
> Appreciate any feedback!
> 
> Cheers,
> Andre.
> 
> ---
> 
> VGIC design
> ===
> 
> This document describes the design of an ARM Generic Interrupt Controller 
> (GIC)
> emulation. It is meant to emulate a GIC for a guest in an virtual machine,
> the common name for that is VGIC (from "virtual GIC").
> 
> This design was the result of a one-week-long design session with some
> engineers in a room, triggered by ever-increasing difficulties in maintaining
> the existing GIC emulation in the KVM hypervisor. The design eventually
> materialised as an alternative VGIC implementation in the Linux kernel
> (merged into Linux v4.7). As of Linux v4.8 the previous VGIC implementation
> was removed, so it is now the current code used by Linux.
> Although being used in KVM, the actual design of this VGIC is rather 
> hypervisor
> agnostic and can be used by other hypervisors as well, in particular for Xen.
> 
> GIC hardware virtualization support
> ---
> 
> The ARM Generic Interrupt Controller (since v2) supports the virtualization
> extensions, which allows some parts of the interrupt life cycle to be handled
> purely inside the guest without exiting into the hypervisor.
> In the GICv2 and GICv3 architecture this covers mostly the "interrupt
> acknowledgement", "priority drop" and "interrupt deactivate" actions.
> So a guest can handle most of the interrupt processing code without
> leaving EL1 and trapping into the hypervisor. To accomplish
> this, the GIC holds so called "list registers" (LRs), which shadow the
> interrupt state for any virtual interrupt. Injecting an interrupt to a guest
> involves setting up one LR with the interrupt number, its priority and initial
> state (mostly "pending"), then entering the guest. Any EOI related action
> from within the guest just acts on those LRs, the hypervisor can later update
> the virtual interrupt state when the guest exists the next time (for whatever
> reason).
> But despite the GIC hardware helping out here, the whole interrupt
> configuration management is not virtualized at all and needs to be emulated
> by the hypervisor - or another related software component, for instance a
> userland emulator. This so called "distributor" part of the GIC consists of
> memory mapped registers, which can be trapped by the hypervisor, so any guest
> access can be emulated in the usual way.
> 
> VGIC design motivation
> --
> 
> A GIC emulation thus needs to take care of those bits:
> 
> - trap GIC distributor MMIO accesses and shadow the configuration setup
>   (enabled/disabled, level/edge, priority, affinity) for virtual interrupts
> - handle incoming hardware and virtual interrupt requests and inject the
>   associated virtual interrupt by manipulating one of the list registers
> - track the state of a virtual interrupt by inspecting the LRs after the
>   guest has exited, possibly adjusting the shadowed virtual interrupt state
> 
> Despite the distributor MMIO register emulation being a sizeable chunk of
> the emulation, it is actually not dominant if looking at the frequency at
> which it is accessed. Normally the interrupt configuration is done at boot
> time or upon initialising the device (driver), but rarely during the actual
> run time of a system. Injecting and EOI-ing interrupts however happens much
> more often. A good 

Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest

[Christoffer Dall]

On Fri, Feb 3, 2017 at 2:53 PM, Bhupinder Thakur
 wrote:
> Hi,
>
>>> Hi,
>>>
>>> I have done the changes for emulating pl011 in Xen. Currently, I have
>>> verified the emulation code by manually reading/writing data to
>>> /dev/ttyAMA0 which is the device file for pl011 device. The data is
>>> flowing fine between xenconsoled and the guest domain.
>>>
>>> As a next step, I wanted to use /dev/ttyAMA0 as a console.
>>>
>>> For that I tried adding console=ttyAMA0 instead of console=hvc0 in the
>>> "extra" directive in the domU configuration file. However, I do not
>>> see the output on the console once I attached the console using "xl
>>> console ". I tried using "xl console -t serial
>>> " also but that shows the tty1 console and not the
>>> ttyAMA0 one.
>>
>> I would try to verify that when Linux uses the pl011 as a console (as
>> opposed to just writing some characters into /dev/ttyAMA0), that it
>> doesn't error out somewhere.  So like Stefano suggests, make sure you
>> check the guest kernel log after boot (assuming you can SSH or something
>> into the box) and look at 'dmesg' to see if the pl011 driver failed.
>>
>> You could also try adding some log info in you Xen pl011 emulation code
>> to see if the guest kernel driver prods the device in the sequence you
>> expect, based on looking at the linux driver for the pl011.
>
> I tested with the following changes:
>
> 1. Compiled out CONFIG_HVC_XEN
> 2. Specified "console=ttyAMA0" in the guest configuration
>
> With some more changes/fixes, the guest console is working partially
> over ttyAMA0. I can see the guest boot prints on the console. However,
> the prints stop just towards the end when the init scripts are
> executed.
>
> So it seems that during boot, initially ttyAMA0 is used as a
> write-only port, where the console driver keeps writing the output to
> the port. Till this point, only tx has been enabled on ttyAMA0 (driver
> probe() is called at this point). No irq/rx has been enabled yet. Just
> as the init scripts are executed, the irq/rx is also enabled on
> ttyAMA0 (driver startup() is called at this time which enables rx and
> registers IRQ) and then the boot prints stop.
>
> Another observation is that unless you compile out HVC_XEN support,
> the hvc0 console also keeps receiving the data even if the
> console=ttyAMA0.
>

What's your guest distro and file system settings?

This sounds to me like the console bringup job is hard-wired to
/dev/hvc0 as opposed to using the console.  I think most ubuntu
versions prior to using systemd needs the right files in /etc/init.
Check if you have /etc/init/hvc.conf or something like that and if you
have /etc/init/ttyAMA0.conf.  If you're using systemd or a different
distro, figure out what the equivalent settings there are.

-Christoffer

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest

[Christoffer Dall]

On Fri, Jan 27, 2017 at 05:44:15PM +0530, Bhupinder Thakur wrote:
> Hi,
> 
> I have done the changes for emulating pl011 in Xen. Currently, I have
> verified the emulation code by manually reading/writing data to
> /dev/ttyAMA0 which is the device file for pl011 device. The data is
> flowing fine between xenconsoled and the guest domain.
> 
> As a next step, I wanted to use /dev/ttyAMA0 as a console.
> 
> For that I tried adding console=ttyAMA0 instead of console=hvc0 in the
> "extra" directive in the domU configuration file. However, I do not
> see the output on the console once I attached the console using "xl
> console ". I tried using "xl console -t serial
> " also but that shows the tty1 console and not the
> ttyAMA0 one.

I would try to verify that when Linux uses the pl011 as a console (as
opposed to just writing some characters into /dev/ttyAMA0), that it
doesn't error out somewhere.  So like Stefano suggests, make sure you
check the guest kernel log after boot (assuming you can SSH or something
into the box) and look at 'dmesg' to see if the pl011 driver failed.

You could also try adding some log info in you Xen pl011 emulation code
to see if the guest kernel driver prods the device in the sequence you
expect, based on looking at the linux driver for the pl011.

Thanks,
-Christoffer

> 
> 
> On 18 January 2017 at 00:57, Stefano Stabellini <sstabell...@kernel.org> 
> wrote:
> > On Tue, 17 Jan 2017, Julien Grall wrote:
> >> Hi,
> >>
> >> Sorry for the late answer, I am just back from holidays and still 
> >> catching-up
> >> with my e-mails.
> >>
> >> On 03/01/17 20:08, Stefano Stabellini wrote:
> >> > On Thu, 29 Dec 2016, Bhupinder Thakur wrote:
> >> > > On 28 December 2016 at 23:19, Julien Grall <julien.gr...@arm.com> 
> >> > > wrote:
> >> > > > On 21/12/16 22:12, Stefano Stabellini wrote:
> >> > > > >
> >> > > > > On Wed, 21 Dec 2016, Julien Grall wrote:
> >> > > > > >
> >> > > > > > On 20/12/2016 20:53, Stefano Stabellini wrote:
> >> > > > > > >
> >> > > > > > > On Tue, 20 Dec 2016, Julien Grall wrote:
> >> > > > > > > >
> >> > > > > > > > On 19/12/2016 21:24, Stefano Stabellini wrote:
> >> > > > > > > > >
> >> > > > > > > > > On Mon, 19 Dec 2016, Christoffer Dall wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > On Fri, Dec 16, 2016 at 05:03:13PM +, Julien Grall
> >> > > > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > If we use hvm_params for this, we need two new hvm_params 
> >> > > > > > > > > and
> >> > > > > > > > > Xen
> >> > > > > > > > > needs
> >> > > > > > > > > to unmap the pfn from the guest immediately, because we 
> >> > > > > > > > > don't
> >> > > > > > > > > want the
> >> > > > > > > > > guest to have access to it.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > If you unmap the pfn, the PV backend will not be able to 
> >> > > > > > > > request
> >> > > > > > > > the
> >> > > > > > > > page
> >> > > > > > > > because there will be no translation available.
> >> > > > > > > >
> >> > > > > > > > So what you want to do is preventing the guest to at least 
> >> > > > > > > > write
> >> > > > > > > > into
> >> > > > > > > > region
> >> > > > > > > > (not sure if it is worth to restrict read)
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > That's a good idea.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > > and unmap the page via the hypercall
> >> > > > > > > > XENMEM_decrease_reservation.
> >> > > > > > >
> >> > > > > > >
> &

Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest

[Christoffer Dall]

On Tue, Dec 20, 2016 at 01:33:39PM -0800, Stefano Stabellini wrote:
> On Tue, 20 Dec 2016, Christoffer Dall wrote:
> > Hi Stefano,
> > 
> > On Mon, Dec 19, 2016 at 12:24:18PM -0800, Stefano Stabellini wrote:
> > > On Mon, 19 Dec 2016, Christoffer Dall wrote:
> > > > On Fri, Dec 16, 2016 at 05:03:13PM +, Julien Grall wrote:
> > > > > (CC rest maintainers for event channel questions)
> > > > > 
> > > > > On 16/12/16 10:06, Bhupinder Thakur wrote:
> > > > > >Hi,
> > > > > 
> > > > > Hi Bhupinder,
> > > > > 
> > > > > >The idea is for Xen to act as an intermediary as shown below:
> > > > > >
> > > > > >   ring buffers
> > > > > >   rx/tx fifo
> > > > > >dom0 <---> Xen HYP (running pl011 emulation)
> > > > > ><---> domU
> > > > > >event
> > > > > >   interrupts
> > > > > >
> > > > > >Xen will directly manage the in/out console ring buffers (allocated 
> > > > > >by
> > > > > >dom0 for dom0-domU console communication) for reading/writing console
> > > > > >data from/to dom0. On the other side, Xen HYP will emulate pl011 to
> > > > > >read/write data from/to domU and pass it on to/from dom0 over the
> > > > > >in/out console ring buffers. There should be no change in dom0 as it
> > > > > >will still use the same ring buffers. Similarly there should be no
> > > > > >change in domU which would be running a standard pll011 driver.
> > > > > >
> > > > > >Currently, I am working on the interface between dom0 and Xen HYP. I
> > > > > >want to intercept the console events in Xen HYP which pass between
> > > > > >dom0 and domU. For now, I just want to capture console data coming
> > > > > >from dom0 at Xen HYP and loop it back to dom0, to confirm that this
> > > > > >interface is working.
> > > > > >
> > > > > >Since each guest domain will have a unique event channel assigned for
> > > > > >console communication, Xen HYP can find out the event channel for a
> > > > > >given domU from the start_info page of that domU, which should have
> > > > > 
> > > > > The start_info page is x86 specific. If you want to get the console
> > > > > event channel for ARM, you would have to use
> > > > > d->arch.hvm_domain.params[HVM_PARAM_CONSOLE_EVTCHN].
> > > > > 
> > > > > This parameter will be setup by the toolstack (see alloc_magic_pages
> > > > > in libxc/xc_dom_arm.c).
> > > > > 
> > > > > >been allocated by dom0. Whenever, an event is to be dispatched via
> > > > > >evtchn_send() API in Xen, it can check if the event channel is the
> > > > > >console event channel for a given domU. If yes and its source domain
> > > > > >is dom0 and destination domain is domU then it will write the data
> > > > > >back to the console out ring buffer of the domU and raise a console
> > > > > >event to dom0.
> > > > > >
> > > > > >Once this interface is working, Xen HYP can check the source and
> > > > > >destination dom ids and decide which way the event came from and
> > > > > >accordingly process the console data. To allow a mix of PV console
> > > > > >guests and pl011 guests, Xen might have to maintain a flag per 
> > > > > >domain,
> > > > > >which tells whether Xen HYP should intercept and process the data 
> > > > > >(for
> > > > > >pl011 UART case) or let it go transparently (for PV conosle case).
> > > > > 
> > > > > I am not very familiar with the event channel code. I will let the
> > > > > others comment on this bit.
> > > > > 
> > > > > Regardless that, how would you decide whether the hypervisor should
> > > > > intercept the notification?
> > > > > 
> > > > > I can see 2 different cases:
> > > > >   1) The guest is starting to use the pl011 then move to the HVC
> > > > > console (or HVC then pl011)
> > > > >   2) The

Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest

[Christoffer Dall]

Hi Stefano,

On Mon, Dec 19, 2016 at 12:24:18PM -0800, Stefano Stabellini wrote:
> On Mon, 19 Dec 2016, Christoffer Dall wrote:
> > On Fri, Dec 16, 2016 at 05:03:13PM +, Julien Grall wrote:
> > > (CC rest maintainers for event channel questions)
> > > 
> > > On 16/12/16 10:06, Bhupinder Thakur wrote:
> > > >Hi,
> > > 
> > > Hi Bhupinder,
> > > 
> > > >The idea is for Xen to act as an intermediary as shown below:
> > > >
> > > >   ring buffers
> > > >   rx/tx fifo
> > > >dom0 <---> Xen HYP (running pl011 emulation)
> > > ><---> domU
> > > >event
> > > >   interrupts
> > > >
> > > >Xen will directly manage the in/out console ring buffers (allocated by
> > > >dom0 for dom0-domU console communication) for reading/writing console
> > > >data from/to dom0. On the other side, Xen HYP will emulate pl011 to
> > > >read/write data from/to domU and pass it on to/from dom0 over the
> > > >in/out console ring buffers. There should be no change in dom0 as it
> > > >will still use the same ring buffers. Similarly there should be no
> > > >change in domU which would be running a standard pll011 driver.
> > > >
> > > >Currently, I am working on the interface between dom0 and Xen HYP. I
> > > >want to intercept the console events in Xen HYP which pass between
> > > >dom0 and domU. For now, I just want to capture console data coming
> > > >from dom0 at Xen HYP and loop it back to dom0, to confirm that this
> > > >interface is working.
> > > >
> > > >Since each guest domain will have a unique event channel assigned for
> > > >console communication, Xen HYP can find out the event channel for a
> > > >given domU from the start_info page of that domU, which should have
> > > 
> > > The start_info page is x86 specific. If you want to get the console
> > > event channel for ARM, you would have to use
> > > d->arch.hvm_domain.params[HVM_PARAM_CONSOLE_EVTCHN].
> > > 
> > > This parameter will be setup by the toolstack (see alloc_magic_pages
> > > in libxc/xc_dom_arm.c).
> > > 
> > > >been allocated by dom0. Whenever, an event is to be dispatched via
> > > >evtchn_send() API in Xen, it can check if the event channel is the
> > > >console event channel for a given domU. If yes and its source domain
> > > >is dom0 and destination domain is domU then it will write the data
> > > >back to the console out ring buffer of the domU and raise a console
> > > >event to dom0.
> > > >
> > > >Once this interface is working, Xen HYP can check the source and
> > > >destination dom ids and decide which way the event came from and
> > > >accordingly process the console data. To allow a mix of PV console
> > > >guests and pl011 guests, Xen might have to maintain a flag per domain,
> > > >which tells whether Xen HYP should intercept and process the data (for
> > > >pl011 UART case) or let it go transparently (for PV conosle case).
> > > 
> > > I am not very familiar with the event channel code. I will let the
> > > others comment on this bit.
> > > 
> > > Regardless that, how would you decide whether the hypervisor should
> > > intercept the notification?
> > > 
> > > I can see 2 different cases:
> > >   1) The guest is starting to use the pl011 then move to the HVC
> > > console (or HVC then pl011)
> > >   2) The guest is using both the PL011 and the HVC console
> > > 
> > > Should we consider the second case valid? I would say yes, because a
> > > user could specify both on the command line. If we use the same
> > > ring, the output would be a total garbage.
> > > 
> > > So maybe we need to allocate two distinct rings and event channel?
> > 
> > This sounds like the only sensible thing to me.  I think this is really
> > about adding a new device to the Xen virtual platform, and providing the
> > user the option to choose which one he wants the tool in Dom0 to be
> > presented using stdin/out. Presumably the other console/serial can be
> > redirected to a file or socket or something?
> 
> Let me explain how the PV console protocol and drivers work, because
> they are a bit unusual. 

Thanks for this.  As my detailed know

Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest

[Christoffer Dall]

On Fri, Dec 16, 2016 at 05:03:13PM +, Julien Grall wrote:
> (CC rest maintainers for event channel questions)
> 
> On 16/12/16 10:06, Bhupinder Thakur wrote:
> >Hi,
> 
> Hi Bhupinder,
> 
> >The idea is for Xen to act as an intermediary as shown below:
> >
> >   ring buffers
> >   rx/tx fifo
> >dom0 <---> Xen HYP (running pl011 emulation)
> ><---> domU
> >event
> >   interrupts
> >
> >Xen will directly manage the in/out console ring buffers (allocated by
> >dom0 for dom0-domU console communication) for reading/writing console
> >data from/to dom0. On the other side, Xen HYP will emulate pl011 to
> >read/write data from/to domU and pass it on to/from dom0 over the
> >in/out console ring buffers. There should be no change in dom0 as it
> >will still use the same ring buffers. Similarly there should be no
> >change in domU which would be running a standard pll011 driver.
> >
> >Currently, I am working on the interface between dom0 and Xen HYP. I
> >want to intercept the console events in Xen HYP which pass between
> >dom0 and domU. For now, I just want to capture console data coming
> >from dom0 at Xen HYP and loop it back to dom0, to confirm that this
> >interface is working.
> >
> >Since each guest domain will have a unique event channel assigned for
> >console communication, Xen HYP can find out the event channel for a
> >given domU from the start_info page of that domU, which should have
> 
> The start_info page is x86 specific. If you want to get the console
> event channel for ARM, you would have to use
> d->arch.hvm_domain.params[HVM_PARAM_CONSOLE_EVTCHN].
> 
> This parameter will be setup by the toolstack (see alloc_magic_pages
> in libxc/xc_dom_arm.c).
> 
> >been allocated by dom0. Whenever, an event is to be dispatched via
> >evtchn_send() API in Xen, it can check if the event channel is the
> >console event channel for a given domU. If yes and its source domain
> >is dom0 and destination domain is domU then it will write the data
> >back to the console out ring buffer of the domU and raise a console
> >event to dom0.
> >
> >Once this interface is working, Xen HYP can check the source and
> >destination dom ids and decide which way the event came from and
> >accordingly process the console data. To allow a mix of PV console
> >guests and pl011 guests, Xen might have to maintain a flag per domain,
> >which tells whether Xen HYP should intercept and process the data (for
> >pl011 UART case) or let it go transparently (for PV conosle case).
> 
> I am not very familiar with the event channel code. I will let the
> others comment on this bit.
> 
> Regardless that, how would you decide whether the hypervisor should
> intercept the notification?
> 
> I can see 2 different cases:
>   1) The guest is starting to use the pl011 then move to the HVC
> console (or HVC then pl011)
>   2) The guest is using both the PL011 and the HVC console
> 
> Should we consider the second case valid? I would say yes, because a
> user could specify both on the command line. If we use the same
> ring, the output would be a total garbage.
> 
> So maybe we need to allocate two distinct rings and event channel?

This sounds like the only sensible thing to me.  I think this is really
about adding a new device to the Xen virtual platform, and providing the
user the option to choose which one he wants the tool in Dom0 to be
presented using stdin/out.  Presumably the other console/serial can be
redirected to a file or socket or something?


Thanks,
-Christoffer

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest

[Christoffer Dall]

On Wed, Nov 30, 2016 at 02:26:50PM -0800, Stefano Stabellini wrote:
> On Wed, 30 Nov 2016, Julien Grall wrote:
> > Hi all,
> > 
> > Few months ago, Linaro has published the version 2 of the VM specification
> > [1].
> > 
> > For those who don't know, the specification provides guidelines to 
> > guarantee a
> > compliant OS images could run on various hypervisor (e.g Xen, KVM).
> > 
> > Looking at the specification, it will require Xen to expose new devices to 
> > the
> > guest: pl011, rtc, persistent flash (for UEFI variables).
> > 
> > The RTC and persistent will only be used by the UEFI firwmare. The firwmare 
> > is
> > custom made for Xen guest and be loaded by the toolstack, so we could
> > theoretically provide PV drivers for those.
> > 
> > This is not the case for the PL011. The guest will be shipped with a
> > PL011/SBSA UART driver,.This means it will expect to access it through MMIO.
> > 
> > So we have to emulate a PL011. The question is where? Before suggesting some
> > ideas, the guest/user will expect to be able to interact with the console
> > through the UART. This means that the UART and xenconsoled needs to
> > communicate together.
> > 
> > I think we can distinct two places where the PL011 could be emulated:
> > in the hypervisor, or outside the hypervisor.
> > 
> > Emulating the UART in the hypervisor means that we take the risk to increase
> > to the attack surface of Xen if there is a bug in the emulation code. The
> > attack surface could be reduced by emulating the UART in another exception
> > level (e.g EL1, EL0) but still under the control of the hypervisor. Usually
> > the guest is communicating between with xenconsoled using a ring. For the
> > first console this could be discovered using hypercall HVMOP_get_param. For
> > the second and onwards, it described in xenstore. I would not worry too much
> > about emulating multiple PL011s, so we could implement the PV frontend in 
> > Xen.
> > 
> > Emulating the UART outside the hypervisor (e.g in DOM0 or special domain)
> > would require to bring the concept of ioreq server on ARM. Which left the
> > question where do we emulate the PL011? The best place would be xenconsoled.
> > But I am not sure how would be the security impact here. Does all guest
> > consoles are emulated within the same daemon?
> 
> One xenconsoled instance handles all PV console frontends. However QEMU,
> not xenconsoled, handles secondary PV consoles and emulated serials. One
> QEMU instance per VM. The PV console protocol is pretty trivial and not
> easy to exploit.
> 
> Instead of emulating PL011 in xenconsoled we could do it in QEMU (or
> something like it). But I think we should do something else instead, see
> below.
> 
> 
> > I would lean towards the first solution if we implement all the security
> > safety I mentioned. Although, the second solution would be a good move if we
> > decide to implement more devices (e.g RTC, pflash) in the future.
> > 
> > Do you have any opinions?
> 
> As I have just written in this other email:
> 
> http://marc.info/?l=xen-devel=148054285829397
> 
> I don't think we should introduce userspace emulators in Dom0 on ARM.
> The reason is that they are bad both for security and performance. In
> general I think that the best solution is to introduce emulators in EL1
> in Xen.

Slightly hijacking the discussion:

Why would you run stuff in EL1 as opposed to EL0?

My understanding of the HCR_EL2.TGE bit is exactly designed to run
bare-metal applications on top of a hypervisor running in EL2 and has
the added benefit that I think you can switch to your bare-metal
application without having to context-switch any EL1 tate, because TGE
just disabled the whole stage 1 MMU.  Obviously this doesn't give you
virtual memory, unless you map that using stage 2 tables, but that might
actually be preferred, and since the whole point is to separate your
non-privileged hypervisor applications from the hypervisor's memory
anyhow, it may be fine.

I don't know enough about Xen's impementation to say whether this is
doable or not, but I think there are potential advantages in terms of
performance and using the architecture as designed.

Thinking about this, if you run something in EL1 on top of Xen without
setting the TGE bit, what happens if your application misbehaves, for
example causes a page fault?  That would be taken at EL1, right?  So
will you install an EL1 handler for exceptions that somehow forwards the
trap into EL2?  Sounds cleaner to me to catch anything weird the
application does to EL2.

-Christoffer

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Xen ARM - Exposing a PL011 to the guest

[Christoffer Dall]

On Wed, Nov 30, 2016 at 03:29:32PM +, Julien Grall wrote:
> Hi all,
> 
> Few months ago, Linaro has published the version 2 of the VM
> specification [1].
> 
> For those who don't know, the specification provides guidelines to
> guarantee a compliant OS images could run on various hypervisor (e.g
> Xen, KVM).
> 
> Looking at the specification, it will require Xen to expose new
> devices to the guest: pl011, rtc, persistent flash (for UEFI
> variables).
> 
> The RTC and persistent will only be used by the UEFI firwmare. 

Why would a guest booting without UEFI not want to use the RTC directly?

Linux does this on KVM today...

-Christoffer

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH 1/4] Support for building in a Xen binary

[Christoffer Dall]

On Tue, Jul 12, 2016 at 11:40:28AM +0100, Julien Grall wrote:
> Hi Andre,
> 
> On 20/06/16 16:09, Andre Przywara wrote:
> >diff --git a/Makefile.am b/Makefile.am
> >index 692d2cc..1a801c0 100644
> >--- a/Makefile.am
> >+++ b/Makefile.am
> >@@ -85,7 +85,6 @@ TEXT_LIMIT := 0x8008
> >  endif
> >
> >  LD_SCRIPT  := model.lds.S
> >-IMAGE   := linux-system.axf
> >
> >  FS_OFFSET  := 0x1000
> >  FILESYSTEM_START:= $(shell echo $$(($(PHYS_OFFSET) + $(FS_OFFSET
> >@@ -108,6 +107,11 @@ CHOSEN_NODE := chosen { 
> >\
> >};
> >  endif
> >
> >+if XEN
> >+XEN := -DXEN=$(XEN_IMAGE)
> >+XEN_OFFSET  := 0x0820
> >+endif
> >+
> >  CPPFLAGS   += $(INITRD_FLAGS)
> >  CFLAGS += -Iinclude/ -I$(ARCH_SRC)/include/
> >  CFLAGS += -Wall -fomit-frame-pointer
> >@@ -117,11 +121,11 @@ LDFLAGS+= --gc-sections
> >  OFILES += boot_common.o bakery_lock.o platform.o $(GIC) 
> > cache.o lib.o
> >  OFILES += $(addprefix $(ARCH_SRC),boot.o stack.o $(BOOTMETHOD) 
> > utils.o)
> >
> >-all: $(IMAGE)
> >+all: $(IMAGE) $(XIMAGE)
> 
> I cannot find where XIMAGE is set. What XIMAGE is used for?
> 
I can't remember (it's been ages since I wrote this patch, then
something happened that required a more substantial rewrite of the whole
bootwrapper, which I think was then aborted anyhow, and this got lost
somehow, or something like that, and I've sort of lost momentum here),
but I think the idea was that you would build the normal linux Image
IMAGE, and XIMAGE for the xen image, so that a single build would output
both images so you could test with the same settings if the basic build
worked, and then test with Xen.

-Christoffer

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Dom0 crash with apache bench (ab)

[Christoffer Dall]

On Mon, Sep 14, 2015 at 5:20 PM, Ian Campbell <ian.campb...@citrix.com>
wrote:

> On Mon, 2015-09-14 at 14:40 +0200, Christoffer Dall wrote:
> > On Fri, Jul 31, 2015 at 03:17:56PM +0200, Christoffer Dall wrote:
> > > On Fri, Jul 31, 2015 at 12:28 PM, David Vrabel <
> david.vra...@citrix.com
> > > >
> > > wrote:
> > >
> > > > On 31/07/15 11:24, Stefano Stabellini wrote:
> > > > > This is a Linux Dom0 crash on x86 (Dell PowerEdge R320, Xeon E5
> > > > > -2450),
> > > > > CC'ing relevant people. As you can see from the links below the
> > > > > crash
> > > > > is:
> > > > >
> > > > > [ 253.619326] Call Trace:
> > > > > [ 253.619330] 
> > > > > [ 253.619332] [] ? skb_copy_ubufs+0xa5/0x230
> > > > > [ 253.619347] []
> > > > > __netif_receive_skb_core+0x6f5/0x940
> > > > > [ 253.619353] [] __netif_receive_skb+0x18/0x60
> > > > > [ 253.619360] []
> > > > > netif_receive_skb_internal+0x28/0x90
> > > > > [ 253.619366] [] napi_gro_frags+0x125/0x1a0
> > > > > [ 253.619378] []
> > > > > mlx4_en_process_rx_cq+0x753/0xb50
> > > > [mlx4_en]
> > > > > [ 253.619387] [] mlx4_en_poll_rx_cq+0x97/0x160
> > > > [mlx4_en]
> > > >
> > > > What makes you think this is Xen specific?  I suggest raising this
> > > > the
> > > > the mlx4 maintainers.
> > > >
> > > >
> > > Linux native and KVM guests (same hw, same kernel version+config) run
> > > just
> > > fine under the same workload.
> > >
> > Ping?
> >
> > From the fact that bare-metal and KVM works fine with this hardware I
> > still think it's reasonable to assume that it's a Xen issue and not a
> > mlx4 issue.
> >
> > Is this completely flawed?
>
> My (somewhat educated) guess is that this is to do with the difference
> between (pseudo-)physical addresses and machine (AKA real-physical)
> addresses when running under Xen.
>
> The way this often shows up is in drivers which do not make correct use of
> the kernels DMA APIs but which happen to work on native x86 because
> physical==bus address on x86.
>
> Sometimes booting natively with 'iommu=soft swiotlb=force' can expose these
> sorts of issues.
>

Indeed it does, on both v4.0 and v4.3-rc2.


>
> You are running 64-bit so I don't think the recent "config: Enable
> NEED_DMA_MAP_STATE by default when SWIOTLB is selected" is likely to be
> relevant (it's already unconditionally on for 64-bit).
>
> The trace appears to be on rx from a physical nic, there shouldn't be any
> magic Xen stuff (granted pages etc) getting themselves into that path at
> all. If it were tx then maybe it might be an issue with foreign pages. In
> any case I think you are able to repro with just dom0, i.e. never having
> started a domU, is that right?
>
>
Yes, I can reproduce on Dom0.

I will send this to the Mellanox people.

Thanks,
-Christoffer
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH] KVM: arm64: add workaround for Cortex-A57 erratum #852523

[Christoffer Dall]

On Mon, Sep 14, 2015 at 04:46:28PM +0100, Marc Zyngier wrote:
> On 14/09/15 16:06, Will Deacon wrote:
> > When restoring the system register state for an AArch32 guest at EL2,
> > writes to DACR32_EL2 may not be correctly synchronised by Cortex-A57,
> > which can lead to the guest effectively running with junk in the DACR
> > and running into unexpected domain faults.
> > 
> > This patch works around the issue by re-ordering our restoration of the
> > AArch32 register aliases so that they happen before the AArch64 system
> > registers. Ensuring that the registers are restored in this order
> > guarantees that they will be correctly synchronised by the core.
> > 
> > Cc: 
> > Cc: Marc Zyngier 
> > Signed-off-by: Will Deacon 
> 
> Reviewed-by: Marc Zyngier 
> 

Looks good to me too, and I just gave this a spin with my loop test on
Juno as well.

Thanks,
-Christoffer

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Dom0 crash with apache bench (ab)

[Christoffer Dall]

On Fri, Jul 31, 2015 at 03:17:56PM +0200, Christoffer Dall wrote:
> On Fri, Jul 31, 2015 at 12:28 PM, David Vrabel <david.vra...@citrix.com>
> wrote:
> 
> > On 31/07/15 11:24, Stefano Stabellini wrote:
> > > This is a Linux Dom0 crash on x86 (Dell PowerEdge R320, Xeon E5-2450),
> > > CC'ing relevant people. As you can see from the links below the crash
> > > is:
> > >
> > > [ 253.619326] Call Trace:
> > > [ 253.619330] 
> > > [ 253.619332] [] ? skb_copy_ubufs+0xa5/0x230
> > > [ 253.619347] [] __netif_receive_skb_core+0x6f5/0x940
> > > [ 253.619353] [] __netif_receive_skb+0x18/0x60
> > > [ 253.619360] [] netif_receive_skb_internal+0x28/0x90
> > > [ 253.619366] [] napi_gro_frags+0x125/0x1a0
> > > [ 253.619378] [] mlx4_en_process_rx_cq+0x753/0xb50
> > [mlx4_en]
> > > [ 253.619387] [] mlx4_en_poll_rx_cq+0x97/0x160
> > [mlx4_en]
> >
> > What makes you think this is Xen specific?  I suggest raising this the
> > the mlx4 maintainers.
> >
> >
> Linux native and KVM guests (same hw, same kernel version+config) run just
> fine under the same workload.
> 
Ping?

>From the fact that bare-metal and KVM works fine with this hardware I
still think it's reasonable to assume that it's a Xen issue and not a
mlx4 issue.

Is this completely flawed?

Thanks,
-Christoffer

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Dom0 crash with apache bench (ab)

[Christoffer Dall]

On Mon, Sep 14, 2015 at 5:20 PM, Ian Campbell <ian.campb...@citrix.com>
wrote:

> On Mon, 2015-09-14 at 14:40 +0200, Christoffer Dall wrote:
> > On Fri, Jul 31, 2015 at 03:17:56PM +0200, Christoffer Dall wrote:
> > > On Fri, Jul 31, 2015 at 12:28 PM, David Vrabel <
> david.vra...@citrix.com
> > > >
> > > wrote:
> > >
> > > > On 31/07/15 11:24, Stefano Stabellini wrote:
> > > > > This is a Linux Dom0 crash on x86 (Dell PowerEdge R320, Xeon E5
> > > > > -2450),
> > > > > CC'ing relevant people. As you can see from the links below the
> > > > > crash
> > > > > is:
> > > > >
> > > > > [ 253.619326] Call Trace:
> > > > > [ 253.619330] 
> > > > > [ 253.619332] [] ? skb_copy_ubufs+0xa5/0x230
> > > > > [ 253.619347] []
> > > > > __netif_receive_skb_core+0x6f5/0x940
> > > > > [ 253.619353] [] __netif_receive_skb+0x18/0x60
> > > > > [ 253.619360] []
> > > > > netif_receive_skb_internal+0x28/0x90
> > > > > [ 253.619366] [] napi_gro_frags+0x125/0x1a0
> > > > > [ 253.619378] []
> > > > > mlx4_en_process_rx_cq+0x753/0xb50
> > > > [mlx4_en]
> > > > > [ 253.619387] [] mlx4_en_poll_rx_cq+0x97/0x160
> > > > [mlx4_en]
> > > >
> > > > What makes you think this is Xen specific?  I suggest raising this
> > > > the
> > > > the mlx4 maintainers.
> > > >
> > > >
> > > Linux native and KVM guests (same hw, same kernel version+config) run
> > > just
> > > fine under the same workload.
> > >
> > Ping?
> >
> > From the fact that bare-metal and KVM works fine with this hardware I
> > still think it's reasonable to assume that it's a Xen issue and not a
> > mlx4 issue.
> >
> > Is this completely flawed?
>
> My (somewhat educated) guess is that this is to do with the difference
> between (pseudo-)physical addresses and machine (AKA real-physical)
> addresses when running under Xen.
>
> The way this often shows up is in drivers which do not make correct use of
> the kernels DMA APIs but which happen to work on native x86 because
> physical==bus address on x86.
>
> Sometimes booting natively with 'iommu=soft swiotlb=force' can expose these
> sorts of issues.
>

I'll give this a try.


>
> You are running 64-bit so I don't think the recent "config: Enable
> NEED_DMA_MAP_STATE by default when SWIOTLB is selected" is likely to be
> relevant (it's already unconditionally on for 64-bit).
>
> The trace appears to be on rx from a physical nic, there shouldn't be any
> magic Xen stuff (granted pages etc) getting themselves into that path at
> all. If it were tx then maybe it might be an issue with foreign pages. In
> any case I think you are able to repro with just dom0, i.e. never having
> started a domU, is that right?
>

As far as I remember and as far as I can interpret my own e-mail, yes.

Thanks for the feedback, I'll try the suggested approaches and also try
using v4.3-rc1 and take it up with the mlx4 maintainers if I still see the
issue.

-Christoffer
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 5

[Christoffer Dall]

Hi Jan,

On Wed, Sep 02, 2015 at 02:41:36AM -0600, Jan Beulich wrote:
> >>> Shannon Zhao  09/02/15 8:03 AM >>>
> >There are some descriptions in Documentation/arm64/booting.txt of Linux:
> >
> >"The Image must be placed text_offset bytes from a 2MB aligned base
> >address near the start of usable system RAM and called there. Memory
> >below that base address is currently unusable by Linux, and therefore it
> >is strongly recommended that this location is the start of system RAM.
> >At least image_size bytes from the start of the image must be free for
> >use by the kernel."
> >
> >From this, it says "Memory below that base address is currently unusable
> >by Linux". So if we put these tables below Dom0 RAM address and even
> >describe these regions as RAM, the Linux could not use them.
> 
> May I remind you that a design should not take specific guest OS
> implementation details (which even for that one OS may change over time)
> as the basis for decisions?
> 
While I agree that the guest behavior should not dictate an unfortunate
design, surely factoring in the behavior of the expected guests in the
design is a reasonable thing to do?

Changing the boot requirements of Linux for an architecture is a really
invasive change, IMHO, and should be avoided if possible.

Are there other acceptable solutions for placing the EFI tables
somewhere else that would work?

Thanks,
-Christoffer

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 5

[Christoffer Dall]

On Wed, Sep 2, 2015 at 3:54 PM, Ian Campbell  wrote:
> On Wed, 2015-09-02 at 14:48 +0100, Julien Grall wrote:
>> On 02/09/15 14:26, Ian Campbell wrote:
>> > > > > I think the problem is how you reserved this region in the EFI
>> > > > > memory
>> > > > > table. From what I saw, you marked this new memory with
>> > > > > EFI_MEMORY_WB
>> > > > > (which means that the region can be usable by Linux).
>> > > > >
>> > > > Yes, I mark it with EFI_MEMORY_WB. Is this right?
>> > >
>> > > I would say no, but it's only because I looked at the kernel code
>> > > quickly.
>> > >
>> > > You have to looks how ACPI region/UEFI tables are described in the
>> > > host
>> > > EFI memory map and mimicking for the DOM0 EFI memory map.
>> >
>> > Surely it is the type (EfiACPIReclaimMemory, EfiACPIMemoryNVS etc) and
>> > not
>> > the mapping attributes which should control whether an OS considers a
>> > region usable? At least until the OS is done parsing tables neither of
>> > those are usable (which implies we want NVS as our type, unless the
>> > memory
>> > is intended to be reclaimed by dom0, implying it should own it).
>>
>> It looks like that Linux on ARM64 is considering any region with
>> EFI_MEMORY_WB set as normal RAM and will try to add as System RAM (see
>> reserve_regions in arch/arm64/kernel/efi.c).
>
> It's hard to believe this isn't a bug... It's probably worth asking the
> Linux maintainers about this.
>

wasn't this that whole workaround to make sure Linux maps the data as
regular RAM, because otherwise architecture generic code would map it
as IO memory, and generic routines such as memcpy would fault on
unaligned accesses, or am I confusing ACPI with EFI here?

Leif (added to the to-field) had some insight on this earlier on.

-Christoffer

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2

[Christoffer Dall]

On Thu, Aug 13, 2015 at 8:41 AM, Jan Beulich jbeul...@suse.com wrote:
 On 12.08.15 at 18:18, julien.gr...@citrix.com wrote:
 On 12/08/15 16:58, Jan Beulich wrote:
 On 12.08.15 at 17:51, christoffer.d...@linaro.org wrote:
 On Wed, Aug 12, 2015 at 5:45 PM, Jan Beulich jbeul...@suse.com wrote:
 On 07.08.15 at 04:11, zhaoshengl...@huawei.com wrote:
 All these tables will be copied to Dom0 memory except that the reused
 tables(DSDT, SPCR, etc) will be mapped to Dom0.

 Which again seems odd - such tables should be considered MMIO
 (despite living in RAM), and hence not be part of Dom0's memory
 assignment.

 not sure if this applies to the context of your comment, but we had
 issues before when trying to deal with this data as device memory
 (MMIO), because ARM doesn't allow unaligned accesses etc. on device
 memory, and so Dom0 would crash at memory abort exceptions during
 boot, so this really does have to be mapped as RAM to Dom0.  If you
 meant some internal Xen bookkeeping thing, then just ignore me.

 Hmm, how would native Linux avoid such unaligned accesses then?

 ACPI table are living on RAM on ARM. So there is no issue with Linux
 baremetal.

 I'm sure our interpretation of the meaning of RAM here differs:
 While from a physical perspective the tables live in RAM too on
 x86, from a memory map perspective they don't. And since iirc
 ACPI implies UEFI, and since UEFI has special memory types
 for such tables (to prevent the OS from using the space as
 normal memory), I can't believe this to be different under ARM.

 The problem is from Xen where we are mapping memory region with Device
 attribute. This is because until now we never had a reason to directly
 map other thing than MMIO to the domain.

 This could be fixed by adding new helper in Xen to directly map RAM region.

 Which would seem to be the correct route.

 Nonetheless, we still have to copy some table in Xen in order to modify
 them and/or new one. I have in mind the FADT table to set the hypervisor
 field and hiding the hypervisor specific data (GIC hyp, timer hyp...) to
 avoid the kernel thinks there is hyp mode available.

 Have to? Or rather would like to? As said in another reply to
 Shannon, I didn't see any rationale spelled out for this fundamental
 difference to x86.

So the suggestion is to just edit the tables in place rather than
copying them and relying on Xen having already read whatever
'original' information it needs before modifying them?

Is there ever a case where the tables live in ROM or shouldn't be
modified for any other reason?

-Christoffer

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2

[Christoffer Dall]

On Thu, Aug 13, 2015 at 11:22:19AM +0100, Ian Campbell wrote:
 On Thu, 2015-08-13 at 11:13 +0100, Stefano Stabellini wrote:
  
For example it is only natural for the kernel to try to use the GIC 
hyp
functionalities if they are described, while actually they are not
emulated by Xen at all.
   
   See Ian's earlier reply: It can also be considered natural for it to
   be aware that when run in EL2 to not use EL1 functionality.
 
 NB EL2 == Hyp and EL1 == Kernel, so it's the other way round, FWIW.
 
  It is not just about the GIC Hyp functionalities.
 
 What else is there which is not subject to this logic? Timers are too, it
 even applies to IOMMU's which have both stage1 and stage2 bits.
 
 BTW, I think kernels _already_ need to deal with a lot of this because in
 reality nobody modifies the DTB when they use a firmware which launches the
 kernel in EL1. IOW I think the kernel is already aware of which resources
 can be used by which privilege level.
 
Yes, for resources specific to EL2 I believe that is indeed the case
(the GIC driver doesn't look at the hypervisor control register address,
and KVM does not even get that far if you're not booted in EL2, and the
timer only uses the virtual timer if not booted in EL2 - we never
attempt to use the hyp timer until Marc's VHE patches land, but they
also depend on being booted in hyp mode).

However, what about for other resources?  Having code somewhere that
says hide this random piece of hardware if you're Xen dom0 sounds
awful to me.  I know it's only the serial port right now, but still.

-Christoffer

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2

[Christoffer Dall]

On Fri, Aug 7, 2015 at 12:33 PM, Julien Grall julien.gr...@citrix.com wrote:
 Hi Shannon,

 Just some clarification questions.

 On 07/08/15 03:11, Shannon Zhao wrote:
 3. Dom0 gets grant table and event channel irq information
 ---
 As said above, we assign the hypervisor_id be XenVMM to tell Dom0 that
 it runs on Xen hypervisor.

 For grant table, add two new HVM_PARAMs: HVM_PARAM_GNTTAB_START_ADDRESS
 and HVM_PARAM_GNTTAB_SIZE.

 For event channel irq, reuse HVM_PARAM_CALLBACK_IRQ and add a new
 delivery type:
 val[63:56] == 3: val[15:8] is flag: val[7:0] is a PPI (ARM and ARM64
 only)

 Can you describe the content of flag?

 When constructing Dom0 in Xen, save these values. Then Dom0 could get
 them through hypercall HVMOP_get_param.

 4. Map MMIO regions
 ---
 Register a bus_notifier for platform and amba bus in Linux. Add a new
 XENMAPSPACE XENMAPSPACE_dev_mmio. Within the register, check if the
 device is newly added, then call hypercall XENMEM_add_to_physmap to map
 the mmio regions.

 5. Route device interrupts to Dom0
 --
 Route all the SPI interrupts to Dom0 before Dom0 booting.

 Not all the SPI will be routed to DOM0. Some are used by Xen and should
 never be used by any guest. I have in mind the UART and SMMU interrupts.

 You will have to find away to skip them nicely. Note that not all the
 IRQs used by Xen are properly registered when we build DOM0 (see the SMMU).

Just to clarify; Xen should map all SPIs which are not reserved for
Xen to Dom0, right?

-Christoffer

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen

[Christoffer Dall]

Hi Shannon,

On Tue, Aug 4, 2015 at 3:43 PM, Shannon Zhao zhaoshengl...@huawei.com wrote:
 This document is going to explain the design details of Xen booting with
 ACPI on ARM. Maybe parts of it may not be appropriate. Any comments are
 welcome.

 To Xen itself booting with ACPI, this is similar to Linux kernel except
 that Xen doesn't parse DSDT table. So I'll skip this part and focus on
 how Xen prepares ACPI tables for DOM0 and how Xen passes them to DOM0.

 1)copy and change some EFI and ACPI tables.
   a) Copy EFI_SYSTEM_TABLE and change the value of FirmwareVendor,
  VendorGuid, VendorTable, ConfigurationTable. These changes are not
  very special and it just assign values to these members.
   b) Create EFI_MEMORY_DESCRIPTOR table. This will add memory start and
  size information of DOM0. And DOM0 will get the memory information
  through this EFI table.

Are the tables above EFI tables and all the tables below ACPI tables?

   c) Copy FADT table. Change the value of arm_boot_flags to enable PSCI
  and HVC. Let the hypervisor_id be XenVMM in order to tell DOM0
  that it runs on Xen hypervisor, so DOM0 can call hypercall to get
  some informations for booting necessity, such as grant tab start
  address and size. Change header revison, length and checksum as
  well.
   d) Copy GTDT table. Set non_secure_el2_interrupt and
  non_secure_el2_flags to 0 to mask EL2 timer for DOM0.

Is this necessary, doesn't Linux notice that it's booted in EL1 and
will ignore everything related to EL2?

   e) Copy MADT table. According to the value of dom0_max_vcpus to change
  the number GICC entries.
   f) Create STAO table. This table is a new added one that's used to
  define a list of ACPI namespace names that are to be ignored by the
  OSPM in DOM0. Currently we use it to tell OSPM should ignore UART
  defined in SPCR table.
   g) Copy XSDT table. Add a new table entry for STAO and change other
  table's entries.
   h) Change the value of xsdt_physical_address in RSDP table.
   i) The reset of tables are not copied or changed. They are reused

s/reset/rest/ ?

[...]

-Christoffer

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Dom0 crash with apache bench (ab)

[Christoffer Dall]

On Fri, Jul 31, 2015 at 12:28 PM, David Vrabel david.vra...@citrix.com
wrote:

 On 31/07/15 11:24, Stefano Stabellini wrote:
  This is a Linux Dom0 crash on x86 (Dell PowerEdge R320, Xeon E5-2450),
  CC'ing relevant people. As you can see from the links below the crash
  is:
 
  [ 253.619326] Call Trace:
  [ 253.619330] IRQ
  [ 253.619332] [815d7c25] ? skb_copy_ubufs+0xa5/0x230
  [ 253.619347] [815e8525] __netif_receive_skb_core+0x6f5/0x940
  [ 253.619353] [815e8788] __netif_receive_skb+0x18/0x60
  [ 253.619360] [815e87f8] netif_receive_skb_internal+0x28/0x90
  [ 253.619366] [815e91f5] napi_gro_frags+0x125/0x1a0
  [ 253.619378] [a01b1173] mlx4_en_process_rx_cq+0x753/0xb50
 [mlx4_en]
  [ 253.619387] [a01b1657] mlx4_en_poll_rx_cq+0x97/0x160
 [mlx4_en]

 What makes you think this is Xen specific?  I suggest raising this the
 the mlx4 maintainers.


Linux native and KVM guests (same hw, same kernel version+config) run just
fine under the same workload.

Thanks,
-Christoffer
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] Dom0 crash with apache bench (ab)

[Christoffer Dall]

Hi,

I've been doing some performance comparisons lately, and wanted to compare
the performance overhead of using Xen with apache bench, but unfortunately
the Dom0 kernel crashes when hitting it with ab from a remote machine.
Most other workloads seem to be stable, however, I do see similar crashes
if hitting Dom0 mysql with a mysql benchmark with a high level of
parallelism.

I use a 10G Mellanox MX354A Dual port FDR CX3 adapter for networking on a
Dell PowerEdge R320 system with a Xeon E5-2450 and 16 GB of RAM.

Interestingly, we had a similarly looking issue on arm64 recently, but that
was fixed with an APM-soecific fix to the hypervisor, so I am guessing this
is unrelated, see:
http://lists.xenproject.org/archives/html/xen-devel/2015-03/msg02731.html
and the fix:
http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=50dcb3de603927db2fd87ba09e29c817415aaa44

I have tried with several Linux versions, v3.13, v3.18, v4.0-rc4, and v4.1,
same issue.  I have tried with Xen 4.5-0 release, and the Ubuntu packaged
Xen 4.4 release, same issue.

Examples of crash:
http://pastebin.ubuntu.com/11953498/
http://pastebin.ubuntu.com/11953443/

Running DomU with a bridge and running ab against apache running in a DomU
also causes the system to crash.

Note: The server also has an embedded 1G Broadcom NIC (although not
suitable for testing due to it being 1G and on a control network), and
using that for the test does not cause a system crash, so this points to
some difficulties with the Mellanox device and Xen.

Any ideas or advice is greatly appreciated, thanks.

-Christoffer
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Dom0 crash with apache bench (ab)

[Christoffer Dall]

On Tue, Jul 28, 2015 at 4:55 PM, Ian Campbell ian.campb...@citrix.com
wrote:

 On Tue, 2015-07-28 at 10:50 -0400, Konrad Rzeszutek Wilk wrote:
  On Tue, Jul 28, 2015 at 03:09:31PM +0200, Christoffer Dall wrote:
   Hi,
  
   I've been doing some performance comparisons lately, and wanted to
   compare
   the performance overhead of using Xen with apache bench, but
   unfortunately
   the Dom0 kernel crashes when hitting it with ab from a remote machine.
   Most other workloads seem to be stable, however, I do see similar
   crashes
   if hitting Dom0 mysql with a mysql benchmark with a high level of
   parallelism.
  
   I use a 10G Mellanox MX354A Dual port FDR CX3 adapter for networking on
   a
   Dell PowerEdge R320 system with a Xeon E5-2450 and 16 GB of RAM.
  
   Interestingly, we had a similarly looking issue on arm64 recently, but
   that
   was fixed with an APM-soecific fix to the hypervisor, so I am guessing
   this
   is unrelated, see:
   http://lists.xenproject.org/archives/html/xen-devel/2015
   -03/msg02731.html
   and the fix:
  
 http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=50dcb3de603927db2fd
   87ba09e29c817415aaa44
  
   I have tried with several Linux versions, v3.13, v3.18, v4.0-rc4, and
   v4.1,
   same issue.  I have tried with Xen 4.5-0 release, and the Ubuntu
   packaged
   Xen 4.4 release, same issue.
  
   Examples of crash:
   http://pastebin.ubuntu.com/11953498/
   http://pastebin.ubuntu.com/11953443/
 
  4.0-rc4?
 
  Have you tried 4.1?

 According to the previous paragraph, yes he has.

 yes, I have.  Just for clarify, I used 4.0-rc4 because that's a branch
which contained arm64 PCI support and has been used for other measurements,
so this was simply my 'working tree'.

Thanks,
-Christoffer
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH] xen: arm: X-Gene Storm check GIC DIST address for EOI quirk

[Christoffer Dall]

Hi Pranav,

On Mon, Apr 06, 2015 at 02:24:01PM +0530, Pranavkumar Sawargaonkar wrote:
 In old X-Gene Storm firmware and DT, secure mode addresses have been
 mentioned in GICv2 node. In this case maintenance interrupt is used
 instead of EOI HW method.
 
 This patch checks the GIC Distributor Base Address to enable EOI quirk
 for old firmware.
 
 Ref:
 http://lists.xen.org/archives/html/xen-devel/2014-07/msg01263.html
 
 Signed-off-by: Pranavkumar Sawargaonkar pranavku...@linaro.org

I have tested this on the m400 cluster and can confirm that it prevents
the issue with ab trashing the machine:

Tested-by: Christoffer Dall christoffer.d...@linaro.org

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Xen unstability on HP Moonshot m400

[Christoffer Dall]

On Tue, Mar 24, 2015 at 3:00 PM, Mark Salter msal...@redhat.com wrote:

 On Tue, 2015-03-24 at 09:54 -0400, Mark Salter wrote:
  On Mon, 2015-03-23 at 23:58 +, Stefano Stabellini wrote:
   On Mon, 23 Mar 2015, Christoffer Dall wrote:
On Mon, Mar 23, 2015 at 1:36 PM, Ian Campbell 
 ian.campb...@citrix.com wrote:
  On Sat, 2015-03-21 at 13:34 +0100, Christoffer Dall wrote:
   Hi,
  
   I have been experiencing a problematic crash running Xen on
 m400 over
   the last few days.  I already spoke to Ian and Stefano about
 this, but
   thought I'd summarize what I've seen so far and loop in a
 wider
   audience.
  
   The basic setup is this:
- Two m400 nodes, one running Linux bare-metal, the other
 running
   Xen.
- The Xen node runs Dom0 and 1 DomU
- The m400 has a Mellanox Connectx-3 PCIe 10G ethernet card
 with two
   parts on it
- Dom0 uses NAT forwarding from Dom0's eth0 (which is
 connected to
   the internet) and regular bridging to eth1 which is
 connected to a
   private VLAN to the bare-metal node
- Dom0 and DomU are configured with 14GB of ram, 4 cpus each
- DomU runs apache2 serving the GCC manual (see
  
 https://github.com/chazy/kvmperf/blob/master/cmdline_tests/apache_install.sh
 )
  
   The bare-metal node runs apache bench, like this: ab -n
 10 -c 100
  
 http://secure-web.cisco.com/1r5tZ8-7RF8gHRANwFdizEZzgeMsjxVO0yKbYiV4zy7LeiUfYBXMkFq7FGW_SZ1x-VxdzyK-ErDsOUiQ9z2x-N
   
 y7XkL_loHP8ene_BuNFscGyWmQ3r6CtXAYaZCY4xRmmPT1uJOsZDLMu7j-LfCOGmQDSdBwgW7QYukI2bCtTrXM/http%3A%2F%2F10.10.1.120%2F
  gcc%2Findex.html
  
   (10.10.1.120 is the DomU IP address of the bridged interface
 to eth1)
  
   What happens now is that the entire Xen node goes down.  I
 see various
   errors in the kernel log, some examples:
   http://pastebin.ubuntu.com/10642148/
   http://pastebin.ubuntu.com/10642177/
   http://pastebin.ubuntu.com/10642181/
   http://pastebin.ubuntu.com/10635573/
  
  
   All Linux kernels are 3.18 plus some tweaks for the m400
 cartridge:
  
 https://github.com/columbia/linux-kvm-arm/tree/columbia-armvirt-3.18
   
  Is it worth adding
   
 https://git.kernel.org/cgit/linux/kernel/git/arm64/linux.git/commit/?id=285994a62c80f1d72c6924282bcb59608098d5ec
  to your kernel? It isn't Xen specific but it's perhaps
 possible that Xen opens the window wider.
 
  You definitely want that one. Without it, the page table walker could
  end up using a stale pointer to a page being used for something other
  than page tables.
 
   
  How confident are you in
   
 https://github.com/columbia/linux-kvm-arm/commit/5e29cb0478f3d90e4f568d6bea6840960331bcbb
 ?
  (although I suppose you aren't running in ACPI mode if you are
 running
  Xen?)
   
   
I'm not confident at all, but Linux (last I checked was v3.19)
 doesn't boot without it, so not sure if there's an
alternative?  Mark?
  
   This patch is key: it doesn't look like it is setting
   dev-archdata.dma_coherent appropriately, see the implementation of
   set_arch_dma_coherent_ops.
 
  You'd want this if booting with ACPI. You might also need it for
  enumerated PCI devices even if booting with devicetree.

 There's an updated version of this patch for newer kernels in the
 devel branch of git.fedorahosted.org/git/kernel-arm64.git

 There is also this one in Linus' tree which may be of interest to you:

 commit 7132813c384515c9dede1ae20e56f3895feb7f1e
 Author: Suzuki K. Poulose suzuki.poul...@arm.com
 Date:   Thu Mar 19 18:17:09 2015 +

 arm64: Honor __GFP_ZERO in dma allocations


Thanks Mark!

I'll give both a try!

-Christoffer
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Xen unstability on HP Moonshot m400

[Christoffer Dall]

On Mon, Mar 23, 2015 at 1:36 PM, Ian Campbell ian.campb...@citrix.com
wrote:

 On Sat, 2015-03-21 at 13:34 +0100, Christoffer Dall wrote:
  Hi,
 
  I have been experiencing a problematic crash running Xen on m400 over
  the last few days.  I already spoke to Ian and Stefano about this, but
  thought I'd summarize what I've seen so far and loop in a wider
  audience.
 
  The basic setup is this:
   - Two m400 nodes, one running Linux bare-metal, the other running
  Xen.
   - The Xen node runs Dom0 and 1 DomU
   - The m400 has a Mellanox Connectx-3 PCIe 10G ethernet card with two
  parts on it
   - Dom0 uses NAT forwarding from Dom0's eth0 (which is connected to
  the internet) and regular bridging to eth1 which is connected to a
  private VLAN to the bare-metal node
   - Dom0 and DomU are configured with 14GB of ram, 4 cpus each
   - DomU runs apache2 serving the GCC manual (see
 
 https://github.com/chazy/kvmperf/blob/master/cmdline_tests/apache_install.sh
 )
 
  The bare-metal node runs apache bench, like this: ab -n 10 -c 100
 
 http://secure-web.cisco.com/1r5tZ8-7RF8gHRANwFdizEZzgeMsjxVO0yKbYiV4zy7LeiUfYBXMkFq7FGW_SZ1x-VxdzyK-ErDsOUiQ9z2x-Ny7XkL_loHP8ene_BuNFscGyWmQ3r6CtXAYaZCY4xRmmPT1uJOsZDLMu7j-LfCOGmQDSdBwgW7QYukI2bCtTrXM/http%3A%2F%2F10.10.1.120%2Fgcc%2Findex.html
 
 
  (10.10.1.120 is the DomU IP address of the bridged interface to eth1)
 
  What happens now is that the entire Xen node goes down.  I see various
  errors in the kernel log, some examples:
  http://pastebin.ubuntu.com/10642148/
  http://pastebin.ubuntu.com/10642177/
  http://pastebin.ubuntu.com/10642181/
  http://pastebin.ubuntu.com/10635573/
 
 
  All Linux kernels are 3.18 plus some tweaks for the m400 cartridge:
  https://github.com/columbia/linux-kvm-arm/tree/columbia-armvirt-3.18

 Is it worth adding

 https://git.kernel.org/cgit/linux/kernel/git/arm64/linux.git/commit/?id=285994a62c80f1d72c6924282bcb59608098d5ec
 to your kernel? It isn't Xen specific but it's perhaps possible that Xen
 opens the window wider.

 How confident are you in

 https://github.com/columbia/linux-kvm-arm/commit/5e29cb0478f3d90e4f568d6bea6840960331bcbb
 ?
 (although I suppose you aren't running in ACPI mode if you are running
 Xen?)


I'm not confident at all, but Linux (last I checked was v3.19) doesn't boot
without it, so not sure if there's an alternative?  Mark?



 If we think the issue might be to do with coherency of foreign mappings
 undergoing i/o from dom0 and we've already ruled out disk (by using a
 loopback mounted rootfs) then it might be worth bodging netback to
 always copy too.

 Adding a call to skb_orphan_frags right before the netif_receive_skb in
 drivers/net/xen-netback/netback.c:xenvif_tx_submit is a simple but
 rather inefficient way of doing that (so I hope it doesn't perturb the
 issue).


I'll be happy to try this.



 Stefano (who is more familiar with the Linux swiotlb side of things than
 me) is travelling this week so he'll be on West coast time, not sure
 when he gets off a plane nor if he's on email anyway (he's at ELC + this
 ARM ACPI thing)


ok, we'll see what happens.

-Christoffer
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] Xen unstability on HP Moonshot m400

[Christoffer Dall]

Hi,

I have been experiencing a problematic crash running Xen on m400 over the
last few days.  I already spoke to Ian and Stefano about this, but thought
I'd summarize what I've seen so far and loop in a wider audience.

The basic setup is this:
 - Two m400 nodes, one running Linux bare-metal, the other running Xen.
 - The Xen node runs Dom0 and 1 DomU
 - The m400 has a Mellanox Connectx-3 PCIe 10G ethernet card with two parts
on it
 - Dom0 uses NAT forwarding from Dom0's eth0 (which is connected to the
internet) and regular bridging to eth1 which is connected to a private VLAN
to the bare-metal node
 - Dom0 and DomU are configured with 14GB of ram, 4 cpus each
 - DomU runs apache2 serving the GCC manual (see
https://github.com/chazy/kvmperf/blob/master/cmdline_tests/apache_install.sh
)

The bare-metal node runs apache bench, like this: ab -n 10 -c 100
http://10.10.1.120/gcc/index.html;

(10.10.1.120 is the DomU IP address of the bridged interface to eth1)

What happens now is that the entire Xen node goes down.  I see various
errors in the kernel log, some examples:
http://pastebin.ubuntu.com/10642148/
http://pastebin.ubuntu.com/10642177/
http://pastebin.ubuntu.com/10642181/
http://pastebin.ubuntu.com/10635573/

All Linux kernels are 3.18 plus some tweaks for the m400 cartridge:
https://github.com/columbia/linux-kvm-arm/tree/columbia-armvirt-3.18
config: columbia_armvirt_defconfig (from the same tree:
https://github.com/columbia/linux-kvm-arm/blob/columbia-armvirt-3.18/arch/arm64/configs/columbia_armvirt_defconfig
)

I have also tried applying a set of swiotlb fixes provided by Stefano to
both the Dom0 and DomU kernel, like this:
https://github.com/columbia/linux-kvm-arm/commits/columbia-armvirt-3.18-with-xen-fixes

With these patches I sometime also saw this error in the kernel log (but
not always):
http://pastebin.ubuntu.com/10635062/

Other data points of interest:
 - Bare-metal serving apache doesn't exhibit this behavior
 - KVM guests with bridged networking on identical hardware/setup with the
same kernels also don't exhibit this behavior
 - Other physical identical nodes exhibit the same behavior
 - Just running Dom0 serving apache without running DomU doesn't appear to
exhibit this behavior
 - Running apache on Dom0 and benchmarking the system using Dom0's ip
address but running DomU idle in the background causes this behavior (
http://pastebin.ubuntu.com/10642311/), but the system seems to stay alive
(at least for much longer)!

Stefano suggested that this could be related DMA cache coherency, but I'm
not sure how to investigate that further.

This is a somewhat urgent issue for us at Columbia so I would appreciate
any feedback and/or ideas and will be happy to try out any debugging steps
to get to the bottom of this.

Thanks,
-Christoffer
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH 1/4] xen: Correction to module@1 (dom0 kernel) DT node.

[Christoffer Dall]

Hi Ian,

On Mon, Nov 10, 2014 at 02:09:58PM +, Ian Campbell wrote:
 - Include bootargs (kernel command line) property.

Why?

The logic I was going for was to reduce duplicate code/entries in the
DT, and if I read docs/misc/arm/device-tree/booting.txt, the fact that
we have xen,xen-bootargs but no xen,dom0-bootargs then bootargs added
further down in Makefile.am will be used.

Is this not correct or not preferred for some reason?

-Christoffer

 - Update reg property to match #address-cells and #size-cells in the DTB (both
   2).
 
 Signed-off-by: Ian Campbell ian.campb...@citrix.com
 ---
  Makefile.am |3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
 
 diff --git a/Makefile.am b/Makefile.am
 index 9b6c7e3..ed5acbb 100644
 --- a/Makefile.am
 +++ b/Makefile.am
 @@ -69,8 +69,9 @@ XEN_OFFSET  := 0xA0
  DOM0_OFFSET  := $(shell echo $$(($(PHYS_OFFSET) + $(KERNEL_OFFSET
  XEN_BOOTARGS := xen,xen-bootargs = \$(BOOTARGS)\;  \
  module@1 {   \
 + bootargs = \$(CMDLINE)\;  \
   compatible = \xen,linux-zimage\, 
 \xen,multiboot-module\; \
 - reg = $(DOM0_OFFSET) 0x80;\
 + reg = 0 $(DOM0_OFFSET) 0 0x80;\
  };
  endif
  
 -- 
 1.7.10.4
 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel