Re: [Xen-devel] [PATCH] xen: switch to post-init routines in xen mmu.c earlier

[Juergen Gross]

On 12/10/2014 07:07 PM, David Vrabel wrote:

On 10/12/14 15:56, Juergen Gross wrote:

With the virtual mapped linear p2m list the post-init mmu operations
must be used for setting up the p2m mappings, as in case of
CONFIG_FLATMEM the init routines may trigger BUGs.

Reported-by: Boris Ostrovsky 
Signed-off-by: Juergen Gross 
---
  arch/x86/xen/mmu.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
index 6ab6150..a1a429a 100644
--- a/arch/x86/xen/mmu.c
+++ b/arch/x86/xen/mmu.c
@@ -1225,6 +1225,7 @@ static void __init xen_pagetable_p2m_setup(void)
  static void __init xen_pagetable_init(void)
  {
paging_init();
+   xen_post_allocator_init();

xen_pagetable_p2m_setup();



This feels very chicken-and-egg to me:  To setup the P2M we need to use
the MMU ops that use the P2M...

Please explain very clearly why this is all safe.


Okay. paging_init() sets up all infrastructure needed to switch to the
post-init mmu ops done by xen_post_allocator_init(). With the virtual
mapped linear p2m list we need some mmu ops during setup of this list,
so we have to switch to the correct mmu ops as soon as possible.

The p2m list is usable from the beginning, just expansion requires to
have established the new linear mapping. So the call of
xen_remap_memory() had to be introduced, but this is not due to the
mmu ops requiring this.

Summing it up: calling xen_post_allocator_init() not directly after
paging_init() was conceptually wrong in the beginning, it just didn't
matter up to now as no functions used between the two calls needed
some critical mmu ops (e.g. alloc_pte). This has changed now, so I
corrected it.

Juergen




@@ -1236,7 +1237,6 @@ static void __init xen_pagetable_init(void)
xen_remap_memory();

xen_setup_shared_info();
-   xen_post_allocator_init();
  }
  static void xen_write_cr2(unsigned long cr2)
  {



--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH] xen: switch to post-init routines in xen mmu.c earlier

[Juergen Gross]

On 12/10/2014 05:13 PM, Konrad Rzeszutek Wilk wrote:

On Wed, Dec 10, 2014 at 04:56:03PM +0100, Juergen Gross wrote:

With the virtual mapped linear p2m list the post-init mmu operations
must be used for setting up the p2m mappings, as in case of
CONFIG_FLATMEM the init routines may trigger BUGs.


Um, could you explain a bit more of why the CONFIG_FLATMEM
is such unique? What about the other CONFIG_*MEM cases?


CONFIG_FLATMEM is just the configuration hitting a BUG() in
xen_alloc_pte_init() being used after finishing paging_init().

Juergen





Reported-by: Boris Ostrovsky 
Signed-off-by: Juergen Gross 
---
  arch/x86/xen/mmu.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
index 6ab6150..a1a429a 100644
--- a/arch/x86/xen/mmu.c
+++ b/arch/x86/xen/mmu.c
@@ -1225,6 +1225,7 @@ static void __init xen_pagetable_p2m_setup(void)
  static void __init xen_pagetable_init(void)
  {
paging_init();
+   xen_post_allocator_init();

xen_pagetable_p2m_setup();

@@ -1236,7 +1237,6 @@ static void __init xen_pagetable_init(void)
xen_remap_memory();

xen_setup_shared_info();
-   xen_post_allocator_init();
  }
  static void xen_write_cr2(unsigned long cr2)
  {
--
2.1.2


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Frozen dom0 xl commands

[Balbir Singh]

Good catch Ian!

You are absolutely right! I built everything and it put the tools/etc
in /usr/local. I did not see a link to xencommons, I missed it
completely! Is there any thing else I should care about - any other
daemons/bridge to be setup?

Balbir Singh.

On Wed, Dec 10, 2014 at 7:25 PM, Ian Campbell  wrote:
> On Wed, 2014-12-10 at 19:24 +0530, Balbir Singh wrote:
>> I've been facing an issue on my Ubuntu box (acting as dom0) with
>> xen-4.5 (HEAD). I am running dom0 3.13.0.19-generic (ubuntu). When I
>> try and xl command I see the command hangs, after a while I see the
>> guest kernel complain. I've tried a similar setup on another setup and
>> I see the same issue. Any clues on how to debug?
>>
>> FYI: xl dmesg does not show much. I've tried using the console
>> debugger, but not had much luck with it
>
> Is xenstored running?
>
>>
>> [  484.442137] xl  D  0  4590   4122 
>> 0x0004
>> [  484.442146]  88038cf87dd8 0202 8800c458afe0
>> 88038cf87fd8
>> [  484.442156]  00014440 00014440 8800c458afe0
>> 81fc0260
>> [  484.442164]  88038cf87e10 8800c4130058 8800c413004c
>> 8800c413
>> [  484.442172] Call Trace:
>> [  484.442188]  [] schedule+0x29/0x70
>> [  484.442198]  [] read_reply+0x95/0x140
>> [  484.442210]  [] ? prepare_to_wait_event+0x100/0x100
>> [  484.442218]  [] xenbus_dev_request_and_reply+0x8c/0xb0
>> [  484.442226]  [] xenbus_file_write+0x27e/0x540
>> [  484.442236]  [] ? __sb_start_write+0x49/0xe0
>> [  484.442246]  [] ? security_file_permission+0x23/0xa0
>> [  484.442253]  [] vfs_write+0xb4/0x1f0
>> [  484.442261]  [] SyS_write+0x49/0xa0
>> [  484.442269]  [] tracesys+0xe1/0xe6
>> [  484.442276] INFO: task xl:4660 blocked for more than 120 seconds.
>> [  484.442280]   Tainted: PF  O 3.13.0-19-generic #40-Ubuntu
>> [  484.442283] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [  484.442286] xl  D  0  4660   4646 
>> 0x0004
>> [  484.442291]  88038dc41dc0 0202 88038e2797f0
>> 88038dc41fd8
>> [  484.442300]  00014440 00014440 88038e2797f0
>> 81fc0290
>> [  484.442308]  81fc0294 88038e2797f0 
>> 81fc0298
>> [  484.442316] Call Trace:
>> [  484.442325]  [] schedule_preempt_disabled+0x29/0x70
>> [  484.442333]  [] __mutex_lock_slowpath+0x135/0x1b0
>> [  484.442339]  [] mutex_lock+0x1f/0x2f
>> [  484.442347]  [] xenbus_dev_request_and_reply+0x26/0xb0
>> [  484.442355]  [] xenbus_file_write+0x27e/0x540
>> [  484.442364]  [] ? __sb_start_write+0x49/0xe0
>> [  484.442371]  [] ? security_file_permission+0x23/0xa0
>> [  484.442378]  [] vfs_write+0xb4/0x1f0
>> [  484.442386]  [] SyS_write+0x49/0xa0
>> [  484.442393]  [] tracesys+0xe1/0xe6
>>
>> ___
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel
>
>

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] Few Comments on the Xen SMMU ARM code

[manish jaggi]

Based on my experience with PCI passthrough code merging, below are
some comments:
Both require a change in code

a) The current code which is non-pci passthrough requires a devices'
device tree node to be associated with smmu node, if that device has
to be assigned to domU.

In our system there is no platform device which can be passthough. All
devices (including uart) are enumerated using PCI enumeration.
So the device tree looks like

pcie0 {
}

smmu {
mmu-masters = <&pcie0 0x100>;
}

When dom0 boots pcie is assigned to dom0 and a stream ID 0x100 is
created which later conflicts with a valid device enumerated later

b) Current xen code and linux code used rb_tree with key as dt_node to
locate a device and its smmu. With an enumerated device there is no
such thing. So this would not work.


I need your views how PCI passthrough / Non PCI passthrough code can
coexist with the two points mentioned above ?


-Regards
Manish

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] Timed out waiting for device dev-hvc0.device.

[manish jaggi]

I am facing this issue when booting Xen Dom0 (OpenSuse Rootfs)

...

[  OK  ] Reached target Host and Network Name Lookups.
[  OK  ] Started OpenSSH Daemon.
[ TIME ] Timed out waiting for device dev-hvc0.device.
[DEPEND] Dependency failed for Serial Getty on hvc0.
[  OK  ] Reached target Login Prompts.


Any Idea? what could be wrong here


-Manish

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [libvirt bisection] complete build-i386-libvirt

[xen . org]

branch xen-unstable
xen branch xen-unstable
job build-i386-libvirt
test libvirt-build

Tree: gnulib_libvirt 
git://drall.uk.xensource.com:9419/git://git.sv.gnu.org/gnulib.git%20[fetch=try]
Tree: libvirt git://libvirt.org/libvirt.git
Tree: qemu git://xenbits.xen.org/staging/qemu-xen-unstable.git
Tree: qemuu git://xenbits.xen.org/staging/qemu-upstream-unstable.git
Tree: xen git://xenbits.xen.org/xen.git

*** Found and reproduced problem changeset ***

  Bug is in tree:  libvirt git://libvirt.org/libvirt.git
  Bug introduced:  19a5474d04a497dd12928b694c4c3ab9621c9552
  Bug not present: 100b7a72a4cf209f2146610f1860b87331f1d9ad


  commit 19a5474d04a497dd12928b694c4c3ab9621c9552
  Author: Laine Stump 
  Date:   Thu Nov 20 11:55:50 2014 -0500
  
  util: functions to manage bridge fdb (forwarding database)
  
  These two functions use netlink RTM_NEWNEIGH and RTM_DELNEIGH messages
  to add and delete entries from a bridge's fdb. The bridge itself is
  not referenced in the arguments to the functions, only the name of the
  device that is attached to the bridge (since a device can only be
  attached to one bridge at a time, and must be attached for this
  function to make sense, the kernel easily infers which bridge's fdb is
  being modified by looking at the device name/index).


For bisection revision-tuple graph see:
   
http://www.chiark.greenend.org.uk/~xensrcts/results/bisect.libvirt.build-i386-libvirt.libvirt-build.html
Revision IDs in each graph node refer, respectively, to the Trees above.


Searching for failure / basis pass:
 32180 fail [host=itch-mite] / 32137 [host=moss-bug] 32005 [host=gall-mite] 
31928 [host=grain-weevil] 31860 [host=bush-cricket] 31852 [host=grain-weevil] 
31839 [host=grain-weevil] 31827 ok.
Failure / basis pass flights: 32180 / 31827
(tree with no url: seabios)
Tree: gnulib_libvirt 
git://drall.uk.xensource.com:9419/git://git.sv.gnu.org/gnulib.git%20[fetch=try]
Tree: libvirt git://libvirt.org/libvirt.git
Tree: qemu git://xenbits.xen.org/staging/qemu-xen-unstable.git
Tree: qemuu git://xenbits.xen.org/staging/qemu-upstream-unstable.git
Tree: xen git://xenbits.xen.org/xen.git
Latest 3c4e08331004fb2c43d42f521e1b546222ef8d6e 
8a408b869ce343f953a7ca564598985504b9aa7f 
b0d42741f8e9a00854c3b3faca1da84bfc69bf22 
1ebb75b1fee779621b63e84fefa7b07354c43a99 
3a80985b894f54eb3b2e143e4dea737cf139a517
Basis pass 624ea2886cc570788cb01c4fd59315de301b0cd6 
42874fa45f9fab99212d0ba3f66cf4671ddb0a30 
b0d42741f8e9a00854c3b3faca1da84bfc69bf22 
0c94ca5ffeb6d314404ecbc231bef28fe8d3fc51 
0540b854f6733759593e829bc3f13c9b45974e32
Generating revisions with ./adhoc-revtuple-generator  
git://drall.uk.xensource.com:9419/git://git.sv.gnu.org/gnulib.git%20[fetch=try]#624ea2886cc570788cb01c4fd59315de301b0cd6-3c4e08331004fb2c43d42f521e1b546222ef8d6e
 
git://libvirt.org/libvirt.git#42874fa45f9fab99212d0ba3f66cf4671ddb0a30-8a408b869ce343f953a7ca564598985504b9aa7f
 
git://xenbits.xen.org/staging/qemu-xen-unstable.git#b0d42741f8e9a00854c3b3faca1da84bfc69bf22-b0d42741f8e9a00854c3b3faca1da84bfc69bf22
 
git://xenbits.xen.org/staging/qemu-upstream-unstable.git#0c94ca5ffeb6d314404ecbc231bef28fe8d3fc51-1ebb75b1fee779621b63e84fefa7b07354c43a99
 
git://xenbits.xen.org/xen.git#0540b854f6733759593e829bc3f13c9b45974e32-3a80985b894f54eb3b2e143e4dea737cf139a517
+ exec
+ sh -xe
+ cd /export/home/osstest/repos/gnulib
+ git remote set-url origin 
git://drall.uk.xensource.com:9419/git://git.sv.gnu.org/gnulib.git%20[fetch=try]
+ git fetch -p origin +refs/heads/*:refs/remotes/origin/*
+ exec
+ sh -xe
+ cd /export/home/osstest/repos/libvirt
+ git remote set-url origin 
git://drall.uk.xensource.com:9419/git://libvirt.org/libvirt.git
+ git fetch -p origin +refs/heads/*:refs/remotes/origin/*
+ exec
+ sh -xe
+ cd /export/home/osstest/repos/qemu-upstream-unstable
+ git remote set-url origin 
git://drall.uk.xensource.com:9419/git://xenbits.xen.org/staging/qemu-upstream-unstable.git
+ git fetch -p origin +refs/heads/*:refs/remotes/origin/*
+ exec
+ sh -xe
+ cd /export/home/osstest/repos/xen
+ git remote set-url origin 
git://drall.uk.xensource.com:9419/git://xenbits.xen.org/xen.git
+ git fetch -p origin +refs/heads/*:refs/remotes/origin/*
+ exec
+ sh -xe
+ cd /export/home/osstest/repos/gnulib
+ git remote set-url origin 
git://drall.uk.xensource.com:9419/git://git.sv.gnu.org/gnulib.git%20[fetch=try]
+ git fetch -p origin +refs/heads/*:refs/remotes/origin/*
+ exec
+ sh -xe
+ cd /export/home/osstest/repos/libvirt
+ git remote set-url origin 
git://drall.uk.xensource.com:9419/git://libvirt.org/libvirt.git
+ git fetch -p origin +refs/heads/*:refs/remotes/origin/*
+ exec
+ sh -xe
+ cd /export/home/osstest/repos/qemu-upstream-unstable
+ git remote set-url origin 
git://drall.uk.xensource.com:9419/git://xenbits.xen.org/staging/qemu-upstream-unstable.git
+ git fetch -p origin +refs/heads/*:refs/remotes/origin/*
+ exec
+ sh -xe
+ cd /export/home/osstest/repos/xen
+ git remote set-u

Re: [Xen-devel] [v8][PATCH 04/17] update the existing hypercall to support XEN_DOMCTL_set_rdm

[Tian, Kevin]

> From: Tim Deegan [mailto:t...@xen.org]
> Sent: Wednesday, December 10, 2014 7:12 PM
> 
> Hi Kevin,
> 
> Thanks for taking the time to work through this.
> 
> At 03:39 + on 10 Dec (1418179184), Tian, Kevin wrote:
> > 1. It's more efficient for new people to start from a small, well-defined 
> > task
> > in one area, and then spanning to adjacent areas gradually. Patience must
> > be given by the community to help them grow;
> >
> > 2. Unfortunately this RMRR effort grows from original two patches (very
> VT-d
> > focused) to now v8 17 patches spanning many areas (including hypercall,
> mmu,
> > domain builder, hvmloader, etc.), and thus imposing both long learning curve
> > and lots of frustrations being no achievement returned. Though having a
> complete
> > solution is desired, we need to help split the big task into step-by-step
> approach
> > as long as:
> > - the overall design is agreed
> > - each step is self-contained
> > - each step won't be wasted moving forward.
> > That way new people can see incremental achievements, instead of being hit
> > down before final success.
> >
> > Take this RMRR for example. Maybe we can split into steps like:
> >
> > step-1: setup RMRR identify mapping in hypervisor. fail if confliction. 
> > no
> > user space changes
> > step-2: expose RMRR knowledge to userspace and detect confliction in
> > domain builder and hvmloader
> > step-3: reconcile e820/mmio to avoid confliction in a best effort
> > step-4: miscellaneous enhancements, each can be ACK-ed individually:
> > - handle guest CPU access to RMRR regions
> > - handle conflicting RMRR regions
> > - handle group RMRRs
> > - re-enable USB device assignment
> >
> > That way Tiejun can focus on a self-contained task at one time, and then
> observe
> > continuous acknowledgements for his work. We don't need to claim RMRR
> fully
> > ready in Xen until last patch is accepted, but that at least provides a way 
> > to
> ack
> > new people when working on complex features and also allow for partial
> usage
> > on his work.
> 
> We had this discussion before and I think it was clear that the
> maintainers in general are unhappy with a partial solution.  OTOH, if
> we can agree on the roadmap, and Intel will commit to seeing the work
> through, it might be possible.  I think Jan is the man to convince
> here. :)

I think w/ last 8 series Tiejun has sent out, there's no doubt Intel commits
to make a complete work. :-)

> 
> Now since the code's not going to be in 4.5 anyway, it should be
> possible to _develop_ it in this manner, possibly in a private branch,
> even if the early stages aren't getting applied immediately.  We
> should be able to set up an rmrr branch on the public servers if that
> helps.  But again, that relies on having a design worked out in
> advance that both developers and maintainers are (within reason)
> committed to.

that's a good suggestion. We'll send out a design review today, and then
based on discussion we can see whether making sense to do such 
incremental way.

> 
> > 3. Maintainers sometimes didn't give definitive guidance to the new people,
> > and the high-level design is not closed in the 1st place. e.g. when I 
> > thought
> this v8
> > version has everyone agreed on the design, there's another comment from
> Jan
> > about using XENMEM_set_memory_map might be better, while back to Aug.
> > when Tiejun raised that option the answer is "I'm not sure". Note I
> understand
> > as a maintainer he might not have definite answers for all opens. But w/o a
> > definitive guide new people may waste lots of effort on chasing a wrong
> option,
> > which is definitely frustrating.
> 
> This is definitely a problem, and indeed frustrating for the
> developers.  We had similar difficulties with PVH development, where
> even though we planned the architecture up-front, once the code was
> written it became clear that a different approach would have been
> better.  I'm not sure what we can do here to make it more likely that
> we get the design right first time.

understand and reasonable.

> 
> I do think that figuring out the design in advance makes these
> projects much smoother, and it's something we're seeing more of.  For
> example David Vrabel's designs for the new event channel interface, or
> indeed the design doc he wrote about grant table locking, where review
> at the design stage may have avoided a bunch of implementation effort
> altogether.

yes. a formal review in advance would be lot better than mixing design 
comments in scattered in deep coding review comments. For this RMRR
example, Tiejun did send out some summary for his patch set, but not
abstracted enough to catch people's eye on key design opens. And having
a goal for 4.5 window really brought him hard time to balance code 
refactoring and learn new areas when each series was questioned with 
new design inputs.

> 
> > So I'd suggest for such non-t

Re: [Xen-devel] arch arm qemu compile erro

[Mao Mingya]

-- Original Message --
From: "Ian Campbell" 
To: "Mao Mingya" 
Cc: xen-devel@lists.xen.org
Sent: 10/12/2014 5:47:19 PM
Subject: Re: Re[2]: [Xen-devel] arch arm qemu compile erro


Please don't top post.

On Wed, 2014-12-10 at 07:09 +, Mao Mingya wrote:
 From the the arch arm, since the stubdom is not supported now. Does 
the
 device to be shared between doms have to be pv front/backend 
structure.


Yes.

There is no equivalent to the x86 "hvm" type guest, i.e. one which 
looks

to the guest like a real system, in the ARM version of Xen. On x86
stub-qemu is primarily about providing these emulated system devices in
a sandboxed environment. There are no emulated devices on ARM hence no
need for qemu for this purpose, hence no stubdomains.

qemu has a "second hat" when in a Xen system, which is that it can
provide certain PV backends, we do use it for this purpose on ARM.

 To run the Linux on Dom0 and DomU. Could the qemu and pv backend run 
in

 the Dom0 at the same time?


Yes.


  For example: for network, block device, serial console, the pv
 backend provides the device drive for DomU linux
  while qemu provides all other devices driver for DomU linux.


Yes, qemu can be used to provide PV backends for: disk (using qdisk),
framebuffer, kbd and mouse.

Thank you for the detail explanation.
What is the best pv backend on arch arm? qemu-upstream, 
qemu-xen-traditional, or anything else?

Is there any guild doc to get it work on arch arm?


Ian.




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] One question about the hypercall to translate gfn to mfn.

[Tian, Kevin]

> From: Ian Campbell [mailto:ian.campb...@citrix.com]
> Sent: Wednesday, December 10, 2014 6:11 PM
> 
> On Wed, 2014-12-10 at 01:48 +, Tian, Kevin wrote:
> > I'm not familiar with Arm architecture, but based on a brief reading it's
> > for the assigned case where the MMU is exclusive owned by a VM, so
> > some type of MMU virtualization is required and it's straightforward.
> 
> > However XenGT is a shared GPU usage:
> >
> > - a global GPU page table is partitioned among VMs. a shared shadow
> > global page table is maintained, containing translations for multiple
> > VMs simultaneously based on partitioning information
> > - multiple per-process GPU page tables are created by each VM, and
> > multiple shadow per-process GPU page tables are created correspondingly.
> > shadow page table is switched when doing GPU context switch, same as
> > what we did for CPU shadow page table.
> 
> None of that sounds to me to be impossible to do in the remoteproc
> model, perhaps it needs some extensions from its initial core feature
> set but I see no reason why it couldn't maintain multiple sets of page
> tables, each tagged with an owning domain (for validation purposes) and
> a mechanism to switch between them, or to be able to manage partitioning
> of the GPU address space.

here we're talking about multiple GPU page tables on top of a 
IOMMU page table. Instead of one MMU unit concerned here in 
remoteproc.

> 
> > So you can see above shared MMU virtualization usage is very GPU
> > specific,
> 
> AIUI remoteproc is specific to a particular h/w device too, i.e. there
> is a device specific stub in the hypervisor which essentially knows how
> to implement set_pte for that bit of h/w, with appropriate safety and
> validation, as well as a write_cr3 type operation.
> 
> >  that's why we didn't put in Xen hypervisor, and thus additional
> > interface is required to get p2m mapping to assist our shadow GPU
> > page table usage.
> 
> There is a great reluctance among several maintainers to expose real
> hardware MFNs to VMs (including dom0 and backend driver domains).
> 
> I think you need to think very carefully about possible ways of avoiding
> the need for this. Yes, this might require some changes to your current
> mode/design.
> 

We're open to changes if necessary.

Thanks,
Kevin 
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] One question about the hypercall to translate gfn to mfn.

[Tian, Kevin]

> From: Jan Beulich [mailto:jbeul...@suse.com]
> Sent: Wednesday, December 10, 2014 6:36 PM
> 
> >>> On 10.12.14 at 02:14,  wrote:
> >>  From: Tim Deegan [mailto:t...@xen.org]
> >> It's been suggested before that we should revive this hypercall, and I
> >> don't think it's a good idea.  Whenever a domain needs to know the
> >> actual MFN of another domain's memory it's usually because the
> >> security model is problematic.  In particular, finding the MFN is
> >> usually followed by a brute-force mapping from a dom0 process, or by
> >> passing the MFN to a device for unprotected DMA.
> >
> > In our case it's not because the security model is problematic. It's
> > because GPU virtualization is done in Dom0 while the memory virtualization
> > is done in hypervisor.
> 
> Which by itself is a questionable design decision.
> 

I don't think we want to put a ~20K LOC device model in hypervisor.

Thanks
Kevin

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] One question about the hypercall to translate gfn to mfn.

[Tian, Kevin]

> From: Tim Deegan [mailto:t...@xen.org]
> Sent: Wednesday, December 10, 2014 6:55 PM
> 
> At 01:14 + on 10 Dec (1418170461), Tian, Kevin wrote:
> > > From: Tim Deegan [mailto:t...@xen.org]
> > > Sent: Tuesday, December 09, 2014 6:47 PM
> > >
> > > At 18:10 +0800 on 09 Dec (1418145055), Yu, Zhang wrote:
> > > > Hi all,
> > > >
> > > >As you can see, we are pushing our XenGT patches to the upstream.
> One
> > > > feature we need in xen is to translate guests' gfn to mfn in XenGT dom0
> > > > device model.
> > > >
> > > >Here we may have 2 similar solutions:
> > > >1> Paul told me(and thank you, Paul :)) that there used to be a
> > > > hypercall, XENMEM_translate_gpfn_list, which was removed by Keir in
> > > > commit 2d2f7977a052e655db6748be5dabf5a58f5c5e32, because there
> was
> > > no
> > > > usage at that time.
> > >
> > > It's been suggested before that we should revive this hypercall, and I
> > > don't think it's a good idea.  Whenever a domain needs to know the
> > > actual MFN of another domain's memory it's usually because the
> > > security model is problematic.  In particular, finding the MFN is
> > > usually followed by a brute-force mapping from a dom0 process, or by
> > > passing the MFN to a device for unprotected DMA.
> >
> > In our case it's not because the security model is problematic. It's
> > because GPU virtualization is done in Dom0 while the memory virtualization
> > is done in hypervisor. We need a means to query GPFN->MFN so we can
> > setup shadow GPU page table in Dom0 correctly, for a VM.
> 
> I don't think we understand each other.  Let me try to explain what I
> mean.  My apologies if this sounds patronising; I'm just trying to be
> as clear as I can.

Thanks for your explanation. This is a very helpful discussion. :-)

> 
> It is Xen's job to isolate VMs from each other.  As part of that, Xen
> uses the MMU, nested paging, and IOMMUs to control access to RAM.  Any
> software component that can pass a raw MFN to hardware breaks that
> isolation, because Xen has no way of controlling what that component
> can do (including taking over the hypervisor).  This is why I am
> afraid when developers ask for GFN->MFN translation functions.

When I agree Xen's job absolutely, the isolation is also required in different
layers, regarding to who controls the resource and where the virtualization 
happens. For example talking about I/O virtualization, Dom0 or driver domain 
needs to isolate among backend drivers to avoid one backend interfering 
with another. Xen doesn't know such violation, since it only knows it's Dom0
wants to access a VM's page.

btw curious of how worse exposing GFN->MFN translation compared to
allowing mapping other VM's GFN? If exposing GFN->MFN is under the
same permission control as mapping, would it avoid your worry here?

> 
> So if the XenGT model allowed the backend component to (cause the GPU
> to) perform arbitrary DMA without IOMMU checks, then that component
> would have complete access to the system and (from a security pov)
> might as well be running in the hypervisor.  That would be very
> problematic, but AFAICT that's not what's going on.  From your reply
> on the other thread it seems like the GPU is behind the IOMMU, so
> that's OK. :)
> 
> When the backend component gets a GFN from the guest, it wants an
> address that it can give to the GPU for DMA that will map the right
> memory.  That address must be mapped in the IOMMU tables that the GPU
> will be using, which means the IOMMU tables of the backend domain,
> IIUC[1].  So the hypercall it needs is not "give me the MFN that matches
> this GFN" but "please map this GFN into my IOMMU tables".

Here "please map this GFN into my IOMMU tables" actually breaks the
IOMMU isolation. IOMMU is designed for serving DMA requests issued
by an exclusive VM, so IOMMU page table can restrict that VM's attempts
strictly.

To map multiple VM's GFNs into one IOMMU table, the 1st thing is to
avoid GFN conflictions to make it functional. We thought about this approach
previously, e.g. by reserving highest 3 bits of GFN as VMID, so one IOMMU
page table can be used to combine multi-VM's page table together. However
doing so have two limitations:

a) it still requires write-protect guest GPU page table, and maintain a shadow
GPU page table by translate from real GFN to pseudo GFN (plus VMID), which
doesn't save any engineering effort in the device model part

b) it breaks the designed isolation intrinsic of IOMMU. In such case, IOMMU
can't isolate multiple VMs by itself, since a DMA request can target any 
pseudo GFN if valid in the page table. We have to rely on the audit in the 
backend component in Dom0 to ensure the isolation. So even by using IOMMU,
it loses the isolation intention as you described earlier.

c) this introduces tricky logic in IOMMU driver to handle such non-standard
multiplexed page table style. 

w/o a SR-IOV implementation (so each VF has its own IOMMU page table),
I don't see usi

Re: [Xen-devel] [PATCH v2 2/2] x86/xen: allow privcmd hypercalls to be preempted

[Andy Lutomirski]

On Wed, Dec 10, 2014 at 4:55 PM, Luis R. Rodriguez  wrote:
> On Wed, Dec 10, 2014 at 03:51:48PM -0800, Andy Lutomirski wrote:
>> On Wed, Dec 10, 2014 at 3:34 PM, Luis R. Rodriguez
>>  wrote:
>> > From: "Luis R. Rodriguez" 
>> >
>> > Xen has support for splitting heavy work work into a series
>> > of hypercalls, called multicalls, and preempting them through
>> > what Xen calls continuation [0]. Despite this though without
>> > CONFIG_PREEMPT preemption won't happen and while enabling
>> > CONFIG_RT_GROUP_SCHED can at times help its not enough to
>> > make a system usable. Such is the case for example when
>> > creating a > 50 GiB HVM guest, we can get softlockups [1] with:.
>> >
>> > kernel: [  802.084335] BUG: soft lockup - CPU#1 stuck for 22s! [xend:31351]
>> >
>> > The softlock up triggers on the TASK_UNINTERRUPTIBLE hanger check
>> > (default 120 seconds), on the Xen side in this particular case
>> > this happens when the following Xen hypervisor code is used:
>> >
>> > xc_domain_set_pod_target() -->
>> >   do_memory_op() -->
>> > arch_memory_op() -->
>> >   p2m_pod_set_mem_target()
>> > -- long delay (real or emulated) --
>> >
>> > This happens on arch_memory_op() on the XENMEM_set_pod_target memory
>> > op even though arch_memory_op() can handle continuation via
>> > hypercall_create_continuation() for example.
>> >
>> > Machines over 50 GiB of memory are on high demand and hard to come
>> > by so to help replicate this sort of issue long delays on select
>> > hypercalls have been emulated in order to be able to test this on
>> > smaller machines [2].
>> >
>> > On one hand this issue can be considered as expected given that
>> > CONFIG_PREEMPT=n is used however we have forced voluntary preemption
>> > precedent practices in the kernel even for CONFIG_PREEMPT=n through
>> > the usage of cond_resched() sprinkled in many places. To address
>> > this issue with Xen hypercalls though we need to find a way to aid
>> > to the schedular in the middle of hypercalls. We are motivated to
>> > address this issue on CONFIG_PREEMPT=n as otherwise the system becomes
>> > rather unresponsive for long periods of time; in the worst case, at least
>> > only currently by emulating long delays on select io disk bound
>> > hypercalls, this can lead to filesystem corruption if the delay happens
>> > for example on SCHEDOP_remote_shutdown (when we call 'xl  
>> > shutdown').
>> >
>> > We can address this problem by trying to check if we should schedule
>> > on the xen timer in the middle of a hypercall on the return from the
>> > timer interrupt. We want to be careful to not always force voluntary
>> > preemption though so to do this we only selectively enable preemption
>> > on very specific xen hypercalls.
>> >
>> > This enables hypercall preemption by selectively forcing checks for
>> > voluntary preempting only on ioctl initiated private hypercalls
>> > where we know some folks have run into reported issues [1].
>> >
>> > [0] 
>> > http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=42217cbc5b3e84b8c145d8cfb62dd5de0134b9e8;hp=3a0b9c57d5c9e82c55dd967c84dd06cb43c49ee9
>> > [1] https://bugzilla.novell.com/show_bug.cgi?id=861093
>> > [2] 
>> > http://ftp.suse.com/pub/people/mcgrof/xen/emulate-long-xen-hypercalls.patch
>> >
>> > Based on original work by: David Vrabel 
>> > Cc: Borislav Petkov 
>> > Cc: David Vrabel 
>> > Cc: Thomas Gleixner 
>> > Cc: Ingo Molnar 
>> > Cc: "H. Peter Anvin" 
>> > Cc: x...@kernel.org
>> > Cc: Andy Lutomirski 
>> > Cc: Steven Rostedt 
>> > Cc: Masami Hiramatsu 
>> > Cc: Jan Beulich 
>> > Cc: linux-ker...@vger.kernel.org
>> > Signed-off-by: Luis R. Rodriguez 
>> > ---
>> >  arch/x86/kernel/entry_32.S | 21 +
>> >  arch/x86/kernel/entry_64.S | 17 +
>> >  drivers/xen/Makefile   |  2 +-
>> >  drivers/xen/preempt.c  | 17 +
>> >  drivers/xen/privcmd.c  |  2 ++
>> >  include/xen/xen-ops.h  | 26 ++
>> >  6 files changed, 84 insertions(+), 1 deletion(-)
>> >  create mode 100644 drivers/xen/preempt.c
>> >
>> > diff --git a/arch/x86/kernel/entry_32.S b/arch/x86/kernel/entry_32.S
>> > index 344b63f..40b5c0c 100644
>> > --- a/arch/x86/kernel/entry_32.S
>> > +++ b/arch/x86/kernel/entry_32.S
>> > @@ -982,7 +982,28 @@ ENTRY(xen_hypervisor_callback)
>> >  ENTRY(xen_do_upcall)
>> >  1: mov %esp, %eax
>> > call xen_evtchn_do_upcall
>> > +#ifdef CONFIG_PREEMPT
>> > jmp  ret_from_intr
>> > +#else
>> > +   GET_THREAD_INFO(%ebp)
>> > +#ifdef CONFIG_VM86
>> > +   movl PT_EFLAGS(%esp), %eax  # mix EFLAGS and CS
>> > +   movb PT_CS(%esp), %al
>> > +   andl $(X86_EFLAGS_VM | SEGMENT_RPL_MASK), %eax
>> > +#else
>> > +   movl PT_CS(%esp), %eax
>> > +   andl $SEGMENT_RPL_MASK, %eax
>> > +#endif
>> > +   cmpl $USER_RPL, %eax
>> > +   jae resume_userspace# returning to v8086 or userspace
>> > +   DISABLE_INTERRUPTS(CLBR_ANY)
>> > +   cmpb $0,

Re: [Xen-devel] [PATCH v2 2/2] x86/xen: allow privcmd hypercalls to be preempted

[Luis R. Rodriguez]

On Wed, Dec 10, 2014 at 04:29:06PM -0800, H. Peter Anvin wrote:
> On 12/10/2014 03:34 PM, Luis R. Rodriguez wrote:
> > diff --git a/arch/x86/kernel/entry_32.S b/arch/x86/kernel/entry_32.S
> > index 344b63f..40b5c0c 100644
> > --- a/arch/x86/kernel/entry_32.S
> > +++ b/arch/x86/kernel/entry_32.S
> > @@ -982,7 +982,28 @@ ENTRY(xen_hypervisor_callback)
> >  ENTRY(xen_do_upcall)
> >  1: mov %esp, %eax
> > call xen_evtchn_do_upcall
> > +#ifdef CONFIG_PREEMPT
> > jmp  ret_from_intr
> > +#else
> > +   GET_THREAD_INFO(%ebp)
> > +#ifdef CONFIG_VM86
> > +   movl PT_EFLAGS(%esp), %eax  # mix EFLAGS and CS
> > +   movb PT_CS(%esp), %al
> > +   andl $(X86_EFLAGS_VM | SEGMENT_RPL_MASK), %eax
> > +#else
> > +   movl PT_CS(%esp), %eax
> > +   andl $SEGMENT_RPL_MASK, %eax
> > +#endif
> > +   cmpl $USER_RPL, %eax
> > +   jae resume_userspace# returning to v8086 or userspace
> > +   DISABLE_INTERRUPTS(CLBR_ANY)
> > +   cmpb $0,PER_CPU_VAR(xen_in_preemptible_hcall)
> > +   jz resume_kernel
> > +   movb $0,PER_CPU_VAR(xen_in_preemptible_hcall)
> > +   call cond_resched_irq
> > +   movb $1,PER_CPU_VAR(xen_in_preemptible_hcall)
> > +   jmp resume_kernel
> > +#endif /* CONFIG_PREEMPT */
> > CFI_ENDPROC
> >  ENDPROC(xen_hypervisor_callback)
> >  
> > diff --git a/arch/x86/kernel/entry_64.S b/arch/x86/kernel/entry_64.S
> > index c0226ab..0ccdd06 100644
> > --- a/arch/x86/kernel/entry_64.S
> > +++ b/arch/x86/kernel/entry_64.S
> > @@ -1170,7 +1170,23 @@ ENTRY(xen_do_hypervisor_callback)   # 
> > do_hypervisor_callback(struct *pt_regs)
> > popq %rsp
> > CFI_DEF_CFA_REGISTER rsp
> > decl PER_CPU_VAR(irq_count)
> > +#ifdef CONFIG_PREEMPT
> > jmp  error_exit
> > +#else
> > +   movl %ebx, %eax
> > +   RESTORE_REST
> > +   DISABLE_INTERRUPTS(CLBR_NONE)
> > +   TRACE_IRQS_OFF
> > +   GET_THREAD_INFO(%rcx)
> > +   testl %eax, %eax
> > +   je error_exit_user
> > +   cmpb $0,PER_CPU_VAR(xen_in_preemptible_hcall)
> > +   jz retint_kernel
> > +   movb $0,PER_CPU_VAR(xen_in_preemptible_hcall)
> > +   call cond_resched_irq
> > +   movb $1,PER_CPU_VAR(xen_in_preemptible_hcall)
> > +   jmp retint_kernel
> > +#endif /* CONFIG_PREEMPT */
> > CFI_ENDPROC
> >  END(xen_do_hypervisor_callback)
> >  
> > @@ -1398,6 +1414,7 @@ ENTRY(error_exit)
> > GET_THREAD_INFO(%rcx)
> > testl %eax,%eax
> > jne retint_kernel
> > +error_exit_user:
> > LOCKDEP_SYS_EXIT_IRQ
> > movl TI_flags(%rcx),%edx
> > movl $_TIF_WORK_MASK,%edi
> 
> You're adding a bunch of code for the *non*-preemptive case here... why?

This is an issue onloy for for non*-preemptive kernels.

Some of Xen's hypercalls can take a long time and unfortunately for
*non*-preemptive kernels this can be quite a bit of an issue.
We've handled situations like this with cond_resched() before which will
push even *non*-preemptive kernels to behave as voluntarily preemptive,
I was not aware to what extent this was done and precedents set but
its pretety widespread now... this then just addresses once particular
case where this is also an issuefor but now in IRQ context.

I agree its a hack but so are all the other cond_reshed() calls then.
I don't think its a good idea to be spreading use of something like
this everywhere but after careful review and trying toa void this
exact code for a while I have not been able to find any other reasonable
alternative.

  Luis

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v2 2/2] x86/xen: allow privcmd hypercalls to be preempted

[Luis R. Rodriguez]

On Wed, Dec 10, 2014 at 03:51:48PM -0800, Andy Lutomirski wrote:
> On Wed, Dec 10, 2014 at 3:34 PM, Luis R. Rodriguez
>  wrote:
> > From: "Luis R. Rodriguez" 
> >
> > Xen has support for splitting heavy work work into a series
> > of hypercalls, called multicalls, and preempting them through
> > what Xen calls continuation [0]. Despite this though without
> > CONFIG_PREEMPT preemption won't happen and while enabling
> > CONFIG_RT_GROUP_SCHED can at times help its not enough to
> > make a system usable. Such is the case for example when
> > creating a > 50 GiB HVM guest, we can get softlockups [1] with:.
> >
> > kernel: [  802.084335] BUG: soft lockup - CPU#1 stuck for 22s! [xend:31351]
> >
> > The softlock up triggers on the TASK_UNINTERRUPTIBLE hanger check
> > (default 120 seconds), on the Xen side in this particular case
> > this happens when the following Xen hypervisor code is used:
> >
> > xc_domain_set_pod_target() -->
> >   do_memory_op() -->
> > arch_memory_op() -->
> >   p2m_pod_set_mem_target()
> > -- long delay (real or emulated) --
> >
> > This happens on arch_memory_op() on the XENMEM_set_pod_target memory
> > op even though arch_memory_op() can handle continuation via
> > hypercall_create_continuation() for example.
> >
> > Machines over 50 GiB of memory are on high demand and hard to come
> > by so to help replicate this sort of issue long delays on select
> > hypercalls have been emulated in order to be able to test this on
> > smaller machines [2].
> >
> > On one hand this issue can be considered as expected given that
> > CONFIG_PREEMPT=n is used however we have forced voluntary preemption
> > precedent practices in the kernel even for CONFIG_PREEMPT=n through
> > the usage of cond_resched() sprinkled in many places. To address
> > this issue with Xen hypercalls though we need to find a way to aid
> > to the schedular in the middle of hypercalls. We are motivated to
> > address this issue on CONFIG_PREEMPT=n as otherwise the system becomes
> > rather unresponsive for long periods of time; in the worst case, at least
> > only currently by emulating long delays on select io disk bound
> > hypercalls, this can lead to filesystem corruption if the delay happens
> > for example on SCHEDOP_remote_shutdown (when we call 'xl  
> > shutdown').
> >
> > We can address this problem by trying to check if we should schedule
> > on the xen timer in the middle of a hypercall on the return from the
> > timer interrupt. We want to be careful to not always force voluntary
> > preemption though so to do this we only selectively enable preemption
> > on very specific xen hypercalls.
> >
> > This enables hypercall preemption by selectively forcing checks for
> > voluntary preempting only on ioctl initiated private hypercalls
> > where we know some folks have run into reported issues [1].
> >
> > [0] 
> > http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=42217cbc5b3e84b8c145d8cfb62dd5de0134b9e8;hp=3a0b9c57d5c9e82c55dd967c84dd06cb43c49ee9
> > [1] https://bugzilla.novell.com/show_bug.cgi?id=861093
> > [2] 
> > http://ftp.suse.com/pub/people/mcgrof/xen/emulate-long-xen-hypercalls.patch
> >
> > Based on original work by: David Vrabel 
> > Cc: Borislav Petkov 
> > Cc: David Vrabel 
> > Cc: Thomas Gleixner 
> > Cc: Ingo Molnar 
> > Cc: "H. Peter Anvin" 
> > Cc: x...@kernel.org
> > Cc: Andy Lutomirski 
> > Cc: Steven Rostedt 
> > Cc: Masami Hiramatsu 
> > Cc: Jan Beulich 
> > Cc: linux-ker...@vger.kernel.org
> > Signed-off-by: Luis R. Rodriguez 
> > ---
> >  arch/x86/kernel/entry_32.S | 21 +
> >  arch/x86/kernel/entry_64.S | 17 +
> >  drivers/xen/Makefile   |  2 +-
> >  drivers/xen/preempt.c  | 17 +
> >  drivers/xen/privcmd.c  |  2 ++
> >  include/xen/xen-ops.h  | 26 ++
> >  6 files changed, 84 insertions(+), 1 deletion(-)
> >  create mode 100644 drivers/xen/preempt.c
> >
> > diff --git a/arch/x86/kernel/entry_32.S b/arch/x86/kernel/entry_32.S
> > index 344b63f..40b5c0c 100644
> > --- a/arch/x86/kernel/entry_32.S
> > +++ b/arch/x86/kernel/entry_32.S
> > @@ -982,7 +982,28 @@ ENTRY(xen_hypervisor_callback)
> >  ENTRY(xen_do_upcall)
> >  1: mov %esp, %eax
> > call xen_evtchn_do_upcall
> > +#ifdef CONFIG_PREEMPT
> > jmp  ret_from_intr
> > +#else
> > +   GET_THREAD_INFO(%ebp)
> > +#ifdef CONFIG_VM86
> > +   movl PT_EFLAGS(%esp), %eax  # mix EFLAGS and CS
> > +   movb PT_CS(%esp), %al
> > +   andl $(X86_EFLAGS_VM | SEGMENT_RPL_MASK), %eax
> > +#else
> > +   movl PT_CS(%esp), %eax
> > +   andl $SEGMENT_RPL_MASK, %eax
> > +#endif
> > +   cmpl $USER_RPL, %eax
> > +   jae resume_userspace# returning to v8086 or userspace
> > +   DISABLE_INTERRUPTS(CLBR_ANY)
> > +   cmpb $0,PER_CPU_VAR(xen_in_preemptible_hcall)
> > +   jz resume_kernel
> > +   movb $0,PER_CPU_VAR(xen_in_preemptible_hcall)
> > +   call cond_resched_irq
> > +  

Re: [Xen-devel] [PATCH v2 2/2] x86/xen: allow privcmd hypercalls to be preempted

[H. Peter Anvin]

On 12/10/2014 03:34 PM, Luis R. Rodriguez wrote:
> diff --git a/arch/x86/kernel/entry_32.S b/arch/x86/kernel/entry_32.S
> index 344b63f..40b5c0c 100644
> --- a/arch/x86/kernel/entry_32.S
> +++ b/arch/x86/kernel/entry_32.S
> @@ -982,7 +982,28 @@ ENTRY(xen_hypervisor_callback)
>  ENTRY(xen_do_upcall)
>  1:   mov %esp, %eax
>   call xen_evtchn_do_upcall
> +#ifdef CONFIG_PREEMPT
>   jmp  ret_from_intr
> +#else
> + GET_THREAD_INFO(%ebp)
> +#ifdef CONFIG_VM86
> + movl PT_EFLAGS(%esp), %eax  # mix EFLAGS and CS
> + movb PT_CS(%esp), %al
> + andl $(X86_EFLAGS_VM | SEGMENT_RPL_MASK), %eax
> +#else
> + movl PT_CS(%esp), %eax
> + andl $SEGMENT_RPL_MASK, %eax
> +#endif
> + cmpl $USER_RPL, %eax
> + jae resume_userspace# returning to v8086 or userspace
> + DISABLE_INTERRUPTS(CLBR_ANY)
> + cmpb $0,PER_CPU_VAR(xen_in_preemptible_hcall)
> + jz resume_kernel
> + movb $0,PER_CPU_VAR(xen_in_preemptible_hcall)
> + call cond_resched_irq
> + movb $1,PER_CPU_VAR(xen_in_preemptible_hcall)
> + jmp resume_kernel
> +#endif /* CONFIG_PREEMPT */
>   CFI_ENDPROC
>  ENDPROC(xen_hypervisor_callback)
>  
> diff --git a/arch/x86/kernel/entry_64.S b/arch/x86/kernel/entry_64.S
> index c0226ab..0ccdd06 100644
> --- a/arch/x86/kernel/entry_64.S
> +++ b/arch/x86/kernel/entry_64.S
> @@ -1170,7 +1170,23 @@ ENTRY(xen_do_hypervisor_callback)   # 
> do_hypervisor_callback(struct *pt_regs)
>   popq %rsp
>   CFI_DEF_CFA_REGISTER rsp
>   decl PER_CPU_VAR(irq_count)
> +#ifdef CONFIG_PREEMPT
>   jmp  error_exit
> +#else
> + movl %ebx, %eax
> + RESTORE_REST
> + DISABLE_INTERRUPTS(CLBR_NONE)
> + TRACE_IRQS_OFF
> + GET_THREAD_INFO(%rcx)
> + testl %eax, %eax
> + je error_exit_user
> + cmpb $0,PER_CPU_VAR(xen_in_preemptible_hcall)
> + jz retint_kernel
> + movb $0,PER_CPU_VAR(xen_in_preemptible_hcall)
> + call cond_resched_irq
> + movb $1,PER_CPU_VAR(xen_in_preemptible_hcall)
> + jmp retint_kernel
> +#endif /* CONFIG_PREEMPT */
>   CFI_ENDPROC
>  END(xen_do_hypervisor_callback)
>  
> @@ -1398,6 +1414,7 @@ ENTRY(error_exit)
>   GET_THREAD_INFO(%rcx)
>   testl %eax,%eax
>   jne retint_kernel
> +error_exit_user:
>   LOCKDEP_SYS_EXIT_IRQ
>   movl TI_flags(%rcx),%edx
>   movl $_TIF_WORK_MASK,%edi

You're adding a bunch of code for the *non*-preemptive case here... why?

-hpa



___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Question reg. named vif interface support in guest cfg;

[Wei Liu]

On Wed, Dec 10, 2014 at 08:25:25PM +, Bjoern Rennhak wrote:
> Dear Xen Developers,
> 
> I found handling of vifx.y interfaces slightly unintutive and was
> wondering if it is possible to name the interface via the Xen guest
> config file?
> 
> e.g. a vif_name parameter or otherwise maybe vif interface is automatically 
> named
> after set "name" entry in config instead of generic vifx.y.
> 
> So instead of a bunch of vifx.y's I would see x.y ?
> 
> Is that already possible or a bad idea in general?

Check out

http://xenbits.xen.org/docs/4.4-testing/misc/xl-network-configuration.html

vifname

Wei.

> Should I open a wishlist entry on http://bugs.xenproject.org/xen/?
> 
> Thank you!
> 
> Cheers,
> -Bjoern
> 
> 
> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v2 2/2] x86/xen: allow privcmd hypercalls to be preempted

[Andy Lutomirski]

On Wed, Dec 10, 2014 at 3:34 PM, Luis R. Rodriguez
 wrote:
> From: "Luis R. Rodriguez" 
>
> Xen has support for splitting heavy work work into a series
> of hypercalls, called multicalls, and preempting them through
> what Xen calls continuation [0]. Despite this though without
> CONFIG_PREEMPT preemption won't happen and while enabling
> CONFIG_RT_GROUP_SCHED can at times help its not enough to
> make a system usable. Such is the case for example when
> creating a > 50 GiB HVM guest, we can get softlockups [1] with:.
>
> kernel: [  802.084335] BUG: soft lockup - CPU#1 stuck for 22s! [xend:31351]
>
> The softlock up triggers on the TASK_UNINTERRUPTIBLE hanger check
> (default 120 seconds), on the Xen side in this particular case
> this happens when the following Xen hypervisor code is used:
>
> xc_domain_set_pod_target() -->
>   do_memory_op() -->
> arch_memory_op() -->
>   p2m_pod_set_mem_target()
> -- long delay (real or emulated) --
>
> This happens on arch_memory_op() on the XENMEM_set_pod_target memory
> op even though arch_memory_op() can handle continuation via
> hypercall_create_continuation() for example.
>
> Machines over 50 GiB of memory are on high demand and hard to come
> by so to help replicate this sort of issue long delays on select
> hypercalls have been emulated in order to be able to test this on
> smaller machines [2].
>
> On one hand this issue can be considered as expected given that
> CONFIG_PREEMPT=n is used however we have forced voluntary preemption
> precedent practices in the kernel even for CONFIG_PREEMPT=n through
> the usage of cond_resched() sprinkled in many places. To address
> this issue with Xen hypercalls though we need to find a way to aid
> to the schedular in the middle of hypercalls. We are motivated to
> address this issue on CONFIG_PREEMPT=n as otherwise the system becomes
> rather unresponsive for long periods of time; in the worst case, at least
> only currently by emulating long delays on select io disk bound
> hypercalls, this can lead to filesystem corruption if the delay happens
> for example on SCHEDOP_remote_shutdown (when we call 'xl  shutdown').
>
> We can address this problem by trying to check if we should schedule
> on the xen timer in the middle of a hypercall on the return from the
> timer interrupt. We want to be careful to not always force voluntary
> preemption though so to do this we only selectively enable preemption
> on very specific xen hypercalls.
>
> This enables hypercall preemption by selectively forcing checks for
> voluntary preempting only on ioctl initiated private hypercalls
> where we know some folks have run into reported issues [1].
>
> [0] 
> http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=42217cbc5b3e84b8c145d8cfb62dd5de0134b9e8;hp=3a0b9c57d5c9e82c55dd967c84dd06cb43c49ee9
> [1] https://bugzilla.novell.com/show_bug.cgi?id=861093
> [2] 
> http://ftp.suse.com/pub/people/mcgrof/xen/emulate-long-xen-hypercalls.patch
>
> Based on original work by: David Vrabel 
> Cc: Borislav Petkov 
> Cc: David Vrabel 
> Cc: Thomas Gleixner 
> Cc: Ingo Molnar 
> Cc: "H. Peter Anvin" 
> Cc: x...@kernel.org
> Cc: Andy Lutomirski 
> Cc: Steven Rostedt 
> Cc: Masami Hiramatsu 
> Cc: Jan Beulich 
> Cc: linux-ker...@vger.kernel.org
> Signed-off-by: Luis R. Rodriguez 
> ---
>  arch/x86/kernel/entry_32.S | 21 +
>  arch/x86/kernel/entry_64.S | 17 +
>  drivers/xen/Makefile   |  2 +-
>  drivers/xen/preempt.c  | 17 +
>  drivers/xen/privcmd.c  |  2 ++
>  include/xen/xen-ops.h  | 26 ++
>  6 files changed, 84 insertions(+), 1 deletion(-)
>  create mode 100644 drivers/xen/preempt.c
>
> diff --git a/arch/x86/kernel/entry_32.S b/arch/x86/kernel/entry_32.S
> index 344b63f..40b5c0c 100644
> --- a/arch/x86/kernel/entry_32.S
> +++ b/arch/x86/kernel/entry_32.S
> @@ -982,7 +982,28 @@ ENTRY(xen_hypervisor_callback)
>  ENTRY(xen_do_upcall)
>  1: mov %esp, %eax
> call xen_evtchn_do_upcall
> +#ifdef CONFIG_PREEMPT
> jmp  ret_from_intr
> +#else
> +   GET_THREAD_INFO(%ebp)
> +#ifdef CONFIG_VM86
> +   movl PT_EFLAGS(%esp), %eax  # mix EFLAGS and CS
> +   movb PT_CS(%esp), %al
> +   andl $(X86_EFLAGS_VM | SEGMENT_RPL_MASK), %eax
> +#else
> +   movl PT_CS(%esp), %eax
> +   andl $SEGMENT_RPL_MASK, %eax
> +#endif
> +   cmpl $USER_RPL, %eax
> +   jae resume_userspace# returning to v8086 or userspace
> +   DISABLE_INTERRUPTS(CLBR_ANY)
> +   cmpb $0,PER_CPU_VAR(xen_in_preemptible_hcall)
> +   jz resume_kernel
> +   movb $0,PER_CPU_VAR(xen_in_preemptible_hcall)
> +   call cond_resched_irq
> +   movb $1,PER_CPU_VAR(xen_in_preemptible_hcall)
> +   jmp resume_kernel
> +#endif /* CONFIG_PREEMPT */
> CFI_ENDPROC
>  ENDPROC(xen_hypervisor_callback)
>
> diff --git a/arch/x86/kernel/entry_64.S b/arch/x86/kernel/entry_64.S
> index c0226ab..0ccdd06 100644
> --- a/arch/x86/kernel/e

[Xen-devel] [PATCH v2 1/2] sched: add cond_resched_irq()

[Luis R. Rodriguez]

From: "Luis R. Rodriguez" 

Under special circumstances we may want to force
voluntary preemption even for CONFIG_PREEMPT=n
with interrupts disabled. This adds helpers to
let us do that.

Cc: Borislav Petkov 
Cc: David Vrabel 
Cc: Thomas Gleixner 
Cc: Ingo Molnar 
Cc: "H. Peter Anvin" 
Cc: x...@kernel.org
Cc: Andy Lutomirski 
Cc: Steven Rostedt 
Cc: Masami Hiramatsu 
Cc: Jan Beulich 
Cc: linux-ker...@vger.kernel.org
Signed-off-by: Luis R. Rodriguez 
---
 include/linux/sched.h |  7 +++
 kernel/sched/core.c   | 10 ++
 2 files changed, 17 insertions(+)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 5e344bb..92da927 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -2759,6 +2759,13 @@ static inline int signal_pending_state(long state, 
struct task_struct *p)
  */
 extern int _cond_resched(void);
 
+/*
+ * Voluntarily preempting the kernel even for CONFIG_PREEMPT=n kernels
+ * on very special circumstances. This is to be used with interrupts
+ * disabled.
+ */
+extern int cond_resched_irq(void);
+
 #define cond_resched() ({  \
__might_sleep(__FILE__, __LINE__, 0);   \
_cond_resched();\
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 89e7283..573edb1 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -4239,6 +4239,16 @@ int __sched _cond_resched(void)
 }
 EXPORT_SYMBOL(_cond_resched);
 
+int __sched cond_resched_irq(void)
+{
+   if (should_resched()) {
+   preempt_schedule_irq();
+   return 1;
+   }
+   return 0;
+}
+EXPORT_SYMBOL_GPL(cond_resched_irq);
+
 /*
  * __cond_resched_lock() - if a reschedule is pending, drop the given lock,
  * call schedule, and on return reacquire the lock.
-- 
2.1.1


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v2 2/2] x86/xen: allow privcmd hypercalls to be preempted

[Luis R. Rodriguez]

From: "Luis R. Rodriguez" 

Xen has support for splitting heavy work work into a series
of hypercalls, called multicalls, and preempting them through
what Xen calls continuation [0]. Despite this though without
CONFIG_PREEMPT preemption won't happen and while enabling
CONFIG_RT_GROUP_SCHED can at times help its not enough to
make a system usable. Such is the case for example when
creating a > 50 GiB HVM guest, we can get softlockups [1] with:.

kernel: [  802.084335] BUG: soft lockup - CPU#1 stuck for 22s! [xend:31351]

The softlock up triggers on the TASK_UNINTERRUPTIBLE hanger check
(default 120 seconds), on the Xen side in this particular case
this happens when the following Xen hypervisor code is used:

xc_domain_set_pod_target() -->
  do_memory_op() -->
arch_memory_op() -->
  p2m_pod_set_mem_target()
-- long delay (real or emulated) --

This happens on arch_memory_op() on the XENMEM_set_pod_target memory
op even though arch_memory_op() can handle continuation via
hypercall_create_continuation() for example.

Machines over 50 GiB of memory are on high demand and hard to come
by so to help replicate this sort of issue long delays on select
hypercalls have been emulated in order to be able to test this on
smaller machines [2].

On one hand this issue can be considered as expected given that
CONFIG_PREEMPT=n is used however we have forced voluntary preemption
precedent practices in the kernel even for CONFIG_PREEMPT=n through
the usage of cond_resched() sprinkled in many places. To address
this issue with Xen hypercalls though we need to find a way to aid
to the schedular in the middle of hypercalls. We are motivated to
address this issue on CONFIG_PREEMPT=n as otherwise the system becomes
rather unresponsive for long periods of time; in the worst case, at least
only currently by emulating long delays on select io disk bound
hypercalls, this can lead to filesystem corruption if the delay happens
for example on SCHEDOP_remote_shutdown (when we call 'xl  shutdown').

We can address this problem by trying to check if we should schedule
on the xen timer in the middle of a hypercall on the return from the
timer interrupt. We want to be careful to not always force voluntary
preemption though so to do this we only selectively enable preemption
on very specific xen hypercalls.

This enables hypercall preemption by selectively forcing checks for
voluntary preempting only on ioctl initiated private hypercalls
where we know some folks have run into reported issues [1].

[0] 
http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=42217cbc5b3e84b8c145d8cfb62dd5de0134b9e8;hp=3a0b9c57d5c9e82c55dd967c84dd06cb43c49ee9
[1] https://bugzilla.novell.com/show_bug.cgi?id=861093
[2] http://ftp.suse.com/pub/people/mcgrof/xen/emulate-long-xen-hypercalls.patch

Based on original work by: David Vrabel 
Cc: Borislav Petkov 
Cc: David Vrabel 
Cc: Thomas Gleixner 
Cc: Ingo Molnar 
Cc: "H. Peter Anvin" 
Cc: x...@kernel.org
Cc: Andy Lutomirski 
Cc: Steven Rostedt 
Cc: Masami Hiramatsu 
Cc: Jan Beulich 
Cc: linux-ker...@vger.kernel.org
Signed-off-by: Luis R. Rodriguez 
---
 arch/x86/kernel/entry_32.S | 21 +
 arch/x86/kernel/entry_64.S | 17 +
 drivers/xen/Makefile   |  2 +-
 drivers/xen/preempt.c  | 17 +
 drivers/xen/privcmd.c  |  2 ++
 include/xen/xen-ops.h  | 26 ++
 6 files changed, 84 insertions(+), 1 deletion(-)
 create mode 100644 drivers/xen/preempt.c

diff --git a/arch/x86/kernel/entry_32.S b/arch/x86/kernel/entry_32.S
index 344b63f..40b5c0c 100644
--- a/arch/x86/kernel/entry_32.S
+++ b/arch/x86/kernel/entry_32.S
@@ -982,7 +982,28 @@ ENTRY(xen_hypervisor_callback)
 ENTRY(xen_do_upcall)
 1: mov %esp, %eax
call xen_evtchn_do_upcall
+#ifdef CONFIG_PREEMPT
jmp  ret_from_intr
+#else
+   GET_THREAD_INFO(%ebp)
+#ifdef CONFIG_VM86
+   movl PT_EFLAGS(%esp), %eax  # mix EFLAGS and CS
+   movb PT_CS(%esp), %al
+   andl $(X86_EFLAGS_VM | SEGMENT_RPL_MASK), %eax
+#else
+   movl PT_CS(%esp), %eax
+   andl $SEGMENT_RPL_MASK, %eax
+#endif
+   cmpl $USER_RPL, %eax
+   jae resume_userspace# returning to v8086 or userspace
+   DISABLE_INTERRUPTS(CLBR_ANY)
+   cmpb $0,PER_CPU_VAR(xen_in_preemptible_hcall)
+   jz resume_kernel
+   movb $0,PER_CPU_VAR(xen_in_preemptible_hcall)
+   call cond_resched_irq
+   movb $1,PER_CPU_VAR(xen_in_preemptible_hcall)
+   jmp resume_kernel
+#endif /* CONFIG_PREEMPT */
CFI_ENDPROC
 ENDPROC(xen_hypervisor_callback)
 
diff --git a/arch/x86/kernel/entry_64.S b/arch/x86/kernel/entry_64.S
index c0226ab..0ccdd06 100644
--- a/arch/x86/kernel/entry_64.S
+++ b/arch/x86/kernel/entry_64.S
@@ -1170,7 +1170,23 @@ ENTRY(xen_do_hypervisor_callback)   # 
do_hypervisor_callback(struct *pt_regs)
popq %rsp
CFI_DEF_CFA_REGISTER rsp
decl PER_CPU_VAR(irq_count)
+#ifdef CONFIG_PREEMPT
jmp  error_exit
+#e

[Xen-devel] [PATCH v2 0/2] x86: add xen hypercall preemption

[Luis R. Rodriguez]

From: "Luis R. Rodriguez" 

This is my second series which addresses hypercall preemption
on Xen. On the first iteration of this series [0] I tried as
much as possible to avoid cond_resched() type of behaviour
but after good feedback I've determined using something like
cond_resched() but on IRQ context is required for preempting
Xen hypercalls. This introduces and uses the new cond_resched_irq().

[0] https://lkml.org/lkml/2014/11/26/630

Luis R. Rodriguez (2):
  sched: add cond_resched_irq()
  x86/xen: allow privcmd hypercalls to be preempted

 arch/x86/kernel/entry_32.S | 21 +
 arch/x86/kernel/entry_64.S | 17 +
 drivers/xen/Makefile   |  2 +-
 drivers/xen/preempt.c  | 17 +
 drivers/xen/privcmd.c  |  2 ++
 include/linux/sched.h  |  7 +++
 include/xen/xen-ops.h  | 26 ++
 kernel/sched/core.c| 10 ++
 8 files changed, 101 insertions(+), 1 deletion(-)
 create mode 100644 drivers/xen/preempt.c

-- 
2.1.1


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH] treewide: Convert clockevents_notify to use int cpu

[Joe Perches]

As far as I can tell, there's no value indirecting
the cpu passed to this function via a void *.

Update all the callers and called functions from within
clockevents_notify.

Miscellanea:

Add pr_fmt and convert one printk(KERN_ERR to pr_err

Signed-off-by: Joe Perches 
---
 arch/arm/mach-omap2/cpuidle44xx.c  |  7 +++
 arch/arm/mach-tegra/cpuidle-tegra114.c |  4 ++--
 arch/arm/mach-tegra/cpuidle-tegra20.c  |  8 
 arch/arm/mach-tegra/cpuidle-tegra30.c  |  8 
 arch/x86/kernel/process.c  |  6 +++---
 arch/x86/xen/suspend.c |  2 +-
 drivers/acpi/acpi_pad.c|  9 +
 drivers/acpi/processor_idle.c  |  4 ++--
 drivers/cpuidle/driver.c   |  3 +--
 drivers/idle/intel_idle.c  |  7 +++
 include/linux/clockchips.h |  6 +++---
 kernel/sched/idle.c|  4 ++--
 kernel/time/clockevents.c  | 15 +++
 kernel/time/hrtimer.c  |  6 ++
 kernel/time/tick-broadcast.c   | 16 
 kernel/time/tick-common.c  | 16 
 kernel/time/tick-internal.h| 18 +-
 kernel/time/timekeeping.c  |  4 ++--
 18 files changed, 69 insertions(+), 74 deletions(-)

diff --git a/arch/arm/mach-omap2/cpuidle44xx.c 
b/arch/arm/mach-omap2/cpuidle44xx.c
index 01e398a..5d50aa1 100644
--- a/arch/arm/mach-omap2/cpuidle44xx.c
+++ b/arch/arm/mach-omap2/cpuidle44xx.c
@@ -112,7 +112,7 @@ static int omap_enter_idle_coupled(struct cpuidle_device 
*dev,
mpuss_can_lose_context = (cx->mpu_state == PWRDM_POWER_RET) &&
 (cx->mpu_logic_state == PWRDM_POWER_OFF);
 
-   clockevents_notify(CLOCK_EVT_NOTIFY_BROADCAST_ENTER, &cpu_id);
+   clockevents_notify(CLOCK_EVT_NOTIFY_BROADCAST_ENTER, cpu_id);
 
/*
 * Call idle CPU PM enter notifier chain so that
@@ -169,7 +169,7 @@ static int omap_enter_idle_coupled(struct cpuidle_device 
*dev,
if (dev->cpu == 0 && mpuss_can_lose_context)
cpu_cluster_pm_exit();
 
-   clockevents_notify(CLOCK_EVT_NOTIFY_BROADCAST_EXIT, &cpu_id);
+   clockevents_notify(CLOCK_EVT_NOTIFY_BROADCAST_EXIT, cpu_id);
 
 fail:
cpuidle_coupled_parallel_barrier(dev, &abort_barrier);
@@ -184,8 +184,7 @@ fail:
  */
 static void omap_setup_broadcast_timer(void *arg)
 {
-   int cpu = smp_processor_id();
-   clockevents_notify(CLOCK_EVT_NOTIFY_BROADCAST_ON, &cpu);
+   clockevents_notify(CLOCK_EVT_NOTIFY_BROADCAST_ON, smp_processor_id());
 }
 
 static struct cpuidle_driver omap4_idle_driver = {
diff --git a/arch/arm/mach-tegra/cpuidle-tegra114.c 
b/arch/arm/mach-tegra/cpuidle-tegra114.c
index f2b586d..3b2fc3f 100644
--- a/arch/arm/mach-tegra/cpuidle-tegra114.c
+++ b/arch/arm/mach-tegra/cpuidle-tegra114.c
@@ -44,7 +44,7 @@ static int tegra114_idle_power_down(struct cpuidle_device 
*dev,
tegra_set_cpu_in_lp2();
cpu_pm_enter();
 
-   clockevents_notify(CLOCK_EVT_NOTIFY_BROADCAST_ENTER, &dev->cpu);
+   clockevents_notify(CLOCK_EVT_NOTIFY_BROADCAST_ENTER, dev->cpu);
 
call_firmware_op(prepare_idle);
 
@@ -52,7 +52,7 @@ static int tegra114_idle_power_down(struct cpuidle_device 
*dev,
if (call_firmware_op(do_idle, 0) == -ENOSYS)
cpu_suspend(0, tegra30_sleep_cpu_secondary_finish);
 
-   clockevents_notify(CLOCK_EVT_NOTIFY_BROADCAST_EXIT, &dev->cpu);
+   clockevents_notify(CLOCK_EVT_NOTIFY_BROADCAST_EXIT, dev->cpu);
 
cpu_pm_exit();
tegra_clear_cpu_in_lp2();
diff --git a/arch/arm/mach-tegra/cpuidle-tegra20.c 
b/arch/arm/mach-tegra/cpuidle-tegra20.c
index 4f25a7c..ab30758 100644
--- a/arch/arm/mach-tegra/cpuidle-tegra20.c
+++ b/arch/arm/mach-tegra/cpuidle-tegra20.c
@@ -136,11 +136,11 @@ static bool tegra20_cpu_cluster_power_down(struct 
cpuidle_device *dev,
if (tegra20_reset_cpu_1() || !tegra_cpu_rail_off_ready())
return false;
 
-   clockevents_notify(CLOCK_EVT_NOTIFY_BROADCAST_ENTER, &dev->cpu);
+   clockevents_notify(CLOCK_EVT_NOTIFY_BROADCAST_ENTER, dev->cpu);
 
tegra_idle_lp2_last();
 
-   clockevents_notify(CLOCK_EVT_NOTIFY_BROADCAST_EXIT, &dev->cpu);
+   clockevents_notify(CLOCK_EVT_NOTIFY_BROADCAST_EXIT, dev->cpu);
 
if (cpu_online(1))
tegra20_wake_cpu1_from_reset();
@@ -153,13 +153,13 @@ static bool tegra20_idle_enter_lp2_cpu_1(struct 
cpuidle_device *dev,
 struct cpuidle_driver *drv,
 int index)
 {
-   clockevents_notify(CLOCK_EVT_NOTIFY_BROADCAST_ENTER, &dev->cpu);
+   clockevents_notify(CLOCK_EVT_NOTIFY_BROADCAST_ENTER, dev->cpu);
 
cpu_suspend(0, tegra20_sleep_cpu_secondary_finish);
 
tegra20_cpu_clear_resettable();
 
-   clockevents_notify(CLOCK_EVT_NOTIFY_BROADCAST_EXIT, &dev->cpu);
+   clockevents_notify(CLOCK_EVT_NOTIFY_BROADCAST_EX

[Xen-devel] [OSSTEST PATCH v4 6/9] Debian.pm: load flask policy in uboot

[Wei Liu]

Signed-off-by: Wei Liu 
Acked-by: Ian Campbell 
Acked-by: Ian Jackson 
---
 Osstest/Debian.pm |   18 ++
 1 file changed, 18 insertions(+)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 22b40ff..08f0ad1 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -123,6 +123,22 @@ sub setupboot_uboot () {
my $kern = "vmlinuz-$want_kernver";
my $initrd = "initrd.img-$want_kernver";
 
+   my $flask_commands = "";
+   if ($want_xsm) {
+   my $flaskpolicy = $r{flaskpolicy};
+   $flask_commands = <
+echo Loaded $flaskpolicy to \\\${flask_policy_addr_r} (\\\${filesize})
+
+END
+   }
+
my $root= target_guest_lv_name($ho,"root");
 
logm("Xen options: $xenhopt");
@@ -176,6 +192,8 @@ fdt set /chosen/module\@1 compatible "xen,linux-initrd" 
"xen,multiboot-module"
 fdt set /chosen/module\@1 reg <\\\${ramdisk_addr_r} \\\${filesize}>
 echo Loaded $initrd to \\\${ramdisk_addr_r} (\\\${filesize})
 
+${flask_commands}
+
 fdt print /chosen
 
 echo Booting \\\${xen_addr_r} - \\\${fdt_addr}
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST PATCH v4 7/9] ts-xen-install: install Xen with XSM support if requested

[Wei Liu]

Signed-off-by: Wei Liu 
---
Changes in v4:
1. Use "true" instead of "y"
---
 ts-xen-install |4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/ts-xen-install b/ts-xen-install
index 910181e..08b5fe1 100755
--- a/ts-xen-install
+++ b/ts-xen-install
@@ -46,6 +46,8 @@ if (@ARGV and $ARGV[0] eq '--check') {
 
 our $ho;
 
+my $enable_xsm = $r{enable_xsm} =~ m/true/ ? 1 : 0;
+
 my %distpath;
 
 sub packages () {
@@ -171,7 +173,7 @@ sub setupboot () {
 }
 
 my $want_kernver = get_runvar('kernel_ver',$r{'kernbuildjob'});
-debian_boot_setup($ho, $want_kernver, 0, $xenhopt, \%distpath, \@hooks);
+debian_boot_setup($ho, $want_kernver, $enable_xsm, $xenhopt, \%distpath, 
\@hooks);
 
 logm("ready to boot Xen");
 }
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST PATCH v4 4/9] mfi-common: create build-$arch-xsm job

[Wei Liu]

Signed-off-by: Wei Liu 
---
Changes in v4:
1. Use "true" and "false" instead of "y" and "n".
2. Rename xenbranch_wants_xsm_tests to xenbranch_xsm_variants.
---
 mfi-common |   23 ++-
 1 file changed, 22 insertions(+), 1 deletion(-)

diff --git a/mfi-common b/mfi-common
index 5c4f5d5..161847c 100644
--- a/mfi-common
+++ b/mfi-common
@@ -41,6 +41,19 @@ branch_wants_rumpkernel_tests () {
   esac
 }
 
+xenbranch_xsm_variants () {
+# Test XSM from 4.5 onwards
+case "$xenbranch" in
+xen-3.*-testing) echo "false";;
+xen-4.0-testing) echo "false";;
+xen-4.1-testing) echo "false";;
+xen-4.2-testing) echo "false";;
+xen-4.3-testing) echo "false";;
+xen-4.4-testing) echo "false";;
+*) echo "false true";
+esac
+}
+
 create_build_jobs () {
 
   local arch
@@ -139,8 +152,15 @@ create_build_jobs () {
 
 
build_hostflags=share-build-$suite-$arch,arch-$arch,suite-$suite,purpose-build
 
-./cs-job-create $flight build-$arch build\
+for enable_xsm in $(xenbranch_xsm_variants) ; do
+  if [ x$enable_xsm = xtrue ] ; then
+xsm_suffix="-xsm"
+  else
+xsm_suffix=""
+  fi
+  ./cs-job-create $flight build-$arch$xsm_suffix build   \
 arch=$arch enable_xend=$build_defxend enable_ovmf=$enable_ovmf\
+enable_xsm=$enable_xsm   \
 tree_qemu=$TREE_QEMU \
 tree_qemuu=$TREE_QEMU_UPSTREAM   \
 tree_xen=$TREE_XEN   \
@@ -152,6 +172,7 @@ create_build_jobs () {
 revision_qemu=$REVISION_QEMU \
 revision_qemuu=$REVISION_QEMU_UPSTREAM   \
 revision_seabios=$REVISION_SEABIOS
+done
 
 if [ $build_extraxend = "true" ] ; then
 ./cs-job-create $flight build-$arch-xend build   \
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST PATCH v4 2/9] ts-xen-build-prep: install checkpolicy

[Wei Liu]

This is used to complie Flask policy.

Signed-off-by: Wei Liu 
Acked-by: Ian Jackson 
---
 ts-xen-build-prep |2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ts-xen-build-prep b/ts-xen-build-prep
index a7d0d03..4b016ae 100755
--- a/ts-xen-build-prep
+++ b/ts-xen-build-prep
@@ -178,7 +178,7 @@ sub prep () {
autoconf automake libtool xsltproc
libxml2-utils libxml2-dev libnl-dev
libdevmapper-dev w3c-dtd-xhtml libxml-xpath-perl
-  ccache));
+  ccache checkpolicy));
 
 target_cmd_root($ho, "chmod -R a+r /usr/share/git-core/templates");
 # workaround for Debian #595728
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST PATCH v4 1/9] overlay: update overlay/etc/grub.d/20_linux_xen

[Wei Liu]

This file was originally created to work around Debian bug #633127
("/etc/grub/20_linux does not recognise some old Xen kernels").

According to Debian bug tracker [0], #633127 bug is fixed in Wheezy. As
we're now using Wheezy in OSSTest we can safely remove the old overlay
file if there's no further bugs discovered.

However we have another bug #690538 ("grub-common: Please make submenu
creation optional or at least allow users to disable it easily") that
would break OSSTest.  We're now using Wheezy in production. There's no
way to disable submenu in Wheezy. And submenu breaks OSSTest's grub menu
parser.

So update this overlay file to Wheezy's version and take care of Debian
bug #690538 by removing the lines to generate submenu.

Also work around GRUB bug #43420 ("20_linux_xen doesn't support Xen XSM
policy file") by applying a small patch proposed in [2].

Add a note to reference #633127 and #690538 above grub2 setup function.

0: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=633127
1: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=690538
2: https://savannah.gnu.org/bugs/?43420

Signed-off-by: Wei Liu 
Cc: Ian Jackson 
Cc: Ian Campbell 
---
 Osstest/Debian.pm   |5 ++
 overlay/etc/grub.d/20_linux_xen |  117 +++
 2 files changed, 98 insertions(+), 24 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index c8db601..c446e8b 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -274,6 +274,11 @@ sub setupboot_grub1 ($$$) {
 return $bl;
 }
 
+# Note on running OSSTest on Squeeze with old Xen kernel: check out
+# Debian bug #633127 "/etc/grub/20_linux does not recognise some old
+# Xen kernels"
+# Currently setupboot_grub2 relies on Grub menu not having submenu.
+# Check Debian bug #690538.
 sub setupboot_grub2 ($$$) {
 my ($ho,$want_kernver,$xenhopt,$xenkopt) = @_;
 my $bl= { };
diff --git a/overlay/etc/grub.d/20_linux_xen b/overlay/etc/grub.d/20_linux_xen
index 99854d2..001b76d 100755
--- a/overlay/etc/grub.d/20_linux_xen
+++ b/overlay/etc/grub.d/20_linux_xen
@@ -1,7 +1,7 @@
 #! /bin/sh
 
-# Copied from the identically named file in grub-common 1.98+20100804-14
-# i386.  This version fixes #633127 (and has the patch I proposed there).
+# Copied from the identical named file in grub-common 1.99-27+deb7u2.
+# This version fixed Debian bug #690538 and GRUB bug #43420.
 
 set -e
 
@@ -21,14 +21,14 @@ set -e
 # You should have received a copy of the GNU General Public License
 # along with GRUB.  If not, see .
 
-prefix=/usr
-exec_prefix=${prefix}
-bindir=${exec_prefix}/bin
-libdir=${exec_prefix}/lib
-. ${libdir}/grub/grub-mkconfig_lib
+prefix="/usr"
+exec_prefix="${prefix}"
+datarootdir="${prefix}/share"
+
+. "${datarootdir}/grub/grub-mkconfig_lib"
 
 export TEXTDOMAIN=grub
-export TEXTDOMAINDIR=${prefix}/share/locale
+export TEXTDOMAINDIR="${datarootdir}/locale"
 
 CLASS="--class gnu-linux --class gnu --class os --class xen"
 
@@ -36,7 +36,7 @@ if [ "x${GRUB_DISTRIBUTOR}" = "x" ] ; then
   OS=GNU/Linux
 else
   OS="${GRUB_DISTRIBUTOR} GNU/Linux"
-  CLASS="--class $(echo ${GRUB_DISTRIBUTOR} | tr '[A-Z]' '[a-z]' | cut -d' ' 
-f1) ${CLASS}"
+  CLASS="--class $(echo ${GRUB_DISTRIBUTOR} | tr 'A-Z' 'a-z' | cut -d' ' -f1) 
${CLASS}"
 fi
 
 # loop-AES arranges things so that /dev/loop/X can be our root device, but
@@ -44,6 +44,11 @@ fi
 case ${GRUB_DEVICE} in
   /dev/loop/*|/dev/loop[0-9])
 GRUB_DEVICE=`losetup ${GRUB_DEVICE} | sed -e "s/^[^(]*(\([^)]\+\)).*/\1/"`
+# We can't cope with devices loop-mounted from files here.
+case ${GRUB_DEVICE} in
+  /dev/*) ;;
+  *) exit 0 ;;
+esac
   ;;
 esac
 
@@ -55,6 +60,23 @@ else
   LINUX_ROOT_DEVICE=UUID=${GRUB_DEVICE_UUID}
 fi
 
+# Allow overriding GRUB_CMDLINE_LINUX and GRUB_CMDLINE_LINUX_DEFAULT.
+if [ "${GRUB_CMDLINE_LINUX_XEN_REPLACE}" ]; then
+  GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX_XEN_REPLACE}"
+fi
+if [ "${GRUB_CMDLINE_LINUX_XEN_REPLACE_DEFAULT}" ]; then
+  GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_XEN_REPLACE_DEFAULT}"
+fi
+
+if [ "x`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2>/dev/null || 
true`" = xbtrfs ] \
+|| [ "x`stat -f --printf=%T /`" = xbtrfs ]; then
+  rootsubvol="`make_system_path_relative_to_its_root /`"
+  rootsubvol="${rootsubvol#/}"
+  if [ "x${rootsubvol}" != x ]; then
+GRUB_CMDLINE_LINUX="rootflags=subvol=${rootsubvol} ${GRUB_CMDLINE_LINUX}"
+  fi
+fi
+
 linux_entry ()
 {
   os="$1"
@@ -63,22 +85,43 @@ linux_entry ()
   recovery="$4"
   args="$5"
   xen_args="$6"
-  if ${recovery} ; then
-title="$(gettext_quoted "%s, with Linux %s and XEN %s (recovery mode)")"
+  xsm="$7"
+  # If user wants to enable XSM support, make sure there's
+  # corresponding policy file.
+  if ${xsm} ; then
+  xenpolicy=`echo xenpolicy-$xen_version`
+  if test ! -e "${xen_dirname}/${xenpolicy}" ; then
+ return
+  fi
+  xen_args=`echo $xen_args flask_enabled=1 flask_enforcing=1`
+

[Xen-devel] [OSSTEST PATCH v4 8/9] make-flight: factor out do_pv_debian_tests

[Wei Liu]

Signed-off-by: Wei Liu 
---
 make-flight |   21 ++---
 1 file changed, 14 insertions(+), 7 deletions(-)

diff --git a/make-flight b/make-flight
index 9963a46..35904be 100755
--- a/make-flight
+++ b/make-flight
@@ -281,17 +281,24 @@ do_passthrough_tests () {
   done
 }
 
-test_matrix_do_one () {
-
-  # Basic PV Linux test with xl
+do_pv_debian_test_one () {
+  toolstack=$1
 
-  job_create_test test-$xenarch$kern-$dom0arch-xl test-debian xl \
+  job_create_test test-$xenarch$kern-$dom0arch-$toolstack test-debian 
$toolstack \
 $xenarch $dom0arch   \
 $debian_runvars all_hostflags=$most_hostflags
+}
 
-  job_create_test test-$xenarch$kern-$dom0arch-libvirt test-debian libvirt \
-$xenarch $dom0arch   \
-$debian_runvars all_hostflags=$most_hostflags
+do_pv_debian_tests () {
+  for toolstack in xl libvirt; do
+do_pv_debian_test_one $toolstack
+  done
+}
+
+test_matrix_do_one () {
+
+  # Basic PV Linux test with xl
+  do_pv_debian_tests
 
   # No further arm tests at the moment
   if [ $dom0arch = armhf ]; then
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST PATCH v4 9/9] mfi-common, make-flight: create XSM test jobs

[Wei Liu]

Duplicate Debian PV and HVM test jobs for XSM testing.

Signed-off-by: Wei Liu 
---
Changes in v4:
1. Parse runvar to determine xsm suffix
---
 make-flight |   23 +++
 mfi-common  |   12 ++--
 2 files changed, 29 insertions(+), 6 deletions(-)

diff --git a/make-flight b/make-flight
index 35904be..00bc0fc 100755
--- a/make-flight
+++ b/make-flight
@@ -200,27 +200,36 @@ do_hvm_win7_x64_tests () {
 do_hvm_debian_test_one () {
   testname=$1
   bios=$2
+  xsm=$3
+
   job_create_test test-$xenarch$kern-$dom0arch-xl$qemuu_suffix-$testname-amd64\
 test-debianhvm xl $xenarch $dom0arch $qemuu_runvar \
+enable_xsm=$xsm \
 debianhvm_image=debian-7.2.0-amd64-CD-1.iso \
 bios=$bios \
 all_hostflags=$most_hostflags,hvm
 }
 
 do_hvm_debian_tests() {
+  test_xsm=$(xenbranch_xsm_variants)
+
   if [ $xenarch != amd64 ]; then
 return
   fi
 
   # QEMU upstream supports ovmf and seabios
   if [ "x$qemuu_suffix" == "x-qemuu" ]; then
-do_hvm_debian_test_one ovmf ovmf
-do_hvm_debian_test_one debianhvm seabios
+do_hvm_debian_test_one ovmf ovmf false
+for xsm in $test_xsm ; do
+  do_hvm_debian_test_one debianhvm seabios $xsm
+done
   fi
 
   # QEMU traditional supports rombios
   if [ "x$qemuu_suffix" == "x-qemut" ]; then
-do_hvm_debian_test_one debianhvm rombios
+for xsm in $test_xsm ; do
+  do_hvm_debian_test_one debianhvm rombios $xsm
+done
   fi
 }
 
@@ -283,15 +292,21 @@ do_passthrough_tests () {
 
 do_pv_debian_test_one () {
   toolstack=$1
+  xsm=$2
 
   job_create_test test-$xenarch$kern-$dom0arch-$toolstack test-debian 
$toolstack \
 $xenarch $dom0arch   \
+enable_xsm=$xsm  \
 $debian_runvars all_hostflags=$most_hostflags
 }
 
 do_pv_debian_tests () {
+  test_xsm=$(xenbranch_xsm_variants)
+
   for toolstack in xl libvirt; do
-do_pv_debian_test_one $toolstack
+for xsm in $test_xsm ; do
+  do_pv_debian_test_one $toolstack $xsm
+done
   done
 }
 
diff --git a/mfi-common b/mfi-common
index 161847c..3a97a90 100644
--- a/mfi-common
+++ b/mfi-common
@@ -268,8 +268,16 @@ job_create_test () {
   local xenarch=$1; shift
   local dom0arch=$1; shift
 
-  xenbuildjob="${bfi}build-$xenarch"
-  buildjob="${bfi}build-$dom0arch"
+  xsm_suffix=""
+  for rv in $@ ; do
+  case $rv in
+  enable_xsm=true) xsm_suffix="-xsm";;
+  esac
+  done
+
+  job="$job$xsm_suffix"
+  xenbuildjob="${bfi}build-$xenarch$xsm_suffix"
+  buildjob="${bfi}build-$dom0arch$xsm_suffix"
   tsbuildjob=
 
   case "$xenbranch:$toolstack" in
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST PATCH v4 0/9] XSM test case for OSSTest

[Wei Liu]

Hi all

This patch series attempts to duplicate some Debian test cases for XSM. This
is version 4 of this series.

Tests duplicated for xen-unstable branch:
  build-{i386,amd64,armhf}-xsm
  test-amd64-{i386,amd64}-{xl,libvirt}-xsm
  test-armhf-armhf-{xl,libvirt}-xsm
  test-amd64-{i386,amd64}-xl-qemuu-debianhvm-amd64-xsm
  test-amd64-(i386,amd64}-xl-qemut-debianhvm-amd64-xsm

Changes in v4 can be found in individual patch.

See below for output of
  ./standalone-generate-dump-flight-runvars > origin # master
  ./standalone-generate-dump-flight-runvars > xsm # this series applied
  diff -ub ../origin xsm  | grep '-xen-unstable' | sed  's/[ \t]*$//' # nothing
  diff -ub ../origin xsm  | grep '+xen-unstable' | sed  's/[ \t]*$//'

+xen-unstable   test-amd64-amd64-libvirt-xsm  
all_hostflags   arch-amd64,arch-xen-amd64,suite-wheezy,purpose-test
+xen-unstable   test-amd64-amd64-xl-qemut-debianhvm-amd64-xsm 
all_hostflags   
arch-amd64,arch-xen-amd64,suite-wheezy,purpose-test,hvm
+xen-unstable   test-amd64-amd64-xl-qemuu-debianhvm-amd64-xsm 
all_hostflags   
arch-amd64,arch-xen-amd64,suite-wheezy,purpose-test,hvm
+xen-unstable   test-amd64-amd64-xl-xsm   
all_hostflags   arch-amd64,arch-xen-amd64,suite-wheezy,purpose-test
+xen-unstable   test-amd64-i386-libvirt-xsm   
all_hostflags   arch-i386,arch-xen-amd64,suite-wheezy,purpose-test
+xen-unstable   test-amd64-i386-xl-qemut-debianhvm-amd64-xsm  
all_hostflags   
arch-i386,arch-xen-amd64,suite-wheezy,purpose-test,hvm
+xen-unstable   test-amd64-i386-xl-qemuu-debianhvm-amd64-xsm  
all_hostflags   
arch-i386,arch-xen-amd64,suite-wheezy,purpose-test,hvm
+xen-unstable   test-amd64-i386-xl-xsm
all_hostflags   arch-i386,arch-xen-amd64,suite-wheezy,purpose-test
+xen-unstable   test-armhf-armhf-libvirt-xsm  
all_hostflags   arch-armhf,arch-xen-armhf,suite-wheezy,purpose-test
+xen-unstable   test-armhf-armhf-xl-xsm   
all_hostflags   arch-armhf,arch-xen-armhf,suite-wheezy,purpose-test
+xen-unstable   build-amd64-xsm   arch  
  amd64
+xen-unstable   build-armhf-xsm   arch  
  armhf
+xen-unstable   build-i386-xsmarch  
  i386
+xen-unstable   test-amd64-amd64-libvirt-xsm  arch  
  amd64
+xen-unstable   test-amd64-amd64-xl-qemut-debianhvm-amd64-xsm arch  
  amd64
+xen-unstable   test-amd64-amd64-xl-qemuu-debianhvm-amd64-xsm arch  
  amd64
+xen-unstable   test-amd64-amd64-xl-xsm   arch  
  amd64
+xen-unstable   test-amd64-i386-libvirt-xsm   arch  
  i386
+xen-unstable   test-amd64-i386-xl-qemut-debianhvm-amd64-xsm  arch  
  i386
+xen-unstable   test-amd64-i386-xl-qemuu-debianhvm-amd64-xsm  arch  
  i386
+xen-unstable   test-amd64-i386-xl-xsmarch  
  i386
+xen-unstable   test-armhf-armhf-libvirt-xsm  arch  
  armhf
+xen-unstable   test-armhf-armhf-xl-xsm   arch  
  armhf
+xen-unstable   test-amd64-amd64-xl-qemut-debianhvm-amd64-xsm bios  
  rombios
+xen-unstable   test-amd64-amd64-xl-qemuu-debianhvm-amd64-xsm bios  
  seabios
+xen-unstable   test-amd64-i386-xl-qemut-debianhvm-amd64-xsm  bios  
  rombios
+xen-unstable   test-amd64-i386-xl-qemuu-debianhvm-amd64-xsm  bios  
  seabios
+xen-unstable   build-amd64-xsm   
build_lvextend_max  50
+xen-unstable   build-armhf-xsm   
build_lvextend_max  50
+xen-unstable   build-i386-xsm
build_lvextend_max  50
+xen-unstable   test-amd64-amd64-libvirt-xsm  
buildjobbuild-amd64-xsm
+xen-unstable   test-amd64-amd64-xl-qemut-debianhvm-amd64-xsm 
buildjobbuild-amd64-xsm
+xen-unstable   test-amd64-amd64-xl-qemuu-debianhvm-amd64-xsm 
buildjobbuild-amd64-xsm
+xen-unstable   test-amd64-amd64-xl-xsm   
buildjobbuild-amd64-xsm
+xen-unstable   test-amd64-i386-lib

[Xen-devel] [OSSTEST PATCH v4 3/9] ts-xen-build: build with XSM support if requested

[Wei Liu]

Signed-off-by: Wei Liu 
---
Changes in v4:
1. Use "true" instead of "y"
---
 ts-xen-build |   12 
 1 file changed, 12 insertions(+)

diff --git a/ts-xen-build b/ts-xen-build
index 661f186..9ee4522 100755
--- a/ts-xen-build
+++ b/ts-xen-build
@@ -27,6 +27,8 @@ tsreadconfig();
 selectbuildhost(\@ARGV);
 # remaining arguments are passed as targets to "make"
 builddirsprops();
+
+my $enable_xsm = $r{enable_xsm} =~ m/true/ ? 1 : 0;
 
 sub checkout () {
 prepbuilddirs();
@@ -34,6 +36,7 @@ sub checkout () {
 build_clone($ho, 'xen', $builddir, 'xen');
 
 my $debug_build = $r{xen_build_debug} || 'y';
+my $build_xsm = $enable_xsm ? 'y' : 'n';
 
 # Do not set this unless you know what you are doing. This arm
 # option makes the build specific to a particular type of
@@ -47,6 +50,7 @@ sub checkout () {
 cd $builddir/xen
>.config
echo >>.config debug=$debug_build
+   echo >>.config XSM_ENABLE=$build_xsm
echo >>.config GIT_HTTP=y
echo >>.config LIBLEAFDIR_x86_64=lib
echo >>.config QEMU_REMOTE='$r{tree_qemu}'
@@ -114,6 +118,14 @@ END
 buildcmd_stamped_logged(9000, 'build', '',

[Xen-devel] [OSSTEST PATCH v4 5/9] Debian.pm: pass in XSM configuration to bootloader setup routines

[Wei Liu]

Change to Uboot will come in another patch. GRUB 1 is ignored, as
currently OSSTest only has Wheezy which has GRUB 2.

Signed-off-by: Wei Liu 
Acked-by: Ian Jackson 
---
Changes in v4:
1. Modify callsite of debian_boot_setup to avoid regression.
---
 Osstest/Debian.pm |   32 +---
 ts-xen-install|2 +-
 2 files changed, 22 insertions(+), 12 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index c446e8b..22b40ff 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -45,9 +45,9 @@ BEGIN {
 
 #-- manipulation of Debian bootloader setup --
 
-sub debian_boot_setup (;$) {
+sub debian_boot_setup ($;$) {
 # $xenhopt==undef => is actually a guest, do not set up a hypervisor
-my ($ho, $want_kernver, $xenhopt, $distpath, $hooks) = @_;
+my ($ho, $want_kernver, $want_xsm, $xenhopt, $distpath, $hooks) = @_;
 
 target_kernkind_check($ho);
 target_kernkind_console_inittab($ho,$ho,"/");
@@ -72,11 +72,11 @@ sub debian_boot_setup (;$) {
 
 my $bootloader;
 if ( $ho->{Flags}{'need-uboot-bootscr'} ) {
-   $bootloader= setupboot_uboot($ho, $want_kernver, $xenhopt, $kopt);
+   $bootloader= setupboot_uboot($ho, $want_kernver, $want_xsm, $xenhopt, 
$kopt);
 } elsif ($ho->{Suite} =~ m/lenny/) {
-$bootloader= setupboot_grub1($ho, $want_kernver, $xenhopt, $kopt);
+$bootloader= setupboot_grub1($ho, $want_kernver, $want_xsm, $xenhopt, 
$kopt);
 } else {
-$bootloader= setupboot_grub2($ho, $want_kernver, $xenhopt, $kopt);
+$bootloader= setupboot_grub2($ho, $want_kernver, $want_xsm, $xenhopt, 
$kopt);
 }
 
 $bootloader->{UpdateConfig}($ho);
@@ -112,8 +112,8 @@ sub bl_getmenu_open ($$$) {
 return $f;
 }
 
-sub setupboot_uboot ($$$) {
-my ($ho,$want_kernver,$xenhopt,$xenkopt) = @_;
+sub setupboot_uboot () {
+my ($ho,$want_kernver,$want_xsm,$xenhopt,$xenkopt) = @_;
 my $bl= { };
 
 $bl->{UpdateConfig}= sub {
@@ -194,13 +194,17 @@ END
 return $bl;
 }
 
-sub setupboot_grub1 ($$$) {
-my ($ho,$want_kernver,$xenhopt,$xenkopt) = @_;
+sub setupboot_grub1 () {
+my ($ho,$want_kernver,$want_xsm,$xenhopt,$xenkopt) = @_;
 my $bl= { };
 
 my $rmenu= "/boot/grub/menu.lst";
 my $lmenu= "$stash/$ho->{Name}--menu.lst.out";
 
+if ($want_xsm) {
+   die "Enabling XSM with GRUB is not supported";
+}
+
 target_editfile_root($ho, $rmenu, sub {
 while (<::EI>) {
 if (m/^## ## Start Default/ ..
@@ -279,8 +283,8 @@ sub setupboot_grub1 ($$$) {
 # Xen kernels"
 # Currently setupboot_grub2 relies on Grub menu not having submenu.
 # Check Debian bug #690538.
-sub setupboot_grub2 ($$$) {
-my ($ho,$want_kernver,$xenhopt,$xenkopt) = @_;
+sub setupboot_grub2 () {
+my ($ho,$want_kernver,$want_xsm,$xenhopt,$xenkopt) = @_;
 my $bl= { };
 
 my $rmenu= '/boot/grub/grub.cfg';
@@ -307,6 +311,9 @@ sub setupboot_grub2 ($$$) {
 $entry->{KernVer} ne $want_kernver) {
logm("(skipping entry at $entry->{StartLine};".
 " kernel $entry->{KernVer}, not $want_kernver)");
+   } elsif ($want_xsm && !defined $entry->{Xenpolicy}) {
+   logm("(skipping entry at $entry->{StartLine};".
+" XSM policy file not present)");
} else {
# yes!
last;
@@ -339,6 +346,9 @@ sub setupboot_grub2 ($$$) {
 if (m/^\s*module\s*\/(initrd\S+)/) {
 $entry->{Initrd}= $1;
 }
+   if (m/^\s*module\s*\/(xenpolicy\S+)/) {
+$entry->{Xenpolicy}= $1;
+}
 }
 die 'grub 2 bootloader entry not found' unless $entry;
 
diff --git a/ts-xen-install b/ts-xen-install
index 4d34d1f..910181e 100755
--- a/ts-xen-install
+++ b/ts-xen-install
@@ -171,7 +171,7 @@ sub setupboot () {
 }
 
 my $want_kernver = get_runvar('kernel_ver',$r{'kernbuildjob'});
-debian_boot_setup($ho, $want_kernver, $xenhopt, \%distpath, \@hooks);
+debian_boot_setup($ho, $want_kernver, 0, $xenhopt, \%distpath, \@hooks);
 
 logm("ready to boot Xen");
 }
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [rumpuserxen test] 32219: all pass - PUSHED

[xen . org]

flight 32219 rumpuserxen real [real]
http://www.chiark.greenend.org.uk/~xensrcts/logs/32219/

Perfect :-)
All tests in this flight passed
version targeted for testing:
 rumpuserxen  d40acc2019bd352e1de13842459b5fecf5bc565e
baseline version:
 rumpuserxen  b0b7b13cadc971e8c5745880dcbb92bc2185765b


People who touched revisions under test:
  Antti Kantee 


jobs:
 build-amd64  pass
 build-i386   pass
 build-amd64-pvopspass
 build-i386-pvops pass
 build-amd64-rumpuserxen  pass
 build-i386-rumpuserxen   pass
 test-amd64-amd64-rumpuserxen-amd64   pass
 test-amd64-i386-rumpuserxen-i386 pass



sg-report-flight on osstest.cam.xci-test.com
logs: /home/xc_osstest/logs
images: /home/xc_osstest/images

Logs, config files, etc. are available at
http://www.chiark.greenend.org.uk/~xensrcts/logs

Test harness code can be found at
http://xenbits.xensource.com/gitweb?p=osstest.git;a=summary


Pushing revision :

+ branch=rumpuserxen
+ revision=d40acc2019bd352e1de13842459b5fecf5bc565e
+ . cri-lock-repos
++ . cri-common
+++ . cri-getconfig
+++ umask 002
+++ getconfig Repos
+++ perl -e '
use Osstest;
readglobalconfig();
print $c{"Repos"} or die $!;
'
++ repos=/export/home/osstest/repos
++ repos_lock=/export/home/osstest/repos/lock
++ '[' x '!=' x/export/home/osstest/repos/lock ']'
++ OSSTEST_REPOS_LOCK_LOCKED=/export/home/osstest/repos/lock
++ exec with-lock-ex -w /export/home/osstest/repos/lock ./ap-push rumpuserxen 
d40acc2019bd352e1de13842459b5fecf5bc565e
+ branch=rumpuserxen
+ revision=d40acc2019bd352e1de13842459b5fecf5bc565e
+ . cri-lock-repos
++ . cri-common
+++ . cri-getconfig
+++ umask 002
+++ getconfig Repos
+++ perl -e '
use Osstest;
readglobalconfig();
print $c{"Repos"} or die $!;
'
++ repos=/export/home/osstest/repos
++ repos_lock=/export/home/osstest/repos/lock
++ '[' x/export/home/osstest/repos/lock '!=' x/export/home/osstest/repos/lock 
']'
+ . cri-common
++ . cri-getconfig
++ umask 002
+ select_xenbranch
+ case "$branch" in
+ tree=rumpuserxen
+ xenbranch=xen-unstable
+ '[' xrumpuserxen = xlinux ']'
+ linuxbranch=
+ '[' x = x ']'
+ qemuubranch=qemu-upstream-unstable
+ : tested/2.6.39.x
+ . ap-common
++ : osst...@xenbits.xensource.com
++ : git://xenbits.xen.org/xen.git
++ : osst...@xenbits.xensource.com:/home/xen/git/xen.git
++ : git://xenbits.xen.org/staging/qemu-xen-unstable.git
++ : git://git.kernel.org
++ : git://git.kernel.org/pub/scm/linux/kernel/git
++ : git
++ : git://xenbits.xen.org/libvirt.git
++ : osst...@xenbits.xensource.com:/home/xen/git/libvirt.git
++ : git://xenbits.xen.org/libvirt.git
+++ besteffort_repo git://git.sv.gnu.org/gnulib.git
+++ local repo=git://git.sv.gnu.org/gnulib.git
+++ cached_repo git://git.sv.gnu.org/gnulib.git '[fetch=try]'
+++ local repo=git://git.sv.gnu.org/gnulib.git
+++ local 'options=[fetch=try]'
 getconfig GitCacheProxy
 perl -e '
use Osstest;
readglobalconfig();
print $c{"GitCacheProxy"} or die $!;
'
+++ local cache=git://drall.uk.xensource.com:9419/
+++ '[' xgit://drall.uk.xensource.com:9419/ '!=' x ']'
+++ echo 
'git://drall.uk.xensource.com:9419/git://git.sv.gnu.org/gnulib.git%20[fetch=try]'
++ : 
'git://drall.uk.xensource.com:9419/git://git.sv.gnu.org/gnulib.git%20[fetch=try]'
++ : https://github.com/rumpkernel/rumprun-xen
++ : git
++ : git://xenbits.xen.org/rumpuser-xen.git
++ : osst...@xenbits.xensource.com:/home/xen/git/rumpuser-xen.git
+++ besteffort_repo https://github.com/rumpkernel/rumpkernel-netbsd-src
+++ local repo=https://github.com/rumpkernel/rumpkernel-netbsd-src
+++ cached_repo https://github.com/rumpkernel/rumpkernel-netbsd-src 
'[fetch=try]'
+++ local repo=https://github.com/rumpkernel/rumpkernel-netbsd-src
+++ local 'options=[fetch=try]'
 getconfig GitCacheProxy
 perl -e '
use Osstest;
readglobalconfig();
print $c{"GitCacheProxy"} or die $!;
'
+++ local cache=git://drall.uk.xensource.com:9419/
+++ '[' xgit://drall.uk.xensource.com:9419/ '!=' x ']'
+++ echo 
'git://drall.uk.xensource.com:9419/https://github.com/rumpkernel/rumpkernel-netbsd-src%20[fetch=try]'
++ : 
'git://drall.uk.xensource.com:9419/https://github.com/rumpkernel/rumpkernel-netbsd-src%20[fetch=try]'
++ : git
++ : git://git.seabios.org/seabios.git
++ : osst...@xenbits.xensource.com:/home/xen/git/osstest/seabios.git
++ 

Re: [Xen-devel] [GIT PULL] (xen) for-jens-3.19 for v3.19 Xen blkfront driver updates.

[Jens Axboe]

On 12/10/2014 01:57 PM, Konrad Rzeszutek Wilk wrote:
> Hey Jens,
> 
> These are two fixes for Xen blkfront. They harden how it deals with
> broken backends.

Pulled, thanks.


-- 
Jens Axboe


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH] Update it work with Fedora Core 21.

[Konrad Rzeszutek Wilk]

Signed-off-by: Konrad Rzeszutek Wilk 
---
 fedora-live-xen.ks | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/fedora-live-xen.ks b/fedora-live-xen.ks
index 58f94b1..c99e46c 100644
--- a/fedora-live-xen.ks
+++ b/fedora-live-xen.ks
@@ -8,6 +8,7 @@
 repo --name=virt-preview 
--baseurl=http://fedorapeople.org/groups/virt/virt-preview/fedora-$releasever/$basearch
 # What about updates-testing reository? Well, why not!
 repo --name=updates-testing 
--mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=updates-testing-f$releasever&arch=$basearch
+repo --name=fedora 
--mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=$basearch
 # And what about rawhide? Mmm... better not to go that far...
 #%include fedora-repo-rawhide.ks
 
@@ -45,7 +46,7 @@ xen*
 xen-devel
 libvirt-daemon-xen
 -gnome-boxes
-
+firewalld
 # Get rid of a few heavy stuff
 -@libreoffice
 -@printing
-- 
2.1.0


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [GIT PULL] (xen) for-jens-3.19 for v3.19 Xen blkfront driver updates.

[Konrad Rzeszutek Wilk]

Hey Jens,

These are two fixes for Xen blkfront. They harden how it deals with
broken backends.

Please git pull the following branch:

git pull git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git 
for-jens-3.19

in your for-3.19-drivers branch. This branch is based on:
9af8785 NVMe: Fix command setup on IO retry

Thanks!

The changelog:

drivers/block/xen-blkfront.c | 65 ++--
 1 file changed, 39 insertions(+), 26 deletions(-)

Vitaly Kuznetsov (2):
  xen/blkfront: improve protection against issuing unsupported REQ_FUA
  xen/blkfront: remove redundant flush_op



pgpTbLJziRNeC.pgp
Description: PGP signature
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] Question reg. named vif interface support in guest cfg;

[Bjoern Rennhak]

Dear Xen Developers,

I found handling of vifx.y interfaces slightly unintutive and was
wondering if it is possible to name the interface via the Xen guest
config file?

e.g. a vif_name parameter or otherwise maybe vif interface is automatically 
named
after set "name" entry in config instead of generic vifx.y.

So instead of a bunch of vifx.y's I would see x.y ?

Is that already possible or a bad idea in general?
Should I open a wishlist entry on http://bugs.xenproject.org/xen/?

Thank you!

Cheers,
-Bjoern


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH 0/4 v2] tools/hotplug: systemd changes for 4.5

[Konrad Rzeszutek Wilk]

On Mon, Dec 08, 2014 at 11:18:05AM +0100, Olaf Hering wrote:
> This is a resend of this series, with just the low hanging fruits:
> http://lists.xenproject.org/archives/html/xen-devel/2014-12/msg00669.html
> 

This looks like it would fix some of the issues I saw. I will test it
over today.

Please also CC Michael (Fedora Xen maintainer) on these changes (I've CC-ed
him here).

 The mentioned wrapper to run xenstored from systemd without duplicate
> functionality found in the sysv runlevel script will be send in another patch,
> once it is ready.
> 
> Olaf
> 
> Olaf Hering (4):
>   tools/hotplug: remove SELinux options from var-lib-xenstored.mount
>   tools/hotplug: remove XENSTORED_ROOTDIR from service file
>   tools/hotplug: remove EnvironmentFile from
> xen-qemu-dom0-disk-backend.service
>   tools/hotplug: use xencommons as EnvironmentFile
> 
>  tools/hotplug/Linux/systemd/var-lib-xenstored.mount.in| 4 +---
>  tools/hotplug/Linux/systemd/xen-qemu-dom0-disk-backend.service.in | 1 -
>  tools/hotplug/Linux/systemd/xenconsoled.service.in| 2 +-
>  tools/hotplug/Linux/systemd/xenstored.service.in  | 1 -
>  4 files changed, 2 insertions(+), 6 deletions(-)
> 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [v8][PATCH 04/17] update the existing hypercall to support XEN_DOMCTL_set_rdm

[Konrad Rzeszutek Wilk]

On Wed, Dec 10, 2014 at 09:59:12AM +0800, Chen, Tiejun wrote:
> On 2014/12/9 18:22, Jan Beulich wrote:
> On 09.12.14 at 11:11,  wrote:
> >>At 08:19 + on 09 Dec (1418109561), Jan Beulich wrote:
> >>>Why do you always pick other than the simplest possible solution?
> >>
> >>Jan, please don't make personal comments like this in code review.  It
> >>can only offend and demoralize contributors, and deter other people
> >>from joining in.
> >
> >I apologize - I shouldn't have permitted myself to do so.
> >
> >>I understand that it can be frustrating, and I'm sure I have lashed
> 
> Actually myself also have this same feeling but this really isn't helpful to
> step forward.
> 
> >>out at people on-list in the past.  But remember that people who are
> >>new to the project need time to learn, and keep the comments to the
> >>code itself.
> >>
> >>I can see that this series has been going for a long time, and is
> >>still getting hung up on coding style issues.  Might it be useful to
> 
> Something is my fault here. Although I learn more about Xen from this series
> constantly, looks its hard to cover this big thread with that reasonable
> code under Xen circumstance. This is also why I claimed I can't do this
> right from the start.
> 
> And additionally, even some initial designs are not very clear or argued
> between us, so when we discuss something during each revision, it bring a
> new thought again...


> 
> >>have a round of internal review from some of the more xen-experienced
> >>engineers at Intel before Jan looks at it again?
> >
> >I've been suggesting something along those lines a number of times,
> >with (apparently) no success at all.
> >
> 
> Actually we had a little bit discussion internal and some guys really
> brought me some comments previously, but I think they're too busy to review
> each patch carefully one line by one line, especially if I bring something
> new to address yours comments just in the process of public review.
> 
> Anyway, now let me ask if this can deliver to other suitable guy. As I said
> previously I really would like to quit if this can step next properly.

Bummer. I was really looking forward to more patches from you. I understand
the frustration working throught this (my first patches in Linux took 9
months!) and I was hoping that you having to go through a pretty complex
code and getting the it done (with twist and turns at every corner) the
end result would be:
 - You would understand the Xen codebase at an expert level!
 - And would be able to contribute to Xen on other features knowing
   pitfalls, etc.
 - You would be able to help other engineers that start working on Xen
   how to do it.
 - Comfrotably review other folks patches.

Is there something that can be done for you to not step away?

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] xen-netback: make feature-rx-notify mandatory -- Breaks stubdoms

[David Vrabel]

On 10/12/14 16:20, Ian Campbell wrote:
> On Wed, 2014-12-10 at 15:29 +, David Vrabel wrote:
>> On 10/12/14 15:07, Ian Campbell wrote:
>>> On Wed, 2014-12-10 at 14:12 +, David Vrabel wrote:
 On 10/12/14 13:42, John wrote:
> David,
>
> This patch you put into 3.18.0 appears to break the latest version of
> stubdomains. I found this out today when I tried to update a machine to
> 3.18.0 and all of the domUs crashed on start with the dmesg output like
> this:

 Cc'ing the lists and relevant netback maintainers.

 I guess the stubdoms are using minios's netfront?  This is something I
 forgot about when deciding if it was ok to make this feature mandatory.
>>>
>>> Oh bum, me too :/
>>>
 The patch cannot be reverted as it's a prerequisite for a critical
 (security) bug fix.  I am also unconvinced that the no-feature-rx-notify
 support worked correctly anyway.

 This can be resolved by:

 - Fixing minios's netfront to support feature-rx-notify. This should be
 easy but wouldn't help existing Xen deployments.
>>>
>>> I think this is worth doing in its own right, but as you say it doesn't
>>> help existing users.
>>>
 - Reimplement feature-rx-notify support.  I think the easiest way is to
 queue packets on the guest Rx internal queue with a short expiry time.
>>>
>>> Right, I don't think we especially need to make this case good (so long
>>> as it doesn't reintroduce a security hole!).
>>>
>>> In principal we aren't really obliged to queue at all, but since all the
>>> infrastructure for queuing and timing out all exists I suppose it would
>>> be simple enough to implement and a bit less harsh.
>>>
>>> Given we now have XENVIF_RX_QUEUE_BYTES and rx_drain_timeout_jiffies we
>>> don't have the infinite queue any more. So does the expiry in this case
>>> actually need to be shorter than the norm? Does it cause any extra
>>> issues to keep them around for tx_drain_timeout_jiffies rather than some
>>> shorter time?
>>
>> If the internal guest rx queue fills and the (host) tx queue is stopped,
>> it will take tx_drain_timeout for the thread to wake up and notice if
>> the frontend placed any rx requests on the ring.  This could potentially
>> end up where you shovel 512k through stall for 10 s, put another 512k
>> through, stall for 10 s again and so on.
> 
> Ah, true, that's not so great.
> 
> What about if we don't queue at all(*) if rx-notify isn't supported, i.e
> just drop the packet on the floor in start_xmit if the ring is full?
> Would that be so bad? It would surely be simple...

There needs to be a queue between start_xmit and the rx thread so
checking for ring state in start_xmit doesn't help here since the
internal queue can fill before the thread wakes and begins to drain it.

netback could complete the request directly in start_xmit, avoiding the
internal queue but not allowing for any batching but I don't think it is
a good idea to add a different data path for this mode.

> (*) Not counting the "queue" which is the ring itself.
> 
>> The rx stall detection will also need to be disabled since there would
>> be no way for the frontend to signal rx ready.
> 
> Agreed.
> 
> Could be trivially argued to be safe if we were just dropping packets on
> ring overflow...

David


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [xen-unstable test] 32198: tolerable FAIL - PUSHED

[xen . org]

flight 32198 xen-unstable real [real]
http://www.chiark.greenend.org.uk/~xensrcts/logs/32198/

Failures :-/ but no regressions.

Regressions which are regarded as allowable (not blocking):
 test-amd64-i386-pair17 guest-migrate/src_host/dst_host fail like 32153

Tests which did not succeed, but are not blocking:
 test-amd64-i386-libvirt   9 guest-start  fail   never pass
 test-armhf-armhf-xl  10 migrate-support-checkfail   never pass
 test-amd64-amd64-xl-pcipt-intel  9 guest-start fail never pass
 test-armhf-armhf-libvirt  9 guest-start  fail   never pass
 test-amd64-amd64-libvirt  9 guest-start  fail   never pass
 test-amd64-i386-xl-qemut-winxpsp3 14 guest-stopfail never pass
 test-amd64-i386-xl-qemuu-winxpsp3-vcpus1 14 guest-stop fail never pass
 test-amd64-i386-xl-qemuu-winxpsp3 14 guest-stopfail never pass
 test-amd64-i386-xl-qemut-winxpsp3-vcpus1 14 guest-stop fail never pass
 test-amd64-amd64-xl-qemut-win7-amd64 14 guest-stop fail never pass
 test-amd64-amd64-xl-qemuu-win7-amd64 14 guest-stop fail never pass
 test-amd64-amd64-xl-win7-amd64 14 guest-stop   fail never pass
 test-amd64-amd64-xl-qemut-winxpsp3 14 guest-stop   fail never pass
 test-amd64-i386-xl-qemut-win7-amd64 14 guest-stop  fail never pass
 test-amd64-i386-xl-win7-amd64 14 guest-stop   fail  never pass
 test-amd64-i386-xl-winxpsp3-vcpus1 14 guest-stop   fail never pass
 test-amd64-amd64-xl-winxpsp3 14 guest-stop   fail   never pass
 test-amd64-i386-xl-winxpsp3  14 guest-stop   fail   never pass
 test-amd64-i386-xl-qemuu-win7-amd64 14 guest-stop  fail never pass
 test-amd64-amd64-xl-qemuu-winxpsp3 14 guest-stop   fail never pass

version targeted for testing:
 xen  2a549b9c8aa48dc39d7c97e5a93978b781b3a1db
baseline version:
 xen  10e7747bca53820e313574432f231070153b


People who touched revisions under test:
  Andrew Cooper 
  Keir Fraser 


jobs:
 build-amd64  pass
 build-armhf  pass
 build-i386   pass
 build-amd64-libvirt  pass
 build-armhf-libvirt  pass
 build-i386-libvirt   pass
 build-amd64-oldkern  pass
 build-i386-oldkern   pass
 build-amd64-pvopspass
 build-armhf-pvopspass
 build-i386-pvops pass
 build-amd64-rumpuserxen  pass
 build-i386-rumpuserxen   pass
 test-amd64-amd64-xl  pass
 test-armhf-armhf-xl  pass
 test-amd64-i386-xl   pass
 test-amd64-i386-rhel6hvm-amd pass
 test-amd64-i386-qemut-rhel6hvm-amd   pass
 test-amd64-i386-qemuu-rhel6hvm-amd   pass
 test-amd64-amd64-xl-qemut-debianhvm-amd64pass
 test-amd64-i386-xl-qemut-debianhvm-amd64 pass
 test-amd64-amd64-xl-qemuu-debianhvm-amd64pass
 test-amd64-i386-xl-qemuu-debianhvm-amd64 pass
 test-amd64-i386-freebsd10-amd64  pass
 test-amd64-amd64-xl-qemuu-ovmf-amd64 pass
 test-amd64-i386-xl-qemuu-ovmf-amd64  pass
 test-amd64-amd64-rumpuserxen-amd64   pass
 test-amd64-amd64-xl-qemut-win7-amd64 fail
 test-amd64-i386-xl-qemut-win7-amd64  fail
 test-amd64-amd64-xl-qemuu-win7-amd64 fail
 test-amd64-i386-xl-qemuu-win7-amd64  fail
 test-amd64-amd64-xl-win7-amd64   fail
 test-amd64-i386-xl-win7-amd64fail
 test-amd64-i386-xl-credit2   pass
 test-amd64-i386-freebsd10-i386   pass
 test-amd64-i386-rumpuserxen-i386 pass
 test-amd64-amd64-xl-pcipt-intel  fail
 test-amd64-i386-rhel6hvm-intel   pass
 t

[Xen-devel] [PATCH 26/27] Kernbench perf comparison between host and guest

[Dario Faggioli]

From: Dario Faggioli 

Recipes are defined for running kernbench on baremetal,
and on PV and HVM guests. Jobs making use of those recipes
are instantiated too.

Aim is making  investigating performances loss due to
virtualization overhead easy and automatable.

Signed-off-by: Dario Faggioli 
Cc: Wei Liu 
Cc: Ian Campbell 
Cc: Ian Jackson 
---
 make-bench-flight |   14 +-
 sg-run-job|   21 +++--
 2 files changed, 24 insertions(+), 11 deletions(-)

diff --git a/make-bench-flight b/make-bench-flight
index d41cbbd..12ac9f7 100755
--- a/make-bench-flight
+++ b/make-bench-flight
@@ -109,8 +109,9 @@ do_kernbench_tests () {
   all_hostflags=$most_hostflags
 }
 
-do_hostcmp_unixbench_tests () {
-  sched=$1
+do_bench_hostcmp_tests () {
+  bench=$1
+  sched=$2
 
   # x86_64 only (for now)
   if [ $xenarch != amd64 ]; then
@@ -122,10 +123,11 @@ do_hostcmp_unixbench_tests () {
   fi
 
   job_create_test \
-  bench-hostcmp-unixbench-$xenarch-$sched bench-hostcmp-unixbench \
+  bench-hostcmp-$bench-$xenarch-$sched bench-hostcmp-$bench \
   xl $xenarch $dom0arch xen_boot_append="sched=$sched" 
max_bench_cpus=16 \
   max_bench_mem=24576 $debian_runvars bios=seabios 
unixbench_params="-i 3" \
-  debbenchhvm_image=debian-7.2.0-amd64-CD-1.iso 
all_hostflags=$most_hostflags
+  kernbench_params="-n 3" 
debbenchhvm_image=debian-7.2.0-amd64-CD-1.iso \
+  all_hostflags=$most_hostflags
 }
 
 test_matrix_do_one () {
@@ -134,7 +136,9 @@ test_matrix_do_one () {
   do_unixbench_tests $t $s 4 4096 # 4 vcpus, 4GB RAM
   do_kernbench_tests $t $s 4 4096
 done
-do_hostcmp_unixbench_tests $s
+for b in unixbench kernbench; do
+  do_bench_hostcmp_tests $b $s
+done
   done
 }
 
diff --git a/sg-run-job b/sg-run-job
index b16584a..1b3f4d6 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -388,28 +388,37 @@ proc run-job/bench-kernbench-hvm {} {
 bench-kernbench-guest debianhvm
 }
 
-proc need-hosts/bench-hostcmp-unixbench {} { return {REBOOT host} }
-proc run-job/bench-hostcmp-unixbench {} {
+proc bench-hostcmp {bench} {
 # Run benchmark in a PV guest
 set g debbenchpv
 run-ts . = ts-debian-install   + host $g
 run-ts . = ts-debian-fixup + host $g
 run-ts . = ts-bench-hostcmp-guest-prep + host $g
-bench-unixbench-guest $g
+bench-$bench-guest $g
 # Run benchmark in an HVM guest
 set g debbenchhvm
 run-ts . = ts-debian-hvm-install   + host $g
 run-ts . = ts-guest-stop   + host $g
 run-ts . = ts-bench-hostcmp-guest-prep + host $g
-bench-unixbench-guest $g
+bench-$bench-guest $g
 # Run benchmark on the host
 run-ts . = ts-bench-hostcmp-host-prep
 run-ts . = ts-host-reboot
-bench-unixbench-host
-run-ts . = ts-bench-hostcmp-post   + host unixbench
+bench-$bench-host
+run-ts . = ts-bench-hostcmp-post   + host $bench
 run-ts . = ts-host-reboot
 }
 
+proc need-hosts/bench-hostcmp-unixbench {} { return {REBOOT host} }
+proc run-job/bench-hostcmp-unixbench {} {
+bench-hostcmp unixbench
+}
+
+proc need-hosts/bench-hostcmp-kernbench {} { return {REBOOT host} }
+proc run-job/bench-hostcmp-kernbench {} {
+bench-hostcmp kernbench
+}
+
 #-- builds --
 
 proc need-hosts/build {} { return BUILD }


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH 24/27] Recipes and jobs for running unixbench both on host and guest

[Dario Faggioli]

From: Dario Faggioli 

Recipes are defined for running unixbench on baremetal,
PV and HVM guests, with similar HW resources. Jobs making
use of those recipes are instantiated too.

Aim is making  investigating performances loss due to
virtualization overhead easy and automatable.

In this case, rebooting the host is necessary (to
boot it baremetal, rather than on Dom0), and that makes
leak checks impossible. A mechanism is therefore introduced
for specifying, in sg-run-job, that for a particular job,
we don't want the leak checks.

Signed-off-by: Dario Faggioli 
Cc: Wei Liu 
Cc: Ian Campbell 
Cc: Ian Jackson 
---
 README|   11 +++
 make-bench-flight |   20 
 sg-run-job|   33 +++--
 3 files changed, 62 insertions(+), 2 deletions(-)

diff --git a/README b/README
index b3880b5..1e777df 100644
--- a/README
+++ b/README
@@ -200,6 +200,17 @@ bench-$BENCHNAME-$ARCH--
  tells about the guest's configuration (e.g., whether
 it's HVM or PV, number of vCPUs, RAM, etc.).
 
+bench-hostcmp-$BENCHNAME-$ARCH--
+
+A benchmarking job, for comparing baremetal and guest performances.
+This runs benchmark $BENCHNAME on a $ARCH baremetal host, and on
+PV and HVM guests running on a $ARCH hypervisor and dom0.
+
+In the remainder of the job's name,  tells something
+about Xen's configuration (e.g., what scheduler will be used);
+ tells about the guest's configuration (e.g., whether
+it's HVM or PV, number of vCPUs, RAM, etc.).
+
 NB: $ARCH (and $XENARCH etc) are Debian arch names, i386, amd64, armhf.
 
 Standalone Mode
diff --git a/make-bench-flight b/make-bench-flight
index 125f244..d41cbbd 100755
--- a/make-bench-flight
+++ b/make-bench-flight
@@ -109,12 +109,32 @@ do_kernbench_tests () {
   all_hostflags=$most_hostflags
 }
 
+do_hostcmp_unixbench_tests () {
+  sched=$1
+
+  # x86_64 only (for now)
+  if [ $xenarch != amd64 ]; then
+return
+  fi
+  # "homogeneous" tests only (for now)
+  if [ $xenarch != $dom0arch ]; then
+return
+  fi
+
+  job_create_test \
+  bench-hostcmp-unixbench-$xenarch-$sched bench-hostcmp-unixbench \
+  xl $xenarch $dom0arch xen_boot_append="sched=$sched" 
max_bench_cpus=16 \
+  max_bench_mem=24576 $debian_runvars bios=seabios 
unixbench_params="-i 3" \
+  debbenchhvm_image=debian-7.2.0-amd64-CD-1.iso 
all_hostflags=$most_hostflags
+}
+
 test_matrix_do_one () {
   for s in credit credit2; do
 for t in pv hvm; do
   do_unixbench_tests $t $s 4 4096 # 4 vcpus, 4GB RAM
   do_kernbench_tests $t $s 4 4096
 done
+do_hostcmp_unixbench_tests $s
   done
 }
 
diff --git a/sg-run-job b/sg-run-job
index 32c94db..5954032 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -30,10 +30,17 @@ proc run-job {job} {
 set anyfailed 0
 jobdb::prepare $job
 
+set do_leak_check 1
 set nh [need-hosts/$jobinfo(recipe)]
 if {![string compare $nh BUILD]} {
 set need_xen_hosts {}
 set need_build_host 1
+} elseif {[string match REBOOT* $nh]} {
+# no leak checks for tests that reboot the host
+set do_leak_check 0
+set nh [lreplace $nh 0 0]
+set need_xen_hosts $nh
+set need_build_host 0
 } else {
 set need_xen_hosts $nh
 set need_build_host 0
@@ -54,10 +61,10 @@ proc run-job {job} {
 per-host-ts .   xen-install/@ ts-xen-install
 per-host-ts .   xen-boot/@ts-host-reboot
 
-per-host-ts .   =(*) {ts-leak-check basis}
+if {$do_leak_check} {per-host-ts . =(*) {ts-leak-check basis} }
 
 if {$ok} { catching-otherwise fail  run-job/$jobinfo(recipe)  }
-per-host-ts .   ={ts-leak-check check}
+if {$do_leak_check} {per-host-ts . = {ts-leak-check check}}
 
 if {!$need_build_host} {
 per-host-ts !broken capture-logs/@(*) ts-logs-capture
@@ -381,6 +388,28 @@ proc run-job/bench-kernbench-hvm {} {
 bench-kernbench-guest debianhvm
 }
 
+proc need-hosts/bench-hostcmp-unixbench {} { return {REBOOT host} }
+proc run-job/bench-hostcmp-unixbench {} {
+# Run benchmark in a PV guest
+set g debbenchpv
+run-ts . = ts-debian-install   + host $g
+run-ts . = ts-debian-fixup + host $g
+run-ts . = ts-bench-hostcmp-guest-prep + host $g
+bench-unixbench-guest $g
+# Run benchmark in an HVM guest
+set g debbenchhvm
+run-ts . = ts-debian-hvm-install   + host $g
+run-ts . = ts-guest-stop   + host $g
+run-ts . = ts-bench-hostcmp-guest-prep + host $g
+bench-unixbench-guest $g
+# Run benchmark on the host
+run-ts . = ts-bench-hostcmp-host-prep
+run-ts . = ts-host-reboot
+bench-unixbench-host
+run-ts . = ts-bench-hostcmp-post
+run-ts . = ts-host-reboot
+}
+
 #-- builds --
 
 proc need-hosts/build {} { return BUILD }


___

[Xen-devel] [PATCH 25/27] ts-bench-hostcmp-post: add plotting facilities

[Dario Faggioli]

From: Dario Faggioli 

in order to have an additional graph, comparing host and
guests performance when running unixbench.

Signed-off-by: Dario Faggioli 
Cc: Wei Liu 
Cc: Ian Campbell 
Cc: Ian Jackson 
---
 Osstest/Benchmarking.pm |   22 --
 sg-run-job  |2 +-
 ts-bench-hostcmp-post   |   43 ++-
 ts-unixbench-reslts |6 +++---
 4 files changed, 62 insertions(+), 11 deletions(-)

diff --git a/Osstest/Benchmarking.pm b/Osstest/Benchmarking.pm
index ff45766..301af08 100644
--- a/Osstest/Benchmarking.pm
+++ b/Osstest/Benchmarking.pm
@@ -96,21 +96,31 @@ set boxwidth 1 absolute
 EOF
 
 sub unixbench_plot_results ($$$) {
-  my ($dataf,$num_cols,$pfile)= @_;
-  my $h= new IO::File "> $pfile.gp" or die "$!";
+  my ($dfiles,$num_cols,$pfile)= @_;
+  my $f= keys @$dfiles;
+  my $s= join(' ',@$dfiles);
 
-  printf $h < $pfile.gp" or die "$!";
+  print $h < 
+our $whhost= $ARGV[0];
+our $bn= $ARGV[1];
+
+#our ($whhost,$gn,$bn) = @ARGV;
 $whhost ||= 'host';
 our $ho= selecthost($whhost);
 
+sub plot_hostcmp () {
+  opendir my $dir, "$stash" or die "$!";
+
+  my @dfiles = grep(/.*-DATA$/,readdir($dir));
+  closedir($dir);
+
+  # Mangle the data file a bit, so each data set is marked
+  # with the name of the box (either host or a VM) where
+  # the bench did run.
+  my ($fhi,$fho,$ncols);
+  foreach my $i (0 .. $#dfiles) {
+$_= $dfiles[$i];
+my $host= m/(\w*)(\.|--).*/;$host= $1;
+$dfiles[$i]= "$stash/$dfiles[$i]";
+
+open FH, "<", "$dfiles[$i]" or die "!";
+my @lines= ;
+close FH;
+
+open FH, ">", "$dfiles[$i]" or die "!";
+foreach my $line (@lines) {
+  if ($line eq $lines[0]) {
+$line =~ s/"([^"]*\w*[^"]*)"/"$host $1"/g if $line !~ "$host";
+  }
+  print FH $line;
+  # Figure out the number of data columns, for plotting
+  $ncols= () = $line =~ /(\d+\.\d+)/g;
+}
+close FH;
+  }
+  unixbench_plot_results(\@dfiles,$ncols,"$stash/$job-PLOT") if $bn eq 
"unixbench";
+}
+
 sub resetboot () {
   my ($xenhopt,@hooks)= host_bootxen_setup($ho);
   my $want_kernver = get_runvar('kernel_ver',$r{'kernbuildjob'});
@@ -36,4 +76,5 @@ sub resetboot () {
   logm("host reset to booting Xen");
 }
 
+plot_hostcmp();
 resetboot();
diff --git a/ts-unixbench-reslts b/ts-unixbench-reslts
index b480d15..6c3e0a3 100755
--- a/ts-unixbench-reslts
+++ b/ts-unixbench-reslts
@@ -69,15 +69,15 @@ END
 
 sub process () {
   my $resf= "$stash/$gho->{Name}--$lresfile";
-  my $dataf= "$resf-DATA";
+  my @dataf= "$resf-DATA";
   my $plotf= "$resf-PLOT";
 
   unixbench_process_results(\$results,$resf);
-  unixbench_print_results($results,$dataf);
+  unixbench_print_results($results,$dataf[0]);
 
   # For plotting we need to know the number of data columns
   my $ncols= keys $results->{'Double-Precision Whetstone'}{Index};
-  unixbench_plot_results($dataf,$ncols,$plotf);
+  unixbench_plot_results(\@dataf,$ncols,$plotf);
 }
 
 fetch();


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH 27/27] ts-bench-hostcmp-post: add plotting facilities

[Dario Faggioli]

From: Dario Faggioli 

in order to have an additional graph, comparing host and
guests performance when running kernbench.

Signed-off-by: Dario Faggioli 
Cc: Wei Liu 
Cc: Ian Campbell 
Cc: Ian Jackson 
---
 Osstest/Benchmarking.pm |   17 +
 ts-bench-hostcmp-post   |1 +
 ts-kernbench-reslts |6 +++---
 3 files changed, 17 insertions(+), 7 deletions(-)

diff --git a/Osstest/Benchmarking.pm b/Osstest/Benchmarking.pm
index 301af08..1be9c97 100644
--- a/Osstest/Benchmarking.pm
+++ b/Osstest/Benchmarking.pm
@@ -175,7 +175,9 @@ sub kernbench_print_results ($$) {
 }
 
 sub kernbench_plot_results ($$$) {
-  my ($dataf,$num_cols,$pfile)= @_;
+  my ($dfiles,$num_cols,$pfile)= @_;
+  my $f= keys @$dfiles;
+  my $s= join(' ',@$dfiles);
 
   my $h= new IO::File "> $pfile.gp" or die "$!";
   print $h <{'Elapsed Time'}{'Result'};
-  kernbench_plot_results($dataf,$ncols,$plotf);
+  kernbench_plot_results(\@dataf,$ncols,$plotf);
 }
 
 fetch();


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH 22/27] ts-bench-hostcmp-host-prep: new script

[Dario Faggioli]

From: Dario Faggioli 

the goal is to run a benchmark both in a guest and
on baremetal, to investigate the performances loss
due to the virtualization overhead.

In order to help accomplishing this, the new script
introduced by this commit modifies the host's boot
configuration as follows:
 - it makes it boot baremetal Linux, rather than Xen
   and a Dom0 kernel;
 - it limits the host's pcpus and amount of memory
   to the values contained in the specific runvars
   (if defined).

This is done under the assumption that the benchmark
will (or has been already) run on one (or more)
guest(s) too. If the runvars for limiting host's
resources are defined, it is assumed that they will
be (were) defined, and that they will have (had) the
same values, also when prepping the run of the
benchmark in the guest.

The test script only alter the host's boot config;
it is left to the caller to actually reboot the host,
and also to restore the old config, if wanted, after
the benchmark has been run.

Signed-off-by: Dario Faggioli 
Cc: Wei Liu 
Cc: Ian Campbell 
Cc: Ian Jackson 
---
 Osstest/Debian.pm  |   17 --
 ts-bench-hostcmp-host-prep |   74 
 2 files changed, 88 insertions(+), 3 deletions(-)
 create mode 100755 ts-bench-hostcmp-host-prep

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 70afaec..418d9f2 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -32,6 +32,9 @@ BEGIN {
 $VERSION = 1.00;
 @ISA = qw(Exporter);
 @EXPORT  = qw(debian_boot_setup
+  setupboot_uboot
+  setupboot_grub1
+  setupboot_grub2
   %preseed_cmds
   preseed_base
   preseed_create
@@ -112,7 +115,7 @@ sub bl_getmenu_open ($$$) {
 return $f;
 }
 
-sub setupboot_uboot ($$$) {
+sub setupboot_uboot () {
 my ($ho,$want_kernver,$xenhopt,$xenkopt) = @_;
 my $bl= { };
 
@@ -194,7 +197,7 @@ END
 return $bl;
 }
 
-sub setupboot_grub1 ($$$) {
+sub setupboot_grub1 () {
 my ($ho,$want_kernver,$xenhopt,$xenkopt) = @_;
 my $bl= { };
 
@@ -274,7 +277,7 @@ sub setupboot_grub1 ($$$) {
 return $bl;
 }
 
-sub setupboot_grub2 ($$$) {
+sub setupboot_grub2 () {
 my ($ho,$want_kernver,$xenhopt,$xenkopt) = @_;
 my $bl= { };
 
@@ -387,6 +390,14 @@ END
 my $v= $k{$k};
 $v =~ s/\bquiet\b//;
 $v =~ s/\b(?:console|xencons)=[0-9A-Za-z,]+//;
+# Get rid of any host/dom0 resource constraining. Idea is:
+# (1) in the dom0 case, something like that should be achieved
+# via Xen boot parameters; (2) in any case, if it is important
+# to have something like these in Linux's boot cmd line by
+# default, that should be put into the 'linux_boot_append'
+# runvar, which won't be affected by this.
+$v =~ s/\bmaxcpus=[0-9A-Za-z,]+//;
+$v =~ s/\bmem=[0-9A-Za-z,]+//;
 $v .= " $xenkopt" if $k eq 'GRUB_CMDLINE_LINUX';
 print ::EO "$k=\"$v\"\n" or die $!;
 }
diff --git a/ts-bench-hostcmp-host-prep b/ts-bench-hostcmp-host-prep
new file mode 100755
index 000..493d948
--- /dev/null
+++ b/ts-bench-hostcmp-host-prep
@@ -0,0 +1,74 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2014 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see .
+
+use strict qw(vars);
+use DBI;
+use Osstest;
+use Osstest::Debian;
+use Osstest::TestSupport;
+
+tsreadconfig();
+
+our ($whhost) = @ARGV;
+$whhost ||= 'host';
+our $ho= selecthost($whhost);
+
+# We want to boot baremetal Linux on the host,
+# to compare against it.
+sub fixup () {
+  my ($bl,$bootkern);
+  my ($hcpus,$hmem,$hnodes);
+  my ($bcpus,$bmem);
+
+  target_install_packages_norec($ho, "numactl");
+
+  $hcpus= get_host_cpus($ho);
+  $hmem= get_host_memory($ho);
+  die unless (defined $hcpus and defined $hmem);
+
+  $hnodes= get_host_numanodes($ho);
+  if (defined $hnodes and $hnodes > 1) {
+logm("WARNING: the host has $hnodes NUMA nodes. This may spoil results");
+  }
+
+  $bcpus= (!defined($r{'max_bench_cpus'})) ? $hcpus :
+  ($r{'max_bench_cpus'} > $hcpus) ? $hcpus : $r{'max_benc

[Xen-devel] [PATCH 23/27] ts-bench-hostcmp-host-reset: new script

[Dario Faggioli]

From: Dario Faggioli 

Signed-off-by: Dario Faggioli 
Cc: Wei Liu 
Cc: Ian Campbell 
Cc: Ian Jackson 
---
 Osstest/Debian.pm  |   32 +---
 Osstest/TestSupport.pm |   46 +-
 ts-bench-hostcmp-post  |   39 +++
 ts-xen-install |   38 ++
 4 files changed, 103 insertions(+), 52 deletions(-)
 create mode 100755 ts-bench-hostcmp-post

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 418d9f2..447d2d2 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -87,25 +87,27 @@ sub debian_boot_setup (;$) {
 my $kern= $bootloader->{GetBootKern}();
 logm("dom0 kernel is $kern");
 
-system "tar zvtf $distpath->{kern} boot/$kern";
-$? and die "$distpath->{kern} boot/$kern $?";
-
-my $kernver= $kern;
-$kernver =~ s,^/?(?:boot/)?(?:vmlinu[xz]-)?,, or die "$kernver ?";
-my $kernpath= $kern;
-$kernpath =~ s,^(?:boot/)?,/boot/,;
-
-target_cmd_root($ho,
-"update-initramfs -k $kernver -c ||".
-" update-initramfs -k $kernver -u",
-200);
+if (defined $distpath) {
+system "tar zvtf $distpath->{kern} boot/$kern";
+$? and die "$distpath->{kern} boot/$kern $?";
+
+my $kernver= $kern;
+$kernver =~ s,^/?(?:boot/)?(?:vmlinu[xz]-)?,, or die "$kernver ?";
+my $kernpath= $kern;
+$kernpath =~ s,^(?:boot/)?,/boot/,;
+
+target_cmd_root($ho,
+"update-initramfs -k $kernver -c ||".
+" update-initramfs -k $kernver -u",
+200);
+
+store_runvar(target_var_prefix($ho).'xen_kernel_path',$kernpath);
+store_runvar(target_var_prefix($ho).'xen_kernel_ver',$kernver);
+}
 
 $bootloader->{PreFinalUpdate}();
 
 $bootloader->{UpdateConfig}($ho);
-
-store_runvar(target_var_prefix($ho).'xen_kernel_path',$kernpath);
-store_runvar(target_var_prefix($ho).'xen_kernel_ver',$kernver);
 }
 
 sub bl_getmenu_open ($$$) {
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 251668a..c967e4f 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -99,7 +99,7 @@ BEGIN {
   guest_vncsnapshot_begin guest_vncsnapshot_stash
  guest_check_remus_ok guest_editconfig
   host_involves_pcipassthrough host_get_pcipassthrough_devs
-  toolstack guest_create
+  toolstack guest_create host_bootxen_setup
 
   await_webspace_fetch_byleaf create_webfile
   file_link_contents get_timeout
@@ -1012,6 +1012,50 @@ sub get_host_memory ($) {
 return $mem;
 }
 
+#-- bootloader and booting --
+#
+sub host_bootxen_setup ($) {
+my ($ho)= @_;
+
+my $xenhopt= "conswitch=x watchdog";
+
+my $cons= get_host_property($ho, 'XenSerialConsole', 'com1');
+
+if ( $cons eq "com1" ) {
+$xenhopt .= " com1=$c{Baud},8n1 console=com1,vga gdb=com1";
+} elsif ( $cons eq "dtuart" ) {
+$xenhopt .= " console=dtuart";
+my $dtuart= get_host_property($ho, 'XenDTUARTPath', undef);
+$xenhopt .= " dtuart=$dtuart" if $dtuart;
+} else {
+logm("No Xen console device defined for host");
+}
+if (toolstack()->{Dom0MemFixed}) {
+$xenhopt .= " dom0_mem=512M,max:512M";
+}
+my $append= $r{xen_boot_append};
+$xenhopt .= " $append" if defined $append;
+$append = get_host_property($ho, 'xen-commandline-append', undef);
+$xenhopt .= " $append" if defined $append;
+
+my @hooks;
+
+if (host_involves_pcipassthrough($ho)) {
+push @hooks, {
+EditBootOptions => sub {
+my ($ho,$hopt,$kopt) = @_;
+$$hopt .= ' iommu=on';
+my $hide= ' xen-pciback.hide='. join '',map { "($_->{Bdf})" }
+host_get_pcipassthrough_devs($ho);
+logm("pci passthrough: hiding in dom0: $hide");
+$$kopt .= $hide;
+}
+};
+}
+
+return ($xenhopt,@hooks);
+}
+
 #-- stashed files --
 
 sub open_unique_stashfile ($) {
diff --git a/ts-bench-hostcmp-post b/ts-bench-hostcmp-post
new file mode 100755
index 000..26a67f4
--- /dev/null
+++ b/ts-bench-hostcmp-post
@@ -0,0 +1,39 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2014 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or F

[Xen-devel] [PATCH 21/27] ts-bench-hostcmp-guest-prep: new script

[Dario Faggioli]

From: Dario Faggioli 

the goal is to run a benchmark both in a guest and
on baremetal, to investigate the performances loss
due to the virtualization overhead.

In order to help accomplishing this, the new script
introduced by this commit modifies a guest's config
file in order for it to have the same number of
vcpus and (almost) the same amount of memory of the
underlying host. This is done under the assumption
that the benchmark will (or has been already) run
on the host too.

It is possible to make the guest have less vcpus
than the host has pcpus, and less memory than the
host, by defining two specific runvars. In fact, in
case of really "beefy" hosts, one may not want to
run the benchmark in an unrealistically large guest.
Of course, specific measures to also limit the host
resources, when running the benchmark on it, should
be taken, if comparing apples to apples is important.

Signed-off-by: Dario Faggioli 
Cc: Wei Liu 
Cc: Ian Campbell 
Cc: Ian Jackson 
---
 ts-bench-hostcmp-guest-prep |   87 +++
 1 file changed, 87 insertions(+)
 create mode 100755 ts-bench-hostcmp-guest-prep

diff --git a/ts-bench-hostcmp-guest-prep b/ts-bench-hostcmp-guest-prep
new file mode 100755
index 000..ffde981
--- /dev/null
+++ b/ts-bench-hostcmp-guest-prep
@@ -0,0 +1,87 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2014 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see .
+
+use strict qw(vars);
+use DBI;
+use Osstest;
+use Osstest::TestSupport;
+
+tsreadconfig();
+
+our ($whhost,$gn) = @ARGV;
+$whhost ||= 'host';
+$gn ||= 'debianhvm';
+
+our ($ho,$gho) = ts_get_host_guest($whhost,$gn);
+our $toolstack= toolstack()->{Command};
+
+# We want the guest to have:
+#  - the same number of vcpus than the host has pcpus
+#  - (almost) the same amount of memory the host has
+#
+# For both vcpus and memory, a maximum can be specified via
+# runvars.
+sub fixup () {
+  our ($hnodes,$hcpus,$hmem);
+  our ($cfgfile,$bcpus,$bmem,$hfreemem);
+
+  $hcpus= get_host_cpus($ho);
+  $hmem= get_host_memory($ho);
+  die unless (defined $hcpus and defined $hmem);
+
+  $hnodes= get_host_numanodes($ho);
+  if (defined $hnodes and $hnodes > 1) {
+logm("WARNING: the host has $hnodes NUMA nodes. This may spoil results");
+  }
+
+  $bcpus= (!defined($r{'max_bench_cpus'})) ? $hcpus :
+  ($r{'max_bench_cpus'} > $hcpus) ? $hcpus : $r{'max_bench_cpus'};
+  $bmem= (!defined($r{'max_bench_mem'})) ? $hmem :
+  ($r{'max_bench_mem'} > $hmem) ? $hmem : $r{'max_bench_mem'};
+
+  $hfreemem = host_get_free_memory($ho,$toolstack);
+  if ($hfreemem < $bmem) {
+$bmem= $hfreemem - 1024;
+logm("WARNING: Not enough free memory. Using $bmem MB");
+  }
+
+  logm("Will run the benchmark with: $bcpus vcpus and $bmem MB RAM");
+
+  $cfgfile= $r{"$gho->{Guest}_cfgpath"};
+  target_editfile_root($ho, $cfgfile, sub {
+  while () {
+s/^vcpus.*/vcpus=$bcpus/;
+s/^memory.*/memory=$bmem/;
+   print EO or die $!;
+  }
+  });
+}
+
+sub start () {
+  # Start the guest...
+  guest_umount_lv($ho, $gho);
+  my $cmd= toolstack()->{Command}." create ".
+  $r{ $gho->{Guest}.'_'. toolstack()->{CfgPathVar} };
+  target_cmd_root($ho, $cmd, 30);
+  # ...And verify it's up and running
+  guest_checkrunning($ho, $gho) or die "$gho->{Name} not running";
+  guest_await($gho, target_var($gho,'boot_timeout'));
+  guest_check_up($gho);
+}
+
+fixup();
+start();


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH 20/27] Osstest/TestSupport.pm: read hosts' hardware characteristics

[Dario Faggioli]

From: Dario Faggioli 

if defined, in the form of host properties. In standalone
mode, that should happen via the config file.

Methods are introduced to read those host properties or,
if they are not defined, to fetch the information by
querying the host directly.

The host properties always take precedence. This means
that, if they're defined, no command is run on the host,
and the values stored in the properties are used.

This commit also introduces a simple bash script that,
if run on the host, retrieves and prints such host
hardware properties, for convenience and/or testing.

Signed-off-by: Dario Faggioli 
Cc: Wei Liu 
Cc: Ian Campbell 
Cc: Ian Jackson 
---
 Osstest/TestSupport.pm |   77 +++-
 README |   15 +
 mg-host-hw-specs   |   35 ++
 3 files changed, 126 insertions(+), 1 deletion(-)
 create mode 100755 mg-host-hw-specs

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 7cc5be6..251668a 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -58,7 +58,7 @@ BEGIN {
   target_put_guest_image target_editfile
   target_editfile_root target_file_exists
   target_catfile_stash target_catfile_root_stash
-  target_run_apt
+  target_file_contains target_run_apt
   target_install_packages target_install_packages_norec
   target_jobdir target_extract_jobdistpath_subdir
   target_extract_jobdistpath target_guest_lv_name
@@ -67,6 +67,7 @@ BEGIN {
   contents_make_cpio file_simple_write_contents
 
   selecthost get_hostflags get_host_property
+  get_host_cpus get_host_numanodes get_host_memory
   get_host_native_linux_console
   power_state power_cycle power_cycle_time
   serial_fetch_logs
@@ -519,6 +520,15 @@ sub target_file_exists ($$) {
 die "$rfile $out ?";
 }
 
+sub target_file_contains ($$$) {
+  my ($ho,$rfile,$filecont) = @_;
+  return 0 unless target_file_exists($ho,$rfile);
+  my $out= target_cmd_output($ho, "grep $filecont $rfile");
+  return 1 if ($out ne "");
+  return 0 if ($out eq "");
+  die "$rfile $filecont $out ?";
+}
+
 sub teditfileex {
 my $user= shift @_;
 my $code= pop @_;
@@ -866,6 +876,11 @@ sub selecthost ($) {
 }
 $ho->{Ip}= $ho->{IpStatic};
 
+#- HW specs -
+$ho->{Cpus} = get_host_property($ho,'cpus');
+$ho->{Memory} = get_host_property($ho,'memory');
+$ho->{Nodes} = get_host_property($ho,'nodes');
+
 #- tftp -
 
 my $tftpscope = get_host_property($ho, 'TftpScope', $c{TftpDefaultScope});
@@ -937,6 +952,66 @@ sub get_host_method_object ($$$) {
 return $mo;
 }
 
+sub get_host_cpus ($) {
+my ($ho) = @_;
+
+# Let's first try if there's an host property defined;
+# if no, we'll "ask" the host directly.
+my $cpus= get_host_property($ho,'cpus',undef);
+return $cpus if defined $cpus;
+
+# Is the host running Dom0 or baremetal?
+if (target_file_contains($ho,"/proc/xen/capabilities","control_d")) {
+$cpus= target_cmd_output_root($ho,
+"xl info | grep ^nr_cpus | awk '{print \$3}'");
+} else {
+$cpus= target_cmd_output_root($ho,
+"cat /proc/cpuinfo | grep '^processor' | wc -l");
+}
+
+return $cpus;
+}
+
+sub get_host_numanodes ($) {
+my ($ho) = @_;
+
+# Let's first try if there's an host property defined;
+# if no, we'll "ask" the host directly.
+my $nodes= get_host_property($ho,'nodes',undef);
+return $nodes if defined $nodes;
+
+# Is the host running Dom0 or baremetal?
+if (target_file_contains($ho,"/proc/xen/capabilities","control_d")) {
+$nodes= target_cmd_output_root($ho,
+"xl info | grep ^nr_nodes | awk '{print \$3}'");
+} else {
+$nodes= target_cmd_output_root($ho,
+"which numactl && numactl --hardware | grep ^available: | awk 
'{print \$2}'");
+}
+
+return $nodes;
+}
+
+sub get_host_memory ($) {
+my ($ho) = @_;
+
+# Let's first try if there's an host property defined;
+# if no, we'll "ask" the host directly.
+my $mem= get_host_property($ho,'memory',undef);
+return $mem if defined $mem;
+
+# Is the host running Dom0 or baremetal?
+if (target_file_contains($ho,"/proc/xen/capabilities","control_d")) {
+$mem= target_cmd_output_root($ho,
+"xl info | grep ^total_memory | awk '{print \$3}'");
+} else {
+$mem= target_cmd_output_root($ho,
+"free -m | grep ^Mem: | awk '{print \$2}'");
+}
+
+return $mem;
+}
+
 #-- stashed files --
 
 sub open_unique_stashfile ($) {
diff --git a/README b/README
index 45d1498..b3880b5 100644
--- a/README
+++ b/README
@@ -343,6 +343,21 @@ HostProp__TftpScop

[Xen-devel] [PATCH 18/27] sg-run-job: recipes for the kernbench jobs

[Dario Faggioli]

Recipes are defined for prepping and running kernbench
on the host and on Debian PV and HVM guests.

Signed-off-by: Dario Faggioli 
Cc: Wei Liu 
Cc: Ian Campbell 
Cc: Ian Jackson 
---
 sg-run-job |   29 +
 1 file changed, 29 insertions(+)

diff --git a/sg-run-job b/sg-run-job
index 81c2040..32c94db 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -352,6 +352,35 @@ proc run-job/bench-unixbench-hvm {} {
 bench-unixbench-guest debianhvm
 }
 
+proc bench-kernbench-run {args} {
+run-ts . = ts-kernbench-build  + host $args
+run-ts . = ts-kernbench-run+ host $args
+run-ts . = ts-kernbench-reslts + host $args
+}
+
+proc bench-kernbench-host {} {
+bench-kernbench-run
+}
+
+proc bench-kernbench-guest {g} {
+bench-kernbench-run $g
+run-ts . = ts-guest-stop + host $g
+}
+
+proc need-hosts/bench-kernbench-pv {} { return host }
+proc run-job/bench-kernbench-pv {} {
+run-ts . = ts-debian-install + host
+run-ts . = ts-debian-fixup   + host debian
+run-ts . = ts-guest-start+ host debian
+bench-kernbench-guest debian
+}
+
+proc need-hosts/bench-kernbench-hvm {} { return host }
+proc run-job/bench-kernbench-hvm {} {
+run-ts . = ts-debian-hvm-install
+bench-kernbench-guest debianhvm
+}
+
 #-- builds --
 
 proc need-hosts/build {} { return BUILD }


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH 17/27] ts-kernbench-reslts: process and plot bench results

[Dario Faggioli]

From: Dario Faggioli 

Extract the data from the output of kernbench and produce
the tables, the gnuplot script and the plots.

All is saved in $stash, for the running flight and job.

Signed-off-by: Dario Faggioli 
Cc: Wei Liu 
Cc: Ian Campbell 
Cc: Ian Jackson 
---
 Osstest/Benchmarking.pm |   88 ---
 ts-kernbench-reslts |   19 +-
 2 files changed, 99 insertions(+), 8 deletions(-)

diff --git a/Osstest/Benchmarking.pm b/Osstest/Benchmarking.pm
index 0c5c538..ff45766 100644
--- a/Osstest/Benchmarking.pm
+++ b/Osstest/Benchmarking.pm
@@ -34,6 +34,9 @@ BEGIN {
 @EXPORT  = qw(unixbench_process_results
   unixbench_print_results
   unixbench_plot_results
+  kernbench_process_results
+  kernbench_print_results
+  kernbench_plot_results
   );
 %EXPORT_TAGS = ( );
 
@@ -83,6 +86,15 @@ sub unixbench_print_results ($$) {
   close($h);
 }
 
+our $common_plot_opts= < $pfile.gp" or die "$!";
@@ -91,12 +103,7 @@ sub unixbench_plot_results ($$$) {
 set terminal png enhanced font 
"/usr/share/fonts/truetype/ttf-dejavu/DejaVuSans.ttf" 8 size 800,600
 set output '$pfile.png'
 set title 'Unixbench INDEXes for $flight.$job'
-set key outside center top horizontal noreverse noenhanced autotitles nobox
-set xtics mirror rotate by -45 out
-set style data histogram
-set style histogram cluster gap 1
-set style fill solid border lt -1
-set boxwidth 1 absolute
+$common_plot_opts
 set bmargin 13
 set rmargin 14
 SKIP_COL=1
@@ -112,4 +119,73 @@ EOF
   logm("WARNING: plotting file with \"$err\"") unless $ok;
 }
 
+sub kernbench_process_results ($$) {
+  my ($results_ref,$rfilen)= @_;
+  my $h= new IO::File "< $rfilen" or die "$!";
+
+  my $run;
+  while (<$h>) {
+my ($bench,$val,$stdd);
+if (m/.*load -j ([0-9]*)\s?Run.*$/) {
+  $run= $1 || 0;
+}
+if (m/^(\S[a-zA-z\s]*)\s([0-9]+.?[0-9]*)\s\(([0-9]+.?[0-9]*)\)$/) {
+  $bench=$1;
+  next if $bench =~ /Sleeps|Context|Percent/;
+  $val= $2;
+  $stdd= $3;
+  $$results_ref->{"$bench"}{Result}{"$run"}= $val;
+  $$results_ref->{"$bench"}{StdDev}{"$run"}= $stdd;
+}
+next;
+  }
+  close($h);
+}
+
+sub kernbench_print_results ($$) {
+  my ($results,$rfilen)= @_;
+  open my $h, "|-", "tee $rfilen" or die "$!";
+
+  printf $h "%-25s","\"What\"";
+  foreach my $i (sort keys $results->{'Elapsed Time'}{'Result'}) {
+my $col= "kernbench -j";
+$col .= ($i == 0) ? "" : " $i";
+printf $h "%-20s","\"$col\""
+# TODO: Include stddev too in the report
+  }
+  print $h "\n";
+  foreach my $b (keys $results) {
+printf $h "%-25s","\"$b\"";
+foreach my $i (sort keys $results->{"$b"}{'Result'}) {
+  printf $h "%-20s",$results->{$b}{'Result'}{$i};
+}
+print $h "\n";
+  }
+  close($h);
+}
+
+sub kernbench_plot_results ($$$) {
+  my ($dataf,$num_cols,$pfile)= @_;
+
+  my $h= new IO::File "> $pfile.gp" or die "$!";
+  print $h < "$gp $pfile.gp", verbose => 1 );
+  logm("WARNING: plotting file with \"$err\"") unless $ok;
+}
+
 1;
diff --git a/ts-kernbench-reslts b/ts-kernbench-reslts
index dcb279d..113a4ce 100755
--- a/ts-kernbench-reslts
+++ b/ts-kernbench-reslts
@@ -21,6 +21,7 @@ use DBI;
 use IO::File;
 use POSIX;
 use Osstest::TestSupport;
+use Osstest::Benchmarking;
 
 tsreadconfig();
 
@@ -43,9 +44,11 @@ $lresfile .= (defined($r{'kernbench_run_suffix'})) ?
   $r{'kernbench_run_suffix'} : '';
 $lresfile = "kernbench$lresfile";
 
+our $results;
+
 # Kernbench stores results in the linux kernel tree it
 # built, in a file called kernbench.log
-sub results() {
+sub fetch() {
   my $rresults_dir= "/root/linux-kernbench";
   my $rresfile= "$rresults_dir/kernbench.log";
 
@@ -57,4 +60,16 @@ sub results() {
   "$lresfile");
 }
 
-results();
+sub process () {
+  my $resf= "$stash/$gho->{Name}--$lresfile";
+  my $dataf= "$resf-DATA";
+  my $plotf= "$resf-PLOT";
+
+  kernbench_process_results(\$results,$resf);
+  kernbench_print_results($results,$dataf);
+  my $ncols= keys $results->{'Elapsed Time'}{'Result'};
+  kernbench_plot_results($dataf,$ncols,$plotf);
+}
+
+fetch();
+process();


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH 11/27] make-bench-flight: to create a benchmarking flight

[Dario Faggioli]

This is all done in a new script, to keep these jobs
separated from regular testing jobs defined by make-flight.

Signed-off-by: Dario Faggioli 
Cc: Wei Liu 
Cc: Ian Campbell 
Cc: Ian Jackson 
---
 README|   10 +
 make-bench-flight |  100 +
 2 files changed, 110 insertions(+)
 create mode 100755 make-bench-flight

diff --git a/README b/README
index 3fe5ecc..45d1498 100644
--- a/README
+++ b/README
@@ -190,6 +190,16 @@ test-$XENARCH-$DOM0ARCH-
 Some tests also have a -$DOMUARCH suffix indicating the
 obvious thing.
 
+bench-$BENCHNAME-$ARCH--
+
+A benchmarking job, running benchmark $BENCHNAME on a $ARCH
+hypervisor and dom0 (and guest).
+
+In the remainder of the job's name,  tells something
+about Xen's configuration (e.g., what scheduler will be used);
+ tells about the guest's configuration (e.g., whether
+it's HVM or PV, number of vCPUs, RAM, etc.).
+
 NB: $ARCH (and $XENARCH etc) are Debian arch names, i386, amd64, armhf.
 
 Standalone Mode
diff --git a/make-bench-flight b/make-bench-flight
new file mode 100755
index 000..cdb22ff
--- /dev/null
+++ b/make-bench-flight
@@ -0,0 +1,100 @@
+#!/bin/bash
+
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2014 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see .
+
+
+set -e
+
+branch=$1
+xenbranch=$2
+blessing=$3
+buildflight=$4
+
+flight=`./cs-flight-create $blessing $branch`
+
+. ap-common
+. cri-common
+. mfi-common
+
+defsuite=`getconfig DebianSuite`
+defguestsuite=`getconfig GuestDebianSuite`
+
+if [ x$buildflight = x ]; then
+
+  if [ "x$BUILD_LVEXTEND_MAX" != x ]; then
+ BUILD_RUNVARS+=" build_lvextend_max=$BUILD_LVEXTEND_MAX "
+  fi
+
+  create_build_jobs
+
+else
+
+  bfi=$buildflight.
+
+fi
+
+job_create_test_filter_callback () {
+:
+}
+
+test_matrix_branch_filter_callback () {
+:
+}
+
+do_unixbench_tests () {
+  gvcpus=$1
+  gmem=$2
+
+  # x86_64 only (for now)
+  if [ $xenarch != amd64 ]; then
+return
+  fi
+  # "homogeneous" tests only (for now)
+  if [ $xenarch != $dom0arch ]; then
+return
+  fi
+
+  gvcpus_runvars=guests_vcpus=$gvcpus; gvcpus_suffix=-${gvcpus}vcpus
+  gmem_runvars=guests_memory=$gmem; gmem_suffix=-${gmem}ram
+  if [ $gvcpus -ge 2 ];then params="-c $(($gvcpus/2))"; fi
+  params="$params -c $gvcpus -c $(($gvcpus*2)) -i 6"
+
+  for gt in pv hvm; do
+for sched in credit credit2; do
+  job_create_test \
+  bench-unixbench-$xenarch-$sched-$gt$gvcpus_suffix$gmem_suffix \
+  bench-unixbench-$gt xl $xenarch $dom0arch $gvcpus_runvars 
$gmem_runvars \
+  xen_boot_append="sched=$sched" unixbench_params="$params" 
$debian_runvars \
+  bios=seabios debianhvm_image=debian-7.2.0-amd64-CD-1.iso \
+  all_hostflags=$most_hostflags
+done
+  done
+}
+
+test_matrix_do_one () {
+  do_unixbench_tests 4 4096 # 4 vcpus, 4GB RAM
+}
+
+test_matrix_iterate
+
+echo $flight
+
+# Local variables:
+# mode: sh
+# sh-basic-offset: 2
+# indent-tabs-mode: nil
+# End:


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH 12/27] standalone-reset: introduce a new -t option

[Dario Faggioli]

for making it possible to call the new make-bench-flight
script, and generating the benchmarking jobs. It can be
combined with the existing '-f' option, to create a
benchmarking flight containing all the benchmarking jobs.

This is generic, so, when passing '-t sometype', a script
called make-sometype-flight is what will be invoked.

Signed-off-by: Dario Faggioli 
Cc: Wei Liu 
Cc: Ian Campbell 
Cc: Ian Jackson 
---
Changes from RFCv1:
 * this into "standalone make-flight" too, as requested
   during review.
---
 cr-daily-branch  |3 ++-
 standalone   |7 +--
 standalone-reset |9 ++---
 3 files changed, 13 insertions(+), 6 deletions(-)

diff --git a/cr-daily-branch b/cr-daily-branch
index 17bb2c9..521682a 100755
--- a/cr-daily-branch
+++ b/cr-daily-branch
@@ -22,6 +22,7 @@ set -ex
 . cri-args-hostlists
 . ap-common
 branch=$1; shift
+ftype=$1; shift
 select_branch
 info_linux_tree $branch ||:
 
@@ -231,7 +232,7 @@ if [ "x$NEW_REVISION" = "x$OLD_REVISION" ]; then
 fi
 
 $DAILY_BRANCH_PREMAKE_HOOK
-flight=`./make-flight $branch $xenbranch $OSSTEST_BLESSING "$@"`
+flight=`./make-$ftype-flight $branch $xenbranch $OSSTEST_BLESSING "$@"`
 $DAILY_BRANCH_POSTMAKE_HOOK
 
 heading=tmp/$flight.heading-info
diff --git a/standalone b/standalone
index caf3fd5..22540c1 100755
--- a/standalone
+++ b/standalone
@@ -39,6 +39,7 @@ Options:
 
 -c FILE, --config=FILEUse FILE as configuration file
 -f FLIGHT, --flight=FLIGHTOperate on FLIGHT
+-t FL_TYPE, --type=FL_TYPEFlight type (for invoking make--flight)
 -h HOST, --host=HOST  Test host
 -r, --reuse   Do not wipe test host (default)
 -R, --noreuse, --noreinstall  Wipe the test host (if job or test does so)
@@ -63,12 +64,13 @@ if [ x$op = x--help ] ; then
 exit 0
 fi
 
-TEMP=$(getopt -o c:f:h:rR --long 
config:,flight:,host:,reuse,noreuse,reinstall,lvextendmax:,baseline,help -- 
"$@")
+TEMP=$(getopt -o c:f:t:h:rR --long 
config:,flight:,ftype:,host:,reuse,noreuse,reinstall,lvextendmax:,baseline,help 
-- "$@")
 
 eval set -- "$TEMP"
 
 config=${OSSTEST_CONFIG-$HOME/.xen-osstest/config}
 flight="standalone"
+ftype=
 host=
 reuse=1 # Don't blow away machines by default
 lvextendmax=50 # Leave some LVM space free for running tests
@@ -78,6 +80,7 @@ while true ; do
 case "$1" in
-c|--config) config=$2; shift 2;;
-f|--flight) flight=$2; shift 2;;
+   -t|--type)   ftype=$2;  shift 2;;
-h|--host)   host=$2;   shift 2;;
-r|--reuse)  reuse=1;   shift 1;;
-R|--noreuse|--reinstall)reuse=0;shift 1;;
@@ -184,7 +187,7 @@ case $op in
 OSSTEST_FLIGHT=$flight \
 OSSTEST_CONFIG=$config \
 OSSTEST_NO_BASELINE=$nobaseline \
-with_logging logs/$flight/make-flight.log ./cr-daily-branch $@ 
$branch
+with_logging logs/$flight/make-flight.log ./cr-daily-branch $@ 
$branch $ftype
 ;;
 
 set-paths)
diff --git a/standalone-reset b/standalone-reset
index 8555039..f041e6d 100755
--- a/standalone-reset
+++ b/standalone-reset
@@ -23,14 +23,17 @@ usage(){
 usage: ./standalone-reset [] [ [ []]]
  branch and xenbranch default, separately, to xen-unstable
 options:
- -f generate flight "flight", default is "standalone"
+ -f   generate flight "flight", default is "standalone"
+ -t  generate a different type of flight (it calls
+  make--flight)
 END
 }
 
 flight="standalone"
-while getopts "f:" opt; do
+while getopts "f:t:" opt; do
 case "$opt" in
 f) flight=${OPTARG};;
+t) flight_type="-${OPTARG}";;
 *) usage; exit 1;;
 esac
 done
@@ -140,7 +143,7 @@ fi
 export BUILD_LVEXTEND_MAX
 
 OSSTEST_FLIGHT=$flight \
-./make-flight "$branch" "$xenbranch" play $buildflight >/dev/null
+./make${flight_type}-flight "$branch" "$xenbranch" play $buildflight >/dev/null
 
 #-- done --
 


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH 15/27] ts-kernbench-run: kick off the benchmark on the target

[Dario Faggioli]

There is a runvar called 'kernbench_params', for specifying
the benchmark's runtime arguments.

Signed-off-by: Dario Faggioli 
Cc: Wei Liu 
Cc: Ian Campbell 
Cc: Ian Jackson 
---
 ts-kernbench-run |   63 ++
 1 file changed, 63 insertions(+)
 create mode 100755 ts-kernbench-run

diff --git a/ts-kernbench-run b/ts-kernbench-run
new file mode 100755
index 000..389f6b3
--- /dev/null
+++ b/ts-kernbench-run
@@ -0,0 +1,63 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2014 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see .
+
+use strict qw(vars);
+use DBI;
+use Osstest;
+use Osstest::TestSupport;
+
+tsreadconfig();
+
+# what we expect as argument list is:
+#  host= []
+our ($whhost,$gn) = @ARGV;
+$whhost ||= 'host';
+our $ho= selecthost($whhost);
+
+our $gho= $ho;
+if (defined $gn and $gn ne "") {
+  $gho= selectguest($gn, $ho);
+  my $err= guest_check_ip($gho);
+  die "$err $gho->{Name}" if defined $err;
+}
+
+# Runtime parameters for the benchmark. We expect them to be
+# specified via a runvar. If not, let's provide a safe default.
+if (!defined($r{ 'kernbench_params' })) {
+  store_runvar('kernbench_params',"-n 2");
+}
+our $args= $r{'kernbench_params'};
+our $timeout= (defined($r{'kernbench_timeout'})) ?
+  $r{'kernbench_timeout'} : 12000;
+
+sub run() {
+  logm("Running: /root/linux-kernbench/../kernbench $args (timeout: 
$timeout)");
+  target_cmd_root($gho, 

[Xen-devel] [PATCH 09/27] ts-unixbench-reslts: process and plot bench results

[Dario Faggioli]

From: Dario Faggioli 

Mangle the results of a run of unixbench a bit, so that
they can be plotted. This also produces a (gnu)plot script
and the plot itself. All is saved in $stash, for the
running flight and job.


This is done in a new Osstest/Benchmarking.pm module, as
the functions introduced may turn out useful somewhere else
too.

The results are read from the original unixbench results
file and a new file, with basically a table in it is
produced. Gnuplot uses such file as data for the plotting.

A gnuplot script is produced and invoked (and saved in $stash)
rather than using the gnuplot perl binding because, this way,
one can modify the plotting commands a bit, and regen the
plot(s).

Signed-off-by: Dario Faggioli 
Cc: Wei Liu 
Cc: Ian Campbell 
Cc: Ian Jackson 
---
 Osstest/Benchmarking.pm |  115 +++
 ts-unixbench-reslts |   17 +++
 2 files changed, 132 insertions(+)
 create mode 100644 Osstest/Benchmarking.pm

diff --git a/Osstest/Benchmarking.pm b/Osstest/Benchmarking.pm
new file mode 100644
index 000..0c5c538
--- /dev/null
+++ b/Osstest/Benchmarking.pm
@@ -0,0 +1,115 @@
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see .
+#
+
+package Osstest::Benchmarking;
+
+use strict;
+use warnings;
+
+use IO::File;
+use IPC::Cmd qw[can_run run];
+
+use Osstest;
+use Osstest::TestSupport;
+
+BEGIN {
+use Exporter ();
+our ($VERSION, @ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);
+$VERSION = 1.00;
+@ISA = qw(Exporter);
+@EXPORT  = qw(unixbench_process_results
+  unixbench_print_results
+  unixbench_plot_results
+  );
+%EXPORT_TAGS = ( );
+
+@EXPORT_OK   = qw();
+}
+
+#-- manipulation of benchmarks results --
+
+sub unixbench_process_results ($$) {
+  my ($results_ref,$rfilen)= @_;
+  my $h= new IO::File "< $rfilen" or die "$!";
+
+  my $par;
+  while (<$h>) {
+my ($bench,$val,$idx);
+if (m/.*running ([0-9]*) parallel.*$/) {
+  $par= $1;
+}
+if 
(m/^(\S[a-zA-z0-9-\(\)\s]*)\s([0-9]+.[0-9]?)\s*([0-9]+.[0-9]?)\s*([0-9]+.[0-9]?)$/)
 {
+  $val= $3;
+  $idx= $4;
+  ($bench = $1) =~ s/\s+$//;
+  $$results_ref->{"$bench"}{Result}{"$par"}= $val;
+  $$results_ref->{"$bench"}{Index}{"$par"}= $idx;
+}
+next;
+  }
+  close($h);
+}
+
+sub unixbench_print_results ($$) {
+  my ($results,$rfilen)= @_;
+  open my $h, "|-", "tee $rfilen" or die "$!";
+
+  printf $h "%-50s","\"BENCHMARK NAME\"";
+  foreach my $i (sort keys $results->{'Double-Precision Whetstone'}{Index}) {
+printf $h "%-15s","\"$i VCPUs\"";
+  }
+  print $h "\n";
+  foreach my $b (keys $results) {
+printf $h "%-50s","\"$b\"";
+foreach my $i (sort keys $results->{"$b"}{Index}) {
+  printf $h "%-15s",$results->{$b}{Index}{$i};
+}
+print $h "\n";
+  }
+  close($h);
+}
+
+sub unixbench_plot_results ($$$) {
+  my ($dataf,$num_cols,$pfile)= @_;
+  my $h= new IO::File "> $pfile.gp" or die "$!";
+
+  printf $h < "$gp $pfile.gp", verbose => 1 );
+  logm("WARNING: plotting file with \"$err\"") unless $ok;
+}
+
+1;
diff --git a/ts-unixbench-reslts b/ts-unixbench-reslts
index 6e5a9a8..b480d15 100755
--- a/ts-unixbench-reslts
+++ b/ts-unixbench-reslts
@@ -21,6 +21,7 @@ use DBI;
 use IO::File;
 use POSIX;
 use Osstest::TestSupport;
+use Osstest::Benchmarking;
 
 tsreadconfig();
 
@@ -46,6 +47,8 @@ $lresfile .= (defined($r{'unixbench_run_suffix'})) ?
   $r{'unixbench_run_suffix'} : '';
 $lresfile = "unixbench$lresfile";
 
+our $results;
+
 # Unixbench stores results in a subdirectory called 'results'. The file name
 # is made up out of the box's hostname, today's date and a progressive id
 # (e.g., results/benny-2014-07-02-01).
@@ -64,4 +67,18 @@ END
   "$lresfile");
 }
 
+sub process () {
+  my $resf= "$stash/$gho->{Name}--$lresfile";
+  my $dataf= "$resf-DATA";
+  my $plotf= "$resf-PLOT";
+
+  unixbench_process_results(\$results,$resf);
+  unixbench_print_results($results,$dataf);
+
+  # For plotting we need to know the number of data columns
+  my $ncols= keys $results->{'Double-Precision Whetstone'}{Index};
+  unixbench_plot_results($dataf,$ncols,$plotf);
+}
+
 fetch();
+process();


_

[Xen-devel] [PATCH 19/27] make-bench-flight: create kernbench jobs

[Dario Faggioli]

Signed-off-by: Dario Faggioli 
Cc: Wei Liu 
Cc: Ian Campbell 
Cc: Ian Jackson 
---
 make-bench-flight |   55 -
 1 file changed, 42 insertions(+), 13 deletions(-)

diff --git a/make-bench-flight b/make-bench-flight
index cdb22ff..125f244 100755
--- a/make-bench-flight
+++ b/make-bench-flight
@@ -56,8 +56,10 @@ test_matrix_branch_filter_callback () {
 }
 
 do_unixbench_tests () {
-  gvcpus=$1
-  gmem=$2
+  gt=$1
+  sched=$2
+  gvcpus=$3
+  gmem=$4
 
   # x86_64 only (for now)
   if [ $xenarch != amd64 ]; then
@@ -73,20 +75,47 @@ do_unixbench_tests () {
   if [ $gvcpus -ge 2 ];then params="-c $(($gvcpus/2))"; fi
   params="$params -c $gvcpus -c $(($gvcpus*2)) -i 6"
 
-  for gt in pv hvm; do
-for sched in credit credit2; do
-  job_create_test \
-  bench-unixbench-$xenarch-$sched-$gt$gvcpus_suffix$gmem_suffix \
-  bench-unixbench-$gt xl $xenarch $dom0arch $gvcpus_runvars 
$gmem_runvars \
-  xen_boot_append="sched=$sched" unixbench_params="$params" 
$debian_runvars \
-  bios=seabios debianhvm_image=debian-7.2.0-amd64-CD-1.iso \
-  all_hostflags=$most_hostflags
-done
-  done
+  job_create_test \
+  bench-unixbench-$xenarch-$sched-$gt$gvcpus_suffix$gmem_suffix \
+  bench-unixbench-$gt xl $xenarch $dom0arch $gvcpus_runvars 
$gmem_runvars \
+  xen_boot_append="sched=$sched" unixbench_params="$params" 
$debian_runvars \
+  bios=seabios debianhvm_image=debian-7.2.0-amd64-CD-1.iso \
+  all_hostflags=$most_hostflags
+}
+
+do_kernbench_tests () {
+  gt=$1
+  sched=$2
+  gvcpus=$3
+  gmem=$4
+
+  # x86_64 only (for now)
+  if [ $xenarch != amd64 ]; then
+return
+  fi
+  # "homogeneous" tests only (for now)
+  if [ $xenarch != $dom0arch ]; then
+return
+  fi
+
+  gvcpus_runvars=guests_vcpus=$gvcpus; gvcpus_suffix=-${gvcpus}vcpus
+  gmem_runvars=guests_memory=$gmem; gmem_suffix=-${gmem}ram
+
+  job_create_test \
+  bench-kernbench-$xenarch-$sched-$gt$gvcpus_suffix$gmem_suffix \
+  bench-kernbench-$gt xl $xenarch $dom0arch $gvcpus_runvars 
$gmem_runvars \
+  xen_boot_append="sched=$sched" kernbench_params="-n 4" 
$debian_runvars \
+  bios=seabios debianhvm_image=debian-7.2.0-amd64-CD-1.iso \
+  all_hostflags=$most_hostflags
 }
 
 test_matrix_do_one () {
-  do_unixbench_tests 4 4096 # 4 vcpus, 4GB RAM
+  for s in credit credit2; do
+for t in pv hvm; do
+  do_unixbench_tests $t $s 4 4096 # 4 vcpus, 4GB RAM
+  do_kernbench_tests $t $s 4 4096
+done
+  done
 }
 
 test_matrix_iterate


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH 13/27] mg-kernbench-download: new script for downloading kernbench

[Dario Faggioli]

It downloads the benchmark (it's just a script) and a linux
kernel archive, necessary for running the benchmark itself,
and store them in c{Images}/benchs.

Default values for the repo URL and actual filename are embedded
in the script itself, and can be overridden as usual (e.g., via
standalone.config).

Signed-off-by: Dario Faggioli 
Cc: Wei Liu 
Cc: Ian Campbell 
Cc: Ian Jackson 
---
 mg-kernbench-download |   49 +
 1 file changed, 49 insertions(+)
 create mode 100755 mg-kernbench-download

diff --git a/mg-kernbench-download b/mg-kernbench-download
new file mode 100755
index 000..7ed252b
--- /dev/null
+++ b/mg-kernbench-download
@@ -0,0 +1,49 @@
+#!/bin/bash
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2014 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see .
+
+set -e
+
+if [ -f standalone.config ] ; then
+. standalone.config
+fi
+
+. cri-getconfig
+
+fail () { echo >&2 "$0: $1"; exit 1; }
+
+# By default we try to grab version 0.50, and we build linux 3.15.3
+site=${KERNBENCH_REMOTE:-http://ck.kolivas.org/apps/kernbench/kernbench-0.50}
+rfile=${KERNBENCH_FILE:-kernbench}
+site_kern=${KERNBENCH_REMOTE_KERN:-https://www.kernel.org/pub/linux/kernel/v3.x}
+rfile_kern=${KERNBENCH_FILE_KERN:-linux-3.15.3.tar.xz}
+
+lfile=kernbench
+lfile_kern=linux.xz
+
+images=`getconfig Images`;
+dstdir="${images}/benchs"
+mkdir -p $dstdir
+
+wget ${site}/${rfile} -O ${dstdir}/${lfile} || \
+fail "failed downloading the benchmark"
+
+echo >&2 "downloaded $dstdir/$lfile"
+
+wget ${site_kern}/${rfile_kern} -O ${dstdir}/${lfile_kern} || \
+fail "failed downloading the kernel to be built"
+
+echo >&2 "downloaded $dstdir/$lfile"


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH 10/27] sg-run-job: recipes for the unixbench jobs

[Dario Faggioli]

Recipes are defined for prepping and running
the unixbench benchmark on the host and on
Debian PV and HVM guests.

Signed-off-by: Dario Faggioli 
Cc: Wei Liu 
Cc: Ian Campbell 
Cc: Ian Jackson 
---
 sg-run-job |   31 +++
 1 file changed, 31 insertions(+)

diff --git a/sg-run-job b/sg-run-job
index 2cf810a..81c2040 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -321,6 +321,37 @@ proc run-job/test-rumpuserxen {} {
 run-ts . =   ts-guest-destroy-hard  + host + $g
 }
 
+# benchmarks 
+
+proc bench-unixbench-run {args} {
+run-ts . = ts-unixbench-build  + host $args
+run-ts . = ts-unixbench-run+ host $args
+run-ts . = ts-unixbench-reslts + host $args
+}
+
+proc bench-unixbench-host {} {
+bench-unixbench-run
+}
+
+proc bench-unixbench-guest {g} {
+bench-unixbench-run $g
+run-ts . = ts-guest-stop + host $g
+}
+
+proc need-hosts/bench-unixbench-pv {} { return host }
+proc run-job/bench-unixbench-pv {} {
+run-ts . = ts-debian-install + host
+run-ts . = ts-debian-fixup   + host debian
+run-ts . = ts-guest-start+ host debian
+bench-unixbench-guest debian
+}
+
+proc need-hosts/bench-unixbench-hvm {} { return host }
+proc run-job/bench-unixbench-hvm {} {
+run-ts . = ts-debian-hvm-install
+bench-unixbench-guest debianhvm
+}
+
 #-- builds --
 
 proc need-hosts/build {} { return BUILD }


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH 16/27] ts-unixbench-reslts: retrieve and stash kernbench results

[Dario Faggioli]

in a file named according to the following convention:

 $hostname--$benchname-$benchparams

i.e., something like this:

 debian--kernbench-n2

Signed-off-by: Dario Faggioli 
Cc: Wei Liu 
Cc: Ian Campbell 
Cc: Ian Jackson 
---
 ts-kernbench-reslts |   60 +++
 1 file changed, 60 insertions(+)
 create mode 100755 ts-kernbench-reslts

diff --git a/ts-kernbench-reslts b/ts-kernbench-reslts
new file mode 100755
index 000..dcb279d
--- /dev/null
+++ b/ts-kernbench-reslts
@@ -0,0 +1,60 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see .
+
+use strict qw(vars);
+use Osstest;
+use DBI;
+use IO::File;
+use POSIX;
+use Osstest::TestSupport;
+
+tsreadconfig();
+
+# what we expect as argument list is:
+#  host= []
+our ($whhost,$gn) = @ARGV;
+$whhost ||= 'host';
+our $ho= selecthost($whhost);
+
+our $gho= $ho;
+if (defined $gn and $gn ne "") {
+  $gho= selectguest($gn, $ho);
+  my $err= guest_check_ip($gho);
+  die "$err $gho->{Name}" if defined $err;
+}
+
+our $lresfile= $r{'kernbench_params'};
+$lresfile =~ tr/ //ds;
+$lresfile .= (defined($r{'kernbench_run_suffix'})) ?
+  $r{'kernbench_run_suffix'} : '';
+$lresfile = "kernbench$lresfile";
+
+# Kernbench stores results in the linux kernel tree it
+# built, in a file called kernbench.log
+sub results() {
+  my $rresults_dir= "/root/linux-kernbench";
+  my $rresfile= "$rresults_dir/kernbench.log";
+
+  my $gho_hostname= target_cmd_output_root($gho, 'hostname');
+  target_cmd_root($gho, "chmod a+r $rresfile", 60);
+
+  logm("Fetching $rresfile from $gho_hostname");
+  target_getfile_root_stash($gho, 60, "$rresfile",
+  "$lresfile");
+}
+
+results();


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH 14/27] ts-kernbench-build: prep the environment for running kernbench

[Dario Faggioli]

by shipping the benchmark (it's just a script) and the linux
kernel sources to the target. The dependences installed are
the ones required to build a linux kernel, plus a few more
packages required by the kernbench script.

As for the unixbench equivalent, this accepts two parametrs,
in the form 'host=somehost someguest'. If only the first one is
provided, it must be 'host=somehost', and the script will prep
the host.

This also installs some packages so, if host sharing is to
be allowed (which, BTW, is highly *not* recommended), the
following patch from Ian Campbell is required:

  http://lists.xen.org/archives/html/xen-devel/2014-04/msg02978.html

Signed-off-by: Dario Faggioli 
Cc: Wei Liu 
Cc: Ian Campbell 
Cc: Ian Jackson 
---
 ts-kernbench-build |   86 
 1 file changed, 86 insertions(+)
 create mode 100755 ts-kernbench-build

diff --git a/ts-kernbench-build b/ts-kernbench-build
new file mode 100755
index 000..eea844c
--- /dev/null
+++ b/ts-kernbench-build
@@ -0,0 +1,86 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2014 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see .
+
+use strict qw(vars);
+use DBI;
+use Osstest;
+use Osstest::TestSupport;
+use File::Basename;
+
+tsreadconfig();
+
+# what we expect as argument list is:
+#  host= []
+our ($whhost,$gn) = @ARGV;
+$whhost ||= 'host';
+our $ho= selecthost($whhost);
+
+our $gho= $ho;
+if (defined $gn and $gn ne "") {
+  $gho= selectguest($gn, $ho);
+  my $err= guest_check_ip($gho);
+  die "$err $gho->{Name}" if defined $err;
+}
+
+# By default, we expect to find kernbench 0,50, stored in
+# $c{Images}/benchs/kernbench. To use something different, define
+# r{'kernbench_file'}.
+our $kernbench_file= (defined($r{'kernbench_file'})) ? $r{'kernbench_file'} :
+"$c{Images}/benchs/kernbench";
+
+# We also assume to find the linux kernel archive to be used for the benchmark
+# in $c{Images}benchs/linux.xz. To use something different, define
+# $r{'kernbench_kern_file'}.
+our $kernbench_linux_file= (defined($r{'kernbench_kern_file'})) ?
+$r{'kernbench_kern_file'} : "$c{Images}/benchs/linux.xz";
+
+# Some deps...
+sub install_deps() {
+  target_install_packages_norec($gho, qw(build-essential time bc));
+}
+
+# Ship the benchmark and the linux kernel archive to the target machine
+sub ship() {
+  logm("Shipping $kernbench_file and $kernbench_linux_file");
+  target_putfile_root($gho, 60, "$kernbench_file", "/root/kernbench");
+  target_putfile_root($gho, 200, "$kernbench_linux_file", "/root/");
+
+  our $kernbench_linux_filename= basename($kernbench_linux_file);
+  logm("Extracting $kernbench_linux_filename");
+  target_cmd_root($gho, 

[Xen-devel] [PATCH 07/27] ts-unixbench-run: kick off the benchmark on the target

[Dario Faggioli]

There is a runvar called 'unixbench_params', for specifying
the benchmark's runtime arguments.

The commit also adds a couple of generic functions in
TestSupport.pm, for `cat'-ing the content of a file on
the target into a corresponding file in the stash area.

Signed-off-by: Dario Faggioli 
Cc: Wei Liu 
Cc: Ian Campbell 
Cc: Ian Jackson 
---
Changes from RFCv1:
 * add default value for runtime arguments.
---
 Osstest/TestSupport.pm |   23 
 ts-unixbench-run   |   67 
 2 files changed, 90 insertions(+)
 create mode 100755 ts-unixbench-run

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 67befd0..7cc5be6 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -57,6 +57,7 @@ BEGIN {
   target_getfile_stash target_getfile_root_stash
   target_put_guest_image target_editfile
   target_editfile_root target_file_exists
+  target_catfile_stash target_catfile_root_stash
   target_run_apt
   target_install_packages target_install_packages_norec
   target_jobdir target_extract_jobdistpath_subdir
@@ -488,6 +489,28 @@ sub target_getfile_root_stash ($$$;$) {
 tgfs_core(\&target_getfile_root, @_);
 }
 
+sub tcfs_core {
+  my ($tcmdoutf,$ho,$rfile)= @_;
+  my $lleaf;
+
+  target_somefile_getleaf(\$lleaf,$rfile,$ho);
+
+  my $h= new IO::File "$stash/$lleaf", 'w' or die $!;
+  my $fdata= $tcmdoutf->($ho, "cat $rfile");
+  print $h $fdata or die $!;
+  close $h or die $!;
+}
+
+sub target_catfile_stash ($$) {
+  my ($ho,$rfile)= @_;
+  tcfs_core(\&target_cmd_output,@_);
+}
+
+sub target_catfile_root_stash ($$) {
+  my ($ho,$rfile)= @_;
+  tcfs_core(\&target_cmd_output_root,@_);
+}
+
 sub target_file_exists ($$) {
 my ($ho,$rfile) = @_;
 my $out= target_cmd_output($ho, "if test -e $rfile; then echo y; fi");
diff --git a/ts-unixbench-run b/ts-unixbench-run
new file mode 100755
index 000..47e6964
--- /dev/null
+++ b/ts-unixbench-run
@@ -0,0 +1,67 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see .
+
+use strict qw(vars);
+use DBI;
+use Osstest;
+use Osstest::TestSupport;
+
+tsreadconfig();
+
+# what we expect as argument list is:
+#  host= []
+our ($whhost,$gn) = @ARGV;
+$whhost ||= 'host';
+our $ho= selecthost($whhost);
+
+our $gho= $ho;
+if (defined $gn and $gn ne "") {
+  $gho= selectguest($gn, $ho);
+  my $err= guest_check_ip($gho);
+  die "$err $gho->{Name}" if defined $err;
+}
+
+# Runtime parameters for the benchmark. We expect them to be
+# specified via a runvar. If not, let's provide a safe default.
+if (!defined($r{ 'unixbench_params' })) {
+  store_runvar('unixbench_params',"-i 3");
+}
+our $args= $r{'unixbench_params'};
+our $timeout= (defined($r{'unixbench_timeout'})) ?
+  $r{'unixbench_timeout'} : 6000;
+
+sub run() {
+  logm("Running: /root/unixbench/Run $args (timeout: $timeout)");
+  target_cmd_root($gho, 

[Xen-devel] [PATCH 08/27] ts-unixbench-reslts: for retrieving the results

[Dario Faggioli]

and store them in $c{Stash}, in a file named according
to the following convention:

 $hostname--$benchname-$benchparams

i.e., something like this:

 debian--unixbench-i3-c2

Signed-off-by: Dario Faggioli 
Cc: Wei Liu 
Cc: Ian Campbell 
Cc: Ian Jackson 
---
Changes from RFCv1:
 * use target_getfile_root_stash, as suggested during review;
---
 ts-unixbench-reslts |   67 +++
 1 file changed, 67 insertions(+)
 create mode 100755 ts-unixbench-reslts

diff --git a/ts-unixbench-reslts b/ts-unixbench-reslts
new file mode 100755
index 000..6e5a9a8
--- /dev/null
+++ b/ts-unixbench-reslts
@@ -0,0 +1,67 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2014 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see .
+
+use strict qw(vars);
+use Osstest;
+use DBI;
+use IO::File;
+use POSIX;
+use Osstest::TestSupport;
+
+tsreadconfig();
+
+# what we expect as argument list is:
+#  host= []
+our ($whhost,$gn) = @ARGV;
+$whhost ||= 'host';
+our $ho= selecthost($whhost);
+
+our $gho= $ho;
+if (defined $gn and $gn ne "") {
+  $gho= selectguest($gn, $ho);
+  my $err= guest_check_ip($gho);
+  die "$err $gho->{Name}" if defined $err;
+}
+
+our $run_id= (defined($r{'unixbench_run_id'})) ?
+  $r{'unixbench_run_id'} : '01';
+
+our $lresfile= $r{'unixbench_params'};
+$lresfile =~ tr/ //ds;
+$lresfile .= (defined($r{'unixbench_run_suffix'})) ?
+  $r{'unixbench_run_suffix'} : '';
+$lresfile = "unixbench$lresfile";
+
+# Unixbench stores results in a subdirectory called 'results'. The file name
+# is made up out of the box's hostname, today's date and a progressive id
+# (e.g., results/benny-2014-07-02-01).
+sub fetch() {
+  my $rresults_dir= "/root/unixbench/results";
+  my $gho_hostname= target_cmd_output_root($gho, 'hostname');
+  my $resultspat= "$rresults_dir/$gho_hostname*-$run_id";
+
+  my $rresfile= target_cmd_output_root($gho, <&1 ||:
+echo $resultspat
+END
+
+  logm("Fetching $rresfile from $gho_hostname");
+  target_getfile_root_stash($gho, 60, "$rresfile",
+  "$lresfile");
+}
+
+fetch();


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH 01/27] ts-devbian-hvm-install: prune "cdrom:" from install sources

[Dario Faggioli]

in sources.list, so installing packages in the guest with
apt-get does not stall waiting for the install CD to be
inserted.

Signed-off-by: Dario Faggioli 
Cc: Wei Liu 
Cc: Ian Campbell 
Cc: Ian Jackson 
---
 ts-debian-hvm-install |3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/ts-debian-hvm-install b/ts-debian-hvm-install
index 37eade2..e509064 100755
--- a/ts-debian-hvm-install
+++ b/ts-debian-hvm-install
@@ -78,7 +78,8 @@ d-i preseed/late_command string \\
 in-target mkdir -p /boot/efi/EFI/boot; \\
 in-target cp /boot/efi/EFI/debian/grubx64.efi 
/boot/efi/EFI/boot/bootx64.efi ;\\
 in-target mkdir -p /root/.ssh; \\
-in-target sh -c "echo -e '$authkeys'> /root/.ssh/authorized_keys";
+in-target sh -c "echo -e '$authkeys'> /root/.ssh/authorized_keys";\\
+in-target sed -i "/^deb cdrom:/s/^/#/" /etc/apt/sources.list
 END
 return $preseed_file;
 }


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH 05/27] mg-unixbench-download: new script for downloading the unixbench archive

[Dario Faggioli]

The script fetches it, and saves it in c{Images}/benchs.

Default values for the repo URL and actual filename are embedded
in the script itself, and can be overridden as usual (e.g., via
standalone.config).

Signed-off-by: Dario Faggioli 
Cc: Wei Liu 
Cc: Ian Campbell 
Cc: Ian Jackson 
---
Changes from RFCv1:
 * stop using ap-common for default values (for URL and remote
   filename), as requested during review.
---
 mg-unixbench-download |   40 
 1 file changed, 40 insertions(+)
 create mode 100755 mg-unixbench-download

diff --git a/mg-unixbench-download b/mg-unixbench-download
new file mode 100755
index 000..85184c9
--- /dev/null
+++ b/mg-unixbench-download
@@ -0,0 +1,40 @@
+#!/bin/bash
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2014 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see .
+
+set -e
+
+if [ -f standalone.config ] ; then
+. standalone.config
+fi
+
+. cri-getconfig
+
+fail () { echo >&2 "$0: $1"; exit 1; }
+
+# By default we try to grab Unixbench 5.1.3
+site=${UNIXBENCH_REPO:-http://byte-unixbench.googlecode.com/files}
+rfile=${UNIXBENCH_FILE:-unixbench-5.1.3.tgz}
+lfile=unixbench.tgz
+
+images=`getconfig Images`;
+dstdir="${images}/benchs"
+mkdir -p $dstdir
+
+wget ${site}/${rfile} -O ${dstdir}/${lfile} || \
+fail "failed downloading the benchmark"
+
+echo >&2 "downloaded $dstdir/$lfile"


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH 02/27] Osstest/Debian.pm: fix identifying a Linux baremetal grub2 entry

[Dario Faggioli]

From: Dario Faggioli 

In fact, in setupboot_grub2(), if we are interested
in a Linux baremetal entry, there is no point in
asking for the entry to contain an hypervisor line
("Hv").

Also, in such entry, Linux kernel and initrd are to be
found in "linux" and "initrd" lines, rather than in
"multiboot" and "module" ones.

I haven't actually checked whether also grub1 and/or
uboot needs fixing.

Signed-off-by: Dario Faggioli 
Cc: Wei Liu 
Cc: Ian Campbell 
Cc: Ian Jackson 
---
 Osstest/Debian.pm |9 ++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index c8db601..70afaec 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -293,8 +293,8 @@ sub setupboot_grub2 ($$$) {
 my (@missing) =
 grep { !defined $entry->{$_} } 
(defined $xenhopt
-? qw(Title Hv KernDom0 KernVer)
-: qw(Title Hv KernOnly KernVer));
+? qw(Title Hv KernDom0 KernVer)
+: qw(Title KernOnly KernVer));
if (@missing) {
logm("(skipping entry at $entry->{StartLine};".
 " no @missing)");
@@ -321,11 +321,14 @@ sub setupboot_grub2 ($$$) {
 die unless $entry;
 $entry->{Hv}= $1;
 }
-if (m/^\s*multiboot\s*\/(vmlinu[xz]-(\S+))/) {
+if (m/^\s*linux\s*\/(vmlinu[xz]-(\S+))/) {
 die unless $entry;
 $entry->{KernOnly}= $1;
 $entry->{KernVer}= $2;
 }
+if (m/^\s*initrd\s*\/(initrd\S+)/) {
+$entry->{Initrd}= $1;
+}
 if (m/^\s*module\s*\/(vmlinu[xz]-(\S+))/) {
 die unless $entry;
 $entry->{KernDom0}= $1;


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH 04/27] Osstest/TestSupport.pm: Introduce target_getfile_[root_]stash()

[Dario Faggioli]

From: Dario Faggioli 

As an analogue to target_putfilecontents_[root_]stash().

(While at it, fix one whitespace damaged line.)

Signed-off-by: Dario Faggioli 
Cc: Wei Liu 
Cc: Ian Campbell 
Cc: Ian Jackson 
---
Changes from RFCv1:
 * adding this was requested during review.
---
 Osstest/TestSupport.pm |   17 -
 1 file changed, 16 insertions(+), 1 deletion(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index cdff8d5..67befd0 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -53,7 +53,8 @@ BEGIN {
   target_getfile target_getfile_root
   target_putfile target_putfile_root
   target_putfilecontents_stash
- target_putfilecontents_root_stash
+  target_putfilecontents_root_stash
+  target_getfile_stash target_getfile_root_stash
   target_put_guest_image target_editfile
   target_editfile_root target_file_exists
   target_run_apt
@@ -473,6 +474,20 @@ sub target_putfilecontents_root_stash (;$) {
 tpfcs_core(\&target_putfile_root, @_);
 }
 
+sub tgfs_core {
+my ($tgetfilef,$ho,$timeout,$rsrc,$lleaf) = @_;
+target_somefile_getleaf(\$lleaf,$rsrc,$ho);
+$tgetfilef->($ho, $timeout, $rsrc, "$stash/$lleaf");
+}
+sub target_getfile_stash ($$$;$) {
+my ($ho,$timeout,$rsrc,$lleaf) = @_;
+tgfs_core(\&target_getfile, @_);
+}
+sub target_getfile_root_stash ($$$;$) {
+my ($ho,$timeout,$rsrc,$lleaf) = @_;
+tgfs_core(\&target_getfile_root, @_);
+}
+
 sub target_file_exists ($$) {
 my ($ho,$rfile) = @_;
 my $out= target_cmd_output($ho, "if test -e $rfile; then echo y; fi");


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH 06/27] ts-unixbench-build: prep the environment for running unixbench

[Dario Faggioli]

by installing some dependencies, shipping the archive, untaring
and building the sources.

This accepts two parametrs, in the form 'host=somehost someguest',
as most of the ts-guest-xxx scripts. If only the first one is
provided, it must be 'host=somehost', and the script will prep
the host.

As this installs some packages, troubles may arise if running
on a shered test host. This is not a too big deal, as hosts
used for benchmarking should not be shared anyway. In any case,
this patch from Ian Campbell will make the issue go away:

  http://lists.xen.org/archives/html/xen-devel/2014-04/msg02978.html

Signed-off-by: Dario Faggioli 
Cc: Wei Liu 
Cc: Ian Campbell 
Cc: Ian Jackson 
---
Changes from RFCv1:
 * make selecthost() interact directly with @ARGV, as
   requested during review;
 * renamed to -build (from -prep), as suggested during review;
 * use 'xaf' for automatic archive type detection, as suggested
   during review.
---
 ts-unixbench-build |   77 
 1 file changed, 77 insertions(+)
 create mode 100755 ts-unixbench-build

diff --git a/ts-unixbench-build b/ts-unixbench-build
new file mode 100755
index 000..4852609
--- /dev/null
+++ b/ts-unixbench-build
@@ -0,0 +1,77 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2014 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see .
+
+use strict qw(vars);
+use DBI;
+use Osstest;
+use Osstest::TestSupport;
+use File::Basename;
+
+tsreadconfig();
+
+# what we expect as argument list is:
+#  host= []
+our ($whhost,$gn) = @ARGV;
+$whhost ||= 'host';
+our $ho= selecthost($whhost);
+
+our $gho= $ho;
+if (defined $gn and $gn ne "") {
+  $gho= selectguest($gn, $ho);
+  my $err= guest_check_ip($gho);
+  die "$err $gho->{Name}" if defined $err;
+}
+
+# By default, we expect to find UnixBench 5.1.3, stored in
+# $c{Images}/benchs/unixbench.tgz. To use something different, define
+# r{'unixbench_file'}.
+our $unixbench_file= (defined($r{'unixbench_file'})) ?
+  $r{'unixbench_file'} : "$c{Images}/benchs/unixbench.tgz";
+
+# Prepare the target, by installing dependencies, and building the benchmark
+sub install_deps() {
+  target_install_packages_norec($gho, qw(build-essential libx11-dev
+ libgl1-mesa-dev libxext-dev
+ x11-apps));
+}
+
+# Ship the benchmark to the target machine
+sub ship() {
+  logm("Shipping and extracting $unixbench_file");
+  target_putfile_root($gho, 60, "$unixbench_file", "/root");
+
+  our $unixbench_filename= basename $unixbench_file;
+  target_cmd_root($gho, 

[Xen-devel] [PATCH 00/27] Running benchmarks via OSSTest

[Dario Faggioli]

Hello everyone,

This is a highly reworked and much more mature version of this old RFC series:

  http://lists.xen.org/archives/html/xen-devel/2014-06/msg03429.html

It is about integrating benchmarking capabilities into OSSTest. The series is a
lot bigger, because it is now capable of doing a lot more stuff. :-) All the
comments I got during review of the RFC have been addressed, of course... More
comments are of course welcome!

The series is available here:

  git://xenbits.xen.org/people/dariof/osstest.git benchmarking-with-osstest
  
http://xenbits.xen.org/gitweb/?p=people/dariof/osstest.git;a=shortlog;h=refs/heads/benchmarking-with-osstest

So, about the patches. Patches 01 to 04, as well as patch 20, are preliminary
changes and fixes, necessary for the rest of the series to work properly, but
they are general enough to be considered separately from the series, if
interesting.

Patches 05 to 12 make it possible to run an instance of the Unixbench (XXX)
benchmark via OSSTest. Management scripts, test scripts, jobs and recipes are
provided to download the benchmark, prepare the guest (PV or HVM), build the
benchmark in the guest, run it, and retrieve and process (include some super
fancy plots! :-D) the results.

To do so, do as follows (in standalone mode):

  $ ./standalone-reset -t bench
  $ ./mg-unixbench-download
  $ ./standalone run-job -h ultralisk 
bench-unixbench-amd64-credit-pv-4vcpus-4096ram

One thing about patch 12. I tried, but I can't tell how I should invoke
./standalone in order to test the new parameter I'm adding... Can someone help
with this? IanC?

To have an idea of how the 'output' of one of this jobs looks like, have a look
here:

 
http://xenbits.xen.org/people/dariof/osstest-bench/bench-kernbench-amd64-credit-hvm-4vcpus-4096ram.log
 
http://xenbits.xen.org/people/dariof/osstest-bench/bench-kernbench-amd64-credit-hvm-4vcpus-4096ram/
 
http://xenbits.xen.org/people/dariof/osstest-bench/bench-kernbench-amd64-credit-hvm-4vcpus-4096ram/debianhvm.guest.osstest--kernbench-n4-PLOT.png

Or here:

 
http://xenbits.xen.org/people/dariof/osstest-bench/bench-unixbench-amd64-credit-pv-4vcpus-4096ram.log
 
http://xenbits.xen.org/people/dariof/osstest-bench/bench-unixbench-amd64-credit-pv-4vcpus-4096ram/
 
http://xenbits.xen.org/people/dariof/osstest-bench/bench-unixbench-amd64-credit-pv-4vcpus-4096ram/debian.guest.osstest--unixbench-i1-c4-PLOT.png

Patches 13 to 19 does the very same, but for Kernbench (XXX):

  $ ./mg-kernbench-download
  $ ./standalone run-job -h ultralisk 
bench-kernbench-amd64-credit-hvm-4vcpus-4096ram

Patches 21 to 27 aims at allowing running a benchmark both in a guest and on
baremetal, to investigate the performances loss due to the virtualization
overhead. To do that, for Unixbench and Kernbench, one just issue the following
commands (in standalone mode):

  $ ./standalone run-job -h ultralisk bench-hostcmp-unixbench-amd64-credit
  $ ./standalone run-job -h ultralisk bench-hostcmp-kernbench-amd64-credit2

To have an idea of how the 'output' of one of these jobs looks like, have a
look here:

 
http://xenbits.xen.org/people/dariof/osstest-bench/bench-hostcmp-kernbench-amd64-credit.log
 
http://xenbits.xen.org/people/dariof/osstest-bench/bench-hostcmp-kernbench-amd64-credit/
 
http://xenbits.xen.org/people/dariof/osstest-bench/bench-hostcmp-kernbench-amd64-credit/bench-hostcmp-kernbench-amd64-credit-PLOT.png

TODOs:
 * in patch 02/27, I haven't actually checked whether also grub1 and/or uboot
   needs fixing
 * When comparing performance, I think it would be best to always use the same
   kernel, in host and guests. This happens by default for PV guest, while it
   does not for HVM guests. This of course can be solved pretty easily, and I
   plan to do it either in future posting of this series, or with a follow-up
   patch.

Regards,
Dario
---
Dario Faggioli (27):
  ts-devbian-hvm-install: prune "cdrom:" from install sources
  Osstest/Debian.pm: fix identifying a Linux baremetal grub2 entry
  Guest setup: allow the amount of RAM to be a runvar
  Osstest/TestSupport.pm: Introduce target_getfile_[root_]stash()
  mg-unixbench-download: new script for downloading the unixbench archive
  ts-unixbench-build: prep the environment for running unixbench
  ts-unixbench-run: kick off the benchmark on the target
  ts-unixbench-reslts: for retrieving the results
  ts-unixbench-reslts: process and plot bench results
  sg-run-job: recipes for the unixbench jobs
  make-bench-flight: to create a benchmarking flight
  standalone-reset: introduce a new -t option
  mg-kernbench-download: new script for downloading kernbench
  ts-kernbench-build: prep the environment for running kernbench
  ts-kernbench-run: kick off the benchmark on the target
  ts-unixbench-reslts: retrieve and stash kernbench results
  ts-kernbench-reslts: process and plot bench results
  sg-run-job: recipes for the kernbench jobs
  make-benc

Re: [Xen-devel] [PATCH] xen: switch to post-init routines in xen mmu.c earlier

[David Vrabel]

On 10/12/14 15:56, Juergen Gross wrote:
> With the virtual mapped linear p2m list the post-init mmu operations
> must be used for setting up the p2m mappings, as in case of
> CONFIG_FLATMEM the init routines may trigger BUGs.
> 
> Reported-by: Boris Ostrovsky 
> Signed-off-by: Juergen Gross 
> ---
>  arch/x86/xen/mmu.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
> index 6ab6150..a1a429a 100644
> --- a/arch/x86/xen/mmu.c
> +++ b/arch/x86/xen/mmu.c
> @@ -1225,6 +1225,7 @@ static void __init xen_pagetable_p2m_setup(void)
>  static void __init xen_pagetable_init(void)
>  {
>   paging_init();
> + xen_post_allocator_init();
>  
>   xen_pagetable_p2m_setup();
>  

This feels very chicken-and-egg to me:  To setup the P2M we need to use
the MMU ops that use the P2M...

Please explain very clearly why this is all safe.

> @@ -1236,7 +1237,6 @@ static void __init xen_pagetable_init(void)
>   xen_remap_memory();
>  
>   xen_setup_shared_info();
> - xen_post_allocator_init();
>  }
>  static void xen_write_cr2(unsigned long cr2)
>  {
> 


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH 5/5] tools/hotplug: support XENSTORED_TRACE in systemd

[Olaf Hering]

On Wed, Dec 10, Ian Campbell wrote:

> Separately from the above I wonder if it might be worth moving the
> xenstore readiness check into the xen-init-dom0 helper and having most
> things which currently depend on xenstore actually depend on the
> "dom0-is-ready" unit, which itself depends on xenstored, qemu-dom0 and
> whatever else is supposed to be ready in a dom0?

You mean something like this chain of depencencies?

 xenstored
 xen-init-dom0
 xenconsoled | qemu-dom0
 xendomains

Olaf

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [3.16.y-ckt stable] Patch "xen-netfront: Fix handling packets on compound pages with skb_linearize" has been added to staging queue

[Luis Henriques]

On Wed, Dec 10, 2014 at 05:26:28PM +, Luis Henriques wrote:
> This is a note to let you know that I have just added a patch titled
> 
> xen-netfront: Fix handling packets on compound pages with skb_linearize
> 
> to the linux-3.16.y-queue branch of the 3.16.y-ckt extended stable tree 
> which can be found at:
> 
>  
> http://kernel.ubuntu.com/git?p=ubuntu/linux.git;a=shortlog;h=refs/heads/linux-3.16.y-queue
> 
> This patch is scheduled to be released in version 3.16.7-ckt3.
> 
> If you, or anyone else, feels it should not be added to this tree, please 
> reply to this email.
> 

Ups, sorry!  I thought I had this one dropped after David Vrabel
comment.  Now dropped for sure.

Cheers,
--
Luís


> For more information about the 3.16.y-ckt tree, see
> https://wiki.ubuntu.com/Kernel/Dev/ExtendedStable
> 
> Thanks.
> -Luis
> 
> --
> 
> From 079f869ca5082866d276b686721e89a649e622fe Mon Sep 17 00:00:00 2001
> From: Zoltan Kiss 
> Date: Mon, 11 Aug 2014 18:32:23 +0100
> Subject: xen-netfront: Fix handling packets on compound pages with
>  skb_linearize
> 
> commit 97a6d1bb2b658ac85ed88205ccd1ab809899884d upstream.
> 
> There is a long known problem with the netfront/netback interface: if the 
> guest
> tries to send a packet which constitues more than MAX_SKB_FRAGS + 1 ring 
> slots,
> it gets dropped. The reason is that netback maps these slots to a frag in the
> frags array, which is limited by size. Having so many slots can occur since
> compound pages were introduced, as the ring protocol slice them up into
> individual (non-compound) page aligned slots. The theoretical worst case
> scenario looks like this (note, skbs are limited to 64 Kb here):
> linear buffer: at most PAGE_SIZE - 17 * 2 bytes, overlapping page boundary,
> using 2 slots
> first 15 frags: 1 + PAGE_SIZE + 1 bytes long, first and last bytes are at the
> end and the beginning of a page, therefore they use 3 * 15 = 45 slots
> last 2 frags: 1 + 1 bytes, overlapping page boundary, 2 * 2 = 4 slots
> Although I don't think this 51 slots skb can really happen, we need a solution
> which can deal with every scenario. In real life there is only a few slots
> overdue, but usually it causes the TCP stream to be blocked, as the retry will
> most likely have the same buffer layout.
> This patch solves this problem by linearizing the packet. This is not the
> fastest way, and it can fail much easier as it tries to allocate a big linear
> area for the whole packet, but probably easier by an order of magnitude than
> anything else. Probably this code path is not touched very frequently anyway.
> 
> Signed-off-by: Zoltan Kiss 
> Cc: Wei Liu 
> Cc: Ian Campbell 
> Cc: Paul Durrant 
> Cc: net...@vger.kernel.org
> Cc: linux-ker...@vger.kernel.org
> Cc: xen-de...@lists.xenproject.org
> Signed-off-by: David S. Miller 
> Cc: Stefan Bader 
> Signed-off-by: Luis Henriques 
> ---
>  drivers/net/xen-netfront.c | 7 ---
>  1 file changed, 4 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
> index 055222bae6e4..23359aeb1ba0 100644
> --- a/drivers/net/xen-netfront.c
> +++ b/drivers/net/xen-netfront.c
> @@ -628,9 +628,10 @@ static int xennet_start_xmit(struct sk_buff *skb, struct 
> net_device *dev)
>   slots = DIV_ROUND_UP(offset + len, PAGE_SIZE) +
>   xennet_count_skb_frag_slots(skb);
>   if (unlikely(slots > MAX_SKB_FRAGS + 1)) {
> - net_alert_ratelimited(
> - "xennet: skb rides the rocket: %d slots\n", slots);
> - goto drop;
> + net_dbg_ratelimited("xennet: skb rides the rocket: %d slots, %d 
> bytes\n",
> + slots, skb->len);
> + if (skb_linearize(skb))
> + goto drop;
>   }
> 
>   spin_lock_irqsave(&queue->tx_lock, flags);

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH 5/5] tools/hotplug: support XENSTORED_TRACE in systemd

[Olaf Hering]

On Wed, Dec 10, Ian Campbell wrote:

> That results in a wrapper which unconditionally execs, the systemd unit
> just calls that while the sysv script runs the wrapper and then does the
> xenstore-read -s loop. 

Since systemd handles the socket there is already a listener.
http://lists.freedesktop.org/archives/systemd-devel/2014-December/026157.html


I came up with this, works in my testing with SLE11 sysv and Factory
systemd. The beef looks like shown below.  Let me know if thats good
enough to handle XENSTORED_TRACE also in systemd. I will add some
comments to the wrapper why it is there.

Olaf

diff --git a/tools/hotplug/Linux/init.d/xencommons.in 
b/tools/hotplug/Linux/init.d/xencommons.in
index a1095c2..f57bfd3 100644
--- a/tools/hotplug/Linux/init.d/xencommons.in
+++ b/tools/hotplug/Linux/init.d/xencommons.in
@@ -66,11 +66,13 @@ do_start () {
then
test -z "$XENSTORED_ROOTDIR" && 
XENSTORED_ROOTDIR="@XEN_LIB_STORED@"
rm -f "$XENSTORED_ROOTDIR"/tdb* &>/dev/null
-   test -z "$XENSTORED_TRACE" || XENSTORED_ARGS=" -T 
/var/log/xen/xenstored-trace.log"
 
if [ -n "$XENSTORED" ] ; then
echo -n Starting $XENSTORED...
-   $XENSTORED --pid-file /var/run/xenstored.pid $XENSTORED_ARGS
+   XENSTORED=$XENSTORED \
+   XENSTORED_TRACE=$XENSTORED_TRACE \
+   XENSTORED_ARGS=$XENSTORED_ARGS \
+   ${LIBEXEC_BIN}/xenstored.sh --pid-file 
/var/run/xenstored.pid
else
echo "No xenstored found"
exit 1
diff --git a/tools/hotplug/Linux/systemd/xenstored.service.in 
b/tools/hotplug/Linux/systemd/xenstored.service.in
index 0f0ac58..d906ea2 100644
--- a/tools/hotplug/Linux/systemd/xenstored.service.in
+++ b/tools/hotplug/Linux/systemd/xenstored.service.in
@@ -8,13 +8,12 @@ ConditionPathExists=/proc/xen/capabilities
 
 [Service]
 Type=notify
-Environment=XENSTORED_ARGS=
 Environment=XENSTORED=@XENSTORED@
-EnvironmentFile=-@CONFIG_DIR@/@CONFIG_LEAF_DIR@/xencommons
+EnvironmentFile=@CONFIG_DIR@/@CONFIG_LEAF_DIR@/xencommons
 ExecStartPre=/bin/grep -q control_d /proc/xen/capabilities
 ExecStartPre=-/bin/rm -f @XEN_LIB_STORED@/tdb*
 ExecStartPre=/bin/mkdir -p @XEN_RUN_DIR@
-ExecStart=/bin/sh -c "exec $XENSTORED --no-fork $XENSTORED_ARGS"
+ExecStart=-@LIBEXEC_BIN@/xenstored.sh --no-fork
 
 [Install]
 WantedBy=multi-user.target
diff --git a/tools/hotplug/Linux/xenstored.sh.in 
b/tools/hotplug/Linux/xenstored.sh.in
new file mode 100644
index 000..dc806ee
--- /dev/null
+++ b/tools/hotplug/Linux/xenstored.sh.in
@@ -0,0 +1,6 @@
+#!/bin/sh
+if test -n "$XENSTORED_TRACE"
+then
+   XENSTORED_ARGS=" -T /var/log/xen/xenstored-trace.log"
+fi
+exec $XENSTORED $@ $XENSTORED_ARGS

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [3.16.y-ckt stable] Patch "xen-netfront: Fix handling packets on compound pages with skb_linearize" has been added to staging queue

[Luis Henriques]

This is a note to let you know that I have just added a patch titled

xen-netfront: Fix handling packets on compound pages with skb_linearize

to the linux-3.16.y-queue branch of the 3.16.y-ckt extended stable tree 
which can be found at:

 
http://kernel.ubuntu.com/git?p=ubuntu/linux.git;a=shortlog;h=refs/heads/linux-3.16.y-queue

This patch is scheduled to be released in version 3.16.7-ckt3.

If you, or anyone else, feels it should not be added to this tree, please 
reply to this email.

For more information about the 3.16.y-ckt tree, see
https://wiki.ubuntu.com/Kernel/Dev/ExtendedStable

Thanks.
-Luis

--

>From 079f869ca5082866d276b686721e89a649e622fe Mon Sep 17 00:00:00 2001
From: Zoltan Kiss 
Date: Mon, 11 Aug 2014 18:32:23 +0100
Subject: xen-netfront: Fix handling packets on compound pages with
 skb_linearize

commit 97a6d1bb2b658ac85ed88205ccd1ab809899884d upstream.

There is a long known problem with the netfront/netback interface: if the guest
tries to send a packet which constitues more than MAX_SKB_FRAGS + 1 ring slots,
it gets dropped. The reason is that netback maps these slots to a frag in the
frags array, which is limited by size. Having so many slots can occur since
compound pages were introduced, as the ring protocol slice them up into
individual (non-compound) page aligned slots. The theoretical worst case
scenario looks like this (note, skbs are limited to 64 Kb here):
linear buffer: at most PAGE_SIZE - 17 * 2 bytes, overlapping page boundary,
using 2 slots
first 15 frags: 1 + PAGE_SIZE + 1 bytes long, first and last bytes are at the
end and the beginning of a page, therefore they use 3 * 15 = 45 slots
last 2 frags: 1 + 1 bytes, overlapping page boundary, 2 * 2 = 4 slots
Although I don't think this 51 slots skb can really happen, we need a solution
which can deal with every scenario. In real life there is only a few slots
overdue, but usually it causes the TCP stream to be blocked, as the retry will
most likely have the same buffer layout.
This patch solves this problem by linearizing the packet. This is not the
fastest way, and it can fail much easier as it tries to allocate a big linear
area for the whole packet, but probably easier by an order of magnitude than
anything else. Probably this code path is not touched very frequently anyway.

Signed-off-by: Zoltan Kiss 
Cc: Wei Liu 
Cc: Ian Campbell 
Cc: Paul Durrant 
Cc: net...@vger.kernel.org
Cc: linux-ker...@vger.kernel.org
Cc: xen-de...@lists.xenproject.org
Signed-off-by: David S. Miller 
Cc: Stefan Bader 
Signed-off-by: Luis Henriques 
---
 drivers/net/xen-netfront.c | 7 ---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
index 055222bae6e4..23359aeb1ba0 100644
--- a/drivers/net/xen-netfront.c
+++ b/drivers/net/xen-netfront.c
@@ -628,9 +628,10 @@ static int xennet_start_xmit(struct sk_buff *skb, struct 
net_device *dev)
slots = DIV_ROUND_UP(offset + len, PAGE_SIZE) +
xennet_count_skb_frag_slots(skb);
if (unlikely(slots > MAX_SKB_FRAGS + 1)) {
-   net_alert_ratelimited(
-   "xennet: skb rides the rocket: %d slots\n", slots);
-   goto drop;
+   net_dbg_ratelimited("xennet: skb rides the rocket: %d slots, %d 
bytes\n",
+   slots, skb->len);
+   if (skb_linearize(skb))
+   goto drop;
}

spin_lock_irqsave(&queue->tx_lock, flags);

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v2] xen/blkfront: remove redundant flush_op

[Konrad Rzeszutek Wilk]

On Tue, Dec 09, 2014 at 03:56:46PM -0500, Boris Ostrovsky wrote:
> On 12/09/2014 09:25 AM, Vitaly Kuznetsov wrote:
> >flush_op is unambiguously defined by feature_flush:
> > REQ_FUA | REQ_FLUSH -> BLKIF_OP_WRITE_BARRIER
> > REQ_FLUSH -> BLKIF_OP_FLUSH_DISKCACHE
> > 0 -> 0
> >and thus can be removed. This is just a cleanup.
> >
> >The patch was suggested by Boris Ostrovsky.
> >
> >Signed-off-by: Vitaly Kuznetsov 
> 
> 
> Reviewed-by: Boris Ostrovsky 

Thank you.

I am testing this and the xen/blkfront: improve protection against issuing 
unsupported REQ_FUA
right now for 3.19

> 
> 
> >---
> >Changes from v1:
> >Future-proof feature_flush against new flags [Boris Ostrovsky].
> >
> >The patch is supposed to be applied after "xen/blkfront: improve protection
> >against issuing unsupported REQ_FUA".
> >---
> >  drivers/block/xen-blkfront.c | 51 
> > +++-
> >  1 file changed, 31 insertions(+), 20 deletions(-)
> >
> >diff --git a/drivers/block/xen-blkfront.c b/drivers/block/xen-blkfront.c
> >index 2e6c103..2236c6f 100644
> >--- a/drivers/block/xen-blkfront.c
> >+++ b/drivers/block/xen-blkfront.c
> >@@ -126,7 +126,6 @@ struct blkfront_info
> > unsigned int persistent_gnts_c;
> > unsigned long shadow_free;
> > unsigned int feature_flush;
> >-unsigned int flush_op;
> > unsigned int feature_discard:1;
> > unsigned int feature_secdiscard:1;
> > unsigned int discard_granularity;
> >@@ -479,7 +478,19 @@ static int blkif_queue_request(struct request *req)
> >  * way.  (It's also a FLUSH+FUA, since it is
> >  * guaranteed ordered WRT previous writes.)
> >  */
> >-ring_req->operation = info->flush_op;
> >+switch (info->feature_flush &
> >+((REQ_FLUSH|REQ_FUA))) {
> >+case REQ_FLUSH|REQ_FUA:
> >+ring_req->operation =
> >+BLKIF_OP_WRITE_BARRIER;
> >+break;
> >+case REQ_FLUSH:
> >+ring_req->operation =
> >+BLKIF_OP_FLUSH_DISKCACHE;
> >+break;
> >+default:
> >+ring_req->operation = 0;
> >+}
> > }
> > ring_req->u.rw.nr_segments = nseg;
> > }
> >@@ -685,20 +696,26 @@ static int xlvbd_init_blk_queue(struct gendisk *gd, 
> >u16 sector_size,
> > return 0;
> >  }
> >+static const char *flush_info(unsigned int feature_flush)
> >+{
> >+switch (feature_flush & ((REQ_FLUSH | REQ_FUA))) {
> >+case REQ_FLUSH|REQ_FUA:
> >+return "barrier: enabled;";
> >+case REQ_FLUSH:
> >+return "flush diskcache: enabled;";
> >+default:
> >+return "barrier or flush: disabled;";
> >+}
> >+}
> >  static void xlvbd_flush(struct blkfront_info *info)
> >  {
> > blk_queue_flush(info->rq, info->feature_flush);
> >-printk(KERN_INFO "blkfront: %s: %s: %s %s %s %s %s\n",
> >-   info->gd->disk_name,
> >-   info->flush_op == BLKIF_OP_WRITE_BARRIER ?
> >-"barrier" : (info->flush_op == BLKIF_OP_FLUSH_DISKCACHE ?
> >-"flush diskcache" : "barrier or flush"),
> >-   info->feature_flush ? "enabled;" : "disabled;",
> >-   "persistent grants:",
> >-   info->feature_persistent ? "enabled;" : "disabled;",
> >-   "indirect descriptors:",
> >-   info->max_indirect_segments ? "enabled;" : "disabled;");
> >+pr_info("blkfront: %s: %s %s %s %s %s\n",
> >+info->gd->disk_name, flush_info(info->feature_flush),
> >+"persistent grants:", info->feature_persistent ?
> >+"enabled;" : "disabled;", "indirect descriptors:",
> >+info->max_indirect_segments ? "enabled;" : "disabled;");
> >  }
> >  static int xen_translate_vdev(int vdevice, int *minor, unsigned int 
> > *offset)
> >@@ -1190,7 +1207,6 @@ static irqreturn_t blkif_interrupt(int irq, void 
> >*dev_id)
> > if (error == -EOPNOTSUPP)
> > error = 0;
> > info->feature_flush = 0;
> >-info->flush_op = 0;
> > xlvbd_flush(info);
> > }
> > /* fall through */
> >@@ -1810,7 +1826,6 @@ static void blkfront_connect(struct blkfront_info 
> >*info)
> > physical_sector_size = sector_size;
> > info->feature_flush = 0;
> >-info->flush_op = 0;
> > err = xenbus_gather(XBT_NIL, info->xbdev->otherend,
> > "feature-barrier", "%d", &barrier,
> >@@

Re: [Xen-devel] [PATCH v2 for-4.5 1/3] python/xc: Fix multiple issues in pyflask_context_to_sid()

[Konrad Rzeszutek Wilk]

On Tue, Dec 09, 2014 at 04:43:22PM +, Andrew Cooper wrote:
> The error handling from a failed memory allocation should return
> PyErr_SetFromErrno(xc_error_obj); rather than simply calling it and continuing
> to the memcpy() below, with the dest pointer being NULL.
> 
> Coverity also complains about passing a non-NUL terminated string to
> xc_flask_context_to_sid().  xc_flask_context_to_sid() doesn't actually take a
> NUL terminated string, but it does take a char* which, in context, used to be
> a string, which is why Coverity complains.
> 
> One solution would be to use strdup(ctx) which is simpler than a
> strlen()/malloc()/memcpy() combo, which would result in a NUL-terminated
> string being used with xc_flask_context_to_sid().
> 
> However, ctx is strictly an input to the hypercall and is not mutated along
> the way.  Both these issues can be fixed, and the error logic simplified, by
> not duplicating ctx in the first place.
> 
> Signed-off-by: Andrew Cooper 
> Coverity-IDs: 1055305 1055721
> Acked-by: Ian Campbell 
> CC: Ian Jackson 
> CC: Wei Liu 
> CC: Xen Coverity Team 
> 
> ---
> v2: Expand the commit message.  No code change

Thank you.

Release-Acked-by: Konrad Rzeszutek Wilk 

> ---
>  tools/python/xen/lowlevel/xc/xc.c |   21 +++--
>  1 file changed, 3 insertions(+), 18 deletions(-)
> 
> diff --git a/tools/python/xen/lowlevel/xc/xc.c 
> b/tools/python/xen/lowlevel/xc/xc.c
> index f83e33d..2aa0dc7 100644
> --- a/tools/python/xen/lowlevel/xc/xc.c
> +++ b/tools/python/xen/lowlevel/xc/xc.c
> @@ -2125,8 +2125,6 @@ static PyObject *pyflask_context_to_sid(PyObject *self, 
> PyObject *args,
>  {
>  xc_interface *xc_handle;
>  char *ctx;
> -char *buf;
> -uint32_t len;
>  uint32_t sid;
>  int ret;
>  
> @@ -2136,28 +2134,15 @@ static PyObject *pyflask_context_to_sid(PyObject 
> *self, PyObject *args,
>&ctx) )
>  return NULL;
>  
> -len = strlen(ctx);
> -
> -buf = malloc(len);
> -if (!buf) {
> -errno = -ENOMEM;
> -PyErr_SetFromErrno(xc_error_obj);
> -}
> -
> -memcpy(buf, ctx, len);
> -
>  xc_handle = xc_interface_open(0,0,0);
>  if (!xc_handle) {
> -free(buf);
>  return PyErr_SetFromErrno(xc_error_obj);
>  }
> -
> -ret = xc_flask_context_to_sid(xc_handle, buf, len, &sid);
> -
> +
> +ret = xc_flask_context_to_sid(xc_handle, ctx, strlen(ctx), &sid);
> +
>  xc_interface_close(xc_handle);
>  
> -free(buf);
> -
>  if ( ret != 0 ) {
>  errno = -ret;
>  return PyErr_SetFromErrno(xc_error_obj);
> -- 
> 1.7.10.4
> 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v2] VMX: don't allow PVH to reach handle_mmio()

[Konrad Rzeszutek Wilk]

On Tue, Dec 09, 2014 at 06:01:54PM +0100, Roger Pau Monné wrote:
> El 08/12/14 a les 10.12, Jan Beulich ha escrit:
> > PVH guests accessing I/O ports via string ops is not supported yet.
> > 
> > Reported-by: Roger Pau Monné
> > Signed-off-by: Jan Beulich 
> 
> This looks fine to me (at least it doesn't break the existing IN/OUT
> users), and seems the best solution 4.5 wise:
> 
> Acked-by: Roger Pau Monné 

Release-Acked-by: Konrad Rzeszutek Wilk 

> 
> For 4.6 I think we need to start using a different hvm_io_bitmap for PVH
> Dom0 that allows direct access to the IO ports, bypassing the vmexit and
> simplifying the code in Xen (this would also fix INS/OUTS). Still not
> sure what should be done for PVH DomUs, specially when PVH gains support
> for pci-passthrough.
> 
> Roger.
> 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH] libxl: Fix building libxlu_cfg_y.y with bison 3.0

[Konrad Rzeszutek Wilk]

On Wed, Dec 10, 2014 at 04:44:00PM +, Ian Campbell wrote:
> On Wed, 2014-12-10 at 11:41 -0500, Konrad Rzeszutek Wilk wrote:
> > On Tue, Dec 09, 2014 at 03:25:29PM +, Ian Jackson wrote:
> > > Ian Campbell writes ("Re: [Xen-devel] [PATCH] libxl: Fix building 
> > > libxlu_cfg_y.y with bison 3.0"):
> > > > There was a point in time where the prevailing version of bison (or
> > > > maybe flex) in stable distro releases had a bug which meant these files
> > > > could not be regenerated easily on common distros. I don't recall the
> > > > details well enough to know if that time has now passed. Perhaps Ian J
> > > > does.
> > > 
> > > We use (essential) features found only in non-ancient versions of
> > > bison and flex.  The last time it was proposed to remove these
> > > pregenerated files, there were complaints from people who were using
> > > six-year-old bison and flex.
> > > 
> > > I think we should prioritise compatibility with new versions of bison
> > > and flex, and retain the pregenerated versions of people with
> > > incompatibly old version.
> > > 
> > > I think for 4.5 we should:
> > > 
> > >  * Regenerate the flex files with current wheezy's flex.  (I have
> > >reviewed the diff in the generated code.  It produces trivial
> > >changes which add declarations of xlu__cfg_yyget_column and
> > >xlu__cfg_yyset_column, but no code body changes - see below.)
> > > 
> > >  * Take the patch from Ed and regenerate the bison files too.
> > > 
> > > 
> > > Konrad: can we have two freeze exceptions please ?
> > > 
> > > Risk analysis for Ed's patch:
> > > 
> > > Not taking the patch hurts systems with bison 2.7.1 or later:
> > > 
> > >   - Affected systems include Debian jessie, which will be
> > > released during the lifetime of 4.5.
> > 
> > My understanding is that Jessie won't be using Xen 4.5 but Xen 4.4 And the
> > next version of Debian that will be released will be most likely using Xen 
> > 4.6.
> 
> What Jessie (or Fedora etc) has available in packages has no real
> bearing on whether people might run Xen 4.5 on them, plenty of people
> will still build from source, including developers.

My initial reasoning for not giving the exception was that this bug did
not affect users.

But it does affect developers and distros - which in number comparison to
users make a miniscule amount.

However, from an perspective of bug-fixing, they (developers) are the most
important - not users. Hence making the environement for them to fix issues
and develop new features without much fuss is paramount. As in, I do not want
them to start looking at fixing an issue/developing, and this crops up and
they say "WTF? I just want to add this tiny libxl feature and I get this!
This is a quagmire".

Based on that reasoning (And thank you for pointing out the developers
factors), I retract my earlier reason and:

Release-Acked-by: Konrad Rzeszutek Wilk 

P.S.
And I just verified that this does not cause regressions with bison 2.4.1, so
that is OK.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH] libxl: Fix building libxlu_cfg_y.y with bison 3.0

[Ian Campbell]

On Wed, 2014-12-10 at 11:41 -0500, Konrad Rzeszutek Wilk wrote:
> On Tue, Dec 09, 2014 at 03:25:29PM +, Ian Jackson wrote:
> > Ian Campbell writes ("Re: [Xen-devel] [PATCH] libxl: Fix building 
> > libxlu_cfg_y.y with bison 3.0"):
> > > There was a point in time where the prevailing version of bison (or
> > > maybe flex) in stable distro releases had a bug which meant these files
> > > could not be regenerated easily on common distros. I don't recall the
> > > details well enough to know if that time has now passed. Perhaps Ian J
> > > does.
> > 
> > We use (essential) features found only in non-ancient versions of
> > bison and flex.  The last time it was proposed to remove these
> > pregenerated files, there were complaints from people who were using
> > six-year-old bison and flex.
> > 
> > I think we should prioritise compatibility with new versions of bison
> > and flex, and retain the pregenerated versions of people with
> > incompatibly old version.
> > 
> > I think for 4.5 we should:
> > 
> >  * Regenerate the flex files with current wheezy's flex.  (I have
> >reviewed the diff in the generated code.  It produces trivial
> >changes which add declarations of xlu__cfg_yyget_column and
> >xlu__cfg_yyset_column, but no code body changes - see below.)
> > 
> >  * Take the patch from Ed and regenerate the bison files too.
> > 
> > 
> > Konrad: can we have two freeze exceptions please ?
> > 
> > Risk analysis for Ed's patch:
> > 
> > Not taking the patch hurts systems with bison 2.7.1 or later:
> > 
> >   - Affected systems include Debian jessie, which will be
> > released during the lifetime of 4.5.
> 
> My understanding is that Jessie won't be using Xen 4.5 but Xen 4.4 And the
> next version of Debian that will be released will be most likely using Xen 
> 4.6.

What Jessie (or Fedora etc) has available in packages has no real
bearing on whether people might run Xen 4.5 on them, plenty of people
will still build from source, including developers.

Ian.


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH] libxl: Fix building libxlu_cfg_y.y with bison 3.0

[Konrad Rzeszutek Wilk]

On Tue, Dec 09, 2014 at 03:25:29PM +, Ian Jackson wrote:
> Ian Campbell writes ("Re: [Xen-devel] [PATCH] libxl: Fix building 
> libxlu_cfg_y.y with bison 3.0"):
> > There was a point in time where the prevailing version of bison (or
> > maybe flex) in stable distro releases had a bug which meant these files
> > could not be regenerated easily on common distros. I don't recall the
> > details well enough to know if that time has now passed. Perhaps Ian J
> > does.
> 
> We use (essential) features found only in non-ancient versions of
> bison and flex.  The last time it was proposed to remove these
> pregenerated files, there were complaints from people who were using
> six-year-old bison and flex.
> 
> I think we should prioritise compatibility with new versions of bison
> and flex, and retain the pregenerated versions of people with
> incompatibly old version.
> 
> I think for 4.5 we should:
> 
>  * Regenerate the flex files with current wheezy's flex.  (I have
>reviewed the diff in the generated code.  It produces trivial
>changes which add declarations of xlu__cfg_yyget_column and
>xlu__cfg_yyset_column, but no code body changes - see below.)
> 
>  * Take the patch from Ed and regenerate the bison files too.
> 
> 
> Konrad: can we have two freeze exceptions please ?
> 
> Risk analysis for Ed's patch:
> 
> Not taking the patch hurts systems with bison 2.7.1 or later:
> 
>   - Affected systems include Debian jessie, which will be
> released during the lifetime of 4.5.

My understanding is that Jessie won't be using Xen 4.5 but Xen 4.4 And the
next version of Debian that will be released will be most likely using Xen 4.6.

However, this issue (building) would even show up in Xen 4.4 - but
I hadn't seen any Debian bugs about this?

> 
>   - Bison 2.7.1 was released in April 2013, a year and
> a half ago.  So any system which is less than 18 months out of
> date suffers pain from not updating.

Aye. Any other distro that will use Xen 4.5 that is new (Fedora 21
ships with 3.0.2) can hit this when building the RPMs or out of
tree.

> 
> Taking the patch hurts systems with old bison:
> 
>   - Bison 2.4.1 is known to work and was released in December 2008.
> So systems which suffer pain from updating are using at least
> 4-year-old bison.

I am not following that. Are you saying that taking this patch will
break building of the sources when using bison 2.4.1 (like Fedora
Core 13 - which is what I use for my small regression testing vehicles).

That would imply an regression?
> 
>   - I have not investigated whether there are even older bison
> versions which are still OK with updating.

Anything older than Fedora Core 13 is so ancient I would be scared
to run on contemporary hardware.
> 
> On the affected systems:
> 
>   - Attempts to build actually-from-source, or with modified
> bison input, will definitely fail.
> 
>   - Some proportion of other builds will fail anyway due to
> timestamp skew.
> 
> 
> Risk analysis for regenerating with current wheezy's flex:
> 
> There doesn't seem to be any actual change in the generated code apart
> from a few new function declarations and changes to #line directives.
> So the risk of breakage is small.  Furthermore:
> 
> Not taking the patch now means that our flex file will be out of date
> and may be regenerated and updated accidentally (or need to be
> regenerated) at some point in the future.  That means that (a)
> whatever risk of breakage we are taking might be discovered at an
> inconvenient time (b) additional build system trouble might result
> from an out of date file.
> 
> 
> There isn't AFAICT a necessary connection between these two but we
> normally regenerate both the bison and flex files together, if
> necessary, before making any changes to the input files.
> 
> Thanks,
> Ian.
> 
> 
> 
> diff --git a/tools/libxl/libxlu_cfg_l.c b/tools/libxl/libxlu_cfg_l.c
> index df352aa..450863a 100644
> --- a/tools/libxl/libxlu_cfg_l.c
> +++ b/tools/libxl/libxlu_cfg_l.c
> @@ -610,6 +610,10 @@ int xlu__cfg_yyget_lineno (yyscan_t yyscanner );
>  
>  void xlu__cfg_yyset_lineno (int line_number ,yyscan_t yyscanner );
>  
> +int xlu__cfg_yyget_column  (yyscan_t yyscanner );
> +
> +void xlu__cfg_yyset_column (int column_no ,yyscan_t yyscanner );
> +
>  YYSTYPE * xlu__cfg_yyget_lval (yyscan_t yyscanner );
>  
>  void xlu__cfg_yyset_lval (YYSTYPE * yylval_param ,yyscan_t yyscanner );
> @@ -762,7 +766,7 @@ YY_DECL
>  #line 53 "libxlu_cfg_l.l"
>  
>  
> -#line 766 "libxlu_cfg_l.c"
> +#line 770 "libxlu_cfg_l.c"
>  
>  yylval = yylval_param;
>  
> @@ -971,7 +975,7 @@ YY_RULE_SETUP
>  #line 104 "libxlu_cfg_l.l"
>  YY_FATAL_ERROR( "flex scanner jammed" );
>   YY_BREAK
> -#line 975 "libxlu_cfg_l.c"
> +#line 979 "libxlu_cfg_l.c"
>  case YY_STATE_EOF(INITIAL):
>  case YY_STATE_EOF(lexerr):
>   yyterminate();
> diff --git a/tools/libxl/libxlu_cfg_l.h b/tools/libxl/libxlu_cfg_l.h
> index 4078302..151064e 100644
> --- a/tools/libxl

Re: [Xen-devel] [PATCH for 4.5] libxl: Tell qemu to use raw format when using a tapdisk

[Konrad Rzeszutek Wilk]

On Tue, Dec 09, 2014 at 02:48:21PM +, Ian Campbell wrote:
> On Tue, 2014-12-09 at 14:04 +, George Dunlap wrote:
> > At the moment libxl unconditinally passes the underlying file format
> > to qemu in the device string.  However, when tapdisk is in use,
> > tapdisk handles the underlying format and presents qemu with
> > effectively a raw disk.  When qemu looks at the tapdisk block device
> > and doesn't find the image format it was looking for, it will fail.
> > 
> > This effectively means that tapdisk cannot be used with HVM domains at
> > the moment except for raw files.
> > 
> > Instead, if we're using a tapdisk backend, tell qemu to use a raw file
> > format.
> > 
> > Signed-off-by: George Dunlap 
> 
> Acked-by: Ian Campbell 

Release-Acked-by: Konrad Rzeszutek Wilk 
> 
> > ---
> > CC: Ian Campbell 
> > CC: Ian Jackson 
> > CC: Wei Liu 
> > CC: Konrad Wilk 
> > 
> > Release exception justification:
> 
> I agree with your reasoning.
> 
> >  This fixes a bug in functionality, in
> > that at the moment HVM guests cannot boot with tapdisk and vhd format.
> > 
> > This is not a regression in xl functionality per se, since (AFAICT)
> > this has never worked.  However, given that 4.5 is the first release
> > without xend, this *does* represent a regression in functionality for
> > Xen as a whole (since before people using hvm guest with vhd on blktap
> > could use xend).
> > 
> > The fix is very simple and should only affect codepaths that already
> > don't work, so the risk of regressions should be very low.
> > 
> > While preparing this patch, I also noticed that cdroms will ignore the
> > backend parameter and treat everything as a file.  This is a bug but I
> > think it's a much less important one to address this late in the
> > release cycle.
> > ---
> >  tools/libxl/libxl_dm.c | 7 +--
> >  1 file changed, 5 insertions(+), 2 deletions(-)
> > 
> > diff --git a/tools/libxl/libxl_dm.c b/tools/libxl/libxl_dm.c
> > index b25b574..10f3090 100644
> > --- a/tools/libxl/libxl_dm.c
> > +++ b/tools/libxl/libxl_dm.c
> > @@ -797,11 +797,14 @@ static char ** 
> > libxl__build_device_model_args_new(libxl__gc *gc,
> >  continue;
> >  }
> >  
> > -if (disks[i].backend == LIBXL_DISK_BACKEND_TAP)
> > +if (disks[i].backend == LIBXL_DISK_BACKEND_TAP) {
> > +format = 
> > qemu_disk_format_string(LIBXL_DISK_FORMAT_RAW);
> >  pdev_path = libxl__blktap_devpath(gc, 
> > disks[i].pdev_path,
> >disks[i].format);
> > -else
> > +} else {
> >  pdev_path = disks[i].pdev_path;
> > +}
> > +
> >  
> >  /*
> >   * Explicit sd disks are passed through as is.
> 
> 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] xen-netback: make feature-rx-notify mandatory -- Breaks stubdoms

[Ian Campbell]

On Wed, 2014-12-10 at 15:29 +, David Vrabel wrote:
> On 10/12/14 15:07, Ian Campbell wrote:
> > On Wed, 2014-12-10 at 14:12 +, David Vrabel wrote:
> >> On 10/12/14 13:42, John wrote:
> >>> David,
> >>>
> >>> This patch you put into 3.18.0 appears to break the latest version of
> >>> stubdomains. I found this out today when I tried to update a machine to
> >>> 3.18.0 and all of the domUs crashed on start with the dmesg output like
> >>> this:
> >>
> >> Cc'ing the lists and relevant netback maintainers.
> >>
> >> I guess the stubdoms are using minios's netfront?  This is something I
> >> forgot about when deciding if it was ok to make this feature mandatory.
> > 
> > Oh bum, me too :/
> > 
> >> The patch cannot be reverted as it's a prerequisite for a critical
> >> (security) bug fix.  I am also unconvinced that the no-feature-rx-notify
> >> support worked correctly anyway.
> >>
> >> This can be resolved by:
> >>
> >> - Fixing minios's netfront to support feature-rx-notify. This should be
> >> easy but wouldn't help existing Xen deployments.
> > 
> > I think this is worth doing in its own right, but as you say it doesn't
> > help existing users.
> > 
> >> - Reimplement feature-rx-notify support.  I think the easiest way is to
> >> queue packets on the guest Rx internal queue with a short expiry time.
> > 
> > Right, I don't think we especially need to make this case good (so long
> > as it doesn't reintroduce a security hole!).
> > 
> > In principal we aren't really obliged to queue at all, but since all the
> > infrastructure for queuing and timing out all exists I suppose it would
> > be simple enough to implement and a bit less harsh.
> > 
> > Given we now have XENVIF_RX_QUEUE_BYTES and rx_drain_timeout_jiffies we
> > don't have the infinite queue any more. So does the expiry in this case
> > actually need to be shorter than the norm? Does it cause any extra
> > issues to keep them around for tx_drain_timeout_jiffies rather than some
> > shorter time?
> 
> If the internal guest rx queue fills and the (host) tx queue is stopped,
> it will take tx_drain_timeout for the thread to wake up and notice if
> the frontend placed any rx requests on the ring.  This could potentially
> end up where you shovel 512k through stall for 10 s, put another 512k
> through, stall for 10 s again and so on.

Ah, true, that's not so great.

What about if we don't queue at all(*) if rx-notify isn't supported, i.e
just drop the packet on the floor in start_xmit if the ring is full?
Would that be so bad? It would surely be simple...

(*) Not counting the "queue" which is the ring itself.

> The rx stall detection will also need to be disabled since there would
> be no way for the frontend to signal rx ready.

Agreed.

Could be trivially argued to be safe if we were just dropping packets on
ring overflow...

Ian.


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH] xen: switch to post-init routines in xen mmu.c earlier

[Konrad Rzeszutek Wilk]

On Wed, Dec 10, 2014 at 04:56:03PM +0100, Juergen Gross wrote:
> With the virtual mapped linear p2m list the post-init mmu operations
> must be used for setting up the p2m mappings, as in case of
> CONFIG_FLATMEM the init routines may trigger BUGs.

Um, could you explain a bit more of why the CONFIG_FLATMEM
is such unique? What about the other CONFIG_*MEM cases?

> 
> Reported-by: Boris Ostrovsky 
> Signed-off-by: Juergen Gross 
> ---
>  arch/x86/xen/mmu.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
> index 6ab6150..a1a429a 100644
> --- a/arch/x86/xen/mmu.c
> +++ b/arch/x86/xen/mmu.c
> @@ -1225,6 +1225,7 @@ static void __init xen_pagetable_p2m_setup(void)
>  static void __init xen_pagetable_init(void)
>  {
>   paging_init();
> + xen_post_allocator_init();
>  
>   xen_pagetable_p2m_setup();
>  
> @@ -1236,7 +1237,6 @@ static void __init xen_pagetable_init(void)
>   xen_remap_memory();
>  
>   xen_setup_shared_info();
> - xen_post_allocator_init();
>  }
>  static void xen_write_cr2(unsigned long cr2)
>  {
> -- 
> 2.1.2
> 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH] xen: switch to post-init routines in xen mmu.c earlier

[Juergen Gross]

With the virtual mapped linear p2m list the post-init mmu operations
must be used for setting up the p2m mappings, as in case of
CONFIG_FLATMEM the init routines may trigger BUGs.

Reported-by: Boris Ostrovsky 
Signed-off-by: Juergen Gross 
---
 arch/x86/xen/mmu.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
index 6ab6150..a1a429a 100644
--- a/arch/x86/xen/mmu.c
+++ b/arch/x86/xen/mmu.c
@@ -1225,6 +1225,7 @@ static void __init xen_pagetable_p2m_setup(void)
 static void __init xen_pagetable_init(void)
 {
paging_init();
+   xen_post_allocator_init();
 
xen_pagetable_p2m_setup();
 
@@ -1236,7 +1237,6 @@ static void __init xen_pagetable_init(void)
xen_remap_memory();
 
xen_setup_shared_info();
-   xen_post_allocator_init();
 }
 static void xen_write_cr2(unsigned long cr2)
 {
-- 
2.1.2


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [linux-next test] 32192: regressions - FAIL

[xen . org]

flight 32192 linux-next real [real]
http://www.chiark.greenend.org.uk/~xensrcts/logs/32192/

Regressions :-(

Tests which did not succeed and are blocking,
including tests which could not be run:
 test-amd64-i386-xl5 xen-boot  fail REGR. vs. 32141
 test-amd64-i386-libvirt   5 xen-boot  fail REGR. vs. 32141
 test-amd64-amd64-rumpuserxen-amd64  5 xen-bootfail REGR. vs. 32141
 test-amd64-amd64-libvirt  5 xen-boot  fail REGR. vs. 32141
 test-amd64-i386-xl-credit25 xen-boot  fail REGR. vs. 32141
 test-amd64-i386-xl-multivcpu  5 xen-boot  fail REGR. vs. 32141
 test-amd64-amd64-xl   5 xen-boot  fail REGR. vs. 32141
 test-amd64-i386-rumpuserxen-i386  5 xen-boot  fail REGR. vs. 32141
 test-amd64-i386-qemuu-rhel6hvm-amd  5 xen-bootfail REGR. vs. 32141
 test-amd64-i386-qemut-rhel6hvm-amd  5 xen-bootfail REGR. vs. 32141
 test-amd64-i386-rhel6hvm-amd  5 xen-boot  fail REGR. vs. 32141
 test-amd64-i386-freebsd10-amd64  5 xen-boot   fail REGR. vs. 32141
 test-amd64-i386-qemut-rhel6hvm-intel  5 xen-boot  fail REGR. vs. 32141
 test-amd64-i386-freebsd10-i386  5 xen-bootfail REGR. vs. 32141
 test-amd64-i386-xl-qemut-win7-amd64  5 xen-boot   fail REGR. vs. 32141
 test-amd64-i386-xl-qemut-winxpsp3  5 xen-boot fail REGR. vs. 32141
 test-amd64-i386-qemuu-rhel6hvm-intel  5 xen-boot  fail REGR. vs. 32141
 test-amd64-i386-rhel6hvm-intel  5 xen-bootfail REGR. vs. 32141
 test-amd64-amd64-xl-qemut-debianhvm-amd64  5 xen-boot fail REGR. vs. 32141
 test-amd64-i386-xl-qemuu-debianhvm-amd64  5 xen-boot  fail REGR. vs. 32141
 test-amd64-i386-xl-qemuu-ovmf-amd64  5 xen-boot   fail REGR. vs. 32141
 test-amd64-amd64-pair 8 xen-boot/dst_host fail REGR. vs. 32141
 test-amd64-amd64-pair 7 xen-boot/src_host fail REGR. vs. 32141
 test-amd64-amd64-xl-qemut-win7-amd64  5 xen-boot  fail REGR. vs. 32141
 test-amd64-i386-xl-qemut-debianhvm-amd64  5 xen-boot  fail REGR. vs. 32141
 test-amd64-amd64-xl-qemuu-debianhvm-amd64  5 xen-boot fail REGR. vs. 32141
 test-amd64-amd64-xl-qemuu-ovmf-amd64  5 xen-boot  fail REGR. vs. 32141
 test-amd64-i386-xl-qemut-winxpsp3-vcpus1  5 xen-boot  fail REGR. vs. 32141
 test-amd64-i386-pair  8 xen-boot/dst_host fail REGR. vs. 32141
 test-amd64-i386-pair  7 xen-boot/src_host fail REGR. vs. 32141
 test-amd64-i386-xl-qemuu-win7-amd64  5 xen-boot   fail REGR. vs. 32141
 test-amd64-i386-xl-winxpsp3   5 xen-boot  fail REGR. vs. 32141
 test-amd64-i386-xl-qemuu-winxpsp3  5 xen-boot fail REGR. vs. 32141
 test-amd64-amd64-xl-win7-amd64  5 xen-bootfail REGR. vs. 32141
 test-amd64-i386-xl-winxpsp3-vcpus1  5 xen-bootfail REGR. vs. 32141
 test-amd64-amd64-xl-qemuu-winxpsp3  5 xen-bootfail REGR. vs. 32141
 test-amd64-i386-xl-qemuu-winxpsp3-vcpus1  5 xen-boot  fail REGR. vs. 32141
 test-amd64-amd64-xl-qemuu-win7-amd64  5 xen-boot  fail REGR. vs. 32141
 test-amd64-amd64-xl-winxpsp3  5 xen-boot  fail REGR. vs. 32141
 test-amd64-i386-xl-win7-amd64  5 xen-boot fail REGR. vs. 32141
 test-amd64-amd64-xl-qemut-winxpsp3  5 xen-bootfail REGR. vs. 32141

Regressions which are regarded as allowable (not blocking):
 test-amd64-amd64-xl-pcipt-intel  5 xen-boot   fail REGR. vs. 32141
 test-amd64-amd64-xl-sedf-pin  5 xen-boot  fail REGR. vs. 32141
 test-amd64-amd64-xl-sedf  5 xen-boot  fail REGR. vs. 32141
 test-armhf-armhf-xl  12 guest-start.2fail blocked in 32141

Tests which did not succeed, but are not blocking:
 test-armhf-armhf-libvirt  9 guest-start  fail   never pass
 test-armhf-armhf-xl  10 migrate-support-checkfail   never pass

version targeted for testing:
 linuxcf12164be498180dc466ef97194ca7755ea39f3b
baseline version:
 linux19b022572b9e0fa8ce5490db888714ecb2b1293e

jobs:
 build-amd64  pass
 build-armhf  pass
 build-i386   pass
 build-amd64-libvirt  pass
 build-armhf-libvirt  pass
 build-i386-libvirt   pass
 build-amd64-pvopspass
 build-armhf-pvopspass
 build-i386-pvops pass
 build-amd64-rumpuserxen  pass
 build-i386-rumpuserxen   

Re: [Xen-devel] xen-netback: make feature-rx-notify mandatory -- Breaks stubdoms

[David Vrabel]

On 10/12/14 15:07, Ian Campbell wrote:
> On Wed, 2014-12-10 at 14:12 +, David Vrabel wrote:
>> On 10/12/14 13:42, John wrote:
>>> David,
>>>
>>> This patch you put into 3.18.0 appears to break the latest version of
>>> stubdomains. I found this out today when I tried to update a machine to
>>> 3.18.0 and all of the domUs crashed on start with the dmesg output like
>>> this:
>>
>> Cc'ing the lists and relevant netback maintainers.
>>
>> I guess the stubdoms are using minios's netfront?  This is something I
>> forgot about when deciding if it was ok to make this feature mandatory.
> 
> Oh bum, me too :/
> 
>> The patch cannot be reverted as it's a prerequisite for a critical
>> (security) bug fix.  I am also unconvinced that the no-feature-rx-notify
>> support worked correctly anyway.
>>
>> This can be resolved by:
>>
>> - Fixing minios's netfront to support feature-rx-notify. This should be
>> easy but wouldn't help existing Xen deployments.
> 
> I think this is worth doing in its own right, but as you say it doesn't
> help existing users.
> 
>> - Reimplement feature-rx-notify support.  I think the easiest way is to
>> queue packets on the guest Rx internal queue with a short expiry time.
> 
> Right, I don't think we especially need to make this case good (so long
> as it doesn't reintroduce a security hole!).
> 
> In principal we aren't really obliged to queue at all, but since all the
> infrastructure for queuing and timing out all exists I suppose it would
> be simple enough to implement and a bit less harsh.
> 
> Given we now have XENVIF_RX_QUEUE_BYTES and rx_drain_timeout_jiffies we
> don't have the infinite queue any more. So does the expiry in this case
> actually need to be shorter than the norm? Does it cause any extra
> issues to keep them around for tx_drain_timeout_jiffies rather than some
> shorter time?

If the internal guest rx queue fills and the (host) tx queue is stopped,
it will take tx_drain_timeout for the thread to wake up and notice if
the frontend placed any rx requests on the ring.  This could potentially
end up where you shovel 512k through stall for 10 s, put another 512k
through, stall for 10 s again and so on.

The rx stall detection will also need to be disabled since there would
be no way for the frontend to signal rx ready.

David

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] xen-netback: make feature-rx-notify mandatory -- Breaks stubdoms

[Ian Campbell]

On Wed, 2014-12-10 at 14:12 +, David Vrabel wrote:
> On 10/12/14 13:42, John wrote:
> > David,
> > 
> > This patch you put into 3.18.0 appears to break the latest version of
> > stubdomains. I found this out today when I tried to update a machine to
> > 3.18.0 and all of the domUs crashed on start with the dmesg output like
> > this:
> 
> Cc'ing the lists and relevant netback maintainers.
> 
> I guess the stubdoms are using minios's netfront?  This is something I
> forgot about when deciding if it was ok to make this feature mandatory.

Oh bum, me too :/

> The patch cannot be reverted as it's a prerequisite for a critical
> (security) bug fix.  I am also unconvinced that the no-feature-rx-notify
> support worked correctly anyway.
> 
> This can be resolved by:
> 
> - Fixing minios's netfront to support feature-rx-notify. This should be
> easy but wouldn't help existing Xen deployments.

I think this is worth doing in its own right, but as you say it doesn't
help existing users.

> - Reimplement feature-rx-notify support.  I think the easiest way is to
> queue packets on the guest Rx internal queue with a short expiry time.

Right, I don't think we especially need to make this case good (so long
as it doesn't reintroduce a security hole!).

In principal we aren't really obliged to queue at all, but since all the
infrastructure for queuing and timing out all exists I suppose it would
be simple enough to implement and a bit less harsh.

Given we now have XENVIF_RX_QUEUE_BYTES and rx_drain_timeout_jiffies we
don't have the infinite queue any more. So does the expiry in this case
actually need to be shorter than the norm? Does it cause any extra
issues to keep them around for tx_drain_timeout_jiffies rather than some
shorter time?

Ian.



___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Xen 4.5.0rc3 + Linux v3.18 + dom0pvh=1 = BOOM

[Konrad Rzeszutek Wilk]

On Tue, Dec 09, 2014 at 09:02:49AM +, Jan Beulich wrote:
> >>> On 08.12.14 at 22:27,  wrote:
> > [8.761336] [ cut here ]
> > [8.761342] kernel BUG at arch/x86/xen/smp.c:438!
> 
>   if (HYPERVISOR_vcpu_op(VCPUOP_initialise, cpu, ctxt))
>   BUG();
> 
> > [8.761348] invalid opcode:  [#1] SMP 
> > [8.761355] Modules linked in:
> > [8.761362] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.18.0 #1
> > [8.761369] Hardware name: LENOVO 10A6S09R01/SHARKBAY, BIOS FBKTA1AUS 
> > 10/22/2014
> > [8.761377] task: 8803ef058000 ti: 8803ef06 task.ti: 
> > 8803ef06
> > [8.761386] RIP: 0010:[]  [] 
> > xen_cpu_up+0x4c5/0x4d0
> > [8.761396] RSP: :8803ef063dd8  EFLAGS: 00010282
> > [8.761402] RAX: fff4 RBX: 0001 RCX: 
> > 0001
> 
> -ENOMEM
> 
> Very strange. I suppose that Dom0 kernel doesn't exhaust all of the
> hypervisor's memory?

With 'dom0_mem=max:2GB' it boots. Thought it does hit another issue which
is related to TSC (will post a seperate email).


> 
> Jan
> 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH OSSTEST] Add basic PVH flights.

[Ian Campbell]

On Wed, 2014-12-10 at 14:03 +, Ian Jackson wrote:
> Ian Campbell writes ("Re: [PATCH OSSTEST] Add basic PVH flights."):
> > On Wed, 2014-12-10 at 13:56 +, Ian Jackson wrote:
> > > This should probably be
> > > 
> > >   +   $cfg =~ s/^pvh\b.*//mg;
> > > 
> > > unless you deliberately intend to strip out any other phv-related
> > > settings which xen-create-image might put there ?
> > 
> > Nope, your suggest is a good one.
> > 
> > Shall I resent or are you ok for me to do this change as I commit?
> 
> Please go ahead, but can you please first double check that it still
> does actually still edit the config file as desired and cause the test
> failure on your machine ?  It would be annoying if that line ceased to
> take effect and the job spuriously passed.

I checked both the pvh and non-pvh job and they did/didn't contain a
pvh=1 as expected.

> That said,
> 
> Acked-by: Ian Jackson 

Thanks. As discussed on IRC I've added it to my to-push queue which is
pending the current pretest stuff propagating. I'll flush that once I
see a pass of osstest's own gate.

The queue contains:
$ git log --oneline origin/pretest..pretest 
082e565 Add basic PVH flights.
1f110e8 Osstest/Debian: support adding a rootdelay property to bootargs
f254b4d Osstest/Debian: Add support for "ExtraInitramfsModules" host property
7d8be54 Osstest/Debian: Refactor code to set bootargs in u-boot script
1b01799 ts-debian-install: rename cfg_xend to cfg
1f48acb gitignore: ignore images directory
1b95fe3 README: list chiark-utils-bin as requirement
abc19f4 TestSupport: allow overriding of on_* in prepareguest_part_xencfg

It's also in the pretest branch of my osstest tree on xenbits.

(Wei, just FYI since some patches of yours are in there)

Ian.



___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] xen-netback: make feature-rx-notify mandatory -- Breaks stubdoms

[David Vrabel]

On 10/12/14 13:42, John wrote:
> David,
> 
> This patch you put into 3.18.0 appears to break the latest version of
> stubdomains. I found this out today when I tried to update a machine to
> 3.18.0 and all of the domUs crashed on start with the dmesg output like
> this:

Cc'ing the lists and relevant netback maintainers.

I guess the stubdoms are using minios's netfront?  This is something I
forgot about when deciding if it was ok to make this feature mandatory.

The patch cannot be reverted as it's a prerequisite for a critical
(security) bug fix.  I am also unconvinced that the no-feature-rx-notify
support worked correctly anyway.

This can be resolved by:

- Fixing minios's netfront to support feature-rx-notify. This should be
easy but wouldn't help existing Xen deployments.

Or:

- Reimplement feature-rx-notify support.  I think the easiest way is to
queue packets on the guest Rx internal queue with a short expiry time.

> [   83.045785] device vif2.0 entered promiscuous mode
> [   83.059220] vif vif-2-0: 22 feature-rx-notify is mandatory
> [   83.060763] vif vif-2-0: 1 mapping in shared page 2047 from domain 2
> [   83.060861] vif vif-2-0: 1 mapping shared-frames 2047/2046 port tx 4
> rx 4
> 
> This is on the very latest patched version of 4.4.

David


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH OSSTEST] Add basic PVH flights.

[Ian Jackson]

Ian Campbell writes ("Re: [PATCH OSSTEST] Add basic PVH flights."):
> On Wed, 2014-12-10 at 13:56 +, Ian Jackson wrote:
> > This should probably be
> > 
> >   + $cfg =~ s/^pvh\b.*//mg;
> > 
> > unless you deliberately intend to strip out any other phv-related
> > settings which xen-create-image might put there ?
> 
> Nope, your suggest is a good one.
> 
> Shall I resent or are you ok for me to do this change as I commit?

Please go ahead, but can you please first double check that it still
does actually still edit the config file as desired and cause the test
failure on your machine ?  It would be annoying if that line ceased to
take effect and the job spuriously passed.

That said,

Acked-by: Ian Jackson 

Ian.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH OSSTEST] Add basic PVH flights.

[Ian Campbell]

On Wed, 2014-12-10 at 13:56 +, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST] Add basic PVH flights."):
> > These are the usual PV debian flights with pvh=1 added to the
> > configuration file.
> > 
> > A job is created for each of Intel and AMD, although obviously AMD is
> > expected to fail at the moment.
> ...
> > Beyond that I've not tested this at all I fully expect even Intel to
> > fail in the first instance, due to issues such as lack of necessary
> > kernel options etc. I suggest to take this now and iterate on any
> > further changes.
> 
> That seems reasonable.
> 
> > For a xen-unstable flight this results in these runvars:
> > diff --git a/ts-debian-fixup b/ts-debian-fixup
> > index f001418..00477c5 100755
> > --- a/ts-debian-fixup
> > +++ b/ts-debian-fixup
> > @@ -118,6 +118,12 @@ sub otherfixupcfg () {
> ...
> > +my $pvh = guest_var($gho,'pvh',undef);
> > +if ($pvh) {
> > +   $cfg =~ s/^pvh.*//mg;
> 
> This should probably be
> 
>   +   $cfg =~ s/^pvh\b.*//mg;
> 
> unless you deliberately intend to strip out any other phv-related
> settings which xen-create-image might put there ?

Nope, your suggest is a good one.

Shall I resent or are you ok for me to do this change as I commit?



___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH OSSTEST] Add basic PVH flights.

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST] Add basic PVH flights."):
> These are the usual PV debian flights with pvh=1 added to the
> configuration file.
> 
> A job is created for each of Intel and AMD, although obviously AMD is
> expected to fail at the moment.
...
> Beyond that I've not tested this at all I fully expect even Intel to
> fail in the first instance, due to issues such as lack of necessary
> kernel options etc. I suggest to take this now and iterate on any
> further changes.

That seems reasonable.

> For a xen-unstable flight this results in these runvars:
> diff --git a/ts-debian-fixup b/ts-debian-fixup
> index f001418..00477c5 100755
> --- a/ts-debian-fixup
> +++ b/ts-debian-fixup
> @@ -118,6 +118,12 @@ sub otherfixupcfg () {
...
> +my $pvh = guest_var($gho,'pvh',undef);
> +if ($pvh) {
> + $cfg =~ s/^pvh.*//mg;

This should probably be

  + $cfg =~ s/^pvh\b.*//mg;

unless you deliberately intend to strip out any other phv-related
settings which xen-create-image might put there ?

Thanks,
Ian.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Frozen dom0 xl commands

[Ian Campbell]

On Wed, 2014-12-10 at 19:24 +0530, Balbir Singh wrote:
> I've been facing an issue on my Ubuntu box (acting as dom0) with
> xen-4.5 (HEAD). I am running dom0 3.13.0.19-generic (ubuntu). When I
> try and xl command I see the command hangs, after a while I see the
> guest kernel complain. I've tried a similar setup on another setup and
> I see the same issue. Any clues on how to debug?
> 
> FYI: xl dmesg does not show much. I've tried using the console
> debugger, but not had much luck with it

Is xenstored running?

> 
> [  484.442137] xl  D  0  4590   4122 
> 0x0004
> [  484.442146]  88038cf87dd8 0202 8800c458afe0
> 88038cf87fd8
> [  484.442156]  00014440 00014440 8800c458afe0
> 81fc0260
> [  484.442164]  88038cf87e10 8800c4130058 8800c413004c
> 8800c413
> [  484.442172] Call Trace:
> [  484.442188]  [] schedule+0x29/0x70
> [  484.442198]  [] read_reply+0x95/0x140
> [  484.442210]  [] ? prepare_to_wait_event+0x100/0x100
> [  484.442218]  [] xenbus_dev_request_and_reply+0x8c/0xb0
> [  484.442226]  [] xenbus_file_write+0x27e/0x540
> [  484.442236]  [] ? __sb_start_write+0x49/0xe0
> [  484.442246]  [] ? security_file_permission+0x23/0xa0
> [  484.442253]  [] vfs_write+0xb4/0x1f0
> [  484.442261]  [] SyS_write+0x49/0xa0
> [  484.442269]  [] tracesys+0xe1/0xe6
> [  484.442276] INFO: task xl:4660 blocked for more than 120 seconds.
> [  484.442280]   Tainted: PF  O 3.13.0-19-generic #40-Ubuntu
> [  484.442283] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [  484.442286] xl  D  0  4660   4646 
> 0x0004
> [  484.442291]  88038dc41dc0 0202 88038e2797f0
> 88038dc41fd8
> [  484.442300]  00014440 00014440 88038e2797f0
> 81fc0290
> [  484.442308]  81fc0294 88038e2797f0 
> 81fc0298
> [  484.442316] Call Trace:
> [  484.442325]  [] schedule_preempt_disabled+0x29/0x70
> [  484.442333]  [] __mutex_lock_slowpath+0x135/0x1b0
> [  484.442339]  [] mutex_lock+0x1f/0x2f
> [  484.442347]  [] xenbus_dev_request_and_reply+0x26/0xb0
> [  484.442355]  [] xenbus_file_write+0x27e/0x540
> [  484.442364]  [] ? __sb_start_write+0x49/0xe0
> [  484.442371]  [] ? security_file_permission+0x23/0xa0
> [  484.442378]  [] vfs_write+0xb4/0x1f0
> [  484.442386]  [] SyS_write+0x49/0xa0
> [  484.442393]  [] tracesys+0xe1/0xe6
> 
> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel



___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] Frozen dom0 xl commands

[Balbir Singh]

I've been facing an issue on my Ubuntu box (acting as dom0) with
xen-4.5 (HEAD). I am running dom0 3.13.0.19-generic (ubuntu). When I
try and xl command I see the command hangs, after a while I see the
guest kernel complain. I've tried a similar setup on another setup and
I see the same issue. Any clues on how to debug?

FYI: xl dmesg does not show much. I've tried using the console
debugger, but not had much luck with it

[  484.442137] xl  D  0  4590   4122 0x0004
[  484.442146]  88038cf87dd8 0202 8800c458afe0
88038cf87fd8
[  484.442156]  00014440 00014440 8800c458afe0
81fc0260
[  484.442164]  88038cf87e10 8800c4130058 8800c413004c
8800c413
[  484.442172] Call Trace:
[  484.442188]  [] schedule+0x29/0x70
[  484.442198]  [] read_reply+0x95/0x140
[  484.442210]  [] ? prepare_to_wait_event+0x100/0x100
[  484.442218]  [] xenbus_dev_request_and_reply+0x8c/0xb0
[  484.442226]  [] xenbus_file_write+0x27e/0x540
[  484.442236]  [] ? __sb_start_write+0x49/0xe0
[  484.442246]  [] ? security_file_permission+0x23/0xa0
[  484.442253]  [] vfs_write+0xb4/0x1f0
[  484.442261]  [] SyS_write+0x49/0xa0
[  484.442269]  [] tracesys+0xe1/0xe6
[  484.442276] INFO: task xl:4660 blocked for more than 120 seconds.
[  484.442280]   Tainted: PF  O 3.13.0-19-generic #40-Ubuntu
[  484.442283] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[  484.442286] xl  D  0  4660   4646 0x0004
[  484.442291]  88038dc41dc0 0202 88038e2797f0
88038dc41fd8
[  484.442300]  00014440 00014440 88038e2797f0
81fc0290
[  484.442308]  81fc0294 88038e2797f0 
81fc0298
[  484.442316] Call Trace:
[  484.442325]  [] schedule_preempt_disabled+0x29/0x70
[  484.442333]  [] __mutex_lock_slowpath+0x135/0x1b0
[  484.442339]  [] mutex_lock+0x1f/0x2f
[  484.442347]  [] xenbus_dev_request_and_reply+0x26/0xb0
[  484.442355]  [] xenbus_file_write+0x27e/0x540
[  484.442364]  [] ? __sb_start_write+0x49/0xe0
[  484.442371]  [] ? security_file_permission+0x23/0xa0
[  484.442378]  [] vfs_write+0xb4/0x1f0
[  484.442386]  [] SyS_write+0x49/0xa0
[  484.442393]  [] tracesys+0xe1/0xe6

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH OSSTEST v3 04/11] overlay: update overlay/etc/grub.d/20_linux_xen

[Ian Campbell]

On Wed, 2014-12-10 at 13:50 +, Wei Liu wrote:
> On Wed, Dec 10, 2014 at 01:47:06PM +, Ian Campbell wrote:
> > On Wed, 2014-12-10 at 13:41 +, Wei Liu wrote:
> > > On Wed, Dec 10, 2014 at 12:54:05PM +, Ian Campbell wrote:
> > > > #690538 relates to providing an option to remove the submenus. Please
> > > > can the changelog explain why that is relevant to us.
> > > > 
> > > 
> > > Because somebody else thought not making submenu optional is a bug. So
> > > do I.
> > 
> > I meant: why is this important to osstest? does using submenus break
> > something? if so then what? if not then there is no reason to be
> > diverging from the baseline.
> >  
> 
> Submenu breaks OSSTest's grub menu parser, causing test step to fail.

Great, that's a good reason to diverge, but please say it somewhere like
in the commit log.

I think the parser should also gain a comment with a reference to either
the bug or this discussion saying that it relies on grub not using
submenus.

Ian.


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH OSSTEST v3 04/11] overlay: update overlay/etc/grub.d/20_linux_xen

[Wei Liu]

On Wed, Dec 10, 2014 at 01:47:06PM +, Ian Campbell wrote:
> On Wed, 2014-12-10 at 13:41 +, Wei Liu wrote:
> > On Wed, Dec 10, 2014 at 12:54:05PM +, Ian Campbell wrote:
> > > #690538 relates to providing an option to remove the submenus. Please
> > > can the changelog explain why that is relevant to us.
> > > 
> > 
> > Because somebody else thought not making submenu optional is a bug. So
> > do I.
> 
> I meant: why is this important to osstest? does using submenus break
> something? if so then what? if not then there is no reason to be
> diverging from the baseline.
>  

Submenu breaks OSSTest's grub menu parser, causing test step to fail.

> > > Did you fix it by removing/reverting the submenu support altogether, as
> > > opposed to e.g. importing the patch from Eric Fischer in the bug report?
> > > I don't see stuff which I'd expect if you had applied the patch. I think
> > > it would be worth spelling out in a bit more detail what the changes
> > > you've made to the baseline for each bug were.
> > > 
> > 
> > I fixed this by supplying a 20_linux_xen extracted from wheezy with
> > submenu generation removed (2 lines). This is how we dealt with other
> > grub bugs.
> 
> I think precisely what you've changed from the baseline should be
> documented somewhere.
> 

OK.

Wei.

> Ian.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH OSSTEST v3 06/11] ts-xen-build: build with XSM support if requested

[Wei Liu]

On Wed, Dec 10, 2014 at 01:05:22PM +, Ian Campbell wrote:
> On Tue, 2014-10-14 at 22:50 +0100, Wei Liu wrote:
> > Signed-off-by: Wei Liu 
> 
> Looks like Ian J acked v2 in
> <21559.64364.468553.506...@mariner.uk.xensource.com>.
> 
> 
> > ---
> >  ts-xen-build |   12 
> >  1 file changed, 12 insertions(+)
> > 
> > diff --git a/ts-xen-build b/ts-xen-build
> > index 661f186..390c114 100755
> > --- a/ts-xen-build
> > +++ b/ts-xen-build
> > @@ -27,6 +27,8 @@ tsreadconfig();
> >  selectbuildhost(\@ARGV);
> >  # remaining arguments are passed as targets to "make"
> >  builddirsprops();
> > +
> > +my $enable_xsm = $r{enable_xsm} =~ m/y/ ? 1 : 0;
> 
> Existing boolean runvars (enable_ovmf, enable_xend) appear to use
> true/false (which still need laundering into Perl booleans). Using y/n
> made sense when you were poking it straight into XSM_ENABLE, but if you
> are going to have to translate it there anyway (into $build_xsm) you may
> as well go for consistency.
> 

I tried to be consistent with next path (the "n y" one, eh). I'm OK with
changing them to true, false though.

> >  
> >  sub checkout () {
> >  prepbuilddirs();
> > @@ -34,6 +36,7 @@ sub checkout () {
> >  build_clone($ho, 'xen', $builddir, 'xen');
> >  
> >  my $debug_build = $r{xen_build_debug} || 'y';
> > +my $build_xsm = $enable_xsm ? 'y' : 'n';
> >  
> >  # Do not set this unless you know what you are doing. This arm
> >  # option makes the build specific to a particular type of
> > @@ -47,6 +50,7 @@ sub checkout () {
> >  cd $builddir/xen
> > >.config
> > echo >>.config debug=$debug_build
> > +   echo >>.config XSM_ENABLE=$build_xsm
> > echo >>.config GIT_HTTP=y
> > echo >>.config LIBLEAFDIR_x86_64=lib
> > echo >>.config QEMU_REMOTE='$r{tree_qemu}'
> > @@ -114,6 +118,14 @@ END
> >  buildcmd_stamped_logged(9000, 'build', '',< >  $make_prefix make $makeflags @ARGV
> >  END
> > +
> > +if ($enable_xsm) {
> > +   my $xen_version = target_cmd_output_root($ho, < > +   cd $builddir/xen
> > +   $make_prefix make xenversion
> > +END
> > +store_runvar("flaskpolicy", "xenpolicy-" . $xen_version);
> > +}
> >  }
> >  
> >  sub collectversions () {
> 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH OSSTEST v3 04/11] overlay: update overlay/etc/grub.d/20_linux_xen

[Ian Campbell]

On Wed, 2014-12-10 at 13:41 +, Wei Liu wrote:
> On Wed, Dec 10, 2014 at 12:54:05PM +, Ian Campbell wrote:
> > #690538 relates to providing an option to remove the submenus. Please
> > can the changelog explain why that is relevant to us.
> > 
> 
> Because somebody else thought not making submenu optional is a bug. So
> do I.

I meant: why is this important to osstest? does using submenus break
something? if so then what? if not then there is no reason to be
diverging from the baseline.
 
> > Did you fix it by removing/reverting the submenu support altogether, as
> > opposed to e.g. importing the patch from Eric Fischer in the bug report?
> > I don't see stuff which I'd expect if you had applied the patch. I think
> > it would be worth spelling out in a bit more detail what the changes
> > you've made to the baseline for each bug were.
> > 
> 
> I fixed this by supplying a 20_linux_xen extracted from wheezy with
> submenu generation removed (2 lines). This is how we dealt with other
> grub bugs.

I think precisely what you've changed from the baseline should be
documented somewhere.

Ian.


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH OSSTEST v3 04/11] overlay: update overlay/etc/grub.d/20_linux_xen

[Wei Liu]

On Wed, Dec 10, 2014 at 12:54:05PM +, Ian Campbell wrote:
> On Tue, 2014-10-14 at 22:50 +0100, Wei Liu wrote:
> > This file was created to work around Debian bug #633127.
> > 
> > According to Debian bug tracker [0], this bug is fixed in Wheezy. As
> > we're now using Wheezy in OSSTest we can safely remove this overlay
> > file.
> > 
> > Also add a note to reference #633127 above grub2 setup function, in case
> > someone trips over #633127.
> > 
> > As we're now using Wheezy in production, update this file to Wheezy's
> > version and take care of Debian bug #690538 and GRUB bug #43420.
> 
> When reference bugs it would be useful to include the bug title here so
> the reader doesn't have to go and look it up.
> 

OK.

> #690538 relates to providing an option to remove the submenus. Please
> can the changelog explain why that is relevant to us.
> 

Because somebody else thought not making submenu optional is a bug. So
do I.

> Did you fix it by removing/reverting the submenu support altogether, as
> opposed to e.g. importing the patch from Eric Fischer in the bug report?
> I don't see stuff which I'd expect if you had applied the patch. I think
> it would be worth spelling out in a bit more detail what the changes
> you've made to the baseline for each bug were.
> 

I fixed this by supplying a 20_linux_xen extracted from wheezy with
submenu generation removed (2 lines). This is how we dealt with other
grub bugs.

> I suppose constructing overlay/etc/grub.d/20_linux_xen from a baseline
> unmodified version (checked in, not retrieved from the host) and a
> mini-patch-series (also checked in) on the fly is over complexifying
> things?
> 

Certainly more complex than a single file solution. At the very least
new infrastructure to apply patches for overlay files is needed.

Wei.

> Ian.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH OSSTEST v3 10/11] ts-xen-install: install Xen with XSM support if requested

[Wei Liu]

On Wed, Dec 10, 2014 at 01:15:13PM +, Ian Campbell wrote:
> On Tue, 2014-10-14 at 22:50 +0100, Wei Liu wrote:
> > Signed-off-by: Wei Liu 
> > ---
> >  ts-xen-install |4 +++-
> >  1 file changed, 3 insertions(+), 1 deletion(-)
> > 
> > diff --git a/ts-xen-install b/ts-xen-install
> > index 4d34d1f..2e2fcbc 100755
> > --- a/ts-xen-install
> > +++ b/ts-xen-install
> > @@ -46,6 +46,8 @@ if (@ARGV and $ARGV[0] eq '--check') {
> >  
> >  our $ho;
> >  
> > +my $enable_xsm = $r{enable_xsm} =~ m/y/ ? 1 : 0;
> > +
> >  my %distpath;
> >  
> >  sub packages () {
> > @@ -171,7 +173,7 @@ sub setupboot () {
> >  }
> >  
> >  my $want_kernver = get_runvar('kernel_ver',$r{'kernbuildjob'});
> > -debian_boot_setup($ho, $want_kernver, $xenhopt, \%distpath, \@hooks);
> > +debian_boot_setup($ho, $want_kernver, $enable_xsm, $xenhopt, 
> > \%distpath, \@hooks);
> 
> Doesn't this have to be in the same patch as the one which adds the new
> parameter in the function declaration? Or at least that patch needs to
> hardcode a false until now.
> 

Yes, I noticed this after I sent this series.  I will squash this one
into previous commit.

Wei.

> (not sure we care much about bisectability in osstest, Ian?)
> 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH OSSTEST v3 07/11] mfi-common: create build-$arch-xsm job

[Wei Liu]

On Wed, Dec 10, 2014 at 01:12:23PM +, Ian Campbell wrote:
> On Tue, 2014-10-14 at 22:50 +0100, Wei Liu wrote:
> > Signed-off-by: Wei Liu 
> > ---
> >  mfi-common |   23 ++-
> >  1 file changed, 22 insertions(+), 1 deletion(-)
> > 
> > diff --git a/mfi-common b/mfi-common
> > index 5c4f5d5..e772086 100644
> > --- a/mfi-common
> > +++ b/mfi-common
> > @@ -41,6 +41,19 @@ branch_wants_rumpkernel_tests () {
> >esac
> >  }
> >  
> > +xenbranch_wants_xsm_tests () {
> > +# Test XSM from 4.5 onwards
> > +case "$xenbranch" in
> > +xen-3.*-testing) echo "n";;
> > +xen-4.0-testing) echo "n";;
> > +xen-4.1-testing) echo "n";;
> > +xen-4.2-testing) echo "n";;
> > +xen-4.3-testing) echo "n";;
> > +xen-4.4-testing) echo "n";;
> > +*) echo "n y";
> 
> Did you mean just "y" here, or is something very subtle going on?
> 
> Oh I see, you run over the result of this as a list with for. Sneaky!
> 

Ian J's suggestion.

> I think you should name it something like xenbranch_xsm_variants or
> something else which doesn't make the reader expect it to return a
> boolean.
> 

Ack.

Wei.

> Ian.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel