[Xen-devel] [linux-linus test] 110984: regressions - FAIL

[osstest service owner]

flight 110984 linux-linus real [real]
http://logs.test-lab.xenproject.org/osstest/logs/110984/

Regressions :-(

Tests which did not succeed and are blocking,
including tests which could not be run:
 test-amd64-amd64-xl-qemuu-ovmf-amd64 14 guest-saverestore.2 fail REGR. vs. 
110515
 test-amd64-i386-xl-qemuu-debianhvm-amd64-xsm 15 guest-localmigrate/x10 fail 
REGR. vs. 110515
 test-amd64-amd64-xl-qemut-win7-amd64 15 guest-localmigrate/x10 fail REGR. vs. 
110515

Tests which did not succeed, but are not blocking:
 test-amd64-i386-xl-qemuu-win7-amd64 17 guest-start/win.repeat fail blocked in 
110515
 test-armhf-armhf-libvirt 13 saverestore-support-checkfail  like 110515
 test-armhf-armhf-libvirt-xsm 13 saverestore-support-checkfail  like 110515
 test-armhf-armhf-xl-rtds 15 guest-start/debian.repeatfail  like 110515
 test-amd64-amd64-xl-qemuu-win7-amd64 16 guest-stopfail like 110515
 test-amd64-i386-xl-qemut-win7-amd64 16 guest-stop fail like 110515
 test-amd64-amd64-xl-rtds  9 debian-install   fail  like 110515
 test-armhf-armhf-libvirt-raw 12 saverestore-support-checkfail  like 110515
 test-amd64-amd64-xl-qemut-ws16-amd64  9 windows-installfail never pass
 test-amd64-i386-libvirt  12 migrate-support-checkfail   never pass
 test-amd64-amd64-libvirt-xsm 12 migrate-support-checkfail   never pass
 test-amd64-i386-libvirt-xsm  12 migrate-support-checkfail   never pass
 test-amd64-amd64-libvirt 12 migrate-support-checkfail   never pass
 test-arm64-arm64-xl-credit2  12 migrate-support-checkfail   never pass
 test-arm64-arm64-libvirt-xsm 12 migrate-support-checkfail   never pass
 test-arm64-arm64-xl-credit2  13 saverestore-support-checkfail   never pass
 test-arm64-arm64-libvirt-xsm 13 saverestore-support-checkfail   never pass
 test-amd64-amd64-libvirt-qemuu-debianhvm-amd64-xsm 10 migrate-support-check 
fail never pass
 test-arm64-arm64-xl-xsm  12 migrate-support-checkfail   never pass
 test-arm64-arm64-xl-xsm  13 saverestore-support-checkfail   never pass
 test-arm64-arm64-xl  12 migrate-support-checkfail   never pass
 test-arm64-arm64-xl  13 saverestore-support-checkfail   never pass
 test-armhf-armhf-xl-arndale  12 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-arndale  13 saverestore-support-checkfail   never pass
 test-amd64-amd64-libvirt-vhd 11 migrate-support-checkfail   never pass
 test-amd64-amd64-qemuu-nested-amd 16 debian-hvm-install/l1/l2  fail never pass
 test-armhf-armhf-xl-rtds 12 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-rtds 13 saverestore-support-checkfail   never pass
 test-armhf-armhf-xl  12 migrate-support-checkfail   never pass
 test-armhf-armhf-xl  13 saverestore-support-checkfail   never pass
 test-armhf-armhf-libvirt 12 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-cubietruck 12 migrate-support-checkfail never pass
 test-armhf-armhf-xl-cubietruck 13 saverestore-support-checkfail never pass
 test-armhf-armhf-libvirt-xsm 12 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-multivcpu 12 migrate-support-checkfail  never pass
 test-armhf-armhf-xl-multivcpu 13 saverestore-support-checkfail  never pass
 test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm 10 migrate-support-check 
fail never pass
 test-amd64-amd64-xl-qemuu-ws16-amd64  9 windows-installfail never pass
 test-amd64-i386-xl-qemut-ws16-amd64 12 guest-saverestore   fail never pass
 test-armhf-armhf-xl-vhd  11 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-vhd  12 saverestore-support-checkfail   never pass
 test-armhf-armhf-xl-xsm  12 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-xsm  13 saverestore-support-checkfail   never pass
 test-armhf-armhf-xl-credit2  12 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-credit2  13 saverestore-support-checkfail   never pass
 test-armhf-armhf-libvirt-raw 11 migrate-support-checkfail   never pass
 test-amd64-i386-xl-qemuu-ws16-amd64 12 guest-saverestore   fail never pass
 test-amd64-amd64-xl-qemut-win10-i386  9 windows-installfail never pass
 test-amd64-amd64-xl-qemuu-win10-i386  9 windows-installfail never pass
 test-amd64-i386-xl-qemuu-win10-i386  9 windows-install fail never pass
 test-amd64-i386-xl-qemut-win10-i386  9 windows-install fail never pass

version targeted for testing:
 linuxa38371cba67539ce6a5d5324db34bc2ddaf66cc1
baseline version:
 linux1439ccf73d9c07654fdd5b4969fd53c2feb8684d

Last test of basis   110515  2017-06-17 06:48:56 Z6 days
Failing since110536  2017-06-17 23:48:13 Z6 days7 attempts
Testing same since   110984  2017-06-23 04:20:18 Z0 

Re: [Xen-devel] live migration of HVM domUs with more than 32vcpus fails

[Ankur Arora]

On 2017-06-23 10:03 AM, Boris Ostrovsky wrote:



On 06/23/2017 12:54 PM, Olaf Hering wrote:

On Thu, Jun 22, Boris Ostrovsky wrote:


They are queued for 4.13.
git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip.git for-linus-4.13


This works for me. Thanks.

I assume there is no ready-to-pull variant for linux-4.4.x?
nm.



Not that I am aware of. Unless Ankur backported them to OL which is 
(loosely) based on 4.1.

There was an earlier version of this patch by Konrad -- which takes
care of the migration failure: https://patchwork.kernel.org/patch/6768031/

Olaf, can you try this out?

Thanks
Ankur




-boris

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [PATCH] x86: xen: remove unnecessary variable in xen_foreach_remap_area()

[Gustavo A. R. Silva]

Remove unnecessary variable mfn in function xen_foreach_remap_area() and,
refactor the code.

Variable mfn at line 518:mfn = xen_remap_buf.mfns[i];
is only being used to store a value to be passed as
an argument to the xen_update_mem_tables() function.
This value can be passed directly, which makes variable
mfn unnecessary. Also, value assigned to variable mfn
at line 534:mfn = xen_remap_mfn; is never used.

Addresses-Coverity-ID: 1260110
Signed-off-by: Gustavo A. R. Silva 
---
 arch/x86/xen/setup.c | 7 ++-
 1 file changed, 2 insertions(+), 5 deletions(-)

diff --git a/arch/x86/xen/setup.c b/arch/x86/xen/setup.c
index a5bf7c4..c810463 100644
--- a/arch/x86/xen/setup.c
+++ b/arch/x86/xen/setup.c
@@ -499,7 +499,7 @@ static unsigned long __init xen_foreach_remap_area(unsigned 
long nr_pages,
 void __init xen_remap_memory(void)
 {
unsigned long buf = (unsigned long)_remap_buf;
-   unsigned long mfn_save, mfn, pfn;
+   unsigned long mfn_save, pfn;
unsigned long remapped = 0;
unsigned int i;
unsigned long pfn_s = ~0UL;
@@ -515,8 +515,7 @@ void __init xen_remap_memory(void)
 
pfn = xen_remap_buf.target_pfn;
for (i = 0; i < xen_remap_buf.size; i++) {
-   mfn = xen_remap_buf.mfns[i];
-   xen_update_mem_tables(pfn, mfn);
+   xen_update_mem_tables(pfn, xen_remap_buf.mfns[i]);
remapped++;
pfn++;
}
@@ -530,8 +529,6 @@ void __init xen_remap_memory(void)
pfn_s = xen_remap_buf.target_pfn;
len = xen_remap_buf.size;
}
-
-   mfn = xen_remap_mfn;
xen_remap_mfn = xen_remap_buf.next_area_mfn;
}
 
-- 
2.5.0


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [xen-4.9-testing test] 110978: regressions - FAIL

[osstest service owner]

flight 110978 xen-4.9-testing real [real]
http://logs.test-lab.xenproject.org/osstest/logs/110978/

Regressions :-(

Tests which did not succeed and are blocking,
including tests which could not be run:
 test-armhf-armhf-xl   6 xen-boot   fail in 110942 REGR. vs. 110550

Tests which are failing intermittently (not blocking):
 test-armhf-armhf-xl-credit2 15 guest-start/debian.repeat fail in 110942 pass 
in 110978
 test-armhf-armhf-xl-xsm   5 xen-installfail pass in 110942
 test-armhf-armhf-xl   5 xen-installfail pass in 110942

Regressions which are regarded as allowable (not blocking):
 test-amd64-i386-xl-qemuu-win7-amd64 16 guest-stopfail REGR. vs. 110542

Tests which did not succeed, but are not blocking:
 test-amd64-i386-xl-qemut-win7-amd64 16 guest-stop   fail in 110942 like 110499
 test-amd64-amd64-xl-qemuu-win7-amd64 16 guest-stop  fail in 110942 like 110524
 test-amd64-i386-xl-qemut-ws16-amd64 9 windows-install fail in 110942 like 
110550
 test-amd64-i386-xl-qemuu-ws16-amd64 9 windows-install fail in 110942 like 
110550
 test-armhf-armhf-xl-xsm 12 migrate-support-check fail in 110942 never pass
 test-armhf-armhf-xl-xsm 13 saverestore-support-check fail in 110942 never pass
 test-amd64-i386-xl-qemut-win7-amd64 15 guest-localmigrate/x10 fail like 110542
 test-amd64-amd64-xl-qemut-win7-amd64 15 guest-localmigrate/x10 fail like 110550
 test-amd64-amd64-xl-qemuu-win7-amd64 15 guest-localmigrate/x10 fail like 110550
 test-amd64-amd64-xl-rtds  9 debian-install   fail  like 110550
 test-amd64-amd64-xl-qemuu-ws16-amd64  9 windows-installfail never pass
 test-amd64-amd64-xl-qemut-ws16-amd64  9 windows-installfail never pass
 test-amd64-i386-libvirt-xsm  12 migrate-support-checkfail   never pass
 test-amd64-amd64-libvirt 12 migrate-support-checkfail   never pass
 test-amd64-amd64-libvirt-xsm 12 migrate-support-checkfail   never pass
 build-amd64-prev  6 xen-build/dist-test  fail   never pass
 test-arm64-arm64-xl-credit2  12 migrate-support-checkfail   never pass
 test-arm64-arm64-xl-credit2  13 saverestore-support-checkfail   never pass
 test-arm64-arm64-xl  12 migrate-support-checkfail   never pass
 test-arm64-arm64-xl  13 saverestore-support-checkfail   never pass
 test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm 10 migrate-support-check 
fail never pass
 test-amd64-amd64-libvirt-qemuu-debianhvm-amd64-xsm 10 migrate-support-check 
fail never pass
 test-arm64-arm64-xl-xsm  12 migrate-support-checkfail   never pass
 test-arm64-arm64-xl-xsm  13 saverestore-support-checkfail   never pass
 build-i386-prev   6 xen-build/dist-test  fail   never pass
 test-amd64-amd64-libvirt-vhd 11 migrate-support-checkfail   never pass
 test-armhf-armhf-libvirt 12 migrate-support-checkfail   never pass
 test-armhf-armhf-libvirt 13 saverestore-support-checkfail   never pass
 test-armhf-armhf-xl-multivcpu 12 migrate-support-checkfail  never pass
 test-armhf-armhf-xl-multivcpu 13 saverestore-support-checkfail  never pass
 test-armhf-armhf-libvirt-xsm 12 migrate-support-checkfail   never pass
 test-armhf-armhf-libvirt-xsm 13 saverestore-support-checkfail   never pass
 test-armhf-armhf-xl-cubietruck 12 migrate-support-checkfail never pass
 test-armhf-armhf-xl-cubietruck 13 saverestore-support-checkfail never pass
 test-amd64-i386-xl-qemuu-ws16-amd64 12 guest-saverestore   fail never pass
 test-amd64-i386-libvirt  12 migrate-support-checkfail   never pass
 test-amd64-i386-xl-qemut-ws16-amd64 12 guest-saverestore   fail never pass
 test-armhf-armhf-libvirt-raw 11 migrate-support-checkfail   never pass
 test-armhf-armhf-libvirt-raw 12 saverestore-support-checkfail   never pass
 test-armhf-armhf-xl-rtds 12 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-rtds 13 saverestore-support-checkfail   never pass
 test-armhf-armhf-xl-credit2  12 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-credit2  13 saverestore-support-checkfail   never pass
 test-armhf-armhf-xl-arndale  12 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-arndale  13 saverestore-support-checkfail   never pass
 test-arm64-arm64-libvirt-xsm 12 migrate-support-checkfail   never pass
 test-arm64-arm64-libvirt-xsm 13 saverestore-support-checkfail   never pass
 test-armhf-armhf-xl-vhd  11 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-vhd  12 saverestore-support-checkfail   never pass
 test-amd64-amd64-qemuu-nested-amd 16 debian-hvm-install/l1/l2  fail never pass
 test-amd64-i386-xl-qemuu-win10-i386  9 windows-install fail never pass
 test-amd64-amd64-xl-qemut-win10-i386  9 windows-installfail never pass
 

[Xen-devel] [xen-unstable-smoke test] 111020: tolerable trouble: broken/pass - PUSHED

[osstest service owner]

flight 111020 xen-unstable-smoke real [real]
http://logs.test-lab.xenproject.org/osstest/logs/111020/

Failures :-/ but no regressions.

Tests which did not succeed, but are not blocking:
 test-arm64-arm64-xl-xsm   1 build-check(1)   blocked  n/a
 test-amd64-amd64-libvirt 12 migrate-support-checkfail   never pass
 test-armhf-armhf-xl  12 migrate-support-checkfail   never pass
 test-armhf-armhf-xl  13 saverestore-support-checkfail   never pass

version targeted for testing:
 xen  d5f6383d9a0316a37fb3f05a23d4cce936a262b3
baseline version:
 xen  7f919ac31993e95387cac612d0dc0bf0ac7e

Last test of basis   111009  2017-06-23 16:02:07 Z0 days
Testing same since   111020  2017-06-23 19:01:31 Z0 days1 attempts


People who touched revisions under test:
  Andrew Cooper 
  Bhupinder Thakur 
  Jan Beulich 
  Julien Grall 

jobs:
 build-amd64  pass
 build-armhf  pass
 build-amd64-libvirt  pass
 test-armhf-armhf-xl  pass
 test-arm64-arm64-xl-xsm  broken  
 test-amd64-amd64-xl-qemuu-debianhvm-i386 pass
 test-amd64-amd64-libvirt pass



sg-report-flight on osstest.test-lab.xenproject.org
logs: /home/logs/logs
images: /home/logs/images

Logs, config files, etc. are available at
http://logs.test-lab.xenproject.org/osstest/logs

Explanation of these reports, and of osstest in general, is at
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=master
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master

Test harness code can be found at
http://xenbits.xen.org/gitweb?p=osstest.git;a=summary


Pushing revision :

+ branch=xen-unstable-smoke
+ revision=d5f6383d9a0316a37fb3f05a23d4cce936a262b3
+ . ./cri-lock-repos
++ . ./cri-common
+++ . ./cri-getconfig
+++ umask 002
+++ getrepos
 getconfig Repos
 perl -e '
use Osstest;
readglobalconfig();
print $c{"Repos"} or die $!;
'
+++ local repos=/home/osstest/repos
+++ '[' -z /home/osstest/repos ']'
+++ '[' '!' -d /home/osstest/repos ']'
+++ echo /home/osstest/repos
++ repos=/home/osstest/repos
++ repos_lock=/home/osstest/repos/lock
++ '[' x '!=' x/home/osstest/repos/lock ']'
++ OSSTEST_REPOS_LOCK_LOCKED=/home/osstest/repos/lock
++ exec with-lock-ex -w /home/osstest/repos/lock ./ap-push xen-unstable-smoke 
d5f6383d9a0316a37fb3f05a23d4cce936a262b3
+ branch=xen-unstable-smoke
+ revision=d5f6383d9a0316a37fb3f05a23d4cce936a262b3
+ . ./cri-lock-repos
++ . ./cri-common
+++ . ./cri-getconfig
+++ umask 002
+++ getrepos
 getconfig Repos
 perl -e '
use Osstest;
readglobalconfig();
print $c{"Repos"} or die $!;
'
+++ local repos=/home/osstest/repos
+++ '[' -z /home/osstest/repos ']'
+++ '[' '!' -d /home/osstest/repos ']'
+++ echo /home/osstest/repos
++ repos=/home/osstest/repos
++ repos_lock=/home/osstest/repos/lock
++ '[' x/home/osstest/repos/lock '!=' x/home/osstest/repos/lock ']'
+ . ./cri-common
++ . ./cri-getconfig
++ umask 002
+ select_xenbranch
+ case "$branch" in
+ tree=xen
+ xenbranch=xen-unstable-smoke
+ qemuubranch=qemu-upstream-unstable
+ '[' xxen = xlinux ']'
+ linuxbranch=
+ '[' xqemu-upstream-unstable = x ']'
+ select_prevxenbranch
++ ./cri-getprevxenbranch xen-unstable-smoke
+ prevxenbranch=xen-4.9-testing
+ '[' xd5f6383d9a0316a37fb3f05a23d4cce936a262b3 = x ']'
+ : tested/2.6.39.x
+ . ./ap-common
++ : osst...@xenbits.xen.org
+++ getconfig OsstestUpstream
+++ perl -e '
use Osstest;
readglobalconfig();
print $c{"OsstestUpstream"} or die $!;
'
++ :
++ : git://xenbits.xen.org/xen.git
++ : osst...@xenbits.xen.org:/home/xen/git/xen.git
++ : git://xenbits.xen.org/qemu-xen-traditional.git
++ : git://git.kernel.org
++ : git://git.kernel.org/pub/scm/linux/kernel/git
++ : git
++ : git://xenbits.xen.org/xtf.git
++ : osst...@xenbits.xen.org:/home/xen/git/xtf.git
++ : git://xenbits.xen.org/xtf.git
++ : git://xenbits.xen.org/libvirt.git
++ : osst...@xenbits.xen.org:/home/xen/git/libvirt.git
++ : git://xenbits.xen.org/libvirt.git
++ : git://xenbits.xen.org/osstest/rumprun.git
++ : git
++ : git://xenbits.xen.org/osstest/rumprun.git
++ : osst...@xenbits.xen.org:/home/xen/git/osstest/rumprun.git
++ : git://git.seabios.org/seabios.git
++ : osst...@xenbits.xen.org:/home/xen/git/osstest/seabios.git
++ : git://xenbits.xen.org/osstest/seabios.git
++ : 

Re: [Xen-devel] [RFC v2]Proposal to allow setting up shared memory areas between VMs from xl config file

[Jarvis Roach]

> -Original Message-
> From: Stefano Stabellini [mailto:sstabell...@kernel.org]
> Sent: Friday, June 23, 2017 4:09 PM
> To: Jarvis Roach 
> Cc: Stefano Stabellini ; Julien Grall
> ; Zhongze Liu ; xen-
> de...@lists.xenproject.org; Wei Liu ; Ian Jackson
> ; edg...@xilinx.com; Edgar E. Iglesias
> 
> Subject: RE: [RFC v2]Proposal to allow setting up shared memory areas
> between VMs from xl config file
> 
> On Fri, 23 Jun 2017, Jarvis Roach wrote:
> > > -Original Message-
> > > From: Stefano Stabellini [mailto:sstabell...@kernel.org]
> > > Sent: Friday, June 23, 2017 2:21 PM
> > > To: Julien Grall 
> > > Cc: Stefano Stabellini ; Zhongze Liu
> > > ; xen-de...@lists.xenproject.org; Wei Liu
> > > ; Ian Jackson ;
> > > Jarvis Roach ; edg...@xilinx.com;
> > > Edgar E. Iglesias 
> > > Subject: Re: [RFC v2]Proposal to allow setting up shared memory
> > > areas between VMs from xl config file
> > >
> > > On Fri, 23 Jun 2017, Julien Grall wrote:
> > > > Hi,
> > > >
> > > > On 22/06/17 22:05, Stefano Stabellini wrote:
> > > > > > When we encounter an id IDx during "xl create":
> > > > > >
> > > > > >   + If it’s not under /local/shared_mem:
> > > > > > + If the corresponding entry has a "master" tag, create the
> > > > > >   corresponding entries for IDx in xenstore
> > > > > > + If there isn't a "master" tag, say error.
> > > > > >
> > > > > >   + If it’s found under /local/shared_mem:
> > > > > > + If the corresponding entry has a "master" tag, say error
> > > > > > + If there isn't a "master" tag, map the pages to the newly
> > > > > >   created domain, and add the current domain and necessary
> > > information
> > > > > >   under /local/shared_mem/IDx/slaves.
> > > > >
> > > > > Aside from using "gfn" instead of gmfn everywhere, I think it
> > > > > looks pretty good.
> > > > >
> > > > > I would leave out permissions and cacheability attributes from
> > > > > this version of the work. I would just add a note saying that
> > > > > memory will be mapped as RW regular cacheable RAM. Other
> > > > > permissions and cacheability will be possible, but they are not
> implemented yet.
> > > >
> > > > Well, I think we should design the interface correctly from the
> > > > beginning to facilitate future extension.
> > >
> > > Which interface are you speaking about?
> > >
> > > I don't think we should attemp to write how the hypercall interface
> > > might look like in the future to support setting permissions and
> > > cacheability attributes.
> > >
> > >
> > > > Also, you need to clarify what you mean by "regular cacheable RAM".
> > > > Are they write-through, write-back...? But, on ARM, this would
> > > > only be the caching attribute in stage-2 page table. The final
> > > > caching, memory type, shareability would be a combination of stage-2
> and stage-1 attributes.
> > >
> > > The very same that is used today for the ram of virtual machines, do
> > > we need to say any more than that? (For ARM, p2m_ram_rw and
> > > MATTR_MEM, LPAE_SH_INNER. For stage1, we should refer to
> > > xen/include/public/arch-arm.h.)
> >
> > I have customers who need some buffers LPAE_SH_OUTER and others
> who need NORMAL non-cacheable or inner-cacheable buffers, so my
> suggestion is to provide a way to support the full combination of
> configurations.
> >
> > While the stage 1/stage 2 combination results allow guests (via the stage 1
> translation regime) to force the two combinations I specifically mentioned,  
> in
> the first case the customers want LPAE_SH_OUTER for cache coherency with
> a DMA-capable I/O device. In that case, Xen needs to set the shareability
> attribute to OUTER in the stage 2 table since that's what is used for the
> SMMU. In the second case,  NORMAL non-cacheable or inner-cacheable, the
> customers are in a position where they can't trust the guests to disable their
> cache or set it for inner-cacheable, so it would be good for a way to Xen or
> privileged/trusted domain to do so.
> 
> Let me premise that I would be happy to see the whole set of configurations
> implemented in the long run, we might just not get there on day1. We could
> spec out how the VM config option should look like, but leave the
> cacheability and shareability parameteres unimplemented for now (also to
> address Julien't comment on defining future proof interfaces).
> 
> I understand the need for cache-coherent buffers for dma to/from devices,
> but I think that problem should be solved with the iomem config option. This
> project was meant to setup shared memory regions for VM-to-VM
> communications. It doesn't look like that is the kind of requirement that 

Re: [Xen-devel] [PATCH v4 8/9] arm/mem_access: Add short-descriptor based gpt

[Julien Grall]

On 06/23/2017 08:09 PM, Sergej Proskurin wrote:

Hi Julien,


Hi Sergej,



[...]



Looking at the code, I see very limited point of having the offsets
array as you don't use a loop and also use each offset in a single place.


+((paddr_t)(gva >> 20) & ((1ULL << (12 - n)) - 1)),



Don't you think it is more readable to have the GVA offsets at one
place. Also, the functionality is in this way similar to the one shown
in guest_walk_ld. In my opinion, it is more intuitive to have both
functions work in a similar way. I suggest keeping the array, however
using GENMASK to generate it (as you mentioned in your comments below).


I disagree here. The way to lookup short-descriptor and long-descriptor 
are very different, for instance you don't have a loop as each level 
handle the offset differently. See the way you play will the first level 
table offsets.


+paddr = (ttbr & ~mask) | (offsets[level] << 2);

[...]

+mask = ((1ULL << 10) - 1);
+pte = table[offsets[level] & mask];
+

If you find a way to avoiding playing with the offsets another time, 
then maybe it is worth it.




const vaddr_t offsets[2] = {
 (gva & GENMASK((31 - n), 20)) >> 20,
 (gva & GENMASK(19, 12)) >> 12,
};



Furthermore, it would be clearer if you first mask then shift. As it
helps to understand the code.

If you use GENMASK (or GENMASK_ULL if you don't extend GENMASK), this
will make everything more obvious:



[...]



+
+/*
+ * As we have considered up to 2 MSBs of the GVA for mapping the
first
+ * level translation table, we need to make sure that we limit
the table
+ * offset that is is indexed by GVA<31-n:20> to max 10 bits to avoid
+ * exceeding the page size limit.
+ */
+mask = ((1ULL << 10) - 1);
+pte = table[offsets[level] & mask];


This looks quite complex for just reading a pte. I think it would be
worth to leverage the vgic_guest_access_memory for that (same in LPAE).
It would also add safety if the offsets the table is mistakenly computed
(from the current code, I can't convince myself the offset will always
be right).


As far as I understand, we would still need to use the same offsets even
if we used vgic_access_guest_memory. Also, the only significant
difference between my implementation and vgic_access_guest_memory is
that gic_access_guest_memory checks whether we write over page
boundaries, which is quite hard to achieve as the offsets are limited in
number so that they don't cross page boundaries.


+/* Make sure that we consider the bits ttbr<12:14-n> if n > 2. */
+mask = ((1ULL << 12) - 1) & ~((1ULL << (14 - n)) - 1);
+table = (short_desc_t *)((unsigned long)table | (unsigned 
long)(ttbr & mask));


[...]

+mask = ((1ULL << 10) - 1);
+pte = table[offsets[level] & mask];

Are you able to prove me this will never cross a page boundary? 
Personally, when I read that I cannot convince myself that this will 
never cross happen. This is guest memory mapped, so if there is any 
coding error, you may access unmapped memory and then DOS Xen. Not very 
nice :).


We have a function that read/write into guest memory with all safety 
check associated to it. We should take advantage of any helpers that 
will mitigate any potential miscalculations as you would just the wrong 
IPA. It is better than a DOS and also avoid open-coding so it is much 
easier to update any change in the way we access the guest memory.


Cheers,

--
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC v2]Proposal to allow setting up shared memory areas between VMs from xl config file

[Julien Grall]

Hi Stefano,

On 06/23/2017 07:21 PM, Stefano Stabellini wrote:

On Fri, 23 Jun 2017, Julien Grall wrote:

Hi,

On 22/06/17 22:05, Stefano Stabellini wrote:

When we encounter an id IDx during "xl create":

   + If it’s not under /local/shared_mem:
 + If the corresponding entry has a "master" tag, create the
   corresponding entries for IDx in xenstore
 + If there isn't a "master" tag, say error.

   + If it’s found under /local/shared_mem:
 + If the corresponding entry has a "master" tag, say error
 + If there isn't a "master" tag, map the pages to the newly
   created domain, and add the current domain and necessary information
   under /local/shared_mem/IDx/slaves.


Aside from using "gfn" instead of gmfn everywhere, I think it looks
pretty good.

I would leave out permissions and cacheability attributes from this
version of the work. I would just add a note saying that memory will be
mapped as RW regular cacheable RAM. Other permissions and cacheability
will be possible, but they are not implemented yet.


Well, I think we should design the interface correctly from the beginning to
facilitate future extension.


Which interface are you speaking about?


The interface with the user, i.e libxl and xl. The hypercall can be 
added later if necessary as this could be a DOMCTL so not part of a 
stable ABI.




I don't think we should attemp to write how the hypercall interface
might look like in the future to support setting permissions and
cacheability attributes.



Also, you need to clarify what you mean by "regular cacheable RAM". Are they
write-through, write-back...? But, on ARM, this would only be the caching
attribute in stage-2 page table. The final caching, memory type, shareability
would be a combination of stage-2 and stage-1 attributes.


The very same that is used today for the ram of virtual machines, do we
need to say any more than that? (For ARM, p2m_ram_rw and MATTR_MEM,
LPAE_SH_INNER. For stage1, we should refer to
xen/include/public/arch-arm.h.)


 * All memory which is shared with other entities in the system
 * (including the hypervisor and other guests) must reside in memory
 * which is mapped as Normal Inner-cacheable. This applies to:
 *  - hypercall arguments passed via a pointer to guest memory.
 *  - memory shared via the grant table mechanism (including PV I/O
 *rings etc).
 *  - memory shared with the hypervisor (struct shared_info, struct
 *
 * Any Inner cache allocation strategy (Write-Back, Write-Through etc)
 * is acceptable. There is no restriction on the Outer-cacheability.

This does not include memory shared between guest via other method than 
grant-table. So the documentation should be at least updated.


But AFAICT, this does not say anything about the shareability of the 
region. It only speaks about outer-cache and inner-cache.


Cheers,

--
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC v2]Proposal to allow setting up shared memory areas between VMs from xl config file

[Stefano Stabellini]

On Fri, 23 Jun 2017, Jarvis Roach wrote:
> > -Original Message-
> > From: Stefano Stabellini [mailto:sstabell...@kernel.org]
> > Sent: Friday, June 23, 2017 2:21 PM
> > To: Julien Grall 
> > Cc: Stefano Stabellini ; Zhongze Liu
> > ; xen-de...@lists.xenproject.org; Wei Liu
> > ; Ian Jackson ; Jarvis Roach
> > ; edg...@xilinx.com; Edgar E. Iglesias
> > 
> > Subject: Re: [RFC v2]Proposal to allow setting up shared memory areas
> > between VMs from xl config file
> > 
> > On Fri, 23 Jun 2017, Julien Grall wrote:
> > > Hi,
> > >
> > > On 22/06/17 22:05, Stefano Stabellini wrote:
> > > > > When we encounter an id IDx during "xl create":
> > > > >
> > > > >   + If it’s not under /local/shared_mem:
> > > > > + If the corresponding entry has a "master" tag, create the
> > > > >   corresponding entries for IDx in xenstore
> > > > > + If there isn't a "master" tag, say error.
> > > > >
> > > > >   + If it’s found under /local/shared_mem:
> > > > > + If the corresponding entry has a "master" tag, say error
> > > > > + If there isn't a "master" tag, map the pages to the newly
> > > > >   created domain, and add the current domain and necessary
> > information
> > > > >   under /local/shared_mem/IDx/slaves.
> > > >
> > > > Aside from using "gfn" instead of gmfn everywhere, I think it looks
> > > > pretty good.
> > > >
> > > > I would leave out permissions and cacheability attributes from this
> > > > version of the work. I would just add a note saying that memory will
> > > > be mapped as RW regular cacheable RAM. Other permissions and
> > > > cacheability will be possible, but they are not implemented yet.
> > >
> > > Well, I think we should design the interface correctly from the
> > > beginning to facilitate future extension.
> > 
> > Which interface are you speaking about?
> > 
> > I don't think we should attemp to write how the hypercall interface might
> > look like in the future to support setting permissions and cacheability
> > attributes.
> > 
> > 
> > > Also, you need to clarify what you mean by "regular cacheable RAM".
> > > Are they write-through, write-back...? But, on ARM, this would only be
> > > the caching attribute in stage-2 page table. The final caching, memory
> > > type, shareability would be a combination of stage-2 and stage-1 
> > > attributes.
> > 
> > The very same that is used today for the ram of virtual machines, do we need
> > to say any more than that? (For ARM, p2m_ram_rw and MATTR_MEM,
> > LPAE_SH_INNER. For stage1, we should refer to
> > xen/include/public/arch-arm.h.)
> 
> I have customers who need some buffers LPAE_SH_OUTER and others who need 
> NORMAL non-cacheable or inner-cacheable buffers, so my suggestion is to 
> provide a way to support the full combination of configurations. 
> 
> While the stage 1/stage 2 combination results allow guests (via the stage 1 
> translation regime) to force the two combinations I specifically mentioned,  
> in the first case the customers want LPAE_SH_OUTER for cache coherency with a 
> DMA-capable I/O device. In that case, Xen needs to set the shareability 
> attribute to OUTER in the stage 2 table since that's what is used for the 
> SMMU. In the second case,  NORMAL non-cacheable or inner-cacheable, the 
> customers are in a position where they can't trust the guests to disable 
> their cache or set it for inner-cacheable, so it would be good for a way to 
> Xen or privileged/trusted domain to do so.

Let me premise that I would be happy to see the whole set of
configurations implemented in the long run, we might just not get there
on day1. We could spec out how the VM config option should look like,
but leave the cacheability and shareability parameteres unimplemented
for now (also to address Julien't comment on defining future proof
interfaces).

I understand the need for cache-coherent buffers for dma to/from
devices, but I think that problem should be solved with the iomem config
option. This project was meant to setup shared memory regions for
VM-to-VM communications. It doesn't look like that is the kind of
requirement that this framework is meant to meet, unless I am missing
something?

Normal non-cacheable buffers are more interesting: do you actually see
guests running on non-cacheable memory? If not, could you make an
example of a use-case for two VMs sharing a non-cacheable page?___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH 04/17 v5] xen/arm: vpl011: Add SBSA UART emulation in Xen

[Julien Grall]

Hi,

On 06/23/2017 07:28 PM, Stefano Stabellini wrote:

On Fri, 23 Jun 2017, Julien Grall wrote:

Hi Stefano,

On 22/06/17 23:53, Stefano Stabellini wrote:

On Thu, 22 Jun 2017, Bhupinder Thakur wrote:

+static void vpl011_write_data(struct domain *d, uint8_t data)
+{
+unsigned long flags;
+struct vpl011 *vpl011 = >arch.vpl011;
+struct xencons_interface *intf = vpl011->ring_buf;
+XENCONS_RING_IDX out_cons, out_prod;
+
+VPL011_LOCK(d, flags);
+
+out_cons = intf->out_cons;
+out_prod = intf->out_prod;
+
+smp_rmb();


This should be
smp_mb();


To speed up discussion, it would have been nice to give a bit more details why
you think smp_rmb() is not enough.

In this case, I think smp_rmb() is fine because all the write we care depends
on out_cons and out_prod. So the processor cannot re-order it.


We discussed these barriers at length when I published the pvcalls and
xen 9pfs protocols, see for example
alpine.DEB.2.10.1612021318340.2777@sstabellini-ThinkPad-X260. Please
refer to "Ring Usage" in docs/misc/9pfs.markdown and "Workflow" in
docs/misc/pvcalls.markdown. I would like to keep them consistent across
protocols (the console protocol works exactly like pvcalls and xen 9pfs
in that respect).


None of the people involved in this series were CCed on this thread and 
looking at docs/misc/9pfs.markdown or docs/misc/pvcalls.markdown to 
check what barrier usage for all the PV protocols seem a bit odd...


Anyway, you should probably think of writing a common PV document to 
avoid similar question in the future.


Cheers,

--
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] live migration of HVM domUs with more than 32vcpus fails

[Ankur Arora]

On 2017-06-23 10:03 AM, Boris Ostrovsky wrote:



On 06/23/2017 12:54 PM, Olaf Hering wrote:

On Thu, Jun 22, Boris Ostrovsky wrote:


They are queued for 4.13.
git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip.git for-linus-4.13


This works for me. Thanks.

I assume there is no ready-to-pull variant for linux-4.4.x?
nm.



Not that I am aware of. Unless Ankur backported them to OL which is 
(loosely) based on 4.1.

Sorry, no haven't backported this series at all.

Ankur




-boris

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC v2]Proposal to allow setting up shared memory areas between VMs from xl config file

[Jarvis Roach]

> -Original Message-
> From: Stefano Stabellini [mailto:sstabell...@kernel.org]
> Sent: Friday, June 23, 2017 2:21 PM
> To: Julien Grall 
> Cc: Stefano Stabellini ; Zhongze Liu
> ; xen-de...@lists.xenproject.org; Wei Liu
> ; Ian Jackson ; Jarvis Roach
> ; edg...@xilinx.com; Edgar E. Iglesias
> 
> Subject: Re: [RFC v2]Proposal to allow setting up shared memory areas
> between VMs from xl config file
> 
> On Fri, 23 Jun 2017, Julien Grall wrote:
> > Hi,
> >
> > On 22/06/17 22:05, Stefano Stabellini wrote:
> > > > When we encounter an id IDx during "xl create":
> > > >
> > > >   + If it’s not under /local/shared_mem:
> > > > + If the corresponding entry has a "master" tag, create the
> > > >   corresponding entries for IDx in xenstore
> > > > + If there isn't a "master" tag, say error.
> > > >
> > > >   + If it’s found under /local/shared_mem:
> > > > + If the corresponding entry has a "master" tag, say error
> > > > + If there isn't a "master" tag, map the pages to the newly
> > > >   created domain, and add the current domain and necessary
> information
> > > >   under /local/shared_mem/IDx/slaves.
> > >
> > > Aside from using "gfn" instead of gmfn everywhere, I think it looks
> > > pretty good.
> > >
> > > I would leave out permissions and cacheability attributes from this
> > > version of the work. I would just add a note saying that memory will
> > > be mapped as RW regular cacheable RAM. Other permissions and
> > > cacheability will be possible, but they are not implemented yet.
> >
> > Well, I think we should design the interface correctly from the
> > beginning to facilitate future extension.
> 
> Which interface are you speaking about?
> 
> I don't think we should attemp to write how the hypercall interface might
> look like in the future to support setting permissions and cacheability
> attributes.
> 
> 
> > Also, you need to clarify what you mean by "regular cacheable RAM".
> > Are they write-through, write-back...? But, on ARM, this would only be
> > the caching attribute in stage-2 page table. The final caching, memory
> > type, shareability would be a combination of stage-2 and stage-1 attributes.
> 
> The very same that is used today for the ram of virtual machines, do we need
> to say any more than that? (For ARM, p2m_ram_rw and MATTR_MEM,
> LPAE_SH_INNER. For stage1, we should refer to
> xen/include/public/arch-arm.h.)

I have customers who need some buffers LPAE_SH_OUTER and others who need NORMAL 
non-cacheable or inner-cacheable buffers, so my suggestion is to provide a way 
to support the full combination of configurations. 

While the stage 1/stage 2 combination results allow guests (via the stage 1 
translation regime) to force the two combinations I specifically mentioned,  in 
the first case the customers want LPAE_SH_OUTER for cache coherency with a 
DMA-capable I/O device. In that case, Xen needs to set the shareability 
attribute to OUTER in the stage 2 table since that's what is used for the SMMU. 
In the second case,  NORMAL non-cacheable or inner-cacheable, the customers are 
in a position where they can't trust the guests to disable their cache or set 
it for inner-cacheable, so it would be good for a way to Xen or 
privileged/trusted domain to do so.




___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 8/9] arm/mem_access: Add short-descriptor based gpt

[Sergej Proskurin]

Hi Julien,

[...]

> 
> Looking at the code, I see very limited point of having the offsets
> array as you don't use a loop and also use each offset in a single place.
> 
>> +((paddr_t)(gva >> 20) & ((1ULL << (12 - n)) - 1)),
> 
Don't you think it is more readable to have the GVA offsets at one
place. Also, the functionality is in this way similar to the one shown
in guest_walk_ld. In my opinion, it is more intuitive to have both
functions work in a similar way. I suggest keeping the array, however
using GENMASK to generate it (as you mentioned in your comments below).

const vaddr_t offsets[2] = {
(gva & GENMASK((31 - n), 20)) >> 20,
(gva & GENMASK(19, 12)) >> 12,
};

> 
> Furthermore, it would be clearer if you first mask then shift. As it
> helps to understand the code.
> 
> If you use GENMASK (or GENMASK_ULL if you don't extend GENMASK), this
> will make everything more obvious:
> 

[...]


>> +
>> +/*
>> + * As we have considered up to 2 MSBs of the GVA for mapping the
>> first
>> + * level translation table, we need to make sure that we limit
>> the table
>> + * offset that is is indexed by GVA<31-n:20> to max 10 bits to avoid
>> + * exceeding the page size limit.
>> + */
>> +mask = ((1ULL << 10) - 1);
>> +pte = table[offsets[level] & mask];
> 
> This looks quite complex for just reading a pte. I think it would be
> worth to leverage the vgic_guest_access_memory for that (same in LPAE).
> It would also add safety if the offsets the table is mistakenly computed
> (from the current code, I can't convince myself the offset will always
> be right).

As far as I understand, we would still need to use the same offsets even
if we used vgic_access_guest_memory. Also, the only significant
difference between my implementation and vgic_access_guest_memory is
that gic_access_guest_memory checks whether we write over page
boundaries, which is quite hard to achieve as the offsets are limited in
number so that they don't cross page boundaries.

Yet, if you insist, I will try to incorporate vgic_access_guest_memory
into my implementation.

Thanks,
~Sergej



___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH] xen/disk: don't leak stack data via response ring

[Stefano Stabellini]

On Fri, 23 Jun 2017, Jan Beulich wrote:
> >>> On 22.06.17 at 20:52,  wrote:
> > On Thu, 22 Jun 2017, Jan Beulich wrote:
> >> >>> On 21.06.17 at 20:46,  wrote:
> >> > On Wed, 21 Jun 2017, Jan Beulich wrote:
> >> >> >>> On 20.06.17 at 23:48,  wrote:
> >> >> > On Tue, 20 Jun 2017, Jan Beulich wrote:
> >> >> >> @@ -36,13 +33,7 @@ struct blkif_x86_32_request_discard {
> >> >> >>  blkif_sector_t sector_number;/* start sector idx on disk 
> >> >> >> (r/w 
> >> > only)  */
> >> >> >>  uint64_t   nr_sectors;   /* # of contiguous sectors to 
> >> >> >> discard 
> >> >   */
> >> >> >>  };
> >> >> >> -struct blkif_x86_32_response {
> >> >> >> -uint64_tid;  /* copied from request */
> >> >> >> -uint8_t operation;   /* copied from request */
> >> >> >> -int16_t status;  /* BLKIF_RSP_???   */
> >> >> >> -};
> >> >> >>  typedef struct blkif_x86_32_request blkif_x86_32_request_t;
> >> >> >> -typedef struct blkif_x86_32_response blkif_x86_32_response_t;
> >> >> >>  #pragma pack(pop)
> >> >> >>  
> >> >> >>  /* x86_64 protocol version */
> >> >> >> @@ -62,20 +53,14 @@ struct blkif_x86_64_request_discard {
> >> >> >>  blkif_sector_t sector_number;/* start sector idx on disk 
> >> >> >> (r/w 
> >> > only)  */
> >> >> >>  uint64_t   nr_sectors;   /* # of contiguous sectors to 
> >> >> >> discard 
> >> >   */
> >> >> >>  };
> >> >> >> -struct blkif_x86_64_response {
> >> >> >> -uint64_t   __attribute__((__aligned__(8))) id;
> >> >> >> -uint8_t operation;   /* copied from request */
> >> >> >> -int16_t status;  /* BLKIF_RSP_???   */
> >> >> >> -};
> >> >> >>
> >> >> >>  typedef struct blkif_x86_64_request blkif_x86_64_request_t;
> >> >> >> -typedef struct blkif_x86_64_response blkif_x86_64_response_t;
> >> >> >>  
> >> >> >>  DEFINE_RING_TYPES(blkif_common, struct blkif_common_request,
> >> >> >> -  struct blkif_common_response);
> >> >> >> +  struct blkif_response);
> >> >> >>  DEFINE_RING_TYPES(blkif_x86_32, struct blkif_x86_32_request,
> >> >> >> -  struct blkif_x86_32_response);
> >> >> >> +  struct blkif_response QEMU_PACKED);
> >> >> > 
> >> >> > In my test, the previous sizes and alignments of the response structs
> >> >> > were (on both x86_32 and x86_64):
> >> >> > 
> >> >> > sizeof(blkif_x86_32_response)=12   sizeof(blkif_x86_64_response)=16
> >> >> > align(blkif_x86_32_response)=4 align(blkif_x86_64_response)=8
> >> >> > 
> >> >> > While with these changes are now, when compiled on x86_64:
> >> >> > sizeof(blkif_x86_32_response)=11   sizeof(blkif_x86_64_response)=16
> >> >> > align(blkif_x86_32_response)=1 align(blkif_x86_64_response)=8
> >> >> > 
> >> >> > when compiled on x86_32:
> >> >> > sizeof(blkif_x86_32_response)=11   sizeof(blkif_x86_64_response)=12
> >> >> > align(blkif_x86_32_response)=1 align(blkif_x86_64_response)=4
> >> >> > 
> >> >> > Did I do my tests wrong?
> >> >> > 
> >> >> > QEMU_PACKED is not the same as #pragma pack(push, 4). In fact, it is 
> >> >> > the
> >> >> > same as #pragma pack(push, 1), causing the struct to be densely 
> >> >> > packed,
> >> >> > leaving no padding whatsever.
> >> >> > 
> >> >> > In addition, without __attribute__((__aligned__(8))),
> >> >> > blkif_x86_64_response won't be 8 bytes aligned when built on x86_32.
> >> >> > 
> >> >> > Am I missing something?
> >> >> 
> >> >> Well, you're mixing attribute application upon structure
> >> >> declaration with attribute application upon structure use. It's
> >> >> the latter here, and hence the attribute doesn't affect
> >> >> structure layout at all. All it does is avoid the _containing_
> >> >> 32-bit union to become 8-byte aligned (and tail padding to be
> >> >> inserted).
> >> > 
> >> > Thanks for the explanation. I admit it's the first time I see the
> >> > aligned attribute being used at structure usage only. I think it's the
> >> > first time QEMU_PACKED is used this way in QEMU too.
> >> > 
> >> > Anyway, even taking that into account, things are still not completely
> >> > right: the alignment of struct blkif_x86_32_response QEMU_PACKED is 4
> >> > bytes as you wrote, but the size of struct blkif_x86_32_response is
> >> > still 16 bytes instead of 12 bytes in my test. I suspect it worked for
> >> > you because the other member of the union (blkif_x86_32_request) is
> >> > larger than that. However, I think is not a good idea to rely on this
> >> > implementation detail. The implementation of DEFINE_RING_TYPES should be
> >> > opaque from our point of view. We shouldn't have to know that there is a
> >> > union there.
> >> 
> >> I don't follow - why should we not rely on this? It is a fundamental
> >> aspect of the shared ring model that requests and responses share
> >> space.
> >> 
> >> > Moreover, the other problem is still 

Re: [Xen-devel] [PATCH 04/17 v5] xen/arm: vpl011: Add SBSA UART emulation in Xen

[Stefano Stabellini]

On Fri, 23 Jun 2017, Julien Grall wrote:
> Hi Stefano,
> 
> On 22/06/17 23:53, Stefano Stabellini wrote:
> > On Thu, 22 Jun 2017, Bhupinder Thakur wrote:
> > > +static void vpl011_write_data(struct domain *d, uint8_t data)
> > > +{
> > > +unsigned long flags;
> > > +struct vpl011 *vpl011 = >arch.vpl011;
> > > +struct xencons_interface *intf = vpl011->ring_buf;
> > > +XENCONS_RING_IDX out_cons, out_prod;
> > > +
> > > +VPL011_LOCK(d, flags);
> > > +
> > > +out_cons = intf->out_cons;
> > > +out_prod = intf->out_prod;
> > > +
> > > +smp_rmb();
> > 
> > This should be
> >smp_mb();
> 
> To speed up discussion, it would have been nice to give a bit more details why
> you think smp_rmb() is not enough.
> 
> In this case, I think smp_rmb() is fine because all the write we care depends
> on out_cons and out_prod. So the processor cannot re-order it.

We discussed these barriers at length when I published the pvcalls and
xen 9pfs protocols, see for example
alpine.DEB.2.10.1612021318340.2777@sstabellini-ThinkPad-X260. Please
refer to "Ring Usage" in docs/misc/9pfs.markdown and "Workflow" in
docs/misc/pvcalls.markdown. I would like to keep them consistent across
protocols (the console protocol works exactly like pvcalls and xen 9pfs
in that respect).


> > > +/*
> > > + * It is expected that the ring is not full when this function is
> > > called
> > > + * as the guest is expected to write to the data register only when
> > > the
> > > + * TXFF flag is not set.
> > > + * In case the guest does write even when the TXFF flag is set then
> > > the
> > > + * data will be silently dropped.
> > > + */
> > > +if ( xencons_queued(out_prod, out_cons, sizeof(intf->out)) !=
> > > + sizeof (intf->out) )
> > > +{
> > > +intf->out[xencons_mask(out_prod, sizeof(intf->out))] = data;
> > > +out_prod += 1;
> > > +smp_wmb();
> > > +intf->out_prod = out_prod;
> > > +}
> > > +else
> > > +gprintk(XENLOG_ERR, "vpl011: Unexpected OUT ring buffer full\n");
> > > +
> > > +if ( xencons_queued(out_prod, out_cons, sizeof(intf->out)) ==
> > > + sizeof (intf->out) )
> > > +{
> > > +vpl011->uartfr |= TXFF;
> > > +vpl011->uartris &= ~TXI;
> > > +}
> > > +
> > > +vpl011->uartfr |= BUSY;
> > > +
> > > +vpl011->uartfr &= ~TXFE;
> > > +
> > > +vpl011_update_interrupt_status(d);
> > > +
> > > +VPL011_UNLOCK(d, flags);
> > > +
> > > +/*
> > > + * Send an event to console backend to indicate that there is
> > > + * data in the OUT ring buffer.
> > > + */
> > > +notify_via_xen_event_channel(d, vpl011->evtchn);
> > > +}
> 
> Cheers,
> 
> -- 
> Julien Grall
> 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v2] docs: improve ARM passthrough doc

[Stefano Stabellini]

On Fri, 23 Jun 2017, Julien Grall wrote:
> Hi,
> 
> On 21/06/17 14:10, Julien Grall wrote:
> > Hi Stefano.
> > 
> > On 21/06/17 00:04, Stefano Stabellini wrote:
> > > Add a warning: use passthrough with care.
> > > 
> > > Add a pointer to the gic device tree bindings. Add an explanation on how
> > > to calculate irq numbers from device tree.
> > > 
> > > Add a brief explanation of the reg property and a pointer to the xl docs
> > > for a description of the iomem property. Add a note that in the example
> > > we are using different memory addresses for guests and host.
> > > 
> > > Signed-off-by: Stefano Stabellini 
> > 
> > Acked-by: Julien Grall 
> 
> I would also consider this doc improvement for Xen 4.9.

Yes, done

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC v2]Proposal to allow setting up shared memory areas between VMs from xl config file

[Stefano Stabellini]

On Fri, 23 Jun 2017, Julien Grall wrote:
> Hi,
> 
> On 22/06/17 22:05, Stefano Stabellini wrote:
> > > When we encounter an id IDx during "xl create":
> > > 
> > >   + If it’s not under /local/shared_mem:
> > > + If the corresponding entry has a "master" tag, create the
> > >   corresponding entries for IDx in xenstore
> > > + If there isn't a "master" tag, say error.
> > > 
> > >   + If it’s found under /local/shared_mem:
> > > + If the corresponding entry has a "master" tag, say error
> > > + If there isn't a "master" tag, map the pages to the newly
> > >   created domain, and add the current domain and necessary information
> > >   under /local/shared_mem/IDx/slaves.
> > 
> > Aside from using "gfn" instead of gmfn everywhere, I think it looks
> > pretty good.
> > 
> > I would leave out permissions and cacheability attributes from this
> > version of the work. I would just add a note saying that memory will be
> > mapped as RW regular cacheable RAM. Other permissions and cacheability
> > will be possible, but they are not implemented yet.
> 
> Well, I think we should design the interface correctly from the beginning to
> facilitate future extension.

Which interface are you speaking about?

I don't think we should attemp to write how the hypercall interface
might look like in the future to support setting permissions and
cacheability attributes.


> Also, you need to clarify what you mean by "regular cacheable RAM". Are they
> write-through, write-back...? But, on ARM, this would only be the caching
> attribute in stage-2 page table. The final caching, memory type, shareability
> would be a combination of stage-2 and stage-1 attributes.

The very same that is used today for the ram of virtual machines, do we
need to say any more than that? (For ARM, p2m_ram_rw and MATTR_MEM,
LPAE_SH_INNER. For stage1, we should refer to
xen/include/public/arch-arm.h.)___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [xen-unstable-smoke test] 111009: tolerable trouble: broken/pass - PUSHED

[osstest service owner]

flight 111009 xen-unstable-smoke real [real]
http://logs.test-lab.xenproject.org/osstest/logs/111009/

Failures :-/ but no regressions.

Tests which did not succeed, but are not blocking:
 test-arm64-arm64-xl-xsm   1 build-check(1)   blocked  n/a
 test-amd64-amd64-libvirt 12 migrate-support-checkfail   never pass
 test-armhf-armhf-xl  12 migrate-support-checkfail   never pass
 test-armhf-armhf-xl  13 saverestore-support-checkfail   never pass

version targeted for testing:
 xen  7f919ac31993e95387cac612d0dc0bf0ac7e
baseline version:
 xen  a15516c0cf21d7ac84799f1e2e500b0bb22d2300

Last test of basis   111006  2017-06-23 14:01:19 Z0 days
Testing same since   111009  2017-06-23 16:02:07 Z0 days1 attempts


People who touched revisions under test:
  Andrew Cooper 
  Jan Beulich 
  Konrad Rzeszutek Wilk 
  Konrad Rzeszutek Wilk  [x86 and arm32]
  Razvan Cojocaru 
  Wei Liu 

jobs:
 build-amd64  pass
 build-armhf  pass
 build-amd64-libvirt  pass
 test-armhf-armhf-xl  pass
 test-arm64-arm64-xl-xsm  broken  
 test-amd64-amd64-xl-qemuu-debianhvm-i386 pass
 test-amd64-amd64-libvirt pass



sg-report-flight on osstest.test-lab.xenproject.org
logs: /home/logs/logs
images: /home/logs/images

Logs, config files, etc. are available at
http://logs.test-lab.xenproject.org/osstest/logs

Explanation of these reports, and of osstest in general, is at
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=master
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master

Test harness code can be found at
http://xenbits.xen.org/gitweb?p=osstest.git;a=summary


Pushing revision :

+ branch=xen-unstable-smoke
+ revision=7f919ac31993e95387cac612d0dc0bf0ac7e
+ . ./cri-lock-repos
++ . ./cri-common
+++ . ./cri-getconfig
+++ umask 002
+++ getrepos
 getconfig Repos
 perl -e '
use Osstest;
readglobalconfig();
print $c{"Repos"} or die $!;
'
+++ local repos=/home/osstest/repos
+++ '[' -z /home/osstest/repos ']'
+++ '[' '!' -d /home/osstest/repos ']'
+++ echo /home/osstest/repos
++ repos=/home/osstest/repos
++ repos_lock=/home/osstest/repos/lock
++ '[' x '!=' x/home/osstest/repos/lock ']'
++ OSSTEST_REPOS_LOCK_LOCKED=/home/osstest/repos/lock
++ exec with-lock-ex -w /home/osstest/repos/lock ./ap-push xen-unstable-smoke 
7f919ac31993e95387cac612d0dc0bf0ac7e
+ branch=xen-unstable-smoke
+ revision=7f919ac31993e95387cac612d0dc0bf0ac7e
+ . ./cri-lock-repos
++ . ./cri-common
+++ . ./cri-getconfig
+++ umask 002
+++ getrepos
 getconfig Repos
 perl -e '
use Osstest;
readglobalconfig();
print $c{"Repos"} or die $!;
'
+++ local repos=/home/osstest/repos
+++ '[' -z /home/osstest/repos ']'
+++ '[' '!' -d /home/osstest/repos ']'
+++ echo /home/osstest/repos
++ repos=/home/osstest/repos
++ repos_lock=/home/osstest/repos/lock
++ '[' x/home/osstest/repos/lock '!=' x/home/osstest/repos/lock ']'
+ . ./cri-common
++ . ./cri-getconfig
++ umask 002
+ select_xenbranch
+ case "$branch" in
+ tree=xen
+ xenbranch=xen-unstable-smoke
+ qemuubranch=qemu-upstream-unstable
+ '[' xxen = xlinux ']'
+ linuxbranch=
+ '[' xqemu-upstream-unstable = x ']'
+ select_prevxenbranch
++ ./cri-getprevxenbranch xen-unstable-smoke
+ prevxenbranch=xen-4.9-testing
+ '[' x7f919ac31993e95387cac612d0dc0bf0ac7e = x ']'
+ : tested/2.6.39.x
+ . ./ap-common
++ : osst...@xenbits.xen.org
+++ getconfig OsstestUpstream
+++ perl -e '
use Osstest;
readglobalconfig();
print $c{"OsstestUpstream"} or die $!;
'
++ :
++ : git://xenbits.xen.org/xen.git
++ : osst...@xenbits.xen.org:/home/xen/git/xen.git
++ : git://xenbits.xen.org/qemu-xen-traditional.git
++ : git://git.kernel.org
++ : git://git.kernel.org/pub/scm/linux/kernel/git
++ : git
++ : git://xenbits.xen.org/xtf.git
++ : osst...@xenbits.xen.org:/home/xen/git/xtf.git
++ : git://xenbits.xen.org/xtf.git
++ : git://xenbits.xen.org/libvirt.git
++ : osst...@xenbits.xen.org:/home/xen/git/libvirt.git
++ : git://xenbits.xen.org/libvirt.git
++ : git://xenbits.xen.org/osstest/rumprun.git
++ : git
++ : git://xenbits.xen.org/osstest/rumprun.git
++ : osst...@xenbits.xen.org:/home/xen/git/osstest/rumprun.git
++ : git://git.seabios.org/seabios.git
++ : 

Re: [Xen-devel] [PATCH] rombios: prevent building with PIC

[Olaf Hering]

On Fri, Jun 23, Wei Liu wrote:

> We support >=4.1. Please check those as well.

Yes, 4.1.2 works as well.

Olaf


signature.asc
Description: PGP signature
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH 00/17 v5] SBSA UART emulation support in Xen

[Stefano Stabellini]

On Fri, 23 Jun 2017, Julien Grall wrote:
> Hi Bhupinder,
> 
> On 22/06/17 08:38, Bhupinder Thakur wrote:
> > There are some TBD items which need to be looked at in the future:
> > 
> > 1. Currently UEFI firmware logs the output to hvc console only. How can
> >UEFI firmware be made aware of pl011 console and how it can use it
> >as a console instead of hvc.
> 
> Would it be possible to summarize the discussions we had a couple of weeks ago
> here?
> 
> > 2. Linux seems to have hvc console as the default console i.e. if no
> >console is specified then it uses hvc as the console. How can an
> >option be provided in Linux to select either hvc or pl011 as the
> >default console.
> 
> I am wondering what would happen if you use stdout-path in the device-tree.
> Does it override the default console?
> 
> IHMO, the best way to select the default console would be using either the
> SPCR (for ACPI) or stdout-path (for DT). But the HVC console does not have any
> description in the firmware. It might be worth considering adding description.
> 
> The drawback is the user would always have to specify the console on the
> command line. I think this is not too bad for a first approach.
> 
> > 
> > 3. ACPI support for pl011 device.
> 
> I would be ok to defer this after PL011 series is merged.
> 
> > Bhupinder Thakur (17):
> >   xen/arm: vpl011: Move vgic register access functions to vreg.h
> >   xen/arm: vpl011: Rename vgic_reg* functions definitions and calls to
> > vreg_reg*
> 
> Stefano, would it be possible to commit those patches? They are both acked and
> it would avoid Bhupinder to carry them and rebase them if the vGIC code has
> changed.

Sure

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH 08/17 v5] xen/arm: vpl011: Add a new domctl API to initialize vpl011

[Stefano Stabellini]

On Fri, 23 Jun 2017, Julien Grall wrote:
> On 23/06/17 14:17, Julien Grall wrote:
> > 
> > 
> > On 23/06/17 00:04, Stefano Stabellini wrote:
> > > > diff --git a/tools/libxc/include/xenctrl.h
> > > > b/tools/libxc/include/xenctrl.h
> > > > index 1629f41..26f3d1e 100644
> > > > --- a/tools/libxc/include/xenctrl.h
> > > > +++ b/tools/libxc/include/xenctrl.h
> > > > @@ -885,6 +885,26 @@ int xc_vcpu_getcontext(xc_interface *xch,
> > > > uint32_t vcpu,
> > > > vcpu_guest_context_any_t *ctxt);
> > > > 
> > > > +#if defined (__arm__) || defined(__aarch64__)
> > > > +/**
> > > > + * This function initializes the vpl011 emulation and returns
> > > > + * the event to be used by the backend for communicating with
> > > > + * the emulation code.
> > > > + *
> > > > + * @parm xch a handle to an open hypervisor interface
> > > > + * @parm domid the domain to get information from
> > > > + * @parm console_domid the domid of the backend console
> > > > + * @parm gfn the guest pfn to be used as the ring buffer
> > > > + * @parm evtchn the event channel to be used for events
> > > > + * @return 0 on success, negative error on failure
> > > > + */
> > > > +int xc_dom_vpl011_init(xc_interface *xch,
> > > > +   uint32_t domid,
> > > > +   uint32_t console_domid,
> > > > +   xen_pfn_t gfn,
> > > > +   evtchn_port_t *evtchn);
> > > > +#endif
> > > 
> > > Actually, the pattern is to define the xc_ function on all architecture
> > > but only return ENOSYS where it's not implemented, see
> > > xc_vcpu_get_extstate.
> > 
> > Well, I think the main reason behind if to avoid dummy call to the
> > hypervisor. But effectively the hypervisor will return a proper error.
> 
> Actually, looking at the public header. This is because vcpu_get_extstate
> structure is only available on x86. Whereas vpl011_init is available for all
> the architecture even though only ARM effectively implementing it.
> 
> But my point stands below, there is no harm to implement it for x86 as it
> would compile on any platform.

Sounds good to me


> > 
> > As the call is not made in common code, I would make this function
> > compile on all the platform (there are nothing arch specific in it).
 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH] rombios: prevent building with PIC

[Olaf Hering]

On Fri, Jun 23, Wei Liu wrote:

> We support >=4.1. Please check those as well.

According to the PDF manuals at https://gcc.gnu.org/onlinedocs/ a
"-fno-foo" is mentioned, so I think -fno-pic is recognized. I will see
if I find a copy of SLE10 to verify with 4.1.2.


Olaf


signature.asc
Description: PGP signature
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v7 34/36] x86/mm: Add support to encrypt the kernel in-place

[Tom Lendacky]

On 6/23/2017 5:00 AM, Borislav Petkov wrote:

On Fri, Jun 16, 2017 at 01:56:19PM -0500, Tom Lendacky wrote:

Add the support to encrypt the kernel in-place. This is done by creating
new page mappings for the kernel - a decrypted write-protected mapping
and an encrypted mapping. The kernel is encrypted by copying it through
a temporary buffer.

Signed-off-by: Tom Lendacky 
---
  arch/x86/include/asm/mem_encrypt.h |6 +
  arch/x86/mm/Makefile   |2
  arch/x86/mm/mem_encrypt.c  |  314 
  arch/x86/mm/mem_encrypt_boot.S |  150 +
  4 files changed, 472 insertions(+)
  create mode 100644 arch/x86/mm/mem_encrypt_boot.S

diff --git a/arch/x86/include/asm/mem_encrypt.h 
b/arch/x86/include/asm/mem_encrypt.h
index af835cf..7da6de3 100644
--- a/arch/x86/include/asm/mem_encrypt.h
+++ b/arch/x86/include/asm/mem_encrypt.h
@@ -21,6 +21,12 @@
  
  extern unsigned long sme_me_mask;
  
+void sme_encrypt_execute(unsigned long encrypted_kernel_vaddr,

+unsigned long decrypted_kernel_vaddr,
+unsigned long kernel_len,
+unsigned long encryption_wa,
+unsigned long encryption_pgd);
+
  void __init sme_early_encrypt(resource_size_t paddr,
  unsigned long size);
  void __init sme_early_decrypt(resource_size_t paddr,
diff --git a/arch/x86/mm/Makefile b/arch/x86/mm/Makefile
index 9e13841..0633142 100644
--- a/arch/x86/mm/Makefile
+++ b/arch/x86/mm/Makefile
@@ -38,3 +38,5 @@ obj-$(CONFIG_NUMA_EMU)+= numa_emulation.o
  obj-$(CONFIG_X86_INTEL_MPX)   += mpx.o
  obj-$(CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS) += pkeys.o
  obj-$(CONFIG_RANDOMIZE_MEMORY) += kaslr.o
+
+obj-$(CONFIG_AMD_MEM_ENCRYPT)  += mem_encrypt_boot.o
diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c
index 842c8a6..6e87662 100644
--- a/arch/x86/mm/mem_encrypt.c
+++ b/arch/x86/mm/mem_encrypt.c
@@ -24,6 +24,8 @@
  #include 
  #include 
  #include 
+#include 
+#include 
  
  /*

   * Since SME related variables are set early in the boot process they must
@@ -209,8 +211,320 @@ void swiotlb_set_mem_attributes(void *vaddr, unsigned 
long size)
set_memory_decrypted((unsigned long)vaddr, size >> PAGE_SHIFT);
  }
  
+static void __init sme_clear_pgd(pgd_t *pgd_base, unsigned long start,

+unsigned long end)
+{
+   unsigned long pgd_start, pgd_end, pgd_size;
+   pgd_t *pgd_p;
+
+   pgd_start = start & PGDIR_MASK;
+   pgd_end = end & PGDIR_MASK;
+
+   pgd_size = (((pgd_end - pgd_start) / PGDIR_SIZE) + 1);
+   pgd_size *= sizeof(pgd_t);
+
+   pgd_p = pgd_base + pgd_index(start);
+
+   memset(pgd_p, 0, pgd_size);
+}
+
+#ifndef CONFIG_X86_5LEVEL
+#define native_make_p4d(_x)(p4d_t) { .pgd = native_make_pgd(_x) }
+#endif


Huh, why isn't this in arch/x86/include/asm/pgtable_types.h in the #else
branch of #if CONFIG_PGTABLE_LEVELS > 4 ?


Normally the __p4d() macro would be used and that would be ok whether
CONFIG_X86_5LEVEL is defined or not. But since __p4d() is part of the
paravirt ops path I have to use native_make_p4d(). I'd be the only user
of the function and thought it would be best to localize it this way.



Also

ERROR: Macros with complex values should be enclosed in parentheses
#105: FILE: arch/x86/mm/mem_encrypt.c:232:
+#define native_make_p4d(_x)(p4d_t) { .pgd = native_make_pgd(_x) }

so why isn't it a function?


I can define it as an inline function.




+
+#define PGD_FLAGS  _KERNPG_TABLE_NOENC
+#define P4D_FLAGS  _KERNPG_TABLE_NOENC
+#define PUD_FLAGS  _KERNPG_TABLE_NOENC
+#define PMD_FLAGS  (__PAGE_KERNEL_LARGE_EXEC & ~_PAGE_GLOBAL)
+
+static void __init *sme_populate_pgd(pgd_t *pgd_base, void *pgtable_area,
+unsigned long vaddr, pmdval_t pmd_val)
+{
+   pgd_t *pgd_p;
+   p4d_t *p4d_p;
+   pud_t *pud_p;
+   pmd_t *pmd_p;
+
+   pgd_p = pgd_base + pgd_index(vaddr);
+   if (native_pgd_val(*pgd_p)) {
+   if (IS_ENABLED(CONFIG_X86_5LEVEL))


Err, I don't understand: so this is a Kconfig symbol and when it is
enabled at build time, you do a 5level pagetable.

But you can't stick a 5level pagetable to a hardware which doesn't know
about it.


True, 5-level will only be turned on for specific hardware which is why
I originally had this as only 4-level pagetables. But in a comment from
you back on the v5 version you said it needed to support 5-level. I
guess we should have discussed this more, but I also thought that should
our hardware ever support 5-level paging in the future then this would
be good to go.



Or do you mean that p4d layer folding at runtime to happen? (I admit, I
haven't looked at that in detail.) But then I'd hope that the generic
macros/functions would give you the ability to not care whether we have
a p4d or not and not add a whole 

[Xen-devel] [ovmf test] 110988: all pass - PUSHED

[osstest service owner]

flight 110988 ovmf real [real]
http://logs.test-lab.xenproject.org/osstest/logs/110988/

Perfect :-)
All tests in this flight passed as required
version targeted for testing:
 ovmf c01f13d52a85f097e1cc6b194df1316a3ed24710
baseline version:
 ovmf 95b5c32fb3d6bf677d4cb6471d6c683939014c89

Last test of basis   110965  2017-06-22 09:31:38 Z1 days
Testing same since   110988  2017-06-23 07:16:18 Z0 days1 attempts


People who touched revisions under test:
  Ard Biesheuvel 
  Fu Siyuan 
  Star Zeng 
  Yonghong Zhu 

jobs:
 build-amd64-xsm  pass
 build-i386-xsm   pass
 build-amd64  pass
 build-i386   pass
 build-amd64-libvirt  pass
 build-i386-libvirt   pass
 build-amd64-pvopspass
 build-i386-pvops pass
 test-amd64-amd64-xl-qemuu-ovmf-amd64 pass
 test-amd64-i386-xl-qemuu-ovmf-amd64  pass



sg-report-flight on osstest.test-lab.xenproject.org
logs: /home/logs/logs
images: /home/logs/images

Logs, config files, etc. are available at
http://logs.test-lab.xenproject.org/osstest/logs

Explanation of these reports, and of osstest in general, is at
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=master
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master

Test harness code can be found at
http://xenbits.xen.org/gitweb?p=osstest.git;a=summary


Pushing revision :

+ branch=ovmf
+ revision=c01f13d52a85f097e1cc6b194df1316a3ed24710
+ . ./cri-lock-repos
++ . ./cri-common
+++ . ./cri-getconfig
+++ umask 002
+++ getrepos
 getconfig Repos
 perl -e '
use Osstest;
readglobalconfig();
print $c{"Repos"} or die $!;
'
+++ local repos=/home/osstest/repos
+++ '[' -z /home/osstest/repos ']'
+++ '[' '!' -d /home/osstest/repos ']'
+++ echo /home/osstest/repos
++ repos=/home/osstest/repos
++ repos_lock=/home/osstest/repos/lock
++ '[' x '!=' x/home/osstest/repos/lock ']'
++ OSSTEST_REPOS_LOCK_LOCKED=/home/osstest/repos/lock
++ exec with-lock-ex -w /home/osstest/repos/lock ./ap-push ovmf 
c01f13d52a85f097e1cc6b194df1316a3ed24710
+ branch=ovmf
+ revision=c01f13d52a85f097e1cc6b194df1316a3ed24710
+ . ./cri-lock-repos
++ . ./cri-common
+++ . ./cri-getconfig
+++ umask 002
+++ getrepos
 getconfig Repos
 perl -e '
use Osstest;
readglobalconfig();
print $c{"Repos"} or die $!;
'
+++ local repos=/home/osstest/repos
+++ '[' -z /home/osstest/repos ']'
+++ '[' '!' -d /home/osstest/repos ']'
+++ echo /home/osstest/repos
++ repos=/home/osstest/repos
++ repos_lock=/home/osstest/repos/lock
++ '[' x/home/osstest/repos/lock '!=' x/home/osstest/repos/lock ']'
+ . ./cri-common
++ . ./cri-getconfig
++ umask 002
+ select_xenbranch
+ case "$branch" in
+ tree=ovmf
+ xenbranch=xen-unstable
+ '[' xovmf = xlinux ']'
+ linuxbranch=
+ '[' x = x ']'
+ qemuubranch=qemu-upstream-unstable
+ select_prevxenbranch
++ ./cri-getprevxenbranch xen-unstable
+ prevxenbranch=xen-4.9-testing
+ '[' xc01f13d52a85f097e1cc6b194df1316a3ed24710 = x ']'
+ : tested/2.6.39.x
+ . ./ap-common
++ : osst...@xenbits.xen.org
+++ getconfig OsstestUpstream
+++ perl -e '
use Osstest;
readglobalconfig();
print $c{"OsstestUpstream"} or die $!;
'
++ :
++ : git://xenbits.xen.org/xen.git
++ : osst...@xenbits.xen.org:/home/xen/git/xen.git
++ : git://xenbits.xen.org/qemu-xen-traditional.git
++ : git://git.kernel.org
++ : git://git.kernel.org/pub/scm/linux/kernel/git
++ : git
++ : git://xenbits.xen.org/xtf.git
++ : osst...@xenbits.xen.org:/home/xen/git/xtf.git
++ : git://xenbits.xen.org/xtf.git
++ : git://xenbits.xen.org/libvirt.git
++ : osst...@xenbits.xen.org:/home/xen/git/libvirt.git
++ : git://xenbits.xen.org/libvirt.git
++ : git://xenbits.xen.org/osstest/rumprun.git
++ : git
++ : git://xenbits.xen.org/osstest/rumprun.git
++ : osst...@xenbits.xen.org:/home/xen/git/osstest/rumprun.git
++ : git://git.seabios.org/seabios.git
++ : osst...@xenbits.xen.org:/home/xen/git/osstest/seabios.git
++ : git://xenbits.xen.org/osstest/seabios.git
++ : https://github.com/tianocore/edk2.git
++ : osst...@xenbits.xen.org:/home/xen/git/osstest/ovmf.git
++ : git://xenbits.xen.org/osstest/ovmf.git
++ : git://xenbits.xen.org/osstest/linux-firmware.git
++ : 

Re: [Xen-devel] [PATCH] rombios: prevent building with PIC

[Wei Liu]

On Fri, Jun 23, 2017 at 07:40:36PM +0200, Olaf Hering wrote:
> On Fri, Jun 23, Wei Liu wrote:
> 
> > Do you need to check if the compiler supports -fno-pic?
> 
> In my testing gcc-4.3 and 4.5 know about this option.
> 

We support >=4.1. Please check those as well.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH] rombios: prevent building with PIC

[Olaf Hering]

On Fri, Jun 23, Wei Liu wrote:

> Do you need to check if the compiler supports -fno-pic?

In my testing gcc-4.3 and 4.5 know about this option.

Olaf


signature.asc
Description: PGP signature
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v7 36/36] x86/mm: Add support to make use of Secure Memory Encryption

[Borislav Petkov]

On Fri, Jun 16, 2017 at 01:56:39PM -0500, Tom Lendacky wrote:
> Add support to check if SME has been enabled and if memory encryption
> should be activated (checking of command line option based on the
> configuration of the default state).  If memory encryption is to be
> activated, then the encryption mask is set and the kernel is encrypted
> "in place."
> 
> Signed-off-by: Tom Lendacky 
> ---
>  arch/x86/include/asm/mem_encrypt.h |6 ++-
>  arch/x86/kernel/head64.c   |4 +-
>  arch/x86/mm/mem_encrypt.c  |   86 
> +++-
>  3 files changed, 90 insertions(+), 6 deletions(-)

...

> +/*
> + * Some SME functions run very early causing issues with the stack-protector
> + * support. Provide a way to turn off this support on a per-function basis.
> + */
> +#define SME_NOSTACKP __attribute__((__optimize__("no-stack-protector")))

__nostackp

just like the others in include/linux/compiler-gcc.h.

> +
> +static char sme_cmdline_arg[] __initdata = "mem_encrypt";
> +static char sme_cmdline_on[]  __initdata = "on";
> +static char sme_cmdline_off[] __initdata = "off";
>  
>  /*
>   * Since SME related variables are set early in the boot process they must
> @@ -200,6 +215,8 @@ void __init mem_encrypt_init(void)
>  
>   /* Call into SWIOTLB to update the SWIOTLB DMA buffers */
>   swiotlb_update_mem_attributes();
> +
> + pr_info("AMD Secure Memory Encryption (SME) active\n");
>  }
>  
>  void swiotlb_set_mem_attributes(void *vaddr, unsigned long size)
> @@ -527,8 +544,73 @@ void __init sme_encrypt_kernel(void)
>   native_write_cr3(__native_read_cr3());
>  }
>  
> -void __init sme_enable(void)
> +void __init SME_NOSTACKP sme_enable(struct boot_params *bp)
>  {
> + const char *cmdline_ptr, *cmdline_arg, *cmdline_on, *cmdline_off;
> + unsigned int eax, ebx, ecx, edx;
> + bool active_by_default;
> + unsigned long me_mask;
> + char buffer[16];
> + u64 msr;
> +
> + /* Check for the SME support leaf */
> + eax = 0x8000;
> + ecx = 0;
> + native_cpuid(, , , );
> + if (eax < 0x801f)
> + return;
> +
> + /*
> +  * Check for the SME feature:
> +  *   CPUID Fn8000_001F[EAX] - Bit 0
> +  * Secure Memory Encryption support
> +  *   CPUID Fn8000_001F[EBX] - Bits 5:0
> +  * Pagetable bit position used to indicate encryption
> +  */
> + eax = 0x801f;
> + ecx = 0;
> + native_cpuid(, , , );
> + if (!(eax & 1))
> + return;
> +
> + me_mask = 1UL << (ebx & 0x3f);
> +
> + /* Check if SME is enabled */
> + msr = __rdmsr(MSR_K8_SYSCFG);
> + if (!(msr & MSR_K8_SYSCFG_MEM_ENCRYPT))
> + return;
> +
> + /*
> +  * Fixups have not been applied to phys_base yet and we're running
> +  * identity mapped, so we must obtain the address to the SME command
> +  * line argument data using rip-relative addressing.
> +  */
> + asm ("lea sme_cmdline_arg(%%rip), %0"
> +  : "=r" (cmdline_arg)
> +  : "p" (sme_cmdline_arg));
> + asm ("lea sme_cmdline_on(%%rip), %0"
> +  : "=r" (cmdline_on)
> +  : "p" (sme_cmdline_on));
> + asm ("lea sme_cmdline_off(%%rip), %0"
> +  : "=r" (cmdline_off)
> +  : "p" (sme_cmdline_off));
> +
> + if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT_ACTIVE_BY_DEFAULT))
> + active_by_default = true;
> + else
> + active_by_default = false;
> +
> + cmdline_ptr = (const char *)((u64)bp->hdr.cmd_line_ptr |
> +  ((u64)bp->ext_cmd_line_ptr << 32));
> +
> + cmdline_find_option(cmdline_ptr, cmdline_arg, buffer, sizeof(buffer));
> +
> + if (strncmp(buffer, cmdline_on, sizeof(buffer)) == 0)

if (!strncmp(...

> + sme_me_mask = me_mask;
> + else if (strncmp(buffer, cmdline_off, sizeof(buffer)) == 0)

else if (!strncmp(...

-- 
Regards/Gruss,
Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [PATCH] vtpmmgr: make inline functions static

[Olaf Hering]

gcc7 is more strict with functions marked as inline. They are not
automatically inlined. Instead a function call is generated, but the
actual code is not visible by the linker.

Do a mechanical change and mark every 'inline' as 'static inline'. For
simpler review the static goes into an extra line.

Signed-off-by: Olaf Hering 
---
 stubdom/vtpmmgr/marshal.h  | 76 ++
 stubdom/vtpmmgr/tcg.h  | 14 
 stubdom/vtpmmgr/tpm2_marshal.h | 58 
 stubdom/vtpmmgr/tpmrsa.h   |  1 +
 4 files changed, 149 insertions(+)

diff --git a/stubdom/vtpmmgr/marshal.h b/stubdom/vtpmmgr/marshal.h
index d826f19d89..dce19c6439 100644
--- a/stubdom/vtpmmgr/marshal.h
+++ b/stubdom/vtpmmgr/marshal.h
@@ -47,16 +47,19 @@ typedef enum UnpackPtr {
UNPACK_ALLOC
 } UnpackPtr;
 
+static
 inline BYTE* pack_BYTE(BYTE* ptr, BYTE t) {
ptr[0] = t;
return ++ptr;
 }
 
+static
 inline BYTE* unpack_BYTE(BYTE* ptr, BYTE* t) {
t[0] = ptr[0];
return ++ptr;
 }
 
+static
 inline int unpack3_BYTE(BYTE* ptr, UINT32* pos, UINT32 max, BYTE *t)
 {
if (*pos + 1 > max)
@@ -72,18 +75,21 @@ inline int unpack3_BYTE(BYTE* ptr, UINT32* pos, UINT32 max, 
BYTE *t)
 #define unpack3_BOOL(p, x, m, t) unpack3_BYTE(p, x, m, t)
 #define sizeof_BOOL(t) 1
 
+static
 inline BYTE* pack_UINT16(void* ptr, UINT16 t) {
UINT16* p = ptr;
*p = cpu_to_be16(t);
return ptr + sizeof(UINT16);
 }
 
+static
 inline BYTE* unpack_UINT16(void* ptr, UINT16* t) {
UINT16* p = ptr;
*t = be16_to_cpu(*p);
return ptr + sizeof(UINT16);
 }
 
+static
 inline int unpack3_UINT16(BYTE* ptr, UINT32* pos, UINT32 max, UINT16 *t)
 {
if (*pos + 2 > max)
@@ -93,18 +99,21 @@ inline int unpack3_UINT16(BYTE* ptr, UINT32* pos, UINT32 
max, UINT16 *t)
return 0;
 }
 
+static
 inline BYTE* pack_UINT32(void* ptr, UINT32 t) {
UINT32* p = ptr;
*p = cpu_to_be32(t);
return ptr + sizeof(UINT32);
 }
 
+static
 inline BYTE* unpack_UINT32(void* ptr, UINT32* t) {
UINT32* p = ptr;
*t = be32_to_cpu(*p);
return ptr + sizeof(UINT32);
 }
 
+static
 inline int unpack3_UINT32(BYTE* ptr, UINT32* pos, UINT32 max, UINT32 *t)
 {
if (*pos + 4 > max)
@@ -236,16 +245,19 @@ inline int unpack3_UINT32(BYTE* ptr, UINT32* pos, UINT32 
max, UINT32 *t)
 #define sizeof_TCS_KEY_HANDLE(t) sizeof_UINT32(t)
 
 
+static
 inline BYTE* pack_BUFFER(BYTE* ptr, const BYTE* buf, UINT32 size) {
memcpy(ptr, buf, size);
return ptr + size;
 }
 
+static
 inline BYTE* unpack_BUFFER(BYTE* ptr, BYTE* buf, UINT32 size) {
memcpy(buf, ptr, size);
return ptr + size;
 }
 
+static
 inline int unpack3_BUFFER(BYTE* ptr, UINT32* pos, UINT32 max, BYTE* buf, 
UINT32 size) {
if (*pos + size > max)
return TPM_SIZE;
@@ -256,11 +268,13 @@ inline int unpack3_BUFFER(BYTE* ptr, UINT32* pos, UINT32 
max, BYTE* buf, UINT32
 
 #define sizeof_BUFFER(b, s) s
 
+static
 inline BYTE* unpack_ALIAS(BYTE* ptr, BYTE** buf, UINT32 size) {
*buf = ptr;
return ptr + size;
 }
 
+static
 inline BYTE* unpack_ALLOC(BYTE* ptr, BYTE** buf, UINT32 size) {
if(size) {
*buf = malloc(size);
@@ -271,6 +285,7 @@ inline BYTE* unpack_ALLOC(BYTE* ptr, BYTE** buf, UINT32 
size) {
return ptr + size;
 }
 
+static
 inline BYTE* unpack_PTR(BYTE* ptr, BYTE** buf, UINT32 size, UnpackPtr alloc) {
if(alloc == UNPACK_ALLOC) {
return unpack_ALLOC(ptr, buf, size);
@@ -279,6 +294,7 @@ inline BYTE* unpack_PTR(BYTE* ptr, BYTE** buf, UINT32 size, 
UnpackPtr alloc) {
}
 }
 
+static
 inline int unpack3_PTR(BYTE* ptr, UINT32* pos, UINT32 max, BYTE** buf, UINT32 
size, UnpackPtr alloc) {
if (size > max || *pos + size > max)
return TPM_SIZE;
@@ -292,14 +308,17 @@ inline int unpack3_PTR(BYTE* ptr, UINT32* pos, UINT32 
max, BYTE** buf, UINT32 si
 }
 #define unpack3_VPTR(ptr, pos, max, buf, size, alloc) unpack3_PTR(ptr, pos, 
max, (void*)(buf), size, alloc)
 
+static
 inline BYTE* pack_TPM_AUTHDATA(BYTE* ptr, const TPM_AUTHDATA* d) {
return pack_BUFFER(ptr, *d, TPM_DIGEST_SIZE);
 }
 
+static
 inline BYTE* unpack_TPM_AUTHDATA(BYTE* ptr, TPM_AUTHDATA* d) {
return unpack_BUFFER(ptr, *d, TPM_DIGEST_SIZE);
 }
 
+static
 inline int unpack3_TPM_AUTHDATA(BYTE* ptr, UINT32* pos, UINT32 len, 
TPM_AUTHDATA* d) {
return unpack3_BUFFER(ptr, pos, len, *d, TPM_DIGEST_SIZE);
 }
@@ -325,6 +344,7 @@ inline int unpack3_TPM_AUTHDATA(BYTE* ptr, UINT32* pos, 
UINT32 len, TPM_AUTHDATA
 #define sizeof_TPM_TAG(t) sizeof_UINT16(t)
 #define sizeof_TPM_STRUCTURE_TAG(t) sizeof_UINT16(t)
 
+static
 inline BYTE* pack_TPM_VERSION(BYTE* ptr, const TPM_VERSION* t) {
ptr[0] = t->major;
ptr[1] = t->minor;
@@ -333,6 +353,7 @@ inline BYTE* pack_TPM_VERSION(BYTE* ptr, const TPM_VERSION* 
t) {
return ptr + 

Re: [Xen-devel] [PATCH] rombios: prevent building with PIC

[Wei Liu]

On Fri, Jun 23, 2017 at 07:26:01PM +0200, Olaf Hering wrote:
> If the default compiler silently defaults to to -fPIC/-fPIE building
> rombios fails:
> 
>  ld -melf_i386 -s -r 32bitbios.o tcgbios/tcgbiosext.o util.o pmm.o -o 
> 32bitbios_all.o
>  There are undefined symbols in the BIOS:
>   U _GLOBAL_OFFSET_TABLE_
>  make[10]: *** [Makefile:26: 32bitbios_all.o] Error 11
> 
> Prevent the failure by enforcing non-PIC mode.
> 
> Signed-off-by: Olaf Hering 

Do you need to check if the compiler supports -fno-pic?

I know not every version of gcc support -fno-pie, but I'm not sure about
-fno-pic. In any case, there is already something in our build system to
deal with testing gcc options, which would be useful to you.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [PATCH] rombios: prevent building with PIC

[Olaf Hering]

If the default compiler silently defaults to to -fPIC/-fPIE building
rombios fails:

 ld -melf_i386 -s -r 32bitbios.o tcgbios/tcgbiosext.o util.o pmm.o -o 
32bitbios_all.o
 There are undefined symbols in the BIOS:
  U _GLOBAL_OFFSET_TABLE_
 make[10]: *** [Makefile:26: 32bitbios_all.o] Error 11

Prevent the failure by enforcing non-PIC mode.

Signed-off-by: Olaf Hering 
---

build tested with staging-4.9.

PIE is the default now in openSUSE Tumbleweed:
https://lists.opensuse.org/opensuse-factory/2017-06/msg00403.html

 tools/firmware/rombios/32bit/Makefile | 2 +-
 tools/firmware/rombios/32bit/tcgbios/Makefile | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/tools/firmware/rombios/32bit/Makefile 
b/tools/firmware/rombios/32bit/Makefile
index b0583c93df..97657ae7b1 100644
--- a/tools/firmware/rombios/32bit/Makefile
+++ b/tools/firmware/rombios/32bit/Makefile
@@ -3,7 +3,7 @@ include $(XEN_ROOT)/tools/firmware/Rules.mk
 
 TARGET = 32bitbios_flat.h
 
-CFLAGS += $(CFLAGS_xeninclude) -I.. -I../../../libacpi
+CFLAGS += $(CFLAGS_xeninclude) -I.. -I../../../libacpi -fno-pic
 
 SUBDIRS = tcgbios
 
diff --git a/tools/firmware/rombios/32bit/tcgbios/Makefile 
b/tools/firmware/rombios/32bit/tcgbios/Makefile
index f87d13020f..2e3729910f 100644
--- a/tools/firmware/rombios/32bit/tcgbios/Makefile
+++ b/tools/firmware/rombios/32bit/tcgbios/Makefile
@@ -3,7 +3,7 @@ include $(XEN_ROOT)/tools/firmware/Rules.mk
 
 TARGET  = tcgbiosext.o
 
-CFLAGS += $(CFLAGS_xeninclude) -I.. -I../.. -I../../../../libacpi
+CFLAGS += $(CFLAGS_xeninclude) -I.. -I../.. -I../../../../libacpi -fno-pic
 
 .PHONY: all
 all: $(TARGET)

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [OSSTEST PATCH v11 11/20] ts-openstack-deploy: Increase fd and memory limits for rabbitmq

[Ian Jackson]

Anthony PERARD writes ("Re: [OSSTEST PATCH v11 11/20] ts-openstack-deploy: 
Increase fd and memory limits for rabbitmq"):
> On Fri, Jun 23, 2017 at 04:34:41PM +0100, Ian Jackson wrote:
> > And if one isn't using systemd ?
> 
> I guess it would have to fix devstack. Also, the default is 1k, devstack
> increase it to 2k, but rabbitmq on Ubuntu have a limit at 65k.

My point is this: unless openstack intend to support only systemd, it
is a bug that devstack is not fixed in this way.  So that bug ought to
be filed and referenced in the workaround in openstack.

I'm not sure I'm qualified to comment on the distinction between
devstack's and ubuntu's fd limits, but shouldn't there be an openstack
bug about that too ?

> > > As for the memory limit, it was necessary with a host of 4G of RAM. But
> > > I did not try again with the default limit and a host with 6G of RAM.
> > > (The default is 0.4, here I've set it to 0.8)
> > 
> > It sounds like the default calculation is not right, then ?
> 
> I think 0.4 means 40% of the RAM. Sorry, I should have said so. Also,
> 0.4 is the same on the CI loop, with a dom0 of 7G or so.

Yes, I understood that 0.4 to mean 40% of RAM.

My point is that a calculation which gives too-small a value on a 4G
host could be improved.  May the calculation should be "40% of RAM,
but at least 3G" or something ?  Again, this might warrant an upstream bug.

Ian.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] live migration of HVM domUs with more than 32vcpus fails

[Boris Ostrovsky]

On 06/23/2017 12:54 PM, Olaf Hering wrote:

On Thu, Jun 22, Boris Ostrovsky wrote:


They are queued for 4.13.
git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip.git for-linus-4.13


This works for me. Thanks.

I assume there is no ready-to-pull variant for linux-4.4.x?
nm.



Not that I am aware of. Unless Ankur backported them to OL which is 
(loosely) based on 4.1.



-boris

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [OSSTEST PATCH v11 20/20] Introduce flight for stable branches of OpenStack

[Ian Jackson]

Anthony PERARD writes ("[OSSTEST PATCH v11 20/20] Introduce flight for stable 
branches of OpenStack"):
> OpenStack have many different repo which should be in sync, so this
> patch should grab the revisions of the stable branch of every OpenStack
> tree. Tempest does not have stable branch and should be able to test any
> OpenStack version.
...
> +openstack-*-*)
> + os_tree="${branch#openstack-}"
> + os_tree="${os_tree%-*}"
> + branchcore="${branch##*-}"
> + eval repo_tree_rev_fetch_git "openstack-$os_tree" \
> + "\$TREE_OPENSTACK_${os_tree^^}" "stable/$branchcore" \
> +"\$LOCALREV_OPENSTACK_${os_tree^^}"

From your previous email:

  I think this patch is confusing because I originally try to use osstest
  scripts to find which commit to use for every trees and so have add the
  necessary into ./ap-fetch-version. But I could not make that works
  without duplicating some functions and so went with writing
  'origin/stable/ocata' into REVISION_*.

I think I am indeed still confused by some of it.

For example:

> +openstack_rev() {
> +local os_tree="$1"
> +local os_branch
> +
> +if eval [ "x\$REVISION_OPENSTACK_${os_tree^^}" = x ]; then
> +case "$branch" in
> +openstack-*-*)
> +os_branch="openstack-$os_tree-${branch##*-}"
> +os_git_branch="origin/stable/${branch##*-}"
> +;;
> +*)
> +os_branch="openstack-$os_tree"
> +os_git_branch="origin/master"
> +;;
> +esac
> +
> +# Use latest version, even for other openstack
> +# trees so branch openstack-nova-ocata should have
> +# other trees like openstack-neutron have the
> +# revision of the same branch fetch at the same
> +# time
> +if [ "$branch" != "$os_branch" ]; then
> +eval "export 
> REVISION_OPENSTACK_${os_tree^^}=$os_git_branch"
> +return
> +fi
> +determine_version "REVISION_OPENSTACK_${os_tree^^}" \
> +"$os_branch" "OPENSTACK_${os_tree^^}"
> +eval "export REVISION_OPENSTACK_${os_tree^^}"
> +fi
> +}
> +for os_tree in cinder devstack glance keystone neutron nova requirements; do
> +openstack_rev "$os_tree"
> +done

I wonder if this full generality is really necessary ?  If you don't
intend branches like   openstack-ocata-neutron   then it would be
sufficient to call one function for nova and another for the other
trees.

And, frankly, I don't think we could have branches like
`openstack-ocata-neutron'.  That would be too many branches.

So perhaps the branch `openstack-ocata-nova' should be called
`openstack-ocata' ?


Also, right now I think it's clear that we're not intending to add
openstack jobs to existing branches' flights.  But if we were to do
that in the future, we would want all the subtrees to be tracked.

Maybe we should have a way for cr-daily-branch to fetch, and push,
multiple trees.  We could call ap-fetch-version on every tree,
and set the appropriate variable (with determine_version, as you have
above).  And then set a variable to call ap-push multiple times, if we
get a pass.


Ian.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [qemu-upstream-unstable test] 110975: regressions - trouble: broken/fail/pass

[osstest service owner]

flight 110975 qemu-upstream-unstable real [real]
http://logs.test-lab.xenproject.org/osstest/logs/110975/

Regressions :-(

Tests which did not succeed and are blocking,
including tests which could not be run:
 test-armhf-armhf-xl-credit2 15 guest-start/debian.repeat fail REGR. vs. 106833

Tests which are failing intermittently (not blocking):
 test-armhf-armhf-xl-multivcpu  3 host-install(3) broken pass in 110938
 test-amd64-i386-xl-qemuu-win7-amd64 15 guest-localmigrate/x10 fail in 110938 
pass in 110975
 test-amd64-amd64-xl-qemuu-debianhvm-amd64 9 debian-hvm-install fail pass in 
110938

Regressions which are regarded as allowable (not blocking):
 test-armhf-armhf-xl-rtds15 guest-start/debian.repeat fail REGR. vs. 106833

Tests which did not succeed, but are not blocking:
 test-amd64-i386-xl-qemuu-ws16-amd64 9 windows-install fail in 110938 baseline 
untested
 test-armhf-armhf-xl-multivcpu 12 migrate-support-check fail in 110938 never 
pass
 test-armhf-armhf-xl-multivcpu 13 saverestore-support-check fail in 110938 
never pass
 test-armhf-armhf-libvirt-raw 12 saverestore-support-checkfail  like 106813
 test-armhf-armhf-libvirt-xsm 13 saverestore-support-checkfail  like 106833
 test-armhf-armhf-libvirt 13 saverestore-support-checkfail  like 106833
 test-amd64-i386-xl-qemuu-win7-amd64 16 guest-stop fail like 106833
 test-amd64-amd64-xl-qemuu-ws16-amd64  9 windows-installfail never pass
 test-amd64-amd64-libvirt 12 migrate-support-checkfail   never pass
 test-amd64-amd64-libvirt-xsm 12 migrate-support-checkfail   never pass
 test-amd64-i386-libvirt-xsm  12 migrate-support-checkfail   never pass
 test-amd64-i386-libvirt  12 migrate-support-checkfail   never pass
 test-amd64-amd64-libvirt-qemuu-debianhvm-amd64-xsm 10 migrate-support-check 
fail never pass
 test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm 10 migrate-support-check 
fail never pass
 test-arm64-arm64-xl  12 migrate-support-checkfail   never pass
 test-arm64-arm64-xl  13 saverestore-support-checkfail   never pass
 test-arm64-arm64-xl-xsm  12 migrate-support-checkfail   never pass
 test-arm64-arm64-xl-xsm  13 saverestore-support-checkfail   never pass
 test-arm64-arm64-libvirt-xsm 12 migrate-support-checkfail   never pass
 test-arm64-arm64-libvirt-xsm 13 saverestore-support-checkfail   never pass
 test-armhf-armhf-xl-arndale  12 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-arndale  13 saverestore-support-checkfail   never pass
 test-amd64-amd64-libvirt-vhd 11 migrate-support-checkfail   never pass
 test-amd64-amd64-qemuu-nested-amd 16 debian-hvm-install/l1/l2  fail never pass
 test-amd64-i386-xl-qemuu-ws16-amd64 12 guest-saverestore   fail never pass
 test-armhf-armhf-xl-rtds 12 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-rtds 13 saverestore-support-checkfail   never pass
 test-armhf-armhf-xl  12 migrate-support-checkfail   never pass
 test-armhf-armhf-xl  13 saverestore-support-checkfail   never pass
 test-armhf-armhf-libvirt-xsm 12 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-cubietruck 12 migrate-support-checkfail never pass
 test-armhf-armhf-xl-cubietruck 13 saverestore-support-checkfail never pass
 test-armhf-armhf-xl-credit2  12 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-credit2  13 saverestore-support-checkfail   never pass
 test-armhf-armhf-libvirt 12 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-xsm  12 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-xsm  13 saverestore-support-checkfail   never pass
 test-arm64-arm64-xl-credit2  12 migrate-support-checkfail   never pass
 test-armhf-armhf-libvirt-raw 11 migrate-support-checkfail   never pass
 test-arm64-arm64-xl-credit2  13 saverestore-support-checkfail   never pass
 test-armhf-armhf-xl-vhd  11 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-vhd  12 saverestore-support-checkfail   never pass
 test-amd64-i386-xl-qemuu-win10-i386  9 windows-install fail never pass
 test-amd64-amd64-xl-qemuu-win10-i386  9 windows-installfail never pass

version targeted for testing:
 qemuu414d069b38ab114b89085e44989bf57604ea86d7
baseline version:
 qemuue97832ec6b2a7ddd48b8e6d1d848ffdfee6a31c7

Last test of basis   106833  2017-03-22 07:02:01 Z   93 days
Testing same since   110938  2017-06-21 15:39:52 Z2 days2 attempts


People who touched revisions under test:
  Anthony PERARD 
  Jan Beulich 

jobs:
 build-amd64-xsm  pass
 build-arm64-xsm  

Re: [Xen-devel] live migration of HVM domUs with more than 32vcpus fails

[Olaf Hering]

On Thu, Jun 22, Boris Ostrovsky wrote:

> They are queued for 4.13.
> git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip.git for-linus-4.13

This works for me. Thanks.

I assume there is no ready-to-pull variant for linux-4.4.x?
nm.

Olaf


signature.asc
Description: PGP signature
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [OSSTEST PATCH v11 11/20] ts-openstack-deploy: Increase fd and memory limits for rabbitmq

[Anthony PERARD]

On Fri, Jun 23, 2017 at 04:34:41PM +0100, Ian Jackson wrote:
> Anthony PERARD writes ("Re: [OSSTEST PATCH v11 11/20] ts-openstack-deploy: 
> Increase fd and memory limits for rabbitmq"):
> > On Fri, Jun 23, 2017 at 02:41:59PM +0100, Ian Jackson wrote:
> > > Anthony PERARD writes ("[OSSTEST PATCH v11 11/20] ts-openstack-deploy: 
> > > Increase fd and memory limits for rabbitmq"):
> > > > Signed-off-by: Anthony PERARD 
> > > 
> > > Does this not mean that the upstream defaults are wrong ?
> > 
> > That depends on what you meant by upstream.
> > 
> > On Ubuntu, the limit of open fd is set to an higher value via the
> > systemd unit file. It does not look like devstack (OpenStack) is
> > changing this.
> > 
> > In devstack, I've seen an increate of the limit of open fd, but via
> > systemd, so for all systemd services.
> 
> And if one isn't using systemd ?

I guess it would have to fix devstack. Also, the default is 1k, devstack
increase it to 2k, but rabbitmq on Ubuntu have a limit at 65k.

> > As for the memory limit, it was necessary with a host of 4G of RAM. But
> > I did not try again with the default limit and a host with 6G of RAM.
> > (The default is 0.4, here I've set it to 0.8)
> 
> It sounds like the default calculation is not right, then ?

I think 0.4 means 40% of the RAM. Sorry, I should have said so. Also,
0.4 is the same on the CI loop, with a dom0 of 7G or so.

-- 
Anthony PERARD

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v2] passthrough: give XEN_DOMCTL_test_assign_device more sane semantics

[Daniel De Graaf]

On 06/23/2017 11:00 AM, Jan Beulich wrote:

So far callers of the libxc interface passed in a domain ID which was
then ignored in the hypervisor. Instead, make the hypervisor honor it
(accepting DOMID_INVALID to obtain original behavior), allowing to
query whether a device can be assigned to a particular domain.

Signed-off-by: Jan Beulich 
---
v2: Alter the semantics to check whether the device can be assigned to
 the passed in domain.
TBD: It's not clear to me whether the test-assign XSM hooks are still
  useful this way.


As long as the only user of this hypercall is the device assignment
path, I would remove the XSM hook for test_assign and use the
assign_device hook for both test and actual.  That way, if XSM is
actually preventing the assignment, you will get the same AVC denials as
if you tried without doing the test first.  The hook should just skip the
two checks relating to (d) if it is null.

If this test hypercall were to be used as part of a query (such as
populating a list of assignable devices by trying all PCI devices listed
via sysfs), then it would make sense to have either a different hook or
a flag in the XSM hook to silence the audit messages since you aren't
actually trying to do the assignment yet.  That change will make the
cause of the "can't assign" error harder to diagnose, however, so unless
it's being used like that I don't think it's a good idea.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [OSSTEST PATCH v11 20/20] Introduce flight for stable branches of OpenStack

[Ian Jackson]

Anthony PERARD writes ("Re: [OSSTEST PATCH v11 20/20] Introduce flight for 
stable branches of OpenStack"):
> We decided to track only nova.git, and let osstest clone other trees and
> checkout the HEAD (by having REVISION_* empty). That is fine if we track
> "master" of nova.git.

Right.

> The issue now is if we want to track a branch of OpenStack, like
> "stable/ocata" (which is the latest release), we need to have all trees
> (nova.git, devstack.git, ...) checkout the same branch. (tempest.git
> does not have release specific branch.)

Ah.

> I think this patch is confusing because I originally try to use osstest
> scripts to find which commit to use for every trees and so have add the
> necessary into ./ap-fetch-version. But I could not make that works
> without duplicating some functions and so went with writing
> 'origin/stable/ocata' into REVISION_*.

OIC.

I think it is fine to write origin/stable/ocata into REVISION_*, in
this case.  Of course it means that the sub-trees are not revision
controlled by the osstest push-gate.  IIRC you said that bugs due to
those sub-trees would not be very likely.

> In term of osstest branches, in cr-for-branches, I would like only one
> branch. 'openstack-nova-ocata' would be a the branch that test the
> OpenStack release named 'Ocata'. Later, once this branch run smoothly,
> we can add a branch to test 'master', the development branch of
> OpenStack, it was called 'openstack-nova' in the original 3 patch of
> this patch series.

Aha.

I will go and read the patch again.

Ian.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [OSSTEST PATCH v11 20/20] Introduce flight for stable branches of OpenStack

[Anthony PERARD]

On Fri, Jun 23, 2017 at 02:58:25PM +0100, Ian Jackson wrote:
> Anthony PERARD writes ("[OSSTEST PATCH v11 20/20] Introduce flight for stable 
> branches of OpenStack"):
> > OpenStack have many different repo which should be in sync, so this
> > patch should grab the revisions of the stable branch of every OpenStack
> > tree. Tempest does not have stable branch and should be able to test any
> > OpenStack version.
> 
> I'm afraid I don't understand this patch.
> 
> Partly, I think it needs to be squashed with the original patch
> introducing `openstack-nova' as a branch.  While your series has a
> number of things where a thing is introduced and then later patched,
> and this is largely OK, I think it is too confusing to have a whole
> branch appear and disappear like this (without ever having been
> run for real).

Yes, I can do that.

> > -openstack-nova)
> > +openstack-tempest*)
> > +# OpenStack Tempest does not have stable branches and should work 
> > with any
> 
> Your comment lines need wrapping to ~70-75 (here and later).
> 
> > +# version of OpenStack
> > repo_tree_rev_fetch_git openstack-nova \
> > $TREE_OPENSTACK_NOVA master $LOCALREV_OPENSTACK_NOVA
> > ;;
> > +openstack-*-*)
> 
> I think you need to document what your openstack-*-* branch names are
> going to be.
> 
> And, you make provision here for branches openstack-tempest* but you
> don't add any such branches to cr-for-branches.
> 
> I forget what decisions we made about the push gate logic for the
> various openstack branches.  I think it would be worth explicitly
> writing this down in tree (in a comment in ap-fetch-* perhaps).

We decided to track only nova.git, and let osstest clone other trees and
checkout the HEAD (by having REVISION_* empty). That is fine if we track
"master" of nova.git.

The issue now is if we want to track a branch of OpenStack, like
"stable/ocata" (which is the latest release), we need to have all trees
(nova.git, devstack.git, ...) checkout the same branch. (tempest.git
does not have release specific branch.)

I think this patch is confusing because I originally try to use osstest
scripts to find which commit to use for every trees and so have add the
necessary into ./ap-fetch-version. But I could not make that works
without duplicating some functions and so went with writing
'origin/stable/ocata' into REVISION_*.


In term of osstest branches, in cr-for-branches, I would like only one
branch. 'openstack-nova-ocata' would be a the branch that test the
OpenStack release named 'Ocata'. Later, once this branch run smoothly,
we can add a branch to test 'master', the development branch of
OpenStack, it was called 'openstack-nova' in the original 3 patch of
this patch series.

-- 
Anthony PERARD

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH 1/4] xen: credit2: implement utilization cap

[Dario Faggioli]

On Thu, 2017-06-22 at 17:55 +0100, George Dunlap wrote:
> On 08/06/17 13:08, Dario Faggioli wrote:
> > This commit implements the Xen part of the cap mechanism for
> > Credit2.
> > 
> > A cap is how much, in terms of % of physical CPU time, a domain
> > can execute at most.
> > 
> > For instance, a domain that must not use more than 1/4 of one
> > physical CPU, must have a cap of 25%; one that must not use more
> > than 1+1/2 of physical CPU time, must be given a cap of 150%.
> > 
> > Caps are per domain, so it is all a domain's vCPUs, cumulatively,
> > that will be forced to execute no more than the decided amount.
> > 
> > This is implemented by giving each domain a 'budget', and using
> > a (per-domain again) periodic timer. Values of budget and 'period'
> > are chosen so that budget/period is equal to the cap itself.
> > 
> > Budget is burned by the domain's vCPUs, in a similar way to how
> > credits are.
> > 
> > When a domain runs out of budget, its vCPUs can't run any longer.
> > They can gain, when the budget is replenishment by the timer, which
> > event happens once every period.
> > 
> > Blocking the vCPUs because of lack of budget happens by
> > means of a new (_VPF_parked) pause flag, so that, e.g.,
> > vcpu_runnable() still works. This is similar to what is
> > done in sched_rtds.c, as opposed to what happens in
> > sched_credit.c, where vcpu_pause() and vcpu_unpause()
> > (which means, among other things, more overhead).
> > 
> > Note that xenalyze and tools/xentrace/format are also modified,
> > to keep them updated with one modified event.
> > 
> > Signed-off-by: Dario Faggioli 
> 
> Looks really good overall, Dario!  Just a few relatively minor
> comments.
> 
Cool! Thanks for having a look so quickly. :-)

> > --- a/xen/common/sched_credit2.c
> > +++ b/xen/common/sched_credit2.c
> > @@ -92,6 +92,82 @@
> > + *   Budget never expires, so at whatever time a vCPU wants to
> > run, it can
> > + *   check the domain's budget, and if there is some, it can use
> > it.
> 
> I'm not sure what this paragraph is trying to say. Saying budget
> "never
> expires" makes it sound like you continue to accumulate it, such that
> if
> you don't run at all for several periods, you could "save it up" and
> run
> at 100% for one full period.
> 
Yes, I see what you mean, and I agree that it's unclear. The point is
that there exists algorithm where the budget of a replenishment has an
expiry time, before the next replenishment, i.e., it has to be used
right away after the replenishment, or (usually), shortly after, or it
will be thrown away, and we'll have to wait next replenishment.

But again, yes, saying "never expires", here, and in the way I did it,
and without any reference and comparison to those other algorithms,
makes people think what you just said.

I'll rephrase (or cut the paragraph entirely, it's probably not super
informative/useful anyway). :-)

> > + * - when a budget replenishment occurs, if there are vCPUs that
> > had been
> > + *   blocked because of lack of budget, they'll be unblocked, and
> > they will
> > + *   (potentially) be able to run again.
> > + *
> > + * Finally, some even more implementation related detail:
> > + *
> > + * - budget is stored in a domain-wide pool. vCPUs of the domain
> > that want
> > + *   to run go to such pool, and grub some. When they do so, the
> > amount
> > + *   they grabbed is _immediately_ removed from the pool. This
> > happens in
> > + *   vcpu_try_to_get_budget();
> 
> This sounds like a good solution to the "greedy vcpu" problem. :-)
> 
I like it too. It works because it's coupled with the fact that
runqueue is ordered by credits. E.g., doing the same in Credit1, would
probably still not work, because of the round-robin queues (within same
priority), and because of the boosting on wakeup.

> > @@ -1438,6 +1570,217 @@ void burn_credits(struct
> > +static bool vcpu_try_to_get_budget(struct csched2_vcpu *svc)
> 
> This name is OK, but it's a bit long.  What about
> "vcpu_grab_budget()"?
> 
Ok (and the other renaming suggestions too).

> > +{
> > +struct csched2_dom *sdom = svc->sdom;
> > +unsigned int cpu = svc->vcpu->processor;
> > +
> > +ASSERT(spin_is_locked(per_cpu(schedule_data,
> > cpu).schedule_lock));
> > +
> > +if ( svc->budget > 0 )
> > +return true;
> > +
> > +/* budget_lock nests inside runqueue lock. */
> > +spin_lock(>budget_lock);
> > +
> > +/*
> > + * Here, svc->budget is <= 0 (as, if it was > 0, we'd have
> > taken the if
> > + * above!). That basically means the vCPU has overrun a bit --
> > because of
> > + * various reasons-- and we want to take that into account.
> > With the +=,
> > + * we are actually subtracting the amount of budget the vCPU
> > has
> > + * overconsumed, from the total domain budget.
> > + */
> > +sdom->budget += svc->budget;
> > +
> > +if ( sdom->budget > 0 )
> > +{
> > +svc->budget = sdom->budget;
> > +  

[Xen-devel] [xen-unstable-smoke test] 111006: tolerable trouble: broken/pass - PUSHED

[osstest service owner]

flight 111006 xen-unstable-smoke real [real]
http://logs.test-lab.xenproject.org/osstest/logs/111006/

Failures :-/ but no regressions.

Tests which did not succeed, but are not blocking:
 test-arm64-arm64-xl-xsm   1 build-check(1)   blocked  n/a
 test-amd64-amd64-libvirt 12 migrate-support-checkfail   never pass
 test-armhf-armhf-xl  12 migrate-support-checkfail   never pass
 test-armhf-armhf-xl  13 saverestore-support-checkfail   never pass

version targeted for testing:
 xen  a15516c0cf21d7ac84799f1e2e500b0bb22d2300
baseline version:
 xen  579d698da608a24ab334a6a38d932176bac5cecd

Last test of basis   110976  2017-06-22 16:01:47 Z0 days
Testing same since   111006  2017-06-23 14:01:19 Z0 days1 attempts


People who touched revisions under test:
  Jan Beulich 
  Julien Grall 

jobs:
 build-amd64  pass
 build-armhf  pass
 build-amd64-libvirt  pass
 test-armhf-armhf-xl  pass
 test-arm64-arm64-xl-xsm  broken  
 test-amd64-amd64-xl-qemuu-debianhvm-i386 pass
 test-amd64-amd64-libvirt pass



sg-report-flight on osstest.test-lab.xenproject.org
logs: /home/logs/logs
images: /home/logs/images

Logs, config files, etc. are available at
http://logs.test-lab.xenproject.org/osstest/logs

Explanation of these reports, and of osstest in general, is at
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=master
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master

Test harness code can be found at
http://xenbits.xen.org/gitweb?p=osstest.git;a=summary


Pushing revision :

+ branch=xen-unstable-smoke
+ revision=a15516c0cf21d7ac84799f1e2e500b0bb22d2300
+ . ./cri-lock-repos
++ . ./cri-common
+++ . ./cri-getconfig
+++ umask 002
+++ getrepos
 getconfig Repos
 perl -e '
use Osstest;
readglobalconfig();
print $c{"Repos"} or die $!;
'
+++ local repos=/home/osstest/repos
+++ '[' -z /home/osstest/repos ']'
+++ '[' '!' -d /home/osstest/repos ']'
+++ echo /home/osstest/repos
++ repos=/home/osstest/repos
++ repos_lock=/home/osstest/repos/lock
++ '[' x '!=' x/home/osstest/repos/lock ']'
++ OSSTEST_REPOS_LOCK_LOCKED=/home/osstest/repos/lock
++ exec with-lock-ex -w /home/osstest/repos/lock ./ap-push xen-unstable-smoke 
a15516c0cf21d7ac84799f1e2e500b0bb22d2300
+ branch=xen-unstable-smoke
+ revision=a15516c0cf21d7ac84799f1e2e500b0bb22d2300
+ . ./cri-lock-repos
++ . ./cri-common
+++ . ./cri-getconfig
+++ umask 002
+++ getrepos
 getconfig Repos
 perl -e '
use Osstest;
readglobalconfig();
print $c{"Repos"} or die $!;
'
+++ local repos=/home/osstest/repos
+++ '[' -z /home/osstest/repos ']'
+++ '[' '!' -d /home/osstest/repos ']'
+++ echo /home/osstest/repos
++ repos=/home/osstest/repos
++ repos_lock=/home/osstest/repos/lock
++ '[' x/home/osstest/repos/lock '!=' x/home/osstest/repos/lock ']'
+ . ./cri-common
++ . ./cri-getconfig
++ umask 002
+ select_xenbranch
+ case "$branch" in
+ tree=xen
+ xenbranch=xen-unstable-smoke
+ qemuubranch=qemu-upstream-unstable
+ '[' xxen = xlinux ']'
+ linuxbranch=
+ '[' xqemu-upstream-unstable = x ']'
+ select_prevxenbranch
++ ./cri-getprevxenbranch xen-unstable-smoke
+ prevxenbranch=xen-4.9-testing
+ '[' xa15516c0cf21d7ac84799f1e2e500b0bb22d2300 = x ']'
+ : tested/2.6.39.x
+ . ./ap-common
++ : osst...@xenbits.xen.org
+++ getconfig OsstestUpstream
+++ perl -e '
use Osstest;
readglobalconfig();
print $c{"OsstestUpstream"} or die $!;
'
++ :
++ : git://xenbits.xen.org/xen.git
++ : osst...@xenbits.xen.org:/home/xen/git/xen.git
++ : git://xenbits.xen.org/qemu-xen-traditional.git
++ : git://git.kernel.org
++ : git://git.kernel.org/pub/scm/linux/kernel/git
++ : git
++ : git://xenbits.xen.org/xtf.git
++ : osst...@xenbits.xen.org:/home/xen/git/xtf.git
++ : git://xenbits.xen.org/xtf.git
++ : git://xenbits.xen.org/libvirt.git
++ : osst...@xenbits.xen.org:/home/xen/git/libvirt.git
++ : git://xenbits.xen.org/libvirt.git
++ : git://xenbits.xen.org/osstest/rumprun.git
++ : git
++ : git://xenbits.xen.org/osstest/rumprun.git
++ : osst...@xenbits.xen.org:/home/xen/git/osstest/rumprun.git
++ : git://git.seabios.org/seabios.git
++ : osst...@xenbits.xen.org:/home/xen/git/osstest/seabios.git
++ : git://xenbits.xen.org/osstest/seabios.git
++ : https://github.com/tianocore/edk2.git
++ : osst...@xenbits.xen.org:/home/xen/git/osstest/ovmf.git
++ : 

Re: [Xen-devel] [OSSTEST PATCH v11 18/20] ts-logs-capture: Capture OpenStack logs

[Ian Jackson]

Anthony PERARD writes ("Re: [OSSTEST PATCH v11 18/20] ts-logs-capture: Capture 
OpenStack logs"):
> On Fri, Jun 23, 2017 at 02:49:11PM +0100, Ian Jackson wrote:
> > This is not fine.  If a build host is shared, it will collect all the
> > logs from all of the builds.
> 
> This file should not exist on a build host, as it would be created by
> ts-openstack-deploy which I hope is not run on a build host.

Ah.  Sorry.  I think I got confused.

In the future even test hosts might be shared sequentially by
different jobs but I think given what your ts-openstack-deploy does,
we can't make that true for these jobs.

> We are only using using target_jobdir() to have a directory to clone all
> the OpenStack repo and run ./devstack.

Aha.

> But I can find a way to copy this file somewhere else.

I now see that you have an openstack-tempest-test-specific filename so
it is fine to try to collect this file in all jobs.

> > I think build ts-* scripts ought to collect their own logs.
> 
> Do you mean I can capture logs like ts-logs-capture would do, but before
> it is been run?
> 
> ts-openstack-deploy could collect all the configuration file then, like
> /etc/nova/*.

I'm sorry, forget what I said about build ts-* scripts.


So, the upshot is:

Acked-by: Ian Jackson 

and sorry for not paying proper attention.

Ian.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [OSSTEST PATCH v11 16/20] ts-openstack-tempest: Update list of skipped tests

[Ian Jackson]

Anthony PERARD writes ("Re: [OSSTEST PATCH v11 16/20] ts-openstack-tempest: 
Update list of skipped tests"):
> On Fri, Jun 23, 2017 at 02:47:37PM +0100, Ian Jackson wrote:
> > Again, does this not mean we're going to suffer a maintenance burden
> > as tempest grows new inapplicable tests ?
> 
> That exactly what is happening with the OpenStack CI loop, from time to
> time, there are new tests, that can fail, and diffenrent ways to fix
> this (fix the bug, fix a configuration, or just skip the tests).
> 
> Recently, we've actually push the list of tests to skip into nova.git,
> but I think it is only available in master.

Aha.  Well, if the stable branch is stable then the set of tests to
skip there is probably stable too ?  And on master it's built-in ?  So
this actually won't be a problem - in the sense that this will be
approximately the last necessary update to the list of tests to skip ?

> > Which makes me think: maybe the tempest tests want to be substeps
> > anyway.  Is that possible ?  (Does tempest speak subunit or
> > something ?)
> 
> I think it is subunit, yes. And I've got the command that is been run by
> tempest (as it's printed on stdout).
> 
> running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
> OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
> OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
> OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
> ${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} 
> ${OS_TEST_PATH:-./tempest/test_discover} --list
> 
> then several:
> running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
> OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
> OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
> OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
> ${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} 
> ${OS_TEST_PATH:-./tempest/test_discover}  --load-list /tmp/tmpzSNsrB

It would be really good to have those individual subunit results as
substeps.  There is no subunit parser in osstest yet but we could have
one.

What version of subunit does it print out ?  The subunit v1 protocol
is lovely and simple but there is a Second System :-/.

I guess that tempest doesn't stop on the first failed test ?  So
perhaps we could just tolerate the failed-but-not-skipped tests ?

Ian.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [OSSTEST PATCH v11 12/20] make-flight: Increase dom0_mem for openstack flight

[Ian Jackson]

Anthony PERARD writes ("Re: [OSSTEST PATCH v11 12/20] make-flight: Increase 
dom0_mem for openstack flight"):
> On Fri, Jun 23, 2017 at 02:42:53PM +0100, Ian Jackson wrote:
> > Anthony PERARD writes ("[OSSTEST PATCH v11 12/20] make-flight: Increase 
> > dom0_mem for openstack flight"):
> > > With 4G for dom0_mem, a host running devstack is using about 1.5G of
> > > swap.
> > 
> > Is this going to work properly on 8G hosts ?
> 
> Yes, it is fine. I usually do my OpenStack testing on 8G hosts. The CI
> loop is running with testing on 8G, and have a bit more than 7G for
> dom0. Tempest only run small VM (64MB or 128MB) and only a few at a
> time.

OK, great.

Acked-by: Ian Jackson 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [OSSTEST PATCH v11 11/20] ts-openstack-deploy: Increase fd and memory limits for rabbitmq

[Ian Jackson]

Anthony PERARD writes ("Re: [OSSTEST PATCH v11 11/20] ts-openstack-deploy: 
Increase fd and memory limits for rabbitmq"):
> On Fri, Jun 23, 2017 at 02:41:59PM +0100, Ian Jackson wrote:
> > Anthony PERARD writes ("[OSSTEST PATCH v11 11/20] ts-openstack-deploy: 
> > Increase fd and memory limits for rabbitmq"):
> > > Signed-off-by: Anthony PERARD 
> > 
> > Does this not mean that the upstream defaults are wrong ?
> 
> That depends on what you meant by upstream.
> 
> On Ubuntu, the limit of open fd is set to an higher value via the
> systemd unit file. It does not look like devstack (OpenStack) is
> changing this.
> 
> In devstack, I've seen an increate of the limit of open fd, but via
> systemd, so for all systemd services.

And if one isn't using systemd ?

> As for the memory limit, it was necessary with a host of 4G of RAM. But
> I did not try again with the default limit and a host with 6G of RAM.
> (The default is 0.4, here I've set it to 0.8)

It sounds like the default calculation is not right, then ?

Ian.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v3.1 8/8] osstest: hook FreeBSD flight into cr-daily-branch

[Ian Jackson]

Roger Pau Monne writes ("[PATCH v3.1 8/8] osstest: hook FreeBSD flight into 
cr-daily-branch"):
> +++ b/daily-cron-email-real--freebsd
> @@ -0,0 +1,4 @@
> +To: xen-de...@lists.xenproject.org,
> +osstest-ad...@xenproject.org,
> +roy...@freebsd.org
> +Bcc: osstest-out...@lists.xenproject.org

FTR this is good.

Ian.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [OSSTEST PATCH v11 18/20] ts-logs-capture: Capture OpenStack logs

[Anthony PERARD]

On Fri, Jun 23, 2017 at 02:49:11PM +0100, Ian Jackson wrote:
> Anthony PERARD writes ("[OSSTEST PATCH v11 18/20] ts-logs-capture: Capture 
> OpenStack logs"):
> > +  /var/log/openstack/*.log
> > +  /etc/nova/*
> > +  /etc/neutron/*
> > +  /etc/cinder/*
> 
> This is fine:
> 
> > +  
> > /home/osstest/build.*.test-*-devstack/tempest/etc/tempest.conf
> 
> This is not fine.  If a build host is shared, it will collect all the
> logs from all of the builds.

This file should not exist on a build host, as it would be created by
ts-openstack-deploy which I hope is not run on a build host.

We are only using using target_jobdir() to have a directory to clone all
the OpenStack repo and run ./devstack.

But I can find a way to copy this file somewhere else.

> I think build ts-* scripts ought to collect their own logs.

Do you mean I can capture logs like ts-logs-capture would do, but before
it is been run?

ts-openstack-deploy could collect all the configuration file then, like
/etc/nova/*.

-- 
Anthony PERARD

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v3 8/8] osstest: hook FreeBSD flight into cr-daily-branch

[Ian Jackson]

Roger Pau Monne writes ("[PATCH v3 8/8] osstest: hook FreeBSD flight into 
cr-daily-branch"):
> +++ b/daily-cron-email-freebsd
> @@ -0,0 +1 @@
> +To: roy...@freebsd.org

Please at least Bcc osstest-output.  See daily-cron-email-osstest for
an example.

Please also provide information about the changes to flights etc.,
for the whole series, by diffing the output of
  OSSTEST_CONFIG=standalone-config-example eatmydata 
standalone-generate-dump-flight-runvars
or some such.

Thanks,
Ian.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v3 7/8] osstest: introduce a script to create a FreeBSD flight

[Ian Jackson]

Roger Pau Monne writes ("[PATCH v3 7/8] osstest: introduce a script to create a 
FreeBSD flight"):
> +FreeBSDDist
> +   Path to the folder that contains the FreeBSD install image and
> +   the FreeBSD compressed install sets, together with the MANIFEST
> +   file that holds the checksums. This is required in order to run
> +   a FreeBSD host install if no previous FreeBSD buildjob is
> +   available (ie: for example when running in standalone mode).
> +
> +FreeBSDVersion
> +   Numeric value holding the major FreeBSD version of the media
> +   provided in FreeBSDDist (ie: 12).

This means that a user who is setting these manually needs to supply
both.  Is there a way to avoid that ?

> +get_freebsdjob_flags () {
> +arch=$1

You repeated use the word "flags" here for things which are runvars.
flags are, in osstest, strictly boolean.

> +flags=`get_freebsdjob_flags $arch`
> +job_create_build build-$arch-freebsd build-freebsd   \
> +arch=$arch   \
> +$RUNVARS $BUILD_RUNVARS $BUILD_FREEBSD_RUNVARS $arch_runvars \
> +tree_freebsd=$TREE_FREEBSD   \
> +revision_freebsd=$REVISION_FREEBSD   \
> +extra_hostflags=arch-$arch,purpose-build \
> +$flags

See my comments about extra_hostflags in the previous patch.  Use
host_hostflags here.

> diff --git a/sg-run-job b/sg-run-job
> index ceb79800..239446b8 100755
> --- a/sg-run-job
> +++ b/sg-run-job

IWBN to split this out.  Firstly, please split the sg-run-job changes
from the make-flight changes.

Secondly, can you split the sg-run-job patch into two ?  1. change the
meaning of need_build_host but not introduce the FreeBSD versions;
2. add the FreeBSD cases.  sg-run-job is a bit fragile to do this kind
of work in.

Also I'm afraid you're going to find some conflicts with my recent
syslog work.  You will probably want to rebase onto pretest, which I
think is very likely to pass and be pushed soon.

> @@ -53,12 +53,15 @@ proc run-job {job} {
>  set skip_globs [jobdb::read-runvar $flight $job skip_testids]
>  
>  set nh [need-hosts/$jobinfo(recipe)]
> -if {![string compare $nh BUILD]} {
> +if {![string compare $nh BUILD_LINUX]} {

How about
   if {[string match $nh BUILD_*]} {
and then set need_build_host to the RHS ?

> -if {$need_build_host} { catching-otherwise broken prepare-build-host }
> +if {![string compare $need_build_host LINUX]} {
> +catching-otherwise broken prepare-build-host-linux
> +} elseif {![string compare $need_build_host FREEBSD]}  {
> +catching-otherwise broken prepare-build-host-freebsd
> +}

Please use switch(3tcl).

> -proc prepare-build-host {} {
> +proc prepare-build-host-linux {} {

This rename should be in the first patch.

Ian.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [OSSTEST PATCH v11 16/20] ts-openstack-tempest: Update list of skipped tests

[Anthony PERARD]

On Fri, Jun 23, 2017 at 02:47:37PM +0100, Ian Jackson wrote:
> Anthony PERARD writes ("[OSSTEST PATCH v11 16/20] ts-openstack-tempest: 
> Update list of skipped tests"):
> > Signed-off-by: Anthony PERARD 
> 
> Again, does this not mean we're going to suffer a maintenance burden
> as tempest grows new inapplicable tests ?

That exactly what is happening with the OpenStack CI loop, from time to
time, there are new tests, that can fail, and diffenrent ways to fix
this (fix the bug, fix a configuration, or just skip the tests).

Recently, we've actually push the list of tests to skip into nova.git,
but I think it is only available in master.

> Other possibilities that come to my mind: ideally the tempest tests
> would have metadata so that particular classes of tests could be
> skipped.  Alternatively, if they aren't disruptive or very slow, if
> run, we could run them anyway as substeps.
> 
> Which makes me think: maybe the tempest tests want to be substeps
> anyway.  Is that possible ?  (Does tempest speak subunit or
> something ?)

I think it is subunit, yes. And I've got the command that is been run by
tempest (as it's printed on stdout).

running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} 
${OS_TEST_PATH:-./tempest/test_discover} --list

then several:
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} 
${OS_TEST_PATH:-./tempest/test_discover}  --load-list /tmp/tmpzSNsrB



-- 
Anthony PERARD

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v2] x86emul: correct CF output of SHLD/SHRD

[Andrew Cooper]

On 23/06/17 15:38, Jan Beulich wrote:
> CF reflects the last bit shifted out, i.e. can't possibly be derived
> from the result value.
>
> Signed-off-by: Jan Beulich 

Reviewed-by: Andrew Cooper 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v3 6/8] osstest: introduce a script to set the hostflags for FreeBSD jobs

[Ian Jackson]

Roger Pau Monne writes ("[PATCH v3 6/8] osstest: introduce a script to set the 
hostflags for FreeBSD jobs"):
> Due to the nature of the FreeBSD install media, which is
> self-generated from the ts-freebsd-build script, the hostflags set to
> FreeBSD jobs are related to the current version under test.
> 
> The following hostflags might need to be fetched from the runvars of a
> previous build-$arch-freebsd job:

I think it is wrong to have a script set the normally baked-in runvar
host_hostflags.

In fact, I don't know how this script can have worked for you.
Currently many build jobs have the runvar "host_hostflags" including
many flags including "arch-i386" say, which I assume your FreeBSD
build jobs will have from make-flight too.  (It is forbidden, and
prevented, for a ts-* script to use store_runvar to modify a runvar
provided as part of the job definition.)

I think you should probably invent something like
  runtime_IDENT_hostflags
and teach ts-hosts-allocate-Executive about it.

> +sub get_freebsd_image_hash() {
> +my $distpath =  $r{"freebsd_distpath"} ||
> +get_stashed("path_freebsddist", $r{"freebsdbuildjob"});
> +
> +return `sha256sum $distpath/install.img|head -c 16`;

This pattern again.  I commented on it before, but now that you are
repeating it, it should become a helper function.

Now that I think about this some more, why not use Digest::SHA and
$sha->addfile ?

> +store_runvar("host_hostflags", $r{"extra_hostflags"} .
> + ",share-build-freebsd-$arch-$hash,freebsd-$version");

"extra_hostflags" would be the host flags for the host ident extra.

Ian.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [PATCH v2] passthrough: give XEN_DOMCTL_test_assign_device more sane semantics

[Jan Beulich]

So far callers of the libxc interface passed in a domain ID which was
then ignored in the hypervisor. Instead, make the hypervisor honor it
(accepting DOMID_INVALID to obtain original behavior), allowing to
query whether a device can be assigned to a particular domain.

Signed-off-by: Jan Beulich 
---
v2: Alter the semantics to check whether the device can be assigned to
the passed in domain.
TBD: It's not clear to me whether the test-assign XSM hooks are still
 useful this way.

--- a/xen/common/domctl.c
+++ b/xen/common/domctl.c
@@ -391,11 +391,15 @@ long do_domctl(XEN_GUEST_HANDLE_PARAM(xe
 
 switch ( op->cmd )
 {
-case XEN_DOMCTL_createdomain:
 case XEN_DOMCTL_test_assign_device:
+if ( op->domain == DOMID_INVALID )
+{
+case XEN_DOMCTL_createdomain:
 case XEN_DOMCTL_gdbsx_guestmemio:
-d = NULL;
-break;
+d = NULL;
+break;
+}
+/* fall through */
 default:
 d = rcu_lock_domain_by_id(op->domain);
 if ( !d && op->cmd != XEN_DOMCTL_getdomaininfo )
--- a/xen/drivers/passthrough/device_tree.c
+++ b/xen/drivers/passthrough/device_tree.c
@@ -143,12 +143,15 @@ int iommu_do_dt_domctl(struct xen_domctl
 switch ( domctl->cmd )
 {
 case XEN_DOMCTL_assign_device:
+ASSERT(d);
+/* fall through */
+case XEN_DOMCTL_test_assign_device:
 ret = -ENODEV;
 if ( domctl->u.assign_device.dev != XEN_DOMCTL_DEV_DT )
 break;
 
 ret = -EINVAL;
-if ( d->is_dying || domctl->u.assign_device.flags )
+if ( (d && d->is_dying) || domctl->u.assign_device.flags )
 break;
 
 ret = dt_find_node_by_gpath(domctl->u.assign_device.u.dt.path,
@@ -157,10 +160,22 @@ int iommu_do_dt_domctl(struct xen_domctl
 if ( ret )
 break;
 
-ret = xsm_assign_dtdevice(XSM_HOOK, d, dt_node_full_name(dev));
+ret = d ? xsm_assign_dtdevice(XSM_HOOK, d, dt_node_full_name(dev))
+: xsm_test_assign_dtdevice(XSM_HOOK, dt_node_full_name(dev));
 if ( ret )
 break;
 
+if ( domctl->cmd == XEN_DOMCTL_test_assign_device )
+{
+if ( iommu_dt_device_is_assigned(dev) )
+{
+printk(XENLOG_G_ERR "%s already assigned.\n",
+   dt_node_full_name(dev));
+ret = -EINVAL;
+}
+break;
+}
+
 ret = iommu_assign_dt_device(d, dev);
 
 if ( ret )
@@ -194,33 +209,6 @@ int iommu_do_dt_domctl(struct xen_domctl
dt_node_full_name(dev), d->domain_id, ret);
 break;
 
-case XEN_DOMCTL_test_assign_device:
-ret = -ENODEV;
-if ( domctl->u.assign_device.dev != XEN_DOMCTL_DEV_DT )
-break;
-
-ret = -EINVAL;
-if ( domctl->u.assign_device.flags )
-break;
-
-ret = dt_find_node_by_gpath(domctl->u.assign_device.u.dt.path,
-domctl->u.assign_device.u.dt.size,
-);
-if ( ret )
-break;
-
-ret = xsm_test_assign_dtdevice(XSM_HOOK, dt_node_full_name(dev));
-if ( ret )
-break;
-
-if ( iommu_dt_device_is_assigned(dev) )
-{
-printk(XENLOG_G_ERR "%s already assigned.\n",
-   dt_node_full_name(dev));
-ret = -EINVAL;
-}
-break;
-
 default:
 ret = -ENOSYS;
 break;
--- a/xen/drivers/passthrough/pci.c
+++ b/xen/drivers/passthrough/pci.c
@@ -1579,35 +1579,10 @@ int iommu_do_pci_domctl(
 }
 break;
 
-case XEN_DOMCTL_test_assign_device:
-ret = -ENODEV;
-if ( domctl->u.assign_device.dev != XEN_DOMCTL_DEV_PCI )
-break;
-
-ret = -EINVAL;
-if ( domctl->u.assign_device.flags )
-break;
-
-machine_sbdf = domctl->u.assign_device.u.pci.machine_sbdf;
-
-ret = xsm_test_assign_device(XSM_HOOK, machine_sbdf);
-if ( ret )
-break;
-
-seg = machine_sbdf >> 16;
-bus = PCI_BUS(machine_sbdf);
-devfn = PCI_DEVFN2(machine_sbdf);
-
-if ( device_assigned(seg, bus, devfn) )
-{
-printk(XENLOG_G_INFO
-   "%04x:%02x:%02x.%u already assigned, or non-existent\n",
-   seg, bus, PCI_SLOT(devfn), PCI_FUNC(devfn));
-ret = -EINVAL;
-}
-break;
-
 case XEN_DOMCTL_assign_device:
+ASSERT(d);
+/* fall through */
+case XEN_DOMCTL_test_assign_device:
 /* Don't support self-assignment of devices. */
 if ( d == current->domain )
 {
@@ -1621,12 +1596,15 @@ int iommu_do_pci_domctl(
 
 ret = -EINVAL;
 flags = domctl->u.assign_device.flags;
-if ( d->is_dying || (flags & ~XEN_DOMCTL_DEV_RDM_RELAXED) )
+if ( domctl->cmd == 

Re: [Xen-devel] [PATCH v3 5/8] osstest: introduce a FreeBSD build script

[Ian Jackson]

Roger Pau Monne writes ("[PATCH v3 5/8] osstest: introduce a FreeBSD build 
script"):
> The following new helpers are also introduced, that are only used by
> the FreeBSD build script: buildcmd_stamped_logged_root and
> target_cmd_build_root, they behave exactly the same as the non-root
> counterparts.

Please split these out into a separate patch.

> diff --git a/ts-freebsd-build b/ts-freebsd-build
> new file mode 100755
> index ..6c10eece
> --- /dev/null
> +++ b/ts-freebsd-build
...
> +sub install_deps () {
> +target_cmd_root($ho, 'pkg-static install git', 300);
> +}

This needs to be concurrency-safe, I think, since it might run
simultaneously in differnet jobs.  Is it ?

I wonder if the right answer is for you to call
   target_install_packages
and teach target_install_packages a mapping from the Debian package
names to FreeBSD ones, and how to do package installation on FreeBSD.

> +logm("Cleaning up previous builds");
> +buildcmd_stamped_logged(300, 'freebsd', 'cleanworld',
> +$prefix, 'make cleanworld', '');
> +
> +logm("Building world");
> +buildcmd_stamped_logged(25200, 'freebsd', 'buildworld',
> +$prefix, < +make $makeflags buildworld
> +END
> +
> +logm("Building kernel");
> +buildcmd_stamped_logged(3600, 'freebsd', 'buildkernel',
> +$prefix, < +make $makeflags buildkernel
> +END

These are quite formulaic, aren't they ?  Maybe you want to make a
sub for them (either global in this file, or an anon subref).

> +# Set path_freebsddist to point to the build output folder
> +# in order to make ts-build-check happy.
> +store_runvar("path_freebsddist", "build/");

Heh.  Fine by me.

Ian.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 19/27] x86: move hypercall_page_initialise_ring3_kernel to pv/hypercall.c

[Andrew Cooper]

On 23/06/17 15:49, Wei Liu wrote:
> On Fri, Jun 23, 2017 at 01:41:29PM +0100, Andrew Cooper wrote:
>> On 08/06/17 18:11, Wei Liu wrote:
>>> Signed-off-by: Wei Liu 
>>> ---
>>>  xen/arch/x86/pv/hypercall.c | 36 
>>>  xen/arch/x86/x86_64/traps.c | 36 
>>>  xen/include/asm-x86/hypercall.h |  1 +
>>>  3 files changed, 37 insertions(+), 36 deletions(-)
>>>
>>> diff --git a/xen/arch/x86/pv/hypercall.c b/xen/arch/x86/pv/hypercall.c
>>> index 7c5e5a629d..287340e774 100644
>>> --- a/xen/arch/x86/pv/hypercall.c
>>> +++ b/xen/arch/x86/pv/hypercall.c
>>> @@ -255,6 +255,42 @@ enum mc_disposition arch_do_multicall_call(struct 
>>> mc_state *state)
>>>   ? mc_continue : mc_preempt;
>>>  }
>>>  
>>> +void hypercall_page_initialise_ring3_kernel(void *hypercall_page)
>>> +{
>>> +char *p;
>>> +int i;
>> void *p = hypercall_page;
>> unsigned int i;
>>
>>> +
>>> +/* Fill in all the transfer points with template machine code. */
>>> +for ( i = 0; i < (PAGE_SIZE / 32); i++ )
>>> +{
>> i++, p += 32
> Wait, I think pointer arithmetic is not allowed on void* ?

void pointer arithmetic is a GCCism, which we use in plenty of other
locations.

~Andrew

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 19/27] x86: move hypercall_page_initialise_ring3_kernel to pv/hypercall.c

[Wei Liu]

On Fri, Jun 23, 2017 at 01:41:29PM +0100, Andrew Cooper wrote:
> On 08/06/17 18:11, Wei Liu wrote:
> > Signed-off-by: Wei Liu 
> > ---
> >  xen/arch/x86/pv/hypercall.c | 36 
> >  xen/arch/x86/x86_64/traps.c | 36 
> >  xen/include/asm-x86/hypercall.h |  1 +
> >  3 files changed, 37 insertions(+), 36 deletions(-)
> >
> > diff --git a/xen/arch/x86/pv/hypercall.c b/xen/arch/x86/pv/hypercall.c
> > index 7c5e5a629d..287340e774 100644
> > --- a/xen/arch/x86/pv/hypercall.c
> > +++ b/xen/arch/x86/pv/hypercall.c
> > @@ -255,6 +255,42 @@ enum mc_disposition arch_do_multicall_call(struct 
> > mc_state *state)
> >   ? mc_continue : mc_preempt;
> >  }
> >  
> > +void hypercall_page_initialise_ring3_kernel(void *hypercall_page)
> > +{
> > +char *p;
> > +int i;
> 
> void *p = hypercall_page;
> unsigned int i;
> 
> > +
> > +/* Fill in all the transfer points with template machine code. */
> > +for ( i = 0; i < (PAGE_SIZE / 32); i++ )
> > +{
> 
> i++, p += 32

Wait, I think pointer arithmetic is not allowed on void* ?

> 
> Otherwise, Reviewed-by: Andrew Cooper 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [OSSTEST PATCH v11 15/20] ts-openstack-tempest: Fix tempest invocation

[Anthony PERARD]

On Fri, Jun 23, 2017 at 02:45:12PM +0100, Ian Jackson wrote:
> Anthony PERARD writes ("[OSSTEST PATCH v11 15/20] ts-openstack-tempest: Fix 
> tempest invocation"):
> > ./run_tempest.sh is deprecated.
> ...
> >  target_cmd($ho, < >  set -e
> > -$builddir/tempest/run_tempest.sh --virtual-env -- --concurrency=2 '$regex'
> > +cd $builddir/tempest
> > +tempest run --concurrency=2 --regex '$regex'
> 
> Has /usr/local/bin/tempest or something been created by
> ts-openstay-deploy, then ?

Yes.

> If so,
> 
>   Acked-by: Ian Jackson 
> 
> Otherwise I wonder how this works, since I don't see how `tempest'
> would be on PATH.
> 
> Ian.

-- 
Anthony PERARD

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH for-4.9 v3 3/3] xen/livepatch: Don't crash on encountering STN_UNDEF relocations

[Konrad Rzeszutek Wilk]

On Fri, Jun 23, 2017 at 03:36:51PM +0100, Julien Grall wrote:
> 
> 
> On 23/06/17 15:35, Konrad Rzeszutek Wilk wrote:
> > On Fri, Jun 23, 2017 at 02:45:22PM +0100, Andrew Cooper wrote:
> > > On 23/06/17 14:43, Julien Grall wrote:
> > > > Hi,
> > > > 
> > > > On 23/06/17 14:33, Andrew Cooper wrote:
> > > > > On 23/06/17 14:32, Julien Grall wrote:
> > > > > > Hi Andrew,
> > > > > > 
> > > > > > I am a bit confused, the title says "PATCH for-4.9 v3 3/3". I 
> > > > > > haven't
> > > > > > been CCed on the first two patches. Does it mean you are only 
> > > > > > looking
> > > > > > at this patch to be in 4.9?
> > > > > 
> > > > > Sorry - I messed up the CC lists.  The correctness of this patch does
> > > > > depend on the previous two, so all 3 are looking for inclusion.
> > > > 
> > > > Given that we don't have livepatch testing in osstest how much test
> > > > have we done on those 3 patches?
> > > 
> > > There is testing in OSSTest.
> > 
> > Hurray hurray hurray!
> > > 
> > > I've manually run each of the scenarios, including with my livepatch
> > > which has a STN_UNDEF relocation.
> > > 
> > > I don't know what testing Konrad has done.
> > 
> > I run a version of the same tests that are in OSSTest (basically an earlier
> > version of the Perl code) and I have done it on x86 and on ARM32.
> > 
> > And I also run the standalone OSSTest (on x86)
> > 
> > And then I also do a livepatch using the livepatch-build-tools on x86 to
> > patch some silly function.
> > 
> > So from a testing perspective these patches have been tested very 
> > exhaustively.
> 
> Well it has not been tested on ARM64 :). I am about to do that.

/me facepalm.

I really need to get myself a working ARM64 box that is not expensive.


Also attached is the poor-man livepatch_test.perl script that mirrors
what OSSTest does.

[Do adjust the path, it is /usr/lib/debug, but it should be 
/usr/lib/debug/livepatch
with Xen 4.9]

#!/usr/bin/perl

use Data::Dumper;

my @livepatch_files = ("xen_hello_world.livepatch", 
"xen_replace_world.livepatch",
"xen_bye_world.livepatch", "xen_nop.livepatch");

my @livepatch_tests = (
{cmd => "xen-livepatch list", rc => 0},
{cmd => "xl info | grep xen_extra | grep -q \"Hello World\"", rc => 
256},
{cmd => "xen-livepatch revert xen_hello_world", rc => 256},
{cmd => "xen-livepatch load xen_hello_world.livepatch", rc => 0},
{cmd => "xen-livepatch load xen_hello_world.livepatch", rc => 256},
{cmd => "xen-livepatch list | grep -q xen_hello_world", rc => 0},
{cmd => "xl info | grep xen_extra | grep -q \"Hello World\"", rc => 0},
{cmd => "xen-livepatch revert xen_hello_world", rc => 0},
{cmd => "xl info | grep xen_extra | grep -q \"Hello World\"", rc => 
256},
{cmd => "xen-livepatch unload xen_hello_world", rc => 0},
{cmd => "xen-livepatch unload xen_hello_world", rc => 256},
{cmd => "xl info | grep xen_extra | grep -q \"Hello World\"", rc => 
256},
{cmd => "xen-livepatch load xen_hello_world.livepatch", rc => 0},
{cmd => "xen-livepatch load xen_bye_world.livepatch", rc => 0},
{cmd => "xl info | grep xen_extra | grep -q \"Bye World\"", rc => 0},
{cmd => "xen-livepatch upload xen_replace xen_replace_world.livepatch", 
rc => 0},
{cmd => "xen-livepatch replace xen_replace", rc => 0},
{cmd => "xen-livepatch apply xen_hello_world", rc => 256},
{cmd => "xen-livepatch apply xen_bye_world", rc => 256},
{cmd => "xl info | grep xen_extra | grep -q \"Hello Again Wor\"", rc => 
0},
{cmd => "xen-livepatch apply xen_replace", rc => 0},
{cmd => "xen-livepatch revert xen_replace", rc => 0},
{cmd => "xen-livepatch unload xen_replace", rc => 0},
{cmd => "xen-livepatch unload xen_hello_world", rc => 0},
{cmd => "xen-livepatch unload xen_bye_world", rc => 0},
{cmd => "xen-livepatch list | grep -q xen", rc => 256},
# If running this under Xen 4.4, or 5.5 it will fail.
#{cmd => "[ `xl info| grep \"xen_m\" | grep or | sed s/.*:// | uniq | 
wc -l` == 2 ]", rc => 0},
{cmd => "xen-livepatch load xen_nop.livepatch", rc => 0},
{cmd => "xen-livepatch revert xen_nop", rc => 0},
{cmd => "xen-livepatch apply xen_nop", rc => 0},
{cmd => "[ `xl info| grep \"xen_m\" | grep or | sed s/.*:// | uniq | wc 
-l` == 1 ]", rc => 0},
{cmd => "xen-livepatch unload xen_nop", rc => 256},
{cmd => "xen-livepatch revert xen_nop", rc => 0},
{cmd => "xen-livepatch unload xen_nop", rc => 0},
);

chdir("/usr/lib/debug") or die "cannot change: $!\n";

foreach my $file (@livepatch_files)
{
if ( ! -f $file ) {
die "$file is missing!\n";
}
}
print "Have $#livepatch_tests test-cases\n";
my $rc=0;
for my $i ( 0 .. $#livepatch_tests ) {
my $expected_rc = $livepatch_tests[$i]{rc};
my $cmd = $livepatch_tests[$i]{cmd};

Re: [Xen-devel] [PATCH v3 4/8] osstest: add a FreeBSD host install recipe

[Ian Jackson]

Roger Pau Monne writes ("[PATCH v3 4/8] osstest: add a FreeBSD host install 
recipe"):
> The installation is performed using the bsdinstall tool, which is part
> of the FreeBSD base system. The installer image is setup with the
> osstest ssh keys and sshd enabled by default, which allows the test
> harness to just ssh into the box, create the install config file and
> launch the scripted install.

thanks.

> +# The installer image must be named 'install.img', and the sets
> +# 'kernel.txz', 'base.txz' and finally the 'MANIFEST' file that contains
> +# the checksums.

Thanks for the good doc comment.

> +sub setup_netboot_installer () {
> +my $image = "$path_prefix/install.img";
> +my $pxeimg = target_tftp_prefix($ho) . "--freebsd.img";
> +my $hash = `sha256sum $image | head -c 16` or die $!;

You should do this `  -' stripping in Perl.  That way, also, you won't
lose the exit status from sha256sum as you do here.

> +my $tftp_freebsd = 
> "$ho->{Tftp}{Path}/$ho->{Tftp}{TmpDir}/freebsd-images/";
> +my $script = <<'END';
> +basedir=$0
> +imagepath=$1
> +sharedpath=$2
> +targetpath=$3

Please pass a dummy argument to the script (I usually use `x')
and shift all these arguments along.  Passing a real argument as $0 is
IMO strange.

> +cd $basedir
> +if [ ! -f $sharedpath ]; then
> +mkdir -p `dirname $sharedpath`
> +cp $imagepath $sharedpath

Please use the copy-to-tempfile-and-rename pattern.

AFAICT your filename pattern for the per-hash filename is
   $tftp_freebsd/$r{arch}/$hash/install.img

Is there some reason why this needs
 - to be qualified with $r{arch}
 - to have a bunch of per-hash directories containing one file each
?

Why not
   $tftp_freebsd/by-hash/$hash.img
?

> +# Dir format from basedir is $arch/$hash/install.img
> +for hashdir in `find -mindepth 2 -maxdepth 2 -type d`; do

1. use xargs or -exec -rm
2. use find -links
3. add -ctime +7 or something, so we don't delete things which
   have just been added (or used)

> +my @cmd = ( "with-lock-ex", "-w", "$tftp_freebsd/lock",
> +"bash", "-exc", "$script",
> +"$tftp_freebsd", "$image", "$r{arch}/$hash/install.img",
> +"$ho->{Tftp}{Path}/$pxeimg" );

Use of qw() would make this a lot less visually noisy.

> +ensuredir("$tftp_freebsd");

" " are redundant in Perl.  It's not sh :-).

> +logm("Trying to find a disk to install to");
> +$disk = target_cmd_output_root($ho, < +for disk in @disk_names; do

Did you want to add set -e ?

> +if [ -c "/dev/\$disk" ]; then
> +echo \$disk
> +exit 0
> +fi
> +done
> +exit 1
> +END
> +defined($disk) or die "Unable to find a valid disk";

target_cmd_output_root will never return undef.  I think this
error check is redundant, therefore.

> +logm("Trying to figure out primary nic device name");
> +$nic = target_cmd_output_root($ho, < +nics=`ifconfig -l`

I think this definitely needs set -e.

> +for nic in \$nics; do
> +addr=`ifconfig \$nic inet|grep inet|awk {'print \$2'}`
> +if [ "\$addr" = "$ho->{Ip}" ]; then
> +echo \$nic
> +exit 0
> +fi
> +done
> +exit 1
> +END

Is it likely that the disk or network device name, for a particular
device, will change, for example across versions of FreeBSD ?

> +logm("Uploading the install sets to the system");
> +target_cmd_root($ho, < +mkdir -p $target_sets

Missing set -e.

> +logm("Creating the installer script");
> +target_putfilecontents_root_stash($ho, 10, < +set -a
> +BSDINSTALL_DISTDIR="$target_sets"
> +ZFSBOOT_DISKS="$disk"
> +DISTRIBUTIONS="@sets"
> +nonInteractive=1
> +
> +#!/bin/sh
> +set -ex

There's a #! halfway through this script.

> +# Setup serial console
> +printf "%s" "-h -S$c{Baud}" >> /boot.config

Are you deliberately leaving /boot.config with no final newline ?

> +cat << ENDBOOT >> /boot/loader.conf
> +boot_serial="YES"
> +comconsole_speed="$c{Baud}"
> +console="comconsole"
> +boot_verbose="YES"
> +beastie_disable="YES"

:-) re beastie.


Ian.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [OSSTEST PATCH v11 12/20] make-flight: Increase dom0_mem for openstack flight

[Anthony PERARD]

On Fri, Jun 23, 2017 at 02:42:53PM +0100, Ian Jackson wrote:
> Anthony PERARD writes ("[OSSTEST PATCH v11 12/20] make-flight: Increase 
> dom0_mem for openstack flight"):
> > With 4G for dom0_mem, a host running devstack is using about 1.5G of
> > swap.
> 
> Is this going to work properly on 8G hosts ?

Yes, it is fine. I usually do my OpenStack testing on 8G hosts. The CI
loop is running with testing on 8G, and have a bit more than 7G for
dom0. Tempest only run small VM (64MB or 128MB) and only a few at a
time.

So yes.

-- 
Anthony PERARD

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH] x86/mm: Rename d to currd in do_mmuext_op()

[Jan Beulich]

>>> On 23.06.17 at 16:26,  wrote:
> This will make future cleanup more obviously correct.  No functional change.
> 
> Signed-off-by: Andrew Cooper 

Acked-by: Jan Beulich 



___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [PATCH v2] x86emul: correct CF output of SHLD/SHRD

[Jan Beulich]

CF reflects the last bit shifted out, i.e. can't possibly be derived
from the result value.

Signed-off-by: Jan Beulich 
---
v2: Fix 64-bit testcase build.

--- a/tools/tests/x86_emulator/test_x86_emulator.c
+++ b/tools/tests/x86_emulator/test_x86_emulator.c
@@ -783,6 +783,31 @@ int main(int argc, char **argv)
 printf("okay\n");
 #endif
 
+printf("%-40s", "Testing shld $1,%ecx,(%edx)...");
+res[0]  = 0x12345678;
+regs.edx= (unsigned long)res;
+regs.ecx= 0x9abcdef0;
+instr[0] = 0x0f; instr[1] = 0xa4; instr[2] = 0x0a; instr[3] = 0x01;
+for ( i = 0; i < 0x20; ++i )
+{
+uint32_t r = res[0];
+const uint32_t m = X86_EFLAGS_ARITH_MASK & ~X86_EFLAGS_AF;
+unsigned long f;
+
+asm ( "shld $1,%2,%0; pushf; pop %1"
+  : "+rm" (r), "=rm" (f) : "r" ((uint32_t)regs.ecx) );
+regs.eflags = f ^ m;
+regs.eip= (unsigned long)[0];
+rc = x86_emulate(, );
+if ( (rc != X86EMUL_OKAY) ||
+ (regs.eip != (unsigned long)[4]) ||
+ (res[0] != r) ||
+ ((regs.eflags ^ f) & m) )
+goto fail;
+regs.ecx <<= 1;
+}
+printf("okay\n");
+
 printf("%-40s", "Testing movbe (%ecx),%eax...");
 instr[0] = 0x0f; instr[1] = 0x38; instr[2] = 0xf0; instr[3] = 0x01;
 regs.eflags = 0x200;
--- a/xen/arch/x86/x86_emulate/x86_emulate.c
+++ b/xen/arch/x86/x86_emulate/x86_emulate.c
@@ -6424,7 +6424,7 @@ x86_emulate(
(src.val >> (width - shift)));
 _regs.eflags &= ~(X86_EFLAGS_OF | X86_EFLAGS_SF | X86_EFLAGS_ZF |
   X86_EFLAGS_PF | X86_EFLAGS_CF);
-if ( (dst.val >> ((b & 8) ? (shift - 1) : (width - shift))) & 1 )
+if ( (dst.orig_val >> ((b & 8) ? (shift - 1) : (width - shift))) & 1 )
 _regs.eflags |= X86_EFLAGS_CF;
 if ( ((dst.val ^ dst.orig_val) >> (width - 1)) & 1 )
 _regs.eflags |= X86_EFLAGS_OF;



x86emul: correct CF output of SHLD/SHRD

CF reflects the last bit shifted out, i.e. can't possibly be derived
from the result value.

Signed-off-by: Jan Beulich 
---
v2: Fix 64-bit testcase build.

--- a/tools/tests/x86_emulator/test_x86_emulator.c
+++ b/tools/tests/x86_emulator/test_x86_emulator.c
@@ -783,6 +783,31 @@ int main(int argc, char **argv)
 printf("okay\n");
 #endif
 
+printf("%-40s", "Testing shld $1,%ecx,(%edx)...");
+res[0]  = 0x12345678;
+regs.edx= (unsigned long)res;
+regs.ecx= 0x9abcdef0;
+instr[0] = 0x0f; instr[1] = 0xa4; instr[2] = 0x0a; instr[3] = 0x01;
+for ( i = 0; i < 0x20; ++i )
+{
+uint32_t r = res[0];
+const uint32_t m = X86_EFLAGS_ARITH_MASK & ~X86_EFLAGS_AF;
+unsigned long f;
+
+asm ( "shld $1,%2,%0; pushf; pop %1"
+  : "+rm" (r), "=rm" (f) : "r" ((uint32_t)regs.ecx) );
+regs.eflags = f ^ m;
+regs.eip= (unsigned long)[0];
+rc = x86_emulate(, );
+if ( (rc != X86EMUL_OKAY) ||
+ (regs.eip != (unsigned long)[4]) ||
+ (res[0] != r) ||
+ ((regs.eflags ^ f) & m) )
+goto fail;
+regs.ecx <<= 1;
+}
+printf("okay\n");
+
 printf("%-40s", "Testing movbe (%ecx),%eax...");
 instr[0] = 0x0f; instr[1] = 0x38; instr[2] = 0xf0; instr[3] = 0x01;
 regs.eflags = 0x200;
--- a/xen/arch/x86/x86_emulate/x86_emulate.c
+++ b/xen/arch/x86/x86_emulate/x86_emulate.c
@@ -6424,7 +6424,7 @@ x86_emulate(
(src.val >> (width - shift)));
 _regs.eflags &= ~(X86_EFLAGS_OF | X86_EFLAGS_SF | X86_EFLAGS_ZF |
   X86_EFLAGS_PF | X86_EFLAGS_CF);
-if ( (dst.val >> ((b & 8) ? (shift - 1) : (width - shift))) & 1 )
+if ( (dst.orig_val >> ((b & 8) ? (shift - 1) : (width - shift))) & 1 )
 _regs.eflags |= X86_EFLAGS_CF;
 if ( ((dst.val ^ dst.orig_val) >> (width - 1)) & 1 )
 _regs.eflags |= X86_EFLAGS_OF;
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH for-4.9 v3 3/3] xen/livepatch: Don't crash on encountering STN_UNDEF relocations

[Julien Grall]

On 23/06/17 15:35, Konrad Rzeszutek Wilk wrote:

On Fri, Jun 23, 2017 at 02:45:22PM +0100, Andrew Cooper wrote:

On 23/06/17 14:43, Julien Grall wrote:

Hi,

On 23/06/17 14:33, Andrew Cooper wrote:

On 23/06/17 14:32, Julien Grall wrote:

Hi Andrew,

I am a bit confused, the title says "PATCH for-4.9 v3 3/3". I haven't
been CCed on the first two patches. Does it mean you are only looking
at this patch to be in 4.9?


Sorry - I messed up the CC lists.  The correctness of this patch does
depend on the previous two, so all 3 are looking for inclusion.


Given that we don't have livepatch testing in osstest how much test
have we done on those 3 patches?


There is testing in OSSTest.


Hurray hurray hurray!


I've manually run each of the scenarios, including with my livepatch
which has a STN_UNDEF relocation.

I don't know what testing Konrad has done.


I run a version of the same tests that are in OSSTest (basically an earlier
version of the Perl code) and I have done it on x86 and on ARM32.

And I also run the standalone OSSTest (on x86)

And then I also do a livepatch using the livepatch-build-tools on x86 to
patch some silly function.

So from a testing perspective these patches have been tested very exhaustively.


Well it has not been tested on ARM64 :). I am about to do that.

Cheers,

--
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH for-4.9 v3 3/3] xen/livepatch: Don't crash on encountering STN_UNDEF relocations

[Konrad Rzeszutek Wilk]

On Fri, Jun 23, 2017 at 02:45:22PM +0100, Andrew Cooper wrote:
> On 23/06/17 14:43, Julien Grall wrote:
> > Hi,
> >
> > On 23/06/17 14:33, Andrew Cooper wrote:
> >> On 23/06/17 14:32, Julien Grall wrote:
> >>> Hi Andrew,
> >>>
> >>> I am a bit confused, the title says "PATCH for-4.9 v3 3/3". I haven't
> >>> been CCed on the first two patches. Does it mean you are only looking
> >>> at this patch to be in 4.9?
> >>
> >> Sorry - I messed up the CC lists.  The correctness of this patch does
> >> depend on the previous two, so all 3 are looking for inclusion.
> >
> > Given that we don't have livepatch testing in osstest how much test
> > have we done on those 3 patches?
> 
> There is testing in OSSTest.

Hurray hurray hurray!
> 
> I've manually run each of the scenarios, including with my livepatch
> which has a STN_UNDEF relocation.
>
> I don't know what testing Konrad has done.

I run a version of the same tests that are in OSSTest (basically an earlier
version of the Perl code) and I have done it on x86 and on ARM32.

And I also run the standalone OSSTest (on x86)

And then I also do a livepatch using the livepatch-build-tools on x86 to
patch some silly function.

So from a testing perspective these patches have been tested very exhaustively.

> 
> ~Andrew

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 7/9] arm/mem_access: Add long-descriptor based gpt

[Julien Grall]

On 23/06/17 15:23, Sergej Proskurin wrote:

Hi Julien,

[...]


+static bool get_ttbr_and_gran_64bit(uint64_t *ttbr, unsigned int *gran,
+register_t tcr, enum active_ttbr
ttbrx)
+{
+bool disabled;
+
+if ( ttbrx == TTBR0_ACTIVE )
+{
+/* Normalize granule size. */
+switch ( tcr & TCR_TG0_MASK )
+{
+case TCR_TG0_16K:
+*gran = GRANULE_SIZE_INDEX_16K;
+break;
+case TCR_TG0_64K:
+*gran = GRANULE_SIZE_INDEX_64K;
+break;
+default:


Please explain why you use 4K by default.



Fair question. According to ARM DDI 0487B.a D7-2487, if the
TCR_EL1.{TG0|TG1} holds a reserved value, it is implementation defined
how the value is interpreted by the underlying hardware. My
implementation in v4 strongly followed the pseudo-code in ARM DDI
0487B.a, which suggested to use fall back to 4K by default.

However, agree with you at this point. My next patch series explicitly
checks if 4K has to be set or not and returns an error on reserved
values as we cannot be know how the hardware behaves on reserved values.


No. You should not return an error here as you would not be compliant 
with the ARM ARM. I just asked to add a comment explain why you choose 
4K, even if it says "We follow the pseudo-code".




[...]


+
+/*
+ * According to to ARM DDI 0487B.a J1-5927, we return an error if
the found
+ * PTE is invalid or holds a reserved entry (PTE<1:0> == x0)) or
if the PTE
+ * maps a memory block at level 3 (PTE<1:0> == 01).
+ */
+if ( !lpae_valid(pte) || ((level == 3) && lpae_mapping(pte)) )


If you look at the comment on top of lpae_mapping, it should only be
used for L0..L2 ptes. So why are you using it on L3 ptes?



Yes, I saw the comment. Yet, I would like to check exactly for this
mapping. If the mapping would as in the check above, it would be an
error, which is treated accordingly. In v3, you have suggested to look
at the the lpae_is_superpage macro, which, in my opinion, is not the
right construct to use at this point. If you should not like this check,
I could fall back to my previous implementation:

if ( !p2m_valid(pte) || ((level == 3) && !p2m_table(pte)) )
return -EFAULT;


If you look at the ARM ARM (D4.3.1 and D4.3.2 in DDI0487B.a)
* level 0,1,2 will have bit 1 set for table and cleared for mapping.
* level 3 will have bit 1 set for mapping

If you use p2m_table to check it the bit is set, then it mislead 
completely the user as this entry is not a table at all.


In any, this is totally wrong to use p2m_table and p2m_mapping here as 
it would not report correctly the value. So please don't use an helper 
that does not make sense here. You should just open-code it or introduce 
a helper (such as lpae_page with appropriate) to properly check the bit.


Cheers,

--
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [OSSTEST PATCH v11 11/20] ts-openstack-deploy: Increase fd and memory limits for rabbitmq

[Anthony PERARD]

On Fri, Jun 23, 2017 at 02:41:59PM +0100, Ian Jackson wrote:
> Anthony PERARD writes ("[OSSTEST PATCH v11 11/20] ts-openstack-deploy: 
> Increase fd and memory limits for rabbitmq"):
> > Signed-off-by: Anthony PERARD 
> 
> Does this not mean that the upstream defaults are wrong ?

That depends on what you meant by upstream.

On Ubuntu, the limit of open fd is set to an higher value via the
systemd unit file. It does not look like devstack (OpenStack) is
changing this.

In devstack, I've seen an increate of the limit of open fd, but via
systemd, so for all systemd services.


As for the memory limit, it was necessary with a host of 4G of RAM. But
I did not try again with the default limit and a host with 6G of RAM.
(The default is 0.4, here I've set it to 0.8)

> In general I like to see at least an upstream bug report url or number
> mentioned in a comment near the workaround.
> 
> Ian.

-- 
Anthony PERARD

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH] xen-detect: handle asprintf error

[Andrew Cooper]

On 21/06/17 15:41, Wei Liu wrote:
> Otherwise gcc with -Wunused will complain the return value is not
> used.
>
> Reported-by: Olaf Hering 
> Signed-off-by: Wei Liu 

Acked-by: Andrew Cooper 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [PATCH] x86/mm: Rename d to currd in do_mmuext_op()

[Andrew Cooper]

This will make future cleanup more obviously correct.  No functional change.

Signed-off-by: Andrew Cooper 
---
CC: Jan Beulich 
---
 xen/arch/x86/mm.c | 56 ---
 1 file changed, 29 insertions(+), 27 deletions(-)

diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index 8200b11..b20f37f 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -3138,7 +3138,7 @@ long do_mmuext_op(
 unsigned long type;
 unsigned int i, done = 0;
 struct vcpu *curr = current;
-struct domain *d = curr->domain;
+struct domain *currd = curr->domain;
 struct domain *pg_owner;
 int rc = put_old_guest_table(curr);
 
@@ -3182,7 +3182,7 @@ long do_mmuext_op(
 return -EINVAL;
 }
 
-rc = xsm_mmuext_op(XSM_TARGET, d, pg_owner);
+rc = xsm_mmuext_op(XSM_TARGET, currd, pg_owner);
 if ( rc )
 {
 put_pg_owner(pg_owner);
@@ -3203,7 +3203,7 @@ long do_mmuext_op(
 break;
 }
 
-if ( is_hvm_domain(d) )
+if ( is_hvm_domain(currd) )
 {
 switch ( op.cmd )
 {
@@ -3272,7 +3272,7 @@ long do_mmuext_op(
 break;
 }
 
-rc = xsm_memory_pin_page(XSM_HOOK, d, pg_owner, page);
+rc = xsm_memory_pin_page(XSM_HOOK, currd, pg_owner, page);
 if ( !rc && unlikely(test_and_set_bit(_PGT_pinned,
   >u.inuse.type_info)) )
 {
@@ -3288,7 +3288,7 @@ long do_mmuext_op(
 paging_mark_dirty(pg_owner, _mfn(page_to_mfn(page)));
 
 /* We can race domain destruction (domain_relinquish_resources). */
-if ( unlikely(pg_owner != d) )
+if ( unlikely(pg_owner != currd) )
 {
 bool drop_ref;
 
@@ -3349,9 +3349,9 @@ long do_mmuext_op(
 break;
 
 case MMUEXT_NEW_BASEPTR:
-if ( unlikely(d != pg_owner) )
+if ( unlikely(currd != pg_owner) )
 rc = -EPERM;
-else if ( unlikely(paging_mode_translate(d)) )
+else if ( unlikely(paging_mode_translate(currd)) )
 rc = -EINVAL;
 else
 rc = new_guest_cr3(op.arg1.mfn);
@@ -3360,9 +3360,9 @@ long do_mmuext_op(
 case MMUEXT_NEW_USER_BASEPTR: {
 unsigned long old_mfn;
 
-if ( unlikely(d != pg_owner) )
+if ( unlikely(currd != pg_owner) )
 rc = -EPERM;
-else if ( unlikely(paging_mode_translate(d)) )
+else if ( unlikely(paging_mode_translate(currd)) )
 rc = -EINVAL;
 if ( unlikely(rc) )
 break;
@@ -3379,7 +3379,7 @@ long do_mmuext_op(
 {
 rc = get_page_and_type_from_pagenr(op.arg1.mfn,
PGT_root_page_table,
-   d, 0, 1);
+   currd, 0, 1);
 
 if ( unlikely(rc) )
 {
@@ -3391,7 +3391,8 @@ long do_mmuext_op(
  rc, op.arg1.mfn);
 break;
 }
-if ( VM_ASSIST(d, m2p_strict) )
+
+if ( VM_ASSIST(currd, m2p_strict) )
 zap_ro_mpt(op.arg1.mfn);
 }
 
@@ -3419,14 +3420,14 @@ long do_mmuext_op(
 }
 
 case MMUEXT_TLB_FLUSH_LOCAL:
-if ( likely(d == pg_owner) )
+if ( likely(currd == pg_owner) )
 flush_tlb_local();
 else
 rc = -EPERM;
 break;
 
 case MMUEXT_INVLPG_LOCAL:
-if ( unlikely(d != pg_owner) )
+if ( unlikely(currd != pg_owner) )
 rc = -EPERM;
 else
 paging_invlpg(curr, op.arg1.linear_addr);
@@ -3437,9 +3438,9 @@ long do_mmuext_op(
 {
 cpumask_t *mask = this_cpu(scratch_cpumask);
 
-if ( unlikely(d != pg_owner) )
+if ( unlikely(currd != pg_owner) )
 rc = -EPERM;
-else if ( unlikely(vcpumask_to_pcpumask(d,
+else if ( unlikely(vcpumask_to_pcpumask(currd,
guest_handle_to_param(op.arg2.vcpumask,
  const_void),
mask)) )
@@ -3455,32 +3456,33 @@ long do_mmuext_op(
 }
 
 case MMUEXT_TLB_FLUSH_ALL:
-if ( likely(d == pg_owner) )
-flush_tlb_mask(d->domain_dirty_cpumask);
+if ( likely(currd == pg_owner) )
+flush_tlb_mask(currd->domain_dirty_cpumask);
 else
 rc = -EPERM;
 break;
 
 case MMUEXT_INVLPG_ALL:
-if ( unlikely(d != pg_owner) )
+if ( unlikely(currd != 

Re: [Xen-devel] [PATCH v3 3/8] osstest: introduce helper to get per-host tftp prefix

[Ian Jackson]

Roger Pau Monne writes ("[PATCH v3 3/8] osstest: introduce helper to get 
per-host tftp prefix"):
> This is used in order to get the per-host tftp prefix, used to store
> the host initrd file.

Acked-by: Ian Jackson 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH RFC] Live migration for VMs with QEMU backed local storage

[Ian Jackson]

Konrad Rzeszutek Wilk writes ("Re: [Xen-devel] [PATCH RFC] Live migration for 
VMs with QEMU backed local storage"):
> On Fri, Jun 23, 2017 at 03:31:16AM -0400, Bruno Alvisio wrote:
> > disks). This are the ones I can think of:
> > - Fully Virtualized HVM: QEMU emulation
> > - blkback
> > - blktap / blktap2 
> 
> You are missing 'qdisk' which is the QEMU implemenation of blkback.

Some people use drbd for this but that has its own mirroring built-in.

Doing some kind of poor-man's mirroring with blkback, devmapper and
nbd might well be a useful approach too.

Ian.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [OSSTEST PATCH v11 10/20] ts-openstack-deploy: Switch to Neutron for network

[Ian Jackson]

Anthony PERARD writes ("Re: [OSSTEST PATCH v11 10/20] ts-openstack-deploy: 
Switch to Neutron for network"):
> On Fri, Jun 23, 2017 at 02:41:02PM +0100, Ian Jackson wrote:
> > Is this kind of thing going to be common ?  If so then it will be a
> > constant maintenance burden in osstest.
> 
> No, that's not common. I don't think it's going to happen again anytime
> soon.

OK, fine.

> > Is there some way we can get the list of components out of the
> > top-level tree ?
> 
> I don't know if there is a simple way to get this list. And if we start
> looking at the list of all OpenStack git trees, there are hundreds of
> them.

Yow.  Let's not.

Acked-by: Ian Jackson 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 14/27] x86: move do_iret to pv/iret.c

[Andrew Cooper]

On 23/06/17 15:17, Wei Liu wrote:
> On Fri, Jun 23, 2017 at 01:12:15PM +0100, Andrew Cooper wrote:
>> On 08/06/17 18:11, Wei Liu wrote:
>>> Signed-off-by: Wei Liu 
>>> ---
>>> There is no copyright header in the original file. Use the default
>>> one?
>> It should at least gain a basic GPLv2 header.
>>
>> Otherwise, LGTM.
>>
> I'm going to insert the following license from CONTRIBUTING:\
>
> /*
>  * pv/iret.c
>  *
>  * iret hypercall handling code
>  *
>  * This program is free software; you can redistribute it and/or
>  * modify it under the terms and conditions of the GNU General Public
>  * License, version 2, as published by the Free Software Foundation.
>  *
>  * This program is distributed in the hope that it will be useful,
>  * but WITHOUT ANY WARRANTY; without even the implied warranty of
>  * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
>  * General Public License for more details.
>  *
>  * You should have received a copy of the GNU General Public
>  * License along with this program; If not, see
>  * .
>  */
>
>
> Can I have your ack or rb?

Reviewed-by: Andrew Cooper 



___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 14/27] x86: move do_iret to pv/iret.c

[Wei Liu]

On Fri, Jun 23, 2017 at 01:12:15PM +0100, Andrew Cooper wrote:
> On 08/06/17 18:11, Wei Liu wrote:
> > Signed-off-by: Wei Liu 
> > ---
> > There is no copyright header in the original file. Use the default
> > one?
> 
> It should at least gain a basic GPLv2 header.
> 
> Otherwise, LGTM.
> 

I'm going to insert the following license from CONTRIBUTING:\

/*
 * pv/iret.c
 *
 * iret hypercall handling code
 *
 * This program is free software; you can redistribute it and/or
 * modify it under the terms and conditions of the GNU General Public
 * License, version 2, as published by the Free Software Foundation.
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
 * General Public License for more details.
 *
 * You should have received a copy of the GNU General Public
 * License along with this program; If not, see
 * .
 */


Can I have your ack or rb?

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 7/9] arm/mem_access: Add long-descriptor based gpt

[Sergej Proskurin]

Hi Julien,

[...]

>> +static bool get_ttbr_and_gran_64bit(uint64_t *ttbr, unsigned int *gran,
>> +register_t tcr, enum active_ttbr
>> ttbrx)
>> +{
>> +bool disabled;
>> +
>> +if ( ttbrx == TTBR0_ACTIVE )
>> +{
>> +/* Normalize granule size. */
>> +switch ( tcr & TCR_TG0_MASK )
>> +{
>> +case TCR_TG0_16K:
>> +*gran = GRANULE_SIZE_INDEX_16K;
>> +break;
>> +case TCR_TG0_64K:
>> +*gran = GRANULE_SIZE_INDEX_64K;
>> +break;
>> +default:
> 
> Please explain why you use 4K by default.
> 

Fair question. According to ARM DDI 0487B.a D7-2487, if the
TCR_EL1.{TG0|TG1} holds a reserved value, it is implementation defined
how the value is interpreted by the underlying hardware. My
implementation in v4 strongly followed the pseudo-code in ARM DDI
0487B.a, which suggested to use fall back to 4K by default.

However, agree with you at this point. My next patch series explicitly
checks if 4K has to be set or not and returns an error on reserved
values as we cannot be know how the hardware behaves on reserved values.

[...]

>> +
>> +/*
>> + * According to to ARM DDI 0487B.a J1-5927, we return an error if
>> the found
>> + * PTE is invalid or holds a reserved entry (PTE<1:0> == x0)) or
>> if the PTE
>> + * maps a memory block at level 3 (PTE<1:0> == 01).
>> + */
>> +if ( !lpae_valid(pte) || ((level == 3) && lpae_mapping(pte)) )
> 
> If you look at the comment on top of lpae_mapping, it should only be
> used for L0..L2 ptes. So why are you using it on L3 ptes?
> 

Yes, I saw the comment. Yet, I would like to check exactly for this
mapping. If the mapping would as in the check above, it would be an
error, which is treated accordingly. In v3, you have suggested to look
at the the lpae_is_superpage macro, which, in my opinion, is not the
right construct to use at this point. If you should not like this check,
I could fall back to my previous implementation:

if ( !p2m_valid(pte) || ((level == 3) && !p2m_table(pte)) )
return -EFAULT;


Thank you,
~Sergej

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH RFC] Live migration for VMs with QEMU backed local storage

[Konrad Rzeszutek Wilk]

On Fri, Jun 23, 2017 at 03:31:16AM -0400, Bruno Alvisio wrote:
> This patch is the first attempt on adding live migration of instances with 
> local
> storage to Xen. This patch just handles very restricted case of fully
> virtualized HVMs. The code uses the "drive-mirror" capability provided by 
> QEMU.
> A new "-l" option is introduced to "xl migrate" command. If provided, the 
> local
> disk should be mirrored during the migration process. If the option is set,
> during the VM creation a qemu NBD server is started on the destination. After
> the instance is suspended on the source, the QMP "disk-mirror" command is 
> issued
> to mirror the disk to destination. Once the mirroring job is complete, the
> migration process continues as before. Finally, the NBD server is stopped 
> after
> the instance is successfully resumed on the destination node.
> 
> A major problem with this patch is that the mirroring of the disk is performed
> only after the memory stream is completed and the VM is suspended on the 
> source;
> thus the instance is frozen for a long period of time. The reason this happens
> is that the QEMU process (needed for the disk mirroring) is started on the
> destination node only after the memory copying is completed. One possibility I
> was considering to solve this issue (if it is decided that this capability
> should be used): Could a "helper" QEMU process be started on the destination
> node at the beginning of the migration sequence with the sole purpose of
> handling the disk mirroring and kill it at the end of the migration sequence? 
> 
> From the suggestions given by Konrad Wilk and Paul Durrant the preferred
> approach would be to handle the mirroring of disks by QEMU instead of directly
> being handled directly by, for example, blkback. It would be very helpful for 
> me
> to have a mental map of all the scenarios that can be encountered regarding
> local disk (Xen could start supporting live migration of certain types of 
> local
> disks). This are the ones I can think of:
> - Fully Virtualized HVM: QEMU emulation
> - blkback
> - blktap / blktap2 

You are missing 'qdisk' which is the QEMU implemenation of blkback.

> 
> 
> I have included TODOs in the code. I am sending this patch as is because I 
> first
> wanted to get an initial feedback if this is the path the should be pursued. 
> Any
> suggestions and ideas on this patch or on how to make a more generic solution
> would be really appreciated.
> 
> Signed-off-by: Bruno Alvisio 
> 
> ---
>  tools/libxl/libxl.h  |  16 -
>  tools/libxl/libxl_create.c   |  87 +-
>  tools/libxl/libxl_internal.h |  16 +
>  tools/libxl/libxl_qmp.c  | 115 
> ++-
>  tools/ocaml/libs/xl/xenlight_stubs.c |   2 +-
>  tools/xl/xl.h|   1 +
>  tools/xl/xl_migrate.c|  79 +---
>  tools/xl/xl_vmcontrol.c  |   2 +-
>  8 files changed, 303 insertions(+), 15 deletions(-)
> 
> diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
> index cf8687a..81fb2dc 100644
> --- a/tools/libxl/libxl.h
> +++ b/tools/libxl/libxl.h
> @@ -1294,6 +1294,15 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
>  xentoollog_logger *lg);
>  int libxl_ctx_free(libxl_ctx *ctx /* 0 is OK */);
>  
> +int libxl__drive_mirror(libxl_ctx *ctx, int domid, const char* device, const 
> char* target, const char* format) LIBXL_EXTERNAL_CALLERS_ONLY;
> +
> +int libxl__query_block_jobs(libxl_ctx *ctx, int domid, bool *is_ready) 
> LIBXL_EXTERNAL_CALLERS_ONLY;
> +
> +int libxl__query_block(libxl_ctx *ctx, int domid, char *device_names) 
> LIBXL_EXTERNAL_CALLERS_ONLY;
> +
> +int libxl__nbd_server_stop(libxl_ctx *ctx, int domid) 
> LIBXL_EXTERNAL_CALLERS_ONLY;
> +
> +
>  /* domain related functions */
>  
>  /* If the result is ERROR_ABORTED, the domain may or may not exist
> @@ -1307,7 +1316,7 @@ int libxl_domain_create_new(libxl_ctx *ctx, 
> libxl_domain_config *d_config,
>  LIBXL_EXTERNAL_CALLERS_ONLY;
>  int libxl_domain_create_restore(libxl_ctx *ctx, libxl_domain_config 
> *d_config,
>  uint32_t *domid, int restore_fd,
> -int send_back_fd,
> +int send_back_fd, int copy_local_disks,
>  const libxl_domain_restore_params *params,
>  const libxl_asyncop_how *ao_how,
>  const libxl_asyncprogress_how 
> *aop_console_how)
> @@ -1348,7 +1357,7 @@ static inline int libxl_domain_create_restore_0x040400(
>  LIBXL_EXTERNAL_CALLERS_ONLY
>  {
>  return libxl_domain_create_restore(ctx, d_config, domid, restore_fd,
> -   -1, params, ao_how, aop_console_how);
> +   -1, 0, params, ao_how, 
> 

Re: [Xen-devel] [PATCH v6 2/3] x86/pt: enable binding of GSIs to a PVH Dom0

[Jan Beulich]

>>> On 23.06.17 at 11:59,  wrote:
> --- a/xen/include/asm-x86/hvm/vioapic.h
> +++ b/xen/include/asm-x86/hvm/vioapic.h
> @@ -69,5 +69,6 @@ void vioapic_update_EOI(struct domain *d, u8 vector);
>  
>  int vioapic_get_mask(struct domain *d, unsigned int gsi);
>  int vioapic_get_vector(struct domain *d, unsigned int gsi);
> +int vioapic_get_trigger_mode(struct domain *d, unsigned int gsi);

const here too and then
Reviewed-by: Jan Beulich 

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [OSSTEST PATCH v11 10/20] ts-openstack-deploy: Switch to Neutron for network

[Anthony PERARD]

On Fri, Jun 23, 2017 at 02:41:02PM +0100, Ian Jackson wrote:
> Anthony PERARD writes ("[OSSTEST PATCH v11 10/20] ts-openstack-deploy: Switch 
> to Neutron for network"):
> > nova-network is not supported anymore and Neutron is the default.
> 
> Is this kind of thing going to be common ?  If so then it will be a
> constant maintenance burden in osstest.

No, that's not common. I don't think it's going to happen again anytime
soon.

> Is there some way we can get the list of components out of the
> top-level tree ?

I don't know if there is a simple way to get this list. And if we start
looking at the list of all OpenStack git trees, there are hundreds of
them.

-- 
Anthony PERARD

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v6 1/3] x86/vioapic: make gsi_vioapic private

[Jan Beulich]

>>> On 23.06.17 at 11:59,  wrote:
> +int vioapic_get_mask(struct domain *d, unsigned int gsi);
> +int vioapic_get_vector(struct domain *d, unsigned int gsi);

Both should have their first parameter const qualified. With that
Reviewed-by: Jan Beulich 

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 07/27] x86: move do_set_trap_table to pv/traps.c

[Andrew Cooper]

On 23/06/17 14:59, Wei Liu wrote:
> On Fri, Jun 23, 2017 at 12:00:35PM +0100, Andrew Cooper wrote:
>> On 08/06/17 18:11, Wei Liu wrote:
>>> Signed-off-by: Wei Liu 
>> I'd suggest folding this into the next patch, and putting the hypercall
>> in misc-hypercalls.c
>>
>> Despite its name, this hypercall is just setting up state in the vcpu.
>>
> This is trivial to do. But if I do it, does your Rb in the next patch
> apply to the new version?

Yes.

~Andrew

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 07/27] x86: move do_set_trap_table to pv/traps.c

[Wei Liu]

On Fri, Jun 23, 2017 at 12:00:35PM +0100, Andrew Cooper wrote:
> On 08/06/17 18:11, Wei Liu wrote:
> > Signed-off-by: Wei Liu 
> 
> I'd suggest folding this into the next patch, and putting the hypercall
> in misc-hypercalls.c
> 
> Despite its name, this hypercall is just setting up state in the vcpu.
> 

This is trivial to do. But if I do it, does your Rb in the next patch
apply to the new version?

> ~Andrew

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [OSSTEST PATCH v11 20/20] Introduce flight for stable branches of OpenStack

[Ian Jackson]

Anthony PERARD writes ("[OSSTEST PATCH v11 20/20] Introduce flight for stable 
branches of OpenStack"):
> OpenStack have many different repo which should be in sync, so this
> patch should grab the revisions of the stable branch of every OpenStack
> tree. Tempest does not have stable branch and should be able to test any
> OpenStack version.

I'm afraid I don't understand this patch.

Partly, I think it needs to be squashed with the original patch
introducing `openstack-nova' as a branch.  While your series has a
number of things where a thing is introduced and then later patched,
and this is largely OK, I think it is too confusing to have a whole
branch appear and disappear like this (without ever having been
run for real).

> -openstack-nova)
> +openstack-tempest*)
> +# OpenStack Tempest does not have stable branches and should work 
> with any

Your comment lines need wrapping to ~70-75 (here and later).

> +# version of OpenStack
>   repo_tree_rev_fetch_git openstack-nova \
>   $TREE_OPENSTACK_NOVA master $LOCALREV_OPENSTACK_NOVA
>   ;;
> +openstack-*-*)

I think you need to document what your openstack-*-* branch names are
going to be.

And, you make provision here for branches openstack-tempest* but you
don't add any such branches to cr-for-branches.

I forget what decisions we made about the push gate logic for the
various openstack branches.  I think it would be worth explicitly
writing this down in tree (in a comment in ap-fetch-* perhaps).

Thanks,
Ian.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 20/27] x86: move hypercall_page_initialise_ring1_kernel

[Andrew Cooper]

On 23/06/17 14:56, Wei Liu wrote:
> On Fri, Jun 23, 2017 at 01:41:59PM +0100, Andrew Cooper wrote:
>> On 08/06/17 18:11, Wei Liu wrote:
>>> Signed-off-by: Wei Liu 
>> Same review suggestions as the previous patch.
>>
> And then your Rb applies?

Yes.

~Andrew

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 20/27] x86: move hypercall_page_initialise_ring1_kernel

[Wei Liu]

On Fri, Jun 23, 2017 at 01:41:59PM +0100, Andrew Cooper wrote:
> On 08/06/17 18:11, Wei Liu wrote:
> > Signed-off-by: Wei Liu 
> 
> Same review suggestions as the previous patch.
> 

And then your Rb applies?

> ~Andrew

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 16/27] x86/traps: factor out pv_trap_init

[Wei Liu]

On Fri, Jun 23, 2017 at 01:31:22PM +0100, Andrew Cooper wrote:
> On 08/06/17 18:11, Wei Liu wrote:
> > Signed-off-by: Wei Liu 
> > ---
> >  xen/arch/x86/traps.c   | 22 ++
> >  xen/include/asm-x86/pv/traps.h |  4 
> >  2 files changed, 18 insertions(+), 8 deletions(-)
> >
> > diff --git a/xen/arch/x86/traps.c b/xen/arch/x86/traps.c
> > index 8861dfd332..29a83994bd 100644
> > --- a/xen/arch/x86/traps.c
> > +++ b/xen/arch/x86/traps.c
> > @@ -1871,14 +1871,8 @@ void __init init_idt_traps(void)
> >  this_cpu(compat_gdt_table) = boot_cpu_compat_gdt_table;
> >  }
> >  
> > -extern void (*const autogen_entrypoints[NR_VECTORS])(void);
> > -void __init trap_init(void)
> > +void __init pv_trap_init(void)
> >  {
> > -unsigned int vector;
> > -
> > -/* Replace early pagefault with real pagefault handler. */
> > -set_intr_gate(TRAP_page_fault, _fault);
> > -
> >  /* The 32-on-64 hypercall vector is only accessible from ring 1. */
> >  _set_gate(idt_table + HYPERCALL_VECTOR,
> >SYS_DESC_trap_gate, 1, entry_int82);
> > @@ -1886,6 +1880,19 @@ void __init trap_init(void)
> >  /* Fast trap for int80 (faster than taking the #GP-fixup path). */
> >  _set_gate(idt_table + 0x80, SYS_DESC_trap_gate, 3, _direct_trap);
> >  
> > +open_softirq(NMI_MCE_SOFTIRQ, nmi_mce_softirq);
> > +}
> > +
> > +extern void (*const autogen_entrypoints[NR_VECTORS])(void);
> > +void __init trap_init(void)
> > +{
> > +unsigned int vector;
> > +
> > +pv_trap_init();
> 
> This call should be at the end of trap_init(), but I guess you hit the
> assertion?
> 

Yes, that would hit the assertion.

> The !CONFIG_PV case will similarly hit the assertion.
> 
> For !CONFIG_PV, 0x80 and 0x82 would best become general interrupt
> handlers, which means you need to tweak entry.S autogen_stubs, and also
> tweak init_irq_data()
> 

Good point. I will tweak both locations.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [OSSTEST PATCH v11 19/20] ts-openstack-deploy: Increase devstack timeout

[Ian Jackson]

Anthony PERARD writes ("[OSSTEST PATCH v11 19/20] ts-openstack-deploy: Increase 
devstack timeout"):
> Signed-off-by: Anthony PERARD 

Acked-by: Ian Jackson 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST PATCH v11 18/20] ts-logs-capture: Capture OpenStack logs

[Ian Jackson]

Anthony PERARD writes ("[OSSTEST PATCH v11 18/20] ts-logs-capture: Capture 
OpenStack logs"):
> +  /var/log/openstack/*.log
> +  /etc/nova/*
> +  /etc/neutron/*
> +  /etc/cinder/*

This is fine:

> +  
> /home/osstest/build.*.test-*-devstack/tempest/etc/tempest.conf

This is not fine.  If a build host is shared, it will collect all the
logs from all of the builds.

I think build ts-* scripts ought to collect their own logs.

Ian.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [OSSTEST PATCH v11 17/20] ts-openstack-deploy: Move logs to /var/log/openstack

[Ian Jackson]

Anthony PERARD writes ("[OSSTEST PATCH v11 17/20] ts-openstack-deploy: Move 
logs to /var/log/openstack"):
> Signed-off-by: Anthony PERARD 

Acked-by: Ian Jackson 

Ian.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH for-4.9 v3 3/3] xen/livepatch: Don't crash on encountering STN_UNDEF relocations

[Julien Grall]

On 23/06/17 14:45, Andrew Cooper wrote:

On 23/06/17 14:43, Julien Grall wrote:

Hi,

On 23/06/17 14:33, Andrew Cooper wrote:

On 23/06/17 14:32, Julien Grall wrote:

Hi Andrew,

I am a bit confused, the title says "PATCH for-4.9 v3 3/3". I haven't
been CCed on the first two patches. Does it mean you are only looking
at this patch to be in 4.9?


Sorry - I messed up the CC lists.  The correctness of this patch does
depend on the previous two, so all 3 are looking for inclusion.


Given that we don't have livepatch testing in osstest how much test
have we done on those 3 patches?


There is testing in OSSTest.


Oh yes sorry. But only for amd64 and i386. No arm32 nor arm64 test.



I've manually run each of the scenarios, including with my livepatch
which has a STN_UNDEF relocation.

I don't know what testing Konrad has done.


Cheers,

--
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [OSSTEST PATCH v11 16/20] ts-openstack-tempest: Update list of skipped tests

[Ian Jackson]

Anthony PERARD writes ("[OSSTEST PATCH v11 16/20] ts-openstack-tempest: Update 
list of skipped tests"):
> Signed-off-by: Anthony PERARD 

Again, does this not mean we're going to suffer a maintenance burden
as tempest grows new inapplicable tests ?

Other possibilities that come to my mind: ideally the tempest tests
would have metadata so that particular classes of tests could be
skipped.  Alternatively, if they aren't disruptive or very slow, if
run, we could run them anyway as substeps.

Which makes me think: maybe the tempest tests want to be substeps
anyway.  Is that possible ?  (Does tempest speak subunit or
something ?)

Ian.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 27/27] x86: clean up traps.c

[Jan Beulich]

>>> On 23.06.17 at 14:50,  wrote:
> On 08/06/17 18:12, Wei Liu wrote:
>> @@ -1081,8 +1081,8 @@ void do_int3(struct cpu_user_regs *regs)
>>  pv_inject_hw_exception(TRAP_int3, X86_EVENT_NO_EC);
>>  }
>>  
>> -static void reserved_bit_page_fault(
>> -unsigned long addr, struct cpu_user_regs *regs)
>> +static void reserved_bit_page_fault(unsigned long addr,
>> +struct cpu_user_regs *regs)
> 
> Why are these prototypes changing?  For this case, it doesn't matter,
> but the former is necessary if any of them gain more parameters.

I think we should use consistent format, and while originally there
may have been quite a few cases of what is being removed here,
over time we've been switching towards what is being put in
place here. I don't see why adding more parameters would be in
conflict with this. If not even a single parameter declaration fits
on a line, then imo this is a good suggestion that the function
name is too long.

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [OSSTEST PATCH v11 15/20] ts-openstack-tempest: Fix tempest invocation

[Ian Jackson]

Anthony PERARD writes ("[OSSTEST PATCH v11 15/20] ts-openstack-tempest: Fix 
tempest invocation"):
> ./run_tempest.sh is deprecated.
...
>  target_cmd($ho, <  set -e
> -$builddir/tempest/run_tempest.sh --virtual-env -- --concurrency=2 '$regex'
> +cd $builddir/tempest
> +tempest run --concurrency=2 --regex '$regex'

Has /usr/local/bin/tempest or something been created by
ts-openstay-deploy, then ?

If so,

  Acked-by: Ian Jackson 

Otherwise I wonder how this works, since I don't see how `tempest'
would be on PATH.

Ian.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH for-4.9 v3 3/3] xen/livepatch: Don't crash on encountering STN_UNDEF relocations

[Andrew Cooper]

On 23/06/17 14:43, Julien Grall wrote:
> Hi,
>
> On 23/06/17 14:33, Andrew Cooper wrote:
>> On 23/06/17 14:32, Julien Grall wrote:
>>> Hi Andrew,
>>>
>>> I am a bit confused, the title says "PATCH for-4.9 v3 3/3". I haven't
>>> been CCed on the first two patches. Does it mean you are only looking
>>> at this patch to be in 4.9?
>>
>> Sorry - I messed up the CC lists.  The correctness of this patch does
>> depend on the previous two, so all 3 are looking for inclusion.
>
> Given that we don't have livepatch testing in osstest how much test
> have we done on those 3 patches?

There is testing in OSSTest.

I've manually run each of the scenarios, including with my livepatch
which has a STN_UNDEF relocation.

I don't know what testing Konrad has done.

~Andrew

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [OSSTEST PATCH v11 14/20] ts-openstack-deploy: Ignore libvirt-python version and use latest

[Ian Jackson]

Anthony PERARD writes ("[OSSTEST PATCH v11 14/20] ts-openstack-deploy: Ignore 
libvirt-python version and use latest"):
> Devstack is going to try to install a specific version of libvirt-python
> (currently 2.5.0) but this fail with libvirt installed by osstest.
> Remove the requirement and use the latest available instead.

Acked-by: Ian Jackson 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH for-4.9 v3 3/3] xen/livepatch: Don't crash on encountering STN_UNDEF relocations

[Julien Grall]

Hi,

On 23/06/17 14:33, Andrew Cooper wrote:

On 23/06/17 14:32, Julien Grall wrote:

Hi Andrew,

I am a bit confused, the title says "PATCH for-4.9 v3 3/3". I haven't
been CCed on the first two patches. Does it mean you are only looking
at this patch to be in 4.9?


Sorry - I messed up the CC lists.  The correctness of this patch does
depend on the previous two, so all 3 are looking for inclusion.


Given that we don't have livepatch testing in osstest how much test have 
we done on those 3 patches?


Cheers,

--
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [OSSTEST PATCH v11 13/20] ts-openstack-deploy: Apply a Tempest patch

[Ian Jackson]

Anthony PERARD writes ("[OSSTEST PATCH v11 13/20] ts-openstack-deploy: Apply a 
Tempest patch"):
> Signed-off-by: Anthony PERARD 

Acked-by: Ian Jackson 

>  ts-openstack-deploy | 10 ++
>  1 file changed, 10 insertions(+)
> 
> diff --git a/ts-openstack-deploy b/ts-openstack-deploy
> index 04317a0..04053de 100755
> --- a/ts-openstack-deploy
> +++ b/ts-openstack-deploy
> @@ -144,6 +144,16 @@ END
>  <  [{rabbit, [{vm_memory_high_watermark, 0.8}]}].
>  END
> +
> +# Apply https://review.openstack.org/449695/ to tempest to workaround an
> +# issue. Check comments for more information
> +target_cmd($ho, < +set -e
> +cd $builddir/tempest
> +git fetch origin refs/changes/95/449695/1
> +git cherry-pick FETCH_HEAD
> +END

I foresee the need to generalise this...

Ian.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [OSSTEST PATCH v11 12/20] make-flight: Increase dom0_mem for openstack flight

[Ian Jackson]

Anthony PERARD writes ("[OSSTEST PATCH v11 12/20] make-flight: Increase 
dom0_mem for openstack flight"):
> With 4G for dom0_mem, a host running devstack is using about 1.5G of
> swap.

Is this going to work properly on 8G hosts ?

Ian.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [OSSTEST PATCH v11 11/20] ts-openstack-deploy: Increase fd and memory limits for rabbitmq

[Ian Jackson]

Anthony PERARD writes ("[OSSTEST PATCH v11 11/20] ts-openstack-deploy: Increase 
fd and memory limits for rabbitmq"):
> Signed-off-by: Anthony PERARD 

Does this not mean that the upstream defaults are wrong ?

In general I like to see at least an upstream bug report url or number
mentioned in a comment near the workaround.

Ian.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [OSSTEST PATCH v11 10/20] ts-openstack-deploy: Switch to Neutron for network

[Ian Jackson]

Anthony PERARD writes ("[OSSTEST PATCH v11 10/20] ts-openstack-deploy: Switch 
to Neutron for network"):
> nova-network is not supported anymore and Neutron is the default.

Is this kind of thing going to be common ?  If so then it will be a
constant maintenance burden in osstest.

Is there some way we can get the list of components out of the
top-level tree ?

Ian.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 23/27] x86: move the compat callback ops next to the non-compat variant

[Jan Beulich]

>>> On 08.06.17 at 19:11,  wrote:
> +}
> +
> +
> +long compat_callback_op(int cmd, XEN_GUEST_HANDLE(void) arg)

Please avoid double blank lines like above. And preferably, please,
just like you did in earlier patches, use "curr" instead of "v". Then
Acked-by: Jan Beulich 

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [OSSTEST PATCH v11 09/20] ts-kernel-build: Enable network related modules for Neutron

[Ian Jackson]

Anthony PERARD writes ("[OSSTEST PATCH v11 09/20] ts-kernel-build: Enable 
network related modules for Neutron"):
> Those options/modules are needed to run OpenStack Neutron with Open
> vSwitch.

Acked-by: Ian Jackson 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel