from:"longtao.pang"

[Xen-devel] [OSSTEST Nested PATCH v11 0/7] Introduction of netsted HVM test job

2015-05-26 Thread longtao.pang

This patch set adds nested HVM test case for osstest.
In this test case, a Xen hypervisor (L1) runs on top of another Xen 
hypervisor (L0). 
Upon L1 hypervisor, we will then create a nested guest (L2), and test if the 
Linux guest can then be installed and run well.
About nested Xen virtualization, 
refer to http://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xen.

Below patches have passed pretest and been committed; therefore not included 
in this v11 patch set.
(git repo: git://xenbits.xen.org/people/ianc/osstest.git)

commit 699c911e59241350fc210633aba3f53afafee872
Refactor installation of overlays

commit 2545fc639107b2290236bf2ff8e9304b895ebee0
Edit some APIs in TestSupport.pm for nested test

commit 155bdb33b7c0227c1fb8b49ee54610c7a466b59b
Move the code for setting memory size into prep()

v11 patches are based on 'nestedhvm-v10-pretest-reduced' branch.
(git repo: git://xenbits.xen.org/people/ianc/osstest.git)

Test steps
0. To run osstest in standalone mode, write a config file in
   '~/.xen-osstest/config', and then create a standalone.config file 
   to define 'TREE_LINUX', 'REVISION_LINUX' which will be used for
   nested test. The directory path of 'Debian ISO Images' which used
   for installing HVM guest VM could be defined in 
   '~/.xen-osstest/config'. 
1. run './standalone-reset' to generate standalone.db firstly then run 
   'build-amd64' job and then 'build-amd64-pvops', to prepare xen
   installation tarball and hvm guest kernel.
2. run 'test-amd64-amd6-qemuu-nested' job, it does following:
a. invoke test step of 'ts-debain-hvm-install' to install 
   a normal HVM guest
b. invoke test step of 'ts-nested-setup' to make some
   appropriate runvars which selecthost() would recognise and
   prepare the configurations for installing L2 guest VM. 
c. invoke test step of 'ts-xen-install' to install xen on 
   the normal guest, alter it into a L1 hypervisor
d. invoke test step of 'ts-debain-hvm-install' again, but 
   take the L1 hypervisor as host, install the L2 guest on it
e. invoke test step of 'ts-guest-stop', stop L2 guest.
f. invoke test step of 'ts-guest-destroy' to destroy L1 guest.

This patch set reuse 'ts-debian-hvm-install' for both L1 installation
and L2 installation, use 'nestedl1' as L1's guestname and identity 
and use 'nestedl2 as L2's guestname.
It also reuses 'ts-xen-install' with L1's identity 'nestedl1' input parameter 
to differentiate from L0 Xen installation.
This patch series has been tested on test machines of amd64 arch,
Debian-7.2.0-amd64 as guests OS, with hvm domain0 of Linux kernel 3.18.5, 
in standalone mode.
Also, we use linux-stable tree as domain0 kernel source.

Ian Campbell (1):
  grub: remove patch to disable submenu from 20_linux_xen overlay
longtao.pang (6):
  Parsing grub which has 'submenu' primitive
  Changes to support '/boot' leading paths of kernel, xen, in grub
  Changes on test step of Debian hvm guest install
  Add new script to customize nested test configuration
  Compose the main recipe of nested test job
  Add test job for nest test case

 Osstest/Debian.pm   |   32 +-
 make-flight |   31 +
 overlay/etc/grub.d/20_linux_xen |4 ++-
 sg-run-job  |   11 ++
 ts-debian-hvm-install   |   18 +-
 ts-nested-setup |   75 
+
 6 files changed, 160 insertions(+), 11 deletions(-)
 create mode 100755 ts-nested-setup

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v11 5/7] Add new script to customize nested test configuration

2015-05-26 Thread longtao.pang

1. In this script, make some appropriate runvars which selecthost would
recognise.
2. Prepare the configurations for installing L2 guest VM.
3. Create a lv disk in L0 and hot-attach it to L1; Inside L1, using this
new added disk to create a VG which will be used for installing L2 guest.

Signed-off-by: longtao.pang 
Acked-by: Ian Campbell 
---
 ts-nested-setup |   75 +++
 1 file changed, 75 insertions(+)
 create mode 100755 ts-nested-setup

diff --git a/ts-nested-setup b/ts-nested-setup
new file mode 100755
index 000..60ab795
--- /dev/null
+++ b/ts-nested-setup
@@ -0,0 +1,75 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2015 Intel Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use DBI;
+use Osstest;
+use Osstest::Debian;
+use Osstest::TestSupport;
+
+tsreadconfig();
+# Pass L0's ident and L1's guestname
+our ($l0_ident,$l1_gn) = @ARGV;
+our ($l0,$l1) = ts_get_host_guest($l0_ident,$l1_gn);
+
+guest_check_ip($l1);
+
+# L1 guest's ident is same as guestname
+our $l1_ident = $l1->{Guest};
+
+store_runvar($l1_ident,$l1->{Guest});
+store_runvar("${l1_ident}_ip",$l1->{Ip});
+
+target_cmd_root($l1, "update-rc.d osstest-confirm-booted start 99 2 .");
+
+target_install_packages_norec($l1, qw(lvm2 rsync genisoimage));
+
+# We need to attach an extra disk to the L1 guest to be used as L2
+# guest storage.
+#
+# When running in a nested HVM environment the L1 domain is acting
+# as both a guest to L0 and a host to L2 guests and therefore potentially
+# sees connections to two independent xenstore instances, one provided by
+# the L0 host and one which is provided by the L1 instance of xenstore.
+#
+# Unfortunately the kernel is not capable of dealing with this and is only
+# able to cope with a single xenstore connection. Since the L1 toolstack and
+# L2 guests absolutely require xenstore to function we therefore cannot use
+# the L0 xenstore and therefore cannot use PV devices (xvdX etc) in the L1
+# guest and must use emulated devices (sdX etc).
+#
+# However at the moment we have not yet rebooted L1 into Xen and so it does
+# have PV devices available and sdb actually appears as xvdb. We could
+# disable the Xen platform device and use emulated devices for the install
+# phase too but that would be needlessly slow.
+
+my $vgname = $l1->{Vg};
+my $guest_storage_lv_name = "${l1_ident}_guest_storage";
+my $guest_storage_lv_size = guest_var($l1,'guest_storage_size',undef);
+die "guest_storage_lv_size is undefined" unless $guest_storage_lv_size;
+my $guest_storage_lvdev = "/dev/${vgname}/${guest_storage_lv_name}";
+
+die "toolstack $r{toolstack}" unless $r{toolstack} eq "xl";
+target_cmd_root($l0, <{Name} ${guest_storage_lvdev},raw,sdb,rw
+END
+
+# Create a vg in L1 guest and vg name is ${l1_gn}-disk
+target_cmd_root($l1, "pvcreate /dev/xvdb && vgcreate ${l1_gn}-disk /dev/xvdb");
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v11 7/7] Add test job for nest test case

2015-05-26 Thread longtao.pang

1. This patch adds creation of the nested test job, when job creation
procedure is invoked.
2. Set nested L1's vif model, nestedhvm feature, set specific disk
size and memory size for nested test by make-flight.

Signed-off-by: longtao.pang 
Acked-by: Ian Campbell 
---
 make-flight |   31 +++
 1 file changed, 31 insertions(+)

diff --git a/make-flight b/make-flight
index 8a1fceb..318c2db 100755
--- a/make-flight
+++ b/make-flight
@@ -210,6 +210,36 @@ do_hvm_win7_x64_tests () {
 all_hostflags=$most_hostflags,hvm
 }
 
+do_hvm_debian_nested_tests () {
+  bios=$1
+
+  if [ $xenarch != amd64 -o $dom0arch != amd64 \
+  -o "x$qemuu_suffix" != "x-qemuu" ]; then
+return
+  fi
+
+  case $xenbranch in
+xen-3.*-testing)  return;;
+xen-4.0-testing)  return;;
+xen-4.1-testing)  return;;
+xen-4.2-testing)  return;;
+xen-4.3-testing)  return;;
+  esac
+
+  job_create_test test-$xenarch$kern-$dom0arch$qemuu_suffix-nested \
+test-nested xl $xenarch $dom0arch $qemuu_runvar \
+nestedl1_image=debian-7.2.0-amd64-CD-1.iso \
+nestedl1_vifmodel='e1000' \
+nestedl1_disksize='15000' \
+nestedl1_memsize='3072' \
+nestedl1_enable_nestedhvm='true' \
+nestedl1_guest_storage_size='2' \
+nestedl2_image=debian-7.2.0-amd64-CD-1.iso \
+nestedl2_disksize='15000' \
+bios=$bios
+all_hostflags=$most_hostflags,hvm
+}
+
 do_hvm_debian_test_one () {
   testname=$1
   bios=$2
@@ -425,6 +455,7 @@ test_matrix_do_one () {
 do_hvm_rhel6_tests
 
 do_hvm_debian_tests
+do_hvm_debian_nested_tests seabios
 
   done # qemuu_suffix
 
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v11 2/7] Parsing grub which has 'submenu' primitive

2015-05-26 Thread longtao.pang

Now auto-gen kernel grub2 config file's boot menu entries can have
2-level hierarchy, containing 'submenu' primitive, which is comprised by
several sub-menuentries. Xen boot entries are grouped into such kind of
'submenu' block. This patch adds setupboot_grub2() ability to handle
such new grub.cfg format

Signed-off-by: longtao.pang 
---
Changes in v11:
1. Ian corrects our previous implementation of submenu handling. When
selecting submenu's menuentry, it shall be the 2-level format of 'M>N',
M is the 'submenu' entry number in first level, N is the sub-menuentry
numbering inside 'submenu' block, starting from 0.
Ian's correcting patch:
http://lists.xenproject.org/archives/html/xen-devel/2015-05/msg03108.html.
Based on Ian's correcting patch, we dogmatically alternatively use
'pop/push' to handle array of @offsets.
---
 Osstest/Debian.pm |   24 +++-
 1 file changed, 19 insertions(+), 5 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 46388d8..b1d1043 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -405,12 +405,22 @@ sub setupboot_grub2 () {
 my $parsemenu= sub {
 my $f= bl_getmenu_open($ho, $rmenu, "$stash/$ho->{Name}--grub.cfg.1");
 
-my $count= 0;
+my @offsets = (0);
 my $entry;
+my $submenu;
 while (<$f>) {
 next if m/^\s*\#/ || !m/\S/;
 if (m/^\s*\}\s*$/) {
-die unless $entry;
+die unless $entry || $submenu;
+if(!defined $entry && defined $submenu){
+logm("Met end of a submenu starting from ".
+"$submenu->{StartLine}. ".
+"Our want kern is $want_kernver");
+$submenu=undef;
+pop @offsets;
+$offsets[$#offsets]++;
+next;
+}
 my (@missing) =
 grep { !defined $entry->{$_} } 
(defined $xenhopt
@@ -438,8 +448,12 @@ sub setupboot_grub2 () {
 }
 if (m/^menuentry\s+[\'\"](.*)[\'\"].*\{\s*$/) {
 die $entry->{StartLine} if $entry;
-$entry= { Title => $1, StartLine => $., Number => $count };
-$count++;
+$entry= { Title => $1, StartLine => $., MenuEntryPath => join 
">", @offsets };
+$offsets[$#offsets]++;
+}
+if (m/^submenu\s+[\'\"](.*)[\'\"].*\{\s*$/) {
+$submenu={ StartLine =>$., MenuEntryPath => join ">", @offsets 
};
+push @offsets,(0);
 }
 if (m/^\s*multiboot\s*\/(xen\-[0-9][-+.0-9a-z]*\S+)/) {
 die unless $entry;
@@ -500,7 +514,7 @@ sub setupboot_grub2 () {
 }
 print ::EO <{Number}
+GRUB_DEFAULT="$entry->{MenuEntryPath}"
 END
 
 print ::EO <http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v11 6/7] Compose the main recipe of nested test job

2015-05-26 Thread longtao.pang

The ident and guestname are same of 'nestedl1' for L1 guest VM.

Signed-off-by: longtao.pang 
Acked-by: Ian Campbell 
---
 sg-run-job |   11 +++
 1 file changed, 11 insertions(+)

diff --git a/sg-run-job b/sg-run-job
index d53fd83..679761a 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -299,6 +299,17 @@ proc run-job/test-pair {} {
 #run-ts . remus-failover ts-remus-check src_host dst_host + debian
 }
 
+proc need-hosts/test-nested {} {return host}
+proc run-job/test-nested {} {
+run-ts . = ts-debian-hvm-install + host + nestedl1
+run-ts . = ts-nested-setup + host + nestedl1
+run-ts . = ts-xen-install nestedl1
+run-ts . = ts-host-reboot nestedl1
+run-ts . = ts-debian-hvm-install nestedl1 nestedl2
+run-ts . = ts-guest-stop nestedl1 nestedl2
+run-ts . = ts-guest-destroy + host + nestedl1
+}
+
 proc test-guest-migr {g} {
 if {[catch { run-ts . = ts-migrate-support-check + host $g }]} return
 
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v11 4/7] Changes on test step of Debian hvm guest install

2015-05-26 Thread longtao.pang

1. The default disk size for guest is '1M' which is not sufficient
for nested HVM guest, using larger disk size for nested guest
to accommodate to nested test requirement, the specific disk_size is
defined by make-flight.
2. In L1 installation context, assign more memory (defined in runvar) to
it; Since it acts as a nested hypervisor anyway.
3. Comment out CDROM entry in sources.list to make HTTP URL entry
available for L1 hvm guest.
4. Enable nestedhvm feature in 'ExtraConfig' for nested job.

Signed-off-by: longtao.pang 
---
Changes in v11:
1. Since the size of debian-7.2.0-amd64-CD-1.iso is just 624M, 
it's no need to extend nested L1's rootfs size, keep the original
size as '5000M'.
2. Add another use of preseed_hook_command to add new command of 
"in-target sed -i 's/^deb *cdrom/#&/g' /etc/apt/sources.list".
---
 ts-debian-hvm-install |   18 +-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/ts-debian-hvm-install b/ts-debian-hvm-install
index ea2d1ad..66ba5b0 100755
--- a/ts-debian-hvm-install
+++ b/ts-debian-hvm-install
@@ -92,6 +92,13 @@ if [ -e \$b/debian/grubx64.efi ] ; then
 fi
 END
 
+preseed_hook_command($gho, 'late_command', '', <8G free memory, create a guest with 4G memory to catch
 # any error that triggers cross 4G boundary
 my $host_freemem_mb = host_get_free_memory($ho);
@@ -174,13 +185,18 @@ sub prep () {
 if ($host_freemem_mb > $ram_lots * 2 + $ram_minslop) {
 $ram_mb = $ram_lots;
 } else {
-$ram_mb = 768;
+# Use guest_var to get specific memsize, or will use default '768'
+$ram_mb= guest_var($gho,'memsize',768);
 }
 logm("Host has $host_freemem_mb MB free memory, setting guest memory size 
to $ram_mb MB");
 
+# Use guest_var to get specific disk size, or will use default $disk_mb
+$disk_mb= guest_var($gho,'disksize',$disk_mb);
+
 more_prepareguest_hvm($ho,$gho, $ram_mb, $disk_mb,
   OnReboot => 'preserve',
   Bios => $r{bios},
+  ExtraConfig => $extra_config,
   PostImageHook => sub {
 my $cmds = iso_copy_content_from_image($gho, $newiso);
 $cmds .= prepare_initrd($initrddir,$newiso,$preseed_file_path);
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v11 1/7] grub: remove patch to disable submenu from 20_linux_xen overlay

2015-05-26 Thread longtao.pang

setupboot_grub2 now supports submenus, so we can reduce our delta vs
upstream a bit.

I started by extracting 20_linux_xen from
http://snapshot.debian.org/archive/debian/20130703T094657Z/pool/main/g/grub2/grub-common_1.99-27%2Bdeb7u2_amd64.deb
and then applying the patch at
http://savannah.gnu.org/file/grub.patch?file_id=32276 (the patch from
grub bug #42420 at http://savannah.gnu.org/bugs/?43420) and
reinstating the comment at the top of the file (modified to drop the
reference to the Debian bug.

This left me with some spurious changes:

@@ -93,7 +93,7 @@ linux_entry ()
   if test ! -e "${xen_dirname}/${xenpolicy}" ; then
  return
   fi
-  xen_args=`echo $xen_args flask=enforcing`
+  xen_args=`echo $xen_args flask_enabled=1 flask_enforcing=1`
   if ${recovery} ; then
  title="$(gettext_quoted "%s, with Xen %s (XSM enabled) and
Linux %s (recovery mode)")"
   else
@@ -137,7 +137,6 @@ EOF
echo'$message'
module  ${rel_dirname}/${xenpolicy}
 EOF
-  fi
   cat << EOF
 }
 EOF

I think these are bugs in the patch in the grub BTS, which were fixed
while iterating over the XSM series in osstest but didn't make it into
the upstream version, the fixes to those bugs are reverted byu the
above. So I have manually reverted them.

Signed-off-by: Ian Campbell 
Acked-by: Ian Jackson 
Cc: wei@citrix.com
Cc: longtaox.p...@intel.com
---
 overlay/etc/grub.d/20_linux_xen |4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/overlay/etc/grub.d/20_linux_xen b/overlay/etc/grub.d/20_linux_xen
index 5315e2a..aaead1b 100755
--- a/overlay/etc/grub.d/20_linux_xen
+++ b/overlay/etc/grub.d/20_linux_xen
@@ -1,7 +1,7 @@
 #! /bin/sh
 
 # Copied from the identically named file in grub-common 1.99-27+deb7u2.
-# This version fixed Debian bug #690538 and GRUB bug #43420.
+# This version fixes GRUB bug #43420.
 
 set -e
 
@@ -173,6 +173,7 @@ while [ "x${xen_list}" != "x" ] ; do
 xen_dirname=`dirname ${current_xen}`
 rel_xen_dirname=`make_system_path_relative_to_its_root $xen_dirname`
 xen_version=`echo $xen_basename | sed -e "s,.gz$,,g;s,^xen-,,g"`
+echo "submenu \"Xen ${xen_version}\" {"
 while [ "x$list" != "x" ] ; do
linux=`version_find_latest $list`
echo "Found linux image: $linux" >&2
@@ -214,5 +215,6 @@ while [ "x${xen_list}" != "x" ] ; do
 
list=`echo $list | tr ' ' '\n' | grep -vx $linux | tr '\n' ' '`
 done
+echo "}"
 xen_list=`echo $xen_list | tr ' ' '\n' | grep -vx $current_xen | tr '\n' ' 
'`
 done
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v11 3/7] Changes to support '/boot' leading paths of kernel, xen, in grub

2015-05-26 Thread longtao.pang

Support situations of grub that have vmlinuz and other things starting
with path of '/boot' rather than '/'.

Signed-off-by: longtao.pang 
---
 Osstest/Debian.pm |8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index b1d1043..62c4061 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -455,21 +455,21 @@ sub setupboot_grub2 () {
 $submenu={ StartLine =>$., MenuEntryPath => join ">", @offsets 
};
 push @offsets,(0);
 }
-if (m/^\s*multiboot\s*\/(xen\-[0-9][-+.0-9a-z]*\S+)/) {
+if (m/^\s*multiboot\s*(?:\/boot)?\/(xen\S+)/) {
 die unless $entry;
 $entry->{Hv}= $1;
 }
-if (m/^\s*multiboot\s*\/(vmlinu[xz]-(\S+))/) {
+if (m/^\s*multiboot\s*(?:\/boot)?\/(vmlinu[xz]-(\S+))/) {
 die unless $entry;
 $entry->{KernOnly}= $1;
 $entry->{KernVer}= $2;
 }
-if (m/^\s*module\s*\/(vmlinu[xz]-(\S+))/) {
+if (m/^\s*module\s*(?:\/boot)?\/(vmlinu[xz]-(\S+))/) {
 die unless $entry;
 $entry->{KernDom0}= $1;
 $entry->{KernVer}= $2;
 }
-if (m/^\s*module\s*\/(initrd\S+)/) {
+if (m/^\s*module\s*(?:\/boot)?\/(initrd\S+)/) {
 $entry->{Initrd}= $1;
 }
if (m/^\s*module\s*\/(xenpolicy\S+)/) {
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v10 8/9] Compose the main recipe of nested test job

2015-05-12 Thread longtao.pang

The ident and guestname are same of 'nestedl1' for L1 guest VM.

Signed-off-by: longtao.pang 
Acked-by: Ian Campbell 
---
 sg-run-job |   11 +++
 1 file changed, 11 insertions(+)

diff --git a/sg-run-job b/sg-run-job
index eae159d..b19f9f9 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -299,6 +299,17 @@ proc run-job/test-pair {} {
 #run-ts . remus-failover ts-remus-check src_host dst_host + debian
 }
 
+proc need-hosts/test-nested {} {return host}
+proc run-job/test-nested {} {
+run-ts . = ts-debian-hvm-install + host + nestedl1
+run-ts . = ts-nested-setup + host + nestedl1
+run-ts . = ts-xen-install nestedl1
+run-ts . = ts-host-reboot nestedl1
+run-ts . = ts-debian-hvm-install nestedl1 nestedl2
+run-ts . = ts-guest-stop nestedl1 nestedl2
+run-ts . = ts-guest-destroy + host + nestedl1
+}
+
 proc test-guest-migr {g} {
 if {[catch { run-ts . = ts-migrate-support-check + host $g }]} return
 
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v10 9/9] Add test job for nest test case

2015-05-12 Thread longtao.pang

1. This patch adds creation of the nested test job, when job creation
procedure is invoked.
2. Set nested L1's vif model, nestedhvm feature, set specific disk
size and memory size for nested test by make-flight.

Signed-off-by: longtao.pang 
Acked-by: Ian Campbell 
---
 make-flight |   31 +++
 1 file changed, 31 insertions(+)

diff --git a/make-flight b/make-flight
index 70c1900..976fece 100755
--- a/make-flight
+++ b/make-flight
@@ -204,6 +204,36 @@ do_hvm_win7_x64_tests () {
 all_hostflags=$most_hostflags,hvm
 }
 
+do_hvm_debian_nested_tests () {
+  bios=$1
+
+  if [ $xenarch != amd64 -o $dom0arch != amd64 \
+  -o "x$qemuu_suffix" != "x-qemuu" ]; then
+return
+  fi
+
+  case $xenbranch in
+xen-3.*-testing)  return;;
+xen-4.0-testing)  return;;
+xen-4.1-testing)  return;;
+xen-4.2-testing)  return;;
+xen-4.3-testing)  return;;
+  esac
+
+  job_create_test test-$xenarch$kern-$dom0arch$qemuu_suffix-nested \
+test-nested xl $xenarch $dom0arch $qemuu_runvar \
+nestedl1_image=debian-7.2.0-amd64-CD-1.iso \
+nestedl1_vifmodel='e1000' \
+nestedl1_disksize='15000' \
+nestedl1_memsize='3072' \
+nestedl1_enable_nestedhvm='true' \
+nestedl1_guest_storage_size='2' \
+nestedl2_image=debian-7.2.0-amd64-CD-1.iso \
+nestedl2_disksize='15000' \
+bios=$bios
+all_hostflags=$most_hostflags,hvm
+}
+
 do_hvm_debian_test_one () {
   testname=$1
   bios=$2
@@ -408,6 +438,7 @@ test_matrix_do_one () {
 do_hvm_rhel6_tests
 
 do_hvm_debian_tests
+do_hvm_debian_nested_tests seabios
 
   done # qemuu_suffix
 
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v10 7/9] Add new script to customize nested test configuration

2015-05-12 Thread longtao.pang

1. In this script, make some appropriate runvars which selecthost would
recognise.
2. Prepare the configurations for installing L2 guest VM.
3. Create a lv disk in L0 and hot-attach it to L1; Inside L1, using this
new added disk to create a VG which will be used for installing L2 guest.

Signed-off-by: longtao.pang 
---
Changes in v10:
1. Since Ian has fixed the bug of 'late_command' in his v6_patch, remove
the code for editing 'sources.list' file in hvm guest.
2. Currently, reboot is not required after block attach 'lvdev' to L1;
So, remove the related code.
3. Assert "$guest_storage_lv_size" is defined, otherwise die.
4. Assert toolstack is 'xl', otherwise die.
---
 ts-nested-setup |   73 +++
 1 file changed, 73 insertions(+)
 create mode 100755 ts-nested-setup

diff --git a/ts-nested-setup b/ts-nested-setup
new file mode 100755
index 000..308a142
--- /dev/null
+++ b/ts-nested-setup
@@ -0,0 +1,73 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2015 Intel Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use DBI;
+use Osstest;
+use Osstest::Debian;
+use Osstest::TestSupport;
+
+tsreadconfig();
+# Pass L0's ident and L1's guestname
+our ($l0_ident,$l1_gn) = @ARGV;
+our ($l0,$l1) = ts_get_host_guest($l0_ident,$l1_gn);
+
+guest_check_ip($l1);
+
+# L1 guest's ident is same as guestname
+our $l1_ident = $l1->{Guest};
+
+store_runvar($l1_ident,$l1->{Guest});
+store_runvar("${l1_ident}_ip",$l1->{Ip});
+
+target_cmd_root($l1, "update-rc.d osstest-confirm-booted start 99 2 .");
+
+target_install_packages_norec($l1, qw(lvm2 rsync genisoimage));
+
+# We need to attach an extra disk to the L1 guest to be used as L2
+# guest storage.
+#
+# When running in a nested HVM environment the L1 domain is acting
+# as both a guest to L0 and a host to L2 guests and therefore potentially
+# sees connections to two independent xenstore instances, one provided by
+# the L0 host and one which is provided by the L1 instance of xenstore.
+#
+# Unfortunately the kernel is not capable of dealing with this and is only
+# able to cope with a single xenstore connection. Since the L1 toolstack and
+# L2 guests absolutely require xenstore to function we therefore cannot use
+# the L0 xenstore and therefore cannot use PV devices (xvdX etc) in the L1
+# guest and must use emulated devices (sdX etc).
+#
+# However at the moment we have not yet rebooted L1 into Xen and so it does
+# have PV devices available and sdb actually appears as xvdb. We could
+# disable the Xen platform device and use emulated devices for the install
+# phase too but that would be needlessly slow.
+
+my $vgname = $l1->{Vg};
+my $guest_storage_lv_name = "${l1_ident}_guest_storage";
+my $guest_storage_lv_size = guest_var($l1,'guest_storage_size',undef);
+die "guest_storage_lv_size is undefined" unless $guest_storage_lv_size;
+my $guest_storage_lvdev = "/dev/${vgname}/${guest_storage_lv_name}";
+
+die "toolstack $r{toolstack}" unless $r{toolstack} eq "xl";
+target_cmd_root($l0, <{Name} ${guest_storage_lvdev},raw,sdb,rw
+END
+
+# Create a vg in L1 guest and vg name is ${l1_gn}-disk
+target_cmd_root($l1, "pvcreate /dev/xvdb && vgcreate ${l1_gn}-disk /dev/xvdb");
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v10 1/9] Parsing grub which has 'submenu' primitive

2015-05-12 Thread longtao.pang

>From a hvm kernel build from Linux stable Kernel tree,
the auto generated grub2 menu will have 'submenu' primitive, upon the
'menuentry' items. Xen boot entries will be grouped into a submenu. This
patch adds capability to support such grub formats.

Signed-off-by: longtao.pang 
Acked-by: Ian Campbell 
---
 Osstest/Debian.pm |   13 -
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 4476af5..3a1a515 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -407,10 +407,18 @@ sub setupboot_grub2 () {
 
 my $count= 0;
 my $entry;
+my $submenu;
 while (<$f>) {
 next if m/^\s*\#/ || !m/\S/;
 if (m/^\s*\}\s*$/) {
-die unless $entry;
+die unless $entry || $submenu;
+if(!defined $entry && defined $submenu){
+logm("Met end of a submenu starting from ".
+"$submenu->{StartLine}. ".
+"Our want kern is $want_kernver");
+$submenu=undef;
+next;
+}
 my (@missing) =
 grep { !defined $entry->{$_} } 
(defined $xenhopt
@@ -441,6 +449,9 @@ sub setupboot_grub2 () {
 $entry= { Title => $1, StartLine => $., Number => $count };
 $count++;
 }
+if (m/^submenu\s+[\'\"](.*)[\'\"].*\{\s*$/) {
+$submenu={ StartLine =>$.};
+}
 if (m/^\s*multiboot\s*\/(xen\-[0-9][-+.0-9a-z]*\S+)/) {
 die unless $entry;
 $entry->{Hv}= $1;
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v10 2/9] Changes to support '/boot' leading paths of kernel, xen, in grub

2015-05-12 Thread longtao.pang

Support situations of grub that have vmlinuz and other things starting
with path of '/boot' rather than '/'.

Signed-off-by: longtao.pang 
Acked-by: Ian Campbell 
---
 Osstest/Debian.pm |8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 3a1a515..3baacc4 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -452,21 +452,21 @@ sub setupboot_grub2 () {
 if (m/^submenu\s+[\'\"](.*)[\'\"].*\{\s*$/) {
 $submenu={ StartLine =>$.};
 }
-if (m/^\s*multiboot\s*\/(xen\-[0-9][-+.0-9a-z]*\S+)/) {
+if (m/^\s*multiboot\s*(?:\/boot)?\/(xen\S+)/) {
 die unless $entry;
 $entry->{Hv}= $1;
 }
-if (m/^\s*multiboot\s*\/(vmlinu[xz]-(\S+))/) {
+if (m/^\s*multiboot\s*(?:\/boot)?\/(vmlinu[xz]-(\S+))/) {
 die unless $entry;
 $entry->{KernOnly}= $1;
 $entry->{KernVer}= $2;
 }
-if (m/^\s*module\s*\/(vmlinu[xz]-(\S+))/) {
+if (m/^\s*module\s*(?:\/boot)?\/(vmlinu[xz]-(\S+))/) {
 die unless $entry;
 $entry->{KernDom0}= $1;
 $entry->{KernVer}= $2;
 }
-if (m/^\s*module\s*\/(initrd\S+)/) {
+if (m/^\s*module\s*(?:\/boot)?\/(initrd\S+)/) {
 $entry->{Initrd}= $1;
 }
if (m/^\s*module\s*\/(xenpolicy\S+)/) {
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v10 4/9] Edit some APIs in TestSupport.pm for nested test

2015-05-12 Thread longtao.pang

1. If vif model is defined for guest, use it in guest's configuration
2. In L2 installation context, its host (L1) shall not have a static IP
but stored in runvar in L1 installation before; use this stored runvar
IP.

Signed-off-by: longtao.pang 
Acked-by: Ian Campbell 
---
 Osstest/TestSupport.pm |   28 ++--
 1 file changed, 18 insertions(+), 10 deletions(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 3f84110..955bea3 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -888,13 +888,17 @@ sub selecthost ($) {
 serial_host_setup($ho);
 
 $ho->{IpStatic} = get_host_property($ho,'ip-addr');
-if (!defined $ho->{IpStatic}) {
-   my $ip_packed= gethostbyname($ho->{Fqdn});
-   die "$ho->{Fqdn} ?" unless $ip_packed;
-   $ho->{IpStatic}= inet_ntoa($ip_packed);
-   die "$ho->{Fqdn} ?" unless defined $ho->{IpStatic};
+if (defined $r{"${ident}_ip"}) {
+$ho->{Ip} = $r{"${ident}_ip"};
+} else {
+if (!defined $ho->{IpStatic}) {
+   my $ip_packed= gethostbyname($ho->{Fqdn});
+   die "$ho->{Fqdn} ?" unless $ip_packed;
+   $ho->{IpStatic}= inet_ntoa($ip_packed);
+   die "$ho->{Fqdn} ?" unless defined $ho->{IpStatic};
+}
+$ho->{Ip}= $ho->{IpStatic};
 }
-$ho->{Ip}= $ho->{IpStatic};
 
 #- tftp -
 
@@ -1537,13 +1541,15 @@ sub prepareguest ($$) {
 store_runvar("${gn}_tcpcheckport", $tcpcheckport);
 store_runvar("${gn}_boot_timeout", $boot_timeout);
 
+my $gho= selectguest($gn, $ho);
+store_runvar("${gn}_domname", $gho->{Name});
+
+# If we have defined guest specific disksize, use it
+$mb = guest_var($gho,'disksize',$mb);
 if (defined $mb) {
store_runvar("${gn}_disk_lv", $r{"${gn}_hostname"}.'-disk');
 }
 
-my $gho= selectguest($gn, $ho);
-store_runvar("${gn}_domname", $gho->{Name});
-
 if (defined $mb) {
store_runvar("${gn}_vg", '');
if (!length $r{"${gn}_vg"}) {
@@ -1571,11 +1577,13 @@ sub prepareguest_part_xencfg ($) {
 my $oncrash= $xopts->{OnCrash} || 'preserve';
 my $vcpus= guest_var($gho, 'vcpus', $xopts->{DefVcpus} || 2);
 my $xoptcfg= $xopts->{ExtraConfig};
+my $vif= guest_var($gho, 'vifmodel','');
+my $vifmodel= $vif ? ",model=$vif" : '';
 $xoptcfg='' unless defined $xoptcfg;
 my $xencfg= <{Ether}' ]
+vif = [ 'type=ioemu,mac=$gho->{Ether}${vifmodel}' ]
 #
 on_poweroff = '$onpoweroff'
 on_reboot   = '$onreboot'
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v10 6/9] Changes on test step of Debian hvm guest install

2015-05-12 Thread longtao.pang

1. The default disk size for guest is '1M' which is not sufficient
for nested HVM guest, using larger disk size for nested guest
to accommodate to nested test requirement, the specific disk_size is
defined by make-flight.
2. Since 'Debain-xxx-.iso' image will be stored in rootfs of L1 guest,
therefore needs more disk capacity, increase root partition size to
1M.
3. In L1 installation context, assign more memory (defined in runvar) to
it; Since it acts as a nested hypervisor anyway.
4. Comment out CDROM entry in sources.list to make HTTP URL entry
available for L1 hvm guest.
5. Enable nestedhvm feature in 'ExtraConfig' for nested job.

Signed-off-by: longtao.pang 
---
Changes in v10:
Based on Ian's v6_patch[04,05,06], use 'preseed_hook_command' to comment out 
CDROM entry in sources.list to make HTTP URL entry available for hvm guest.
---
 ts-debian-hvm-install |   18 +++---
 1 file changed, 15 insertions(+), 3 deletions(-)

diff --git a/ts-debian-hvm-install b/ts-debian-hvm-install
index d3413e1..e636b1e 100755
--- a/ts-debian-hvm-install
+++ b/ts-debian-hvm-install
@@ -59,7 +59,7 @@ d-i partman-auto/expert_recipe string \\
 use_filesystem{ } filesystem{ vfat } \\
 mountpoint{ /boot/efi } \\
 . \\
-5000 50 5000 ext4 \\
+1 50 1 ext4 \\
 method{ format } format{ } \\
 use_filesystem{ } filesystem{ ext4 } \\
 mountpoint{ / } \\
@@ -80,6 +80,8 @@ if [ -e \$b/debian/grubx64.efi ] ; then
 mkdir -p \$b/boot
 cp \$b/debian/grubx64.efi \$b/boot/bootx64.efi
 fi
+
+in-target sed -i 's/^deb *cdrom/#&/g' /etc/apt/sources.list
 END
 
 $preseed_file .= preseed_hook_cmds();
@@ -156,6 +158,10 @@ sub prep () {
 target_putfilecontents_root_stash($ho, 10, preseed(),
   $preseed_file_path);
 
+my $extra_config='';
+$extra_config .="nestedhvm=1\n"
+if guest_var($gho,"enable_nestedhvm",'false') eq 'true';
+
 # If host has >8G free memory, create a guest with 4G memory to catch
 # any error that triggers cross 4G boundary
 my $host_freemem_mb = host_get_free_memory($ho);
@@ -164,13 +170,19 @@ sub prep () {
 if ($host_freemem_mb > $ram_lots * 2 + $ram_minslop) {
 $ram_mb = $ram_lots;
 } else {
-$ram_mb = 768;
+# Use guest_var to get specific memsize, or will use default '768'
+$ram_mb= guest_var($gho,'memsize',768);
 }
-logm("Host has $host_freemem_mb MB free memory, setting guest memory size 
to $ram_mb MB");
+logm("Host has $host_freemem_mb MB free memory,
+ setting guest memory size to $ram_mb MB");
+
+# Use guest_var to get specific disk size, or will use default $disk_mb
+$disk_mb= guest_var($gho,'disksize',$disk_mb);
 
 more_prepareguest_hvm($ho,$gho, $ram_mb, $disk_mb,
   OnReboot => 'preserve',
   Bios => $r{bios},
+  ExtraConfig => $extra_config,
   PostImageHook => sub {
 my $cmds = iso_copy_content_from_image($gho, $newiso);
 $cmds .= prepare_initrd($initrddir,$newiso,$preseed_file_path);
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v10 3/9] Refactor installation of overlays

2015-05-12 Thread longtao.pang

Apply overlay to guests as well as hosts by moving to preseed_base.

We do this because we want to be able to:
   * Use '/etc/init.d/osstest-confirm-booted' for guests as 
 well as hosts (in particular when testing nestedhvm
 we sometimes want to treat a guest as a host).
   * Use grub's 20_linux_xen which is harmless unless Xen is
 installed in the guest which is the case only for
 nestedhvm testing when we want these changes.
The other things in the overlay are some initscripts
(xenbridge and xenlightdaemons) which are not enabled for guests and
hence are therefore harmless.

Signed-off-by: longtao.pang 
---
Based on Ian Campbell's v6_patch [04,05,06], I create this patch
to refactor installation of overlays for guest as well as host used.

Link of Ian Campbell's patch:
http://lists.xenproject.org/archives/html/xen-devel/2015-05/msg00467.html
http://lists.xenproject.org/archives/html/xen-devel/2015-05/msg00452.html
http://lists.xenproject.org/archives/html/xen-devel/2015-05/msg00459.html
---
 Osstest/Debian.pm |3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 3baacc4..d65f41c 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -630,6 +630,7 @@ sub preseed_base (;@) {
 
 preseed_ssh($ho, $sfx);
 preseed_hook_overlay($ho, $sfx, $c{OverlayLocal}, 'overlay-local.tar');
+preseed_hook_overlay($ho, $sfx, 'overlay', 'overlay.tar');
 
 my $preseed = <<"END";
 d-i mirror/suite string $suite
@@ -766,8 +767,6 @@ ls -l /dev/sd*
 true
 END
 
-preseed_hook_overlay($ho, $sfx, 'overlay', 'overlay.tar');
-
 preseed_hook_command($ho, 'late_command', $sfx, <http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v10 5/9] Move the code for setting memory size into prep()

2015-05-12 Thread longtao.pang

Move the code for setting memory size from outside into prep(),
where these $ram_mb operations are only used; And in next patch,
for L2 guest installation, $ram_mb will be overridden by a runvar value,
which needs $gho in scope.

This is pure code motion. No functional change.

Signed-off-by: longtao.pang 
Acked-by: Ian Campbell 
---
 ts-debian-hvm-install |   24 
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git a/ts-debian-hvm-install b/ts-debian-hvm-install
index b37f0f9..d3413e1 100755
--- a/ts-debian-hvm-install
+++ b/ts-debian-hvm-install
@@ -156,6 +156,18 @@ sub prep () {
 target_putfilecontents_root_stash($ho, 10, preseed(),
   $preseed_file_path);
 
+# If host has >8G free memory, create a guest with 4G memory to catch
+# any error that triggers cross 4G boundary
+my $host_freemem_mb = host_get_free_memory($ho);
+my $ram_minslop = 100;
+my $ram_lots = 5000;
+if ($host_freemem_mb > $ram_lots * 2 + $ram_minslop) {
+$ram_mb = $ram_lots;
+} else {
+$ram_mb = 768;
+}
+logm("Host has $host_freemem_mb MB free memory, setting guest memory size 
to $ram_mb MB");
+
 more_prepareguest_hvm($ho,$gho, $ram_mb, $disk_mb,
   OnReboot => 'preserve',
   Bios => $r{bios},
@@ -173,18 +185,6 @@ sub prep () {
 });
 }
 
-# If host has >8G free memory, create a guest with 4G memory to catch
-# any error that triggers cross 4G boundary
-my $host_freemem_mb = host_get_free_memory($ho);
-my $ram_minslop = 100;
-my $ram_lots = 5000;
-if ($host_freemem_mb > $ram_lots * 2 + $ram_minslop) {
-$ram_mb = $ram_lots;
-} else {
-$ram_mb = 768;
-}
-logm("Host has $host_freemem_mb MB free memory, setting guest memory size to 
$ram_mb MB");
-
 if (!$stage) {
 prep();
 guest_create($gho);
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v10 0/9] Introduction of netsted HVM test job

2015-05-12 Thread longtao.pang

This patch set adds nested HVM test case for osstest.
In this test case, a Xen hypervisor (L1) runs on top of another Xen 
hypervisor (L0). 
Upon L1 hypervisor, we will then create a nested guest (L2), and test if the 
Linux guest can then be installed and run well.
About nested Xen virtualization, 
refer to http://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xen.

Test steps
0. To run osstest in standalone mode, write a config file in
   '~/.xen-osstest/config', and then create a standalone.config file 
   to define 'TREE_LINUX', 'REVISION_LINUX' which will be used for
   nested test. The directory path of 'Debian ISO Images' which used
   for installing HVM guest VM could be defined in 
   '~/.xen-osstest/config'. 
1. run './standalone-reset' to generate standalone.db firstly then run 
   'build-amd64' job and then 'build-amd64-pvops', to prepare xen
   installation tarball and hvm guest kernel.
2. run 'test-amd64-amd6-qemuu-nested' job, it does following:
a. invoke test step of 'ts-debain-hvm-install' to install 
   a normal HVM guest
b. invoke test step of 'ts-nested-setup' to make some
   appropriate runvars which selecthost() would recognise and
   prepare the configurations for installing L2 guest VM. 
c. invoke test step of 'ts-xen-install' to install xen on 
   the normal guest, alter it into a L1 hypervisor
d. invoke test step of 'ts-debain-hvm-install' again, but 
   take the L1 hypervisor as host, install the L2 guest on it
e. invoke test step of 'ts-guest-stop', stop L2 guest.
f. invoke test step of 'ts-guest-destroy' to destroy L1 guest.

This patch set reuse 'ts-debian-hvm-install' for both L1 installation
and L2 installation, use 'nestedl1' as L1's guestname and identity 
and use 'nestedl2 as L2's guestname.
It also reuses 'ts-xen-install' with L1's identity 'nestedl1' input parameter 
to differentiate from L0 Xen installation.
This patch series has been tested on test machines of amd64 arch,
Debian-7.2.0-amd64 as guests OS, with hvm domain0 of Linux kernel 3.18.5, 
in standalone mode.
Also, we use linux-stable tree as domain0 kernel source.

longtao.pang (9):
  parsing grub which has 'submenu' primitive
  Changes to support '/boot' leading paths of kernel, xen, in grub
  Refactor installation of overlays for guest used
  Edit some APIs in TestSupport.pm for nested test
  Move the codes about memory size setting into prep()
  Changes on test step of Debian hvm guest install
  Add new script to customize nested test configuration
  Compose the main recipe of nested test job
  Add test job for nest test case

 Osstest/Debian.pm  |   22 +++-
 Osstest/TestSupport.pm |   27 +++--
 make-flight|   32 ++
 sg-run-job |   11 ++
 ts-debian-hvm-install  |   37 ++
 ts-nested-setup|   64 
+++
 6 files changed, 165 insertions(+), 28 deletions(-)
 create mode 100755 ts-nested-setup

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v9 8/9] Compose the main recipe of nested test job

2015-05-01 Thread longtao.pang

From: "longtao.pang" 

The ident and guestname are same of 'nestedl1' for L1 guest VM.

Signed-off-by: longtao.pang 
---
Changes in v9:
1. For L1, identical its ident and guestname as 'nestedl1';
And same for L2, 'nestedl2' is L2's name as well as its ident.
2. Place '+' to include parameters in the test id for ts-guest-destory
step.
---
 sg-run-job |   11 +++
 1 file changed, 11 insertions(+)

diff --git a/sg-run-job b/sg-run-job
index eae159d..b19f9f9 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -299,6 +299,17 @@ proc run-job/test-pair {} {
 #run-ts . remus-failover ts-remus-check src_host dst_host + debian
 }
 
+proc need-hosts/test-nested {} {return host}
+proc run-job/test-nested {} {
+run-ts . = ts-debian-hvm-install + host + nestedl1
+run-ts . = ts-nested-setup + host + nestedl1
+run-ts . = ts-xen-install nestedl1
+run-ts . = ts-host-reboot nestedl1
+run-ts . = ts-debian-hvm-install nestedl1 nestedl2
+run-ts . = ts-guest-stop nestedl1 nestedl2
+run-ts . = ts-guest-destroy + host + nestedl1
+}
+
 proc test-guest-migr {g} {
 if {[catch { run-ts . = ts-migrate-support-check + host $g }]} return
 
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v9 3/9] Refactor installation of overlays for guest used

2015-05-01 Thread longtao.pang

From: "longtao.pang" 

Based on Ian Campbell's v5_patch [04,05,06], I create this patch
to refactor installation of overlays for guest used.

Link of Ian Campbell's patch:
http://lists.xenproject.org/archives/html/xen-devel/2015-04/msg01452.html
http://lists.xenproject.org/archives/html/xen-devel/2015-04/msg01447.html

Signed-off-by: longtao.pang 
---
 Osstest/Debian.pm |1 +
 1 file changed, 1 insertion(+)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 6691ff6..4af6957 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -624,6 +624,7 @@ sub preseed_base (;@) {
 
 preseed_ssh($ho, $sfx);
 preseed_hook_overlay($ho, $sfx, $c{OverlayLocal}, 'overlay-local.tar');
+preseed_hook_overlay($ho, $sfx, 'overlay', 'overlay.tar');
 
 my $preseed = <<"END";
 d-i mirror/suite string $suite
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v9 4/9] Edit some APIs in TestSupport.pm for nested test

2015-05-01 Thread longtao.pang

From: "longtao.pang" 

1. If vif model is defined for guest, use it in guest's configuration
2. In L2 installation context, its host (L1) shall not have a static IP
but stored in runvar in L1 installation before; use this stored runvar
IP.

Signed-off-by: longtao.pang 
---
Changes in v9:
1. Refactor 'selecthost' to set L1 guest's IpAddr for nested job.
2. Change $vifmodel to a reasonable format.
3. Refactor 'prepareguest' to use a specific disksize for nested L1
guest.
---
 Osstest/TestSupport.pm |   27 +--
 1 file changed, 17 insertions(+), 10 deletions(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 1bde67d..2d7ecc0 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -865,13 +865,17 @@ sub selecthost ($) {
 serial_host_setup($ho);
 
 $ho->{IpStatic} = get_host_property($ho,'ip-addr');
-if (!defined $ho->{IpStatic}) {
-   my $ip_packed= gethostbyname($ho->{Fqdn});
-   die "$ho->{Fqdn} ?" unless $ip_packed;
-   $ho->{IpStatic}= inet_ntoa($ip_packed);
-   die "$ho->{Fqdn} ?" unless defined $ho->{IpStatic};
+if (defined $r{"${ident}_ip"}) {
+$ho->{Ip} = $r{"${ident}_ip"};
+} else {
+if (!defined $ho->{IpStatic}) {
+   my $ip_packed= gethostbyname($ho->{Fqdn});
+   die "$ho->{Fqdn} ?" unless $ip_packed;
+   $ho->{IpStatic}= inet_ntoa($ip_packed);
+   die "$ho->{Fqdn} ?" unless defined $ho->{IpStatic};
+}
+$ho->{Ip}= $ho->{IpStatic};
 }
-$ho->{Ip}= $ho->{IpStatic};
 
 #- tftp -
 
@@ -1513,14 +1517,15 @@ sub prepareguest ($$) {
 store_runvar("${gn}_hostname", $hostname);
 store_runvar("${gn}_tcpcheckport", $tcpcheckport);
 store_runvar("${gn}_boot_timeout", $boot_timeout);
+my $gho= selectguest($gn, $ho);
+store_runvar("${gn}_domname", $gho->{Name});
+#if we have defined guest specific disksize, override it
+$mb = guest_var($gho,'disksize',$mb);
 
 if (defined $mb) {
store_runvar("${gn}_disk_lv", $r{"${gn}_hostname"}.'-disk');
 }
 
-my $gho= selectguest($gn, $ho);
-store_runvar("${gn}_domname", $gho->{Name});
-
 if (defined $mb) {
store_runvar("${gn}_vg", '');
if (!length $r{"${gn}_vg"}) {
@@ -1548,11 +1553,13 @@ sub prepareguest_part_xencfg ($) {
 my $oncrash= $xopts->{OnCrash} || 'preserve';
 my $vcpus= guest_var($gho, 'vcpus', $xopts->{DefVcpus} || 2);
 my $xoptcfg= $xopts->{ExtraConfig};
+my $vif= guest_var($gho, 'vifmodel','');
+my $vifmodel= $vif ? ",model=$vif" : '';
 $xoptcfg='' unless defined $xoptcfg;
 my $xencfg= <{Ether}' ]
+vif = [ 'type=ioemu,mac=$gho->{Ether}${vifmodel}' ]
 #
 on_poweroff = '$onpoweroff'
 on_reboot   = '$onreboot'
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v9 9/9] Add test job for nest test case

2015-05-01 Thread longtao.pang

From: "longtao.pang" 

1. This patch adds creation of the nested test job, when job creation
procedure is invoked.
2. Set nested L1's vif model, nestedhvm feature, set specific disk
size and memory size for nested test by make-flight.

Signed-off-by: longtao.pang 
---
Changes in v9:
1. Group 'nestedl1*' and 'nestedl2*' together in the list.
2. Remove duplicated runvars for nested job.
3. Avoid testing Xen branches before 4.4; Avoid flights of qemu
traditional (i.e. qemu upstream supported only).
---
 make-flight |   32 
 1 file changed, 32 insertions(+)

diff --git a/make-flight b/make-flight
index cc8ecdb..b5c7c27 100755
--- a/make-flight
+++ b/make-flight
@@ -204,6 +204,37 @@ do_hvm_win7_x64_tests () {
 all_hostflags=$most_hostflags,hvm
 }
 
+do_hvm_debian_nested_tests () {
+  if [ $xenarch != amd64 -o $dom0arch != amd64 \
+  -o "x$qemuu_suffix" != "x-qemuu" ]; then
+return
+  fi
+
+  case $xenbranch in
+xen-3.*-testing)  return;;
+xen-4.0-testing)  return;;
+xen-4.1-testing)  return;;
+xen-4.2-testing)  return;;
+xen-4.3-testing)  return;;
+  esac
+  # Define bios type, in 'ts-debian-hvm-install' script will use it,
+  # or will die if "$r{bios}" undefined.
+  bios=$1
+
+  job_create_test test-$xenarch$kern-$dom0arch$qemuu_suffix-nested \
+test-nested xl $xenarch $dom0arch $qemuu_runvar \
+nestedl1_image=debian-7.2.0-amd64-CD-1.iso \
+nestedl1_vifmodel='e1000' \
+nestedl1_disksize='15000' \
+nestedl1_memsize='3072' \
+nestedl1_enable_nestedhvm='true' \
+nestedl1_guest_storage_size='2' \
+nestedl2_image=debian-7.2.0-amd64-CD-1.iso \
+nestedl2_disksize='15000' \
+bios=$bios
+all_hostflags=$most_hostflags,hvm
+}
+
 do_hvm_debian_test_one () {
   testname=$1
   bios=$2
@@ -408,6 +439,7 @@ test_matrix_do_one () {
 do_hvm_rhel6_tests
 
 do_hvm_debian_tests
+do_hvm_debian_nested_tests seabios
 
   done # qemuu_suffix
 
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v9 6/9] Changes on test step of Debian hvm guest install

2015-05-01 Thread longtao.pang

From: "longtao.pang" 

1. Increase disk size to accommodate to nested test requirement.
2. Since 'Debain-xxx-.iso' image will be stored in rootfs of L1 guest,
therefore needs more disk capacity, increase root partition size to
1.
3. In L1 installation context, assign more memory (defined in runvar) to
it; Since it acts as a nested hypervisor anyway.
4. Comment out CDROM entry in sources.list to make HTTP URL entry
available for L1 hvm guest.
5. Enable nestedhvm feature in ExtraConfig for nested job.

Signed-off-by: longtao.pang 
---
Changes in v9:
1. Using 'guest_var' to get nested L1 guest's specific disk size and
specific memory size.
2. Using 'guest_var' to enable or disable nestedhvm feature.
---
 ts-debian-hvm-install |   17 ++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/ts-debian-hvm-install b/ts-debian-hvm-install
index fd28abc..d823233 100755
--- a/ts-debian-hvm-install
+++ b/ts-debian-hvm-install
@@ -59,7 +59,7 @@ d-i partman-auto/expert_recipe string \\
 use_filesystem{ } filesystem{ vfat } \\
 mountpoint{ /boot/efi } \\
 . \\
-5000 50 5000 ext4 \\
+1 50 1 ext4 \\
 method{ format } format{ } \\
 use_filesystem{ } filesystem{ ext4 } \\
 mountpoint{ / } \\
@@ -73,6 +73,7 @@ d-i apt-setup/cdrom/set-first boolean false
 d-i preseed/late_command string \\
 in-target mkdir -p /boot/efi/EFI/boot; \\
 in-target cp /boot/efi/EFI/debian/grubx64.efi 
/boot/efi/EFI/boot/bootx64.efi ;\\
+in-target sed -i 's/^deb *cdrom/#&/g' /etc/apt/sources.list;
 END
 
 $preseed_file .= preseed_hook_cmds();
@@ -149,6 +150,11 @@ sub prep () {
 target_putfilecontents_root_stash($ho, 10, preseed(),
   $preseed_file_path);
 
+my $extra_config='';
+$extra_config .="nestedhvm=1\n"
+# Use guest_var to get boolean value to enable nestedhvm
+# feature or not
+if guest_var($gho,"enable_nestedhvm",'false') eq 'true';
 # If host has >8G free memory, create a guest with 4G memory to catch
 # any error that triggers cross 4G boundary
 my $host_freemem_mb = host_get_free_memory($ho);
@@ -157,13 +163,18 @@ sub prep () {
 if ($host_freemem_mb > $ram_lots * 2 + $ram_minslop) {
 $ram_mb = $ram_lots;
 } else {
-$ram_mb = 768;
+# Use guest_var to get specific memsize, or will use default '768'
+$ram_mb= guest_var($gho,'memsize',768);
 }
-logm("Host has $host_freemem_mb MB free memory, setting guest memory size 
to $ram_mb MB");
+logm("Host has $host_freemem_mb MB free memory,
+ setting guest memory size to $ram_mb MB");
+# Use guest_var to get specific disk size, or will use default $disk_mb
+$disk_mb= guest_var($gho,'disksize',$disk_mb);
 
 more_prepareguest_hvm($ho,$gho, $ram_mb, $disk_mb,
   OnReboot => 'preserve',
   Bios => $r{bios},
+  ExtraConfig => $extra_config,
   PostImageHook => sub {
 my $cmds = iso_copy_content_from_image($gho, $newiso);
 $cmds .= prepare_initrd($initrddir,$newiso,$preseed_file_path);
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v9 7/9] Add new script to customize nested test configuration

2015-05-01 Thread longtao.pang

From: "longtao.pang" 

1. In this script, make some appropriate runvars which selecthost would
recognise.
2. Prepare the configurations for installing L2 guest VM.
3. Create a lv disk in L0 and hot-attach it to L1, need to restart L1 to
make the block disk to be recognized by L1 system, then using this disk
to create a VG that used for installing L2.

Signed-off-by: longtao.pang 
---
Changes in v9:
1. For L1, identical its ident and guestname as 'nestedl1'.
2. Reuse Ian Campbell's v5_patch[04,05,06] to avoid duplicated codes
about 'ssh host keys' and 'overlays' when installing nested guest VM.
3. Refactor LV disk creating for L2 guest.
---
 ts-nested-setup |   66 +++
 1 file changed, 66 insertions(+)
 create mode 100755 ts-nested-setup

diff --git a/ts-nested-setup b/ts-nested-setup
new file mode 100755
index 000..566186c
--- /dev/null
+++ b/ts-nested-setup
@@ -0,0 +1,66 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2015 Intel Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use DBI;
+use Osstest;
+use Osstest::Debian;
+use Osstest::TestSupport;
+
+tsreadconfig();
+# Pass L0's ident and L1's guestname
+our ($l0_ident,$l1_gn) = @ARGV;
+our ($l0,$l1) = ts_get_host_guest($l0_ident,$l1_gn);
+
+guest_check_ip($l1);
+# L1 guest's ident is same as guestname
+our $l1_ident = $l1->{Guest};
+
+store_runvar($l1_ident,$l1->{Guest});
+store_runvar("${l1_ident}_ip",$l1->{Ip});
+
+# Since Ian Campbell's v5_patch[04,05,06] has a bug that the
+# 'preseed/late_command' are unavailable, so add below line here to
+# make HTTP URL entry available in sources.list for L1 hvm guest.
+# Onece the bug is fiexed, this line code could be removed.
+target_cmd_root($l1, "sed -i 's/^deb *cdrom/#&/g' /etc/apt/sources.list");
+target_cmd_root($l1, "update-rc.d osstest-confirm-booted start 99 2 .");
+
+# Inside L0, create storage lv, attach this lv to L1, and then inside L1 using 
the storage for installing L2
+my $vgname = $l1->{Vg};
+my $guest_storage_lv_name = "${l1_ident}_guest_storage";
+my $guest_storage_lv_size = guest_var($l1,'guest_storage_size',undef);
+my $guest_storage_lvdev = "/dev/${vgname}/${guest_storage_lv_name}";
+
+target_cmd_root($l0, <{Name} ${guest_storage_lvdev},raw,sdb,rw
+END
+# This is probably a bug here, if we use 'xvdx' as block 'dev'
+# after installing xen and boot into xen kernel, the disk will not be
+# recognized by OS. Also, if we use 'sde' instead of 'sdb' as block 'dev',
+# after installing xen and boot into xen kernel, the disk will be recognized
+# as 'sdb' not 'sde', so we will have to use 'sdb' here.
+
+target_install_packages_norec($l1, qw(lvm2 rsync genisoimage));
+# After block-attach lvdev to L1, need to reboot L1 to make the attach
+# to take effect. I think this is a bug for kernel.
+target_reboot($l1);
+# Create a vg in L1 guest and vg name is ${l1_gn}-disk
+target_cmd_root($l1, "pvcreate /dev/xvdb && vgcreate ${l1_gn}-disk /dev/xvdb");
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v9 2/9] Changes to support '/boot' leading paths of kernel, xen, in grub

2015-05-01 Thread longtao.pang

From: "longtao.pang" 

Support situations of grub that have vmlinuz and other things starting
with path of '/boot' rather than '/'.

Signed-off-by: longtao.pang 
---
 Osstest/Debian.pm |8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 1e7e8e3..6691ff6 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -452,21 +452,21 @@ sub setupboot_grub2 () {
 if (m/^submenu\s+[\'\"](.*)[\'\"].*\{\s*$/) {
 $submenu={ StartLine =>$.};
 }
-if (m/^\s*multiboot\s*\/(xen\-[0-9][-+.0-9a-z]*\S+)/) {
+if (m/^\s*multiboot\s*(?:\/boot)?\/(xen\S+)/) {
 die unless $entry;
 $entry->{Hv}= $1;
 }
-if (m/^\s*multiboot\s*\/(vmlinu[xz]-(\S+))/) {
+if (m/^\s*multiboot\s*(?:\/boot)?\/(vmlinu[xz]-(\S+))/) {
 die unless $entry;
 $entry->{KernOnly}= $1;
 $entry->{KernVer}= $2;
 }
-if (m/^\s*module\s*\/(vmlinu[xz]-(\S+))/) {
+if (m/^\s*module\s*(?:\/boot)?\/(vmlinu[xz]-(\S+))/) {
 die unless $entry;
 $entry->{KernDom0}= $1;
 $entry->{KernVer}= $2;
 }
-if (m/^\s*module\s*\/(initrd\S+)/) {
+if (m/^\s*module\s*(?:\/boot)?\/(initrd\S+)/) {
 $entry->{Initrd}= $1;
 }
if (m/^\s*module\s*\/(xenpolicy\S+)/) {
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v9 5/9] Move the codes about memory size setting into prep()

2015-05-01 Thread longtao.pang

From: "longtao.pang" 

Move the codes about memory size setting from outside into prep(),
where these $ram_mb operations are only used; and in next patch,
for L2 guest installation, $ram_mb will be override by a runvar value,
which needs $gho in scope.

Signed-off-by: longtao.pang 
---
 ts-debian-hvm-install |   24 
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git a/ts-debian-hvm-install b/ts-debian-hvm-install
index 0085d82..fd28abc 100755
--- a/ts-debian-hvm-install
+++ b/ts-debian-hvm-install
@@ -149,6 +149,18 @@ sub prep () {
 target_putfilecontents_root_stash($ho, 10, preseed(),
   $preseed_file_path);
 
+# If host has >8G free memory, create a guest with 4G memory to catch
+# any error that triggers cross 4G boundary
+my $host_freemem_mb = host_get_free_memory($ho);
+my $ram_minslop = 100;
+my $ram_lots = 5000;
+if ($host_freemem_mb > $ram_lots * 2 + $ram_minslop) {
+$ram_mb = $ram_lots;
+} else {
+$ram_mb = 768;
+}
+logm("Host has $host_freemem_mb MB free memory, setting guest memory size 
to $ram_mb MB");
+
 more_prepareguest_hvm($ho,$gho, $ram_mb, $disk_mb,
   OnReboot => 'preserve',
   Bios => $r{bios},
@@ -166,18 +178,6 @@ sub prep () {
 });
 }
 
-# If host has >8G free memory, create a guest with 4G memory to catch
-# any error that triggers cross 4G boundary
-my $host_freemem_mb = host_get_free_memory($ho);
-my $ram_minslop = 100;
-my $ram_lots = 5000;
-if ($host_freemem_mb > $ram_lots * 2 + $ram_minslop) {
-$ram_mb = $ram_lots;
-} else {
-$ram_mb = 768;
-}
-logm("Host has $host_freemem_mb MB free memory, setting guest memory size to 
$ram_mb MB");
-
 if (!$stage) {
 prep();
 guest_create($gho);
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v9 1/9] parsing grub which has 'submenu' primitive

2015-05-01 Thread longtao.pang

From: "longtao.pang" 

>From a hvm kernel build from Linux stable Kernel tree,
the auto generated grub2 menu will have 'submenu' primitive, upon the
'menuentry' items. Xen boot entries will be grouped into a submenu. This
patch adds capability to support such grub formats.

Signed-off-by: longtao.pang 
---
 Osstest/Debian.pm |   13 -
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index bd506a6..1e7e8e3 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -407,10 +407,18 @@ sub setupboot_grub2 () {
 
 my $count= 0;
 my $entry;
+my $submenu;
 while (<$f>) {
 next if m/^\s*\#/ || !m/\S/;
 if (m/^\s*\}\s*$/) {
-die unless $entry;
+die unless $entry || $submenu;
+if(!defined $entry && defined $submenu){
+logm("Met end of a submenu starting from ".
+"$submenu->{StartLine}. ".
+"Our want kern is $want_kernver");
+$submenu=undef;
+next;
+}
 my (@missing) =
 grep { !defined $entry->{$_} } 
(defined $xenhopt
@@ -441,6 +449,9 @@ sub setupboot_grub2 () {
 $entry= { Title => $1, StartLine => $., Number => $count };
 $count++;
 }
+if (m/^submenu\s+[\'\"](.*)[\'\"].*\{\s*$/) {
+$submenu={ StartLine =>$.};
+}
 if (m/^\s*multiboot\s*\/(xen\-[0-9][-+.0-9a-z]*\S+)/) {
 die unless $entry;
 $entry->{Hv}= $1;
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v9 0/9] Introduction of netsted HVM test job

2015-05-01 Thread longtao.pang

This patch set adds nested HVM test case for osstest.
In this test case, a Xen hypervisor (L1) runs on top of another Xen 
hypervisor (L0). 
Upon L1 hypervisor, we will then create a nested guest (L2), and test if the 
Linux guest can then be installed and run well.
About nested Xen virtualization, 
refer to http://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xen.

Test steps
0. To run osstest in standalone mode, write a config file in
   '~/.xen-osstest/config', and then create a standalone.config file 
   to define 'TREE_LINUX', 'REVISION_LINUX' which will be used for
   nested test. The directory path of 'Debian Images' could be defined 
   in '~/.xen-osstest/config'. 
1. run './standalone-reset' to generate standalone.db firstly then run 
   'build-amd64' job and then 'build-amd64-pvops', to prepare xen
   installation tarball and hvm guest kernel.
2. run 'test-amd64-amd6-qemuu-nested' job, it does following:
a. invoke test step of 'ts-debain-hvm-install' to install 
   a normal HVM guest
b. invoke test step of 'ts-nested-setup' to make some
   appropriate runvars which selecthost() would recognise and
   prepare the configurations for installing L2 guest VM. 
c. invoke test step of 'ts-xen-install' to install xen on 
   the normal guest, alter it into a L1 hypervisor
d. invoke test step of 'ts-debain-hvm-install' again, but 
   take the L1 hypervisor as host, install the L2 guest on it
e. invoke test step of 'ts-guest-stop', stop L2 guest.
f. invoke test step of 'ts-guest-destroy' to destroy L1 guest.

This patch set reuse 'ts-debian-hvm-install' for both L1 installation
and L2 installation, use 'nestedl1' as L1's guestname and identity 
and use 'nestedl2 as L2's guestname.
It also reuses 'ts-xen-install' with L1's identity 'nestedl1' input parameter 
to differentiate from L0 Xen installation.
This patch series has been tested on test machines of amd64 arch,
Debian-7.2.0-amd64 as guests OS, with hvm domain0 of Linux kernel 3.18.5, 
in standalone mode.
Also, we use linux-stable tree as domain0 kernel source.

longtao.pang (9):
  parsing grub which has 'submenu' primitive
  Changes to support '/boot' leading paths of kernel, xen, in grub
  Refactor installation of overlays for guest used
  Edit some APIs in TestSupport.pm for nested test
  Move the codes about memory size setting into prep()
  Changes on test step of Debian hvm guest install
  Add new script to customize nested test configuration
  Compose the main recipe of nested test job
  Add test job for nest test case

 Osstest/Debian.pm  |   22 
 Osstest/TestSupport.pm |   27 
 make-flight|   32 +++
 sg-run-job |   11 
 ts-debian-hvm-install  |   37 +--
 ts-nested-setup|   66 
 6 files changed, 167 insertions(+), 28 deletions(-)
 create mode 100755 ts-nested-setup

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v8 6/7] Compose the main recipe of nested test job

2015-04-13 Thread longtao.pang

Signed-off-by: longtao.pang 
---
Changes in v8:
Change the patch order from 6 to 5.
---
 sg-run-job |   11 +++
 1 file changed, 11 insertions(+)

diff --git a/sg-run-job b/sg-run-job
index eae159d..2ca5ebf 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -299,6 +299,17 @@ proc run-job/test-pair {} {
 #run-ts . remus-failover ts-remus-check src_host dst_host + debian
 }
 
+proc need-hosts/test-nested {} {return host}
+proc run-job/test-nested {} {
+run-ts . = ts-debian-hvm-install + host + nestedl1
+run-ts . = ts-nested-setup + host + nestedl1
+run-ts . = ts-xen-install nested_l1
+run-ts . = ts-host-reboot nested_l1
+run-ts . = ts-debian-hvm-install nested_l1 nestedl2
+run-ts . = ts-guest-stop nested_l1 nestedl2
+run-ts . = ts-guest-destroy host nestedl1
+}
+
 proc test-guest-migr {g} {
 if {[catch { run-ts . = ts-migrate-support-check + host $g }]} return
 
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v8 1/7] parsing grub which has 'submenu' primitive

2015-04-13 Thread longtao.pang

>From a hvm kernel build from Linux stable Kernel tree,
the auto generated grub2 menu will have 'submenu' primitive, upon the
'menuentry' items. Xen boot entries will be grouped into a submenu. This
patch adds capability to support such grub formats.

Signed-off-by: longtao.pang 
---
Changes in v8:
Separate unrelated codes to the use of submenu.
---
 Osstest/Debian.pm |   13 -
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 69530fb..378af54 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -404,10 +404,18 @@ sub setupboot_grub2 () {
 
 my $count= 0;
 my $entry;
+my $submenu;
 while (<$f>) {
 next if m/^\s*\#/ || !m/\S/;
 if (m/^\s*\}\s*$/) {
-die unless $entry;
+die unless $entry || $submenu;
+if(!defined $entry && defined $submenu){
+logm("Met end of a submenu starting from ".
+"$submenu->{StartLine}. ".
+"Our want kern is $want_kernver");
+$submenu=undef;
+next;
+}
 my (@missing) =
 grep { !defined $entry->{$_} } 
(defined $xenhopt
@@ -438,6 +446,9 @@ sub setupboot_grub2 () {
 $entry= { Title => $1, StartLine => $., Number => $count };
 $count++;
 }
+if (m/^submenu\s+[\'\"](.*)[\'\"].*\{\s*$/) {
+$submenu={ StartLine =>$.};
+}
 if (m/^\s*multiboot\s*\/(xen\-[0-9][-+.0-9a-z]*\S+)/) {
 die unless $entry;
 $entry->{Hv}= $1;
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v8 5/7] Add new script to customize nested test configuration

2015-04-13 Thread longtao.pang

1. In this script, make some appropriate runvars which selecthost would
recognise.
2. Prepare the configurations for installing L2 guest VM.
3. Create a lv disk in L0 and hot-attach it to L1, need to restart L1 to
make the block disk to be recognized by L1 system, then using this disk
to create a VG that used for installing L2.

Signed-off-by: longtao.pang 
---
Changes in v8:
1. Replace '$nested_host' by '$l1->{Guest}'.
---
 ts-nested-setup |   52 
 1 file changed, 52 insertions(+)
 create mode 100755 ts-nested-setup

diff --git a/ts-nested-setup b/ts-nested-setup
new file mode 100755
index 000..41d5e80
--- /dev/null
+++ b/ts-nested-setup
@@ -0,0 +1,52 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2015 Intel Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use DBI;
+use Osstest;
+use Osstest::Debian;
+use Osstest::TestSupport;
+
+tsreadconfig();
+our ($l0,$l1) = ts_get_host_guest(@ARGV);
+
+guest_check_ip($l1);
+target_cmd_root($l1, "mkdir -p /home/osstest/.ssh && cp 
/root/.ssh/authorized_keys /home/osstest/.ssh/");
+my $url = $c{WebspaceUrl}.$c{WebspaceCommon}.$l0->{Name}."_".'overlay.tar';
+target_cmd_root($l1, <{Guest});
+store_runvar("$l1->{Guest}_ip",$l1->{Ip});
+
+my $l2_disk_mb = 2;
+my $l2_lv_name = 'nestedl2-disk';
+my $vgname = $l0->{Name};
+$vgname .= ".$c{TestHostDomain}" if ($l0->{Suite} =~ m/lenny/);
+target_cmd_root($l0, <{Name} /dev/${vgname}/${l2_lv_name},raw,sdb,rw
+END
+target_install_packages_norec($l1, qw(lvm2 rsync genisoimage));
+target_reboot($l1);
+target_cmd_root($l1, "pvcreate /dev/xvdb && vgcreate ${l2_lv_name}_vg 
/dev/xvdb");
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v8 2/7] Changes to support '/boot' leading paths of kernel, xen, in grub

2015-04-13 Thread longtao.pang

Support situations of grub that have vmlinuz and other things starting
with path of '/boot' rather than '/'.

Signed-off-by: longtao.pang 
---
 Osstest/Debian.pm |8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 378af54..1a57cdd 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -449,21 +449,21 @@ sub setupboot_grub2 () {
 if (m/^submenu\s+[\'\"](.*)[\'\"].*\{\s*$/) {
 $submenu={ StartLine =>$.};
 }
-if (m/^\s*multiboot\s*\/(xen\-[0-9][-+.0-9a-z]*\S+)/) {
+if (m/^\s*multiboot\s*(?:\/boot)?\/(xen\S+)/) {
 die unless $entry;
 $entry->{Hv}= $1;
 }
-if (m/^\s*multiboot\s*\/(vmlinu[xz]-(\S+))/) {
+if (m/^\s*multiboot\s*(?:\/boot)?\/(vmlinu[xz]-(\S+))/) {
 die unless $entry;
 $entry->{KernOnly}= $1;
 $entry->{KernVer}= $2;
 }
-if (m/^\s*module\s*\/(vmlinu[xz]-(\S+))/) {
+if (m/^\s*module\s*(?:\/boot)?\/(vmlinu[xz]-(\S+))/) {
 die unless $entry;
 $entry->{KernDom0}= $1;
 $entry->{KernVer}= $2;
 }
-if (m/^\s*module\s*\/(initrd\S+)/) {
+if (m/^\s*module\s*(?:\/boot)?\/(initrd\S+)/) {
 $entry->{Initrd}= $1;
 }
if (m/^\s*module\s*\/(xenpolicy\S+)/) {
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v8 3/7] Edit some APIs in TestSupport.pm for nested test

2015-04-13 Thread longtao.pang

1. Designate vif model by make-flight.
2. In L2 installation context, its host (L1) IP address is not queried
from DNS, but after running "ts-nested-setup + host + nested", L1 IP is
stored in runvar.

Signed-off-by: longtao.pang 
---
Changes in v8:
Remove the unnecessary symbol of ';' in 'TestSupport.pm'.
---
 Osstest/TestSupport.pm |8 +++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 1bde67d..4cdacfc 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -818,6 +818,10 @@ sub selecthost ($) {
logm("Host $name is in HostGroup $ho->{Properties}{HostGroup}");
 }
 
+if ( $r{"${name}_ip"} ) {
+$setprop->("IpAddr", $r{"${name}_ip"});
+}
+
 # Next, we use the config file's general properites as defaults
 foreach my $k (keys %c) {
next unless $k =~ m/^HostProp_([A-Z].*)$/;
@@ -1548,11 +1552,13 @@ sub prepareguest_part_xencfg ($) {
 my $oncrash= $xopts->{OnCrash} || 'preserve';
 my $vcpus= guest_var($gho, 'vcpus', $xopts->{DefVcpus} || 2);
 my $xoptcfg= $xopts->{ExtraConfig};
+my $vif= guest_var($gho, 'vifmodel','');
+my $vifmodel= $vif ? "model=$vif" : '';
 $xoptcfg='' unless defined $xoptcfg;
 my $xencfg= <{Ether}' ]
+vif = [ 'type=ioemu,${vifmodel},mac=$gho->{Ether}' ]
 #
 on_poweroff = '$onpoweroff'
 on_reboot   = '$onreboot'
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v8 7/7] Add test job for nest test case

2015-04-13 Thread longtao.pang

1. This patch adds creation of the nested test job, when job creation
procedure is invoked.
2. Set nested L1's vif model as e1000 by make-flight.

Signed-off-by: longtao.pang 
---
Changes in v8:
1. Use 'nestedl1' and 'nestedl2' as L1 and L2's clearer name in nested
test job.
2. Setting Debian_ISO as 'debian-7.2.0-amd64-CD-1.iso' in make-flight.
---
 make-flight |   20 
 1 file changed, 20 insertions(+)

diff --git a/make-flight b/make-flight
index cc8ecdb..40ca808 100755
--- a/make-flight
+++ b/make-flight
@@ -204,6 +204,25 @@ do_hvm_win7_x64_tests () {
 all_hostflags=$most_hostflags,hvm
 }
 
+do_hvm_debian_nested_tests () {
+  if [ $xenarch != amd64 -a $dom0arch != amd64 ]; then
+return
+  fi
+
+  job_create_test test-$xenarch$kern-$dom0arch-nested test-nested xl \
+$xenarch $dom0arch \
+nestedl1_image=debian-7.2.0-amd64-CD-1.iso \
+nestedl2_image=debian-7.2.0-amd64-CD-1.iso \
+bios=seabios \
+kernbuildjob=build-amd64-pvops \
+kernkind=pvops \
+nestedl1_vifmodel='e1000' \
+nested_disk='15000' \
+nestedl1_memory='3072' \
+device_model_version=qemu-xen \
+all_hostflags=$most_hostflags,hvm
+}
+
 do_hvm_debian_test_one () {
   testname=$1
   bios=$2
@@ -430,6 +449,7 @@ test_matrix_do_one () {
 done
 
   fi
+  do_hvm_debian_nested_tests
   #do_passthrough_tests
 }
 
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v8 4/7] Changes on test step of Debian hvm guest install

2015-04-13 Thread longtao.pang

1. Increase disk size to accommodate to nested test requirement.
2. Since 'Debain-xxx-.iso' image will be stored in rootfs of L1 guest,
therefore needs more disk capacity, increase root partition size in
preseed generation.
3. In L1 installation context, assign more memory to it; Since it
acts as a nested hypervisor anyway.
4. Comment out CDROM entry in sources.list to make HTTP URL entry
available for L1 hvm guest.
5. Enable nestedhvm feature in ExtraConfig for nested job.

Signed-off-by: longtao.pang 
---
Changes in v8:
1. Update a conventional way to comment out CDROM entry in sources.list.
2. Enable nestedhvm feature only for the nested job.
3. Set nested disk and memroy size to be driven from runvars in
'ts-debian-hvm-install'.
---
 ts-debian-hvm-install |9 ++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/ts-debian-hvm-install b/ts-debian-hvm-install
index cfd5144..13009d0 100755
--- a/ts-debian-hvm-install
+++ b/ts-debian-hvm-install
@@ -36,7 +36,7 @@ our $ho= selecthost($whhost);
 
 # guest memory size will be set based on host free memory, see below
 our $ram_mb;
-our $disk_mb= 1;
+our $disk_mb= $r{'nested_disk'} ? $r{'nested_disk'} : 1;
 
 our $guesthost= "$gn.guest.osstest";
 our $gho;
@@ -60,7 +60,7 @@ d-i partman-auto/expert_recipe string \\
 use_filesystem{ } filesystem{ vfat } \\
 mountpoint{ /boot/efi } \\
 . \\
-5000 50 5000 ext4 \\
+1 50 1 ext4 \\
 method{ format } format{ } \\
 use_filesystem{ } filesystem{ ext4 } \\
 mountpoint{ / } \\
@@ -75,6 +75,7 @@ d-i preseed/late_command string \\
 in-target mkdir -p /boot/efi/EFI/boot; \\
 in-target cp /boot/efi/EFI/debian/grubx64.efi 
/boot/efi/EFI/boot/bootx64.efi ;\\
 in-target mkdir -p /root/.ssh; \\
+in-target sed -i 's/^deb *cdrom/#&/g' /etc/apt/sources.list; \\
 in-target sh -c "echo -e '$authkeys'> /root/.ssh/authorized_keys";
 END
 return $preseed_file;
@@ -149,9 +150,11 @@ sub prep () {
 target_putfilecontents_root_stash($ho, 10, preseed(),
   $preseed_file_path);
 
+my $nestedhvm_feature = $gn eq 'nestedl1' ? 'nestedhvm=1' : '';
 more_prepareguest_hvm($ho,$gho, $ram_mb, $disk_mb,
   OnReboot => 'preserve',
   Bios => $r{bios},
+  ExtraConfig => $nestedhvm_feature,
   PostImageHook => sub {
 my $cmds = iso_copy_content_from_image($gho, $newiso);
 $cmds .= prepare_initrd($initrddir,$newiso,$preseed_file_path);
@@ -174,7 +177,7 @@ my $ram_lots = 5000;
 if ($host_freemem_mb > $ram_lots * 2 + $ram_minslop) {
 $ram_mb = $ram_lots;
 } else {
-$ram_mb = 768;
+$ram_mb = $r{"${gn}_memory"} ? $r{"${gn}_memory"} : 768;
 }
 logm("Host has $host_freemem_mb MB free memory, setting guest memory size to 
$ram_mb MB");
 
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v8 0/7] Introduction of netsted HVM test job

2015-04-13 Thread longtao.pang

This patch set adds nested HVM test case for osstest.
In this test case, a Xen hypervisor (L1) runs on top of another Xen 
hypervisor (L0). 
Upon L1 hypervisor, we will then create a nested guest (L2), and test if the 
Linux guest can then be installed and run well.
About nested Xen virtualization, 
refer to http://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xen.

Test steps
0. To run osstest in standalone mode, write a config file in
   '~/.xen-osstest/config', and then create a standalone.config file 
   to define 'TREE_LINUX', 'REVISION_LINUX' which will be used for
   nested test. The directory path of 'Debian Images' could be defined 
   in '~/.xen-osstest/config'. 
1. run 'build-amd64' job and then 'build-amd64-pvops', to prepare xen
   installation tarball and hvm guest kernel.
2. run 'test-amd64-amd6-nested' job, it does following:
a. invoke test step of 'ts-debain-hvm-install' to install 
   a normal HVM guest
b. invoke test step of 'ts-nested-setup' to make some
   appropriate runvars which selecthost() would recognise and
   prepare the configurations for installing L2 guest VM. 
c. invoke test step of 'ts-xen-install' to install xen on 
   the normal guest, alter it into a L1 hypervisor
d. invoke test step of 'ts-debain-hvm-install' again, but 
   take the L1 hypervisor as host, install the L2 guest on it
e. invoke test step of 'ts-guest-stop', stop L2 guest.
f. invoke test step of 'ts-guest-destroy' to destroy L1 guest.

This patch set reuse 'ts-debian-hvm-install' for both L1 installation and L2
installation, define 'nestedl1' and 'nestedl2' as L1 and L2's hostname,
define 'nested_l1 as L1's host identity.
It also reuses 'ts-xen-install' with L1's hostname 'nestedl1' input parameter 
to differentiate from L0 Xen installation.
This patch series has been tested on test machines of amd64 arch,
Debian-7.2.0-amd64 as guests OS, with hvm domain0 of Linux kernel 3.18.5, 
in standalone mode.
Also, we use linux-stable tree as domain0 kernel source.

longtao.pang (7):
  parsing grub which has 'submenu' primitive
  Changes to support '/boot' leading paths of kernel, xen, in grub
  Edit some APIs in TestSupport.pm for nested test
  Changes on test step of Debian hvm guest install
  Add new script to customize nested test configuration
  Compose the main recipe of nested test job
  Add test job for nest test case

 Osstest/Debian.pm  |   21 -
 Osstest/TestSupport.pm |8 +++-
 make-flight|   20 
 sg-run-job |   11 +++
 ts-debian-hvm-install  |9 ++---
 ts-nested-setup|   52 

 6 files changed, 112 insertions(+), 9 deletions(-)
 create mode 100755 ts-nested-setup


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v7 1/6] parsing grub which has 'submenu' primitive

2015-03-27 Thread longtao.pang

>From a hvm kernel build from Linux stable Kernel tree,
the auto generated grub2 menu will have 'submenu' primitive, upon the
'menuentry' items. Xen boot entries will be grouped into a submenu. This
patch adds capability to support such grub formats.

Signed-off-by: longtao.pang 
---
Changes in v7:
Remove the reformatting change for Debian.pm and keep the original format.
---
 Osstest/Debian.pm |   21 -
 1 file changed, 16 insertions(+), 5 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 6784024..35163a0 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -398,10 +398,18 @@ sub setupboot_grub2 () {
 
 my $count= 0;
 my $entry;
+my $submenu;
 while (<$f>) {
 next if m/^\s*\#/ || !m/\S/;
 if (m/^\s*\}\s*$/) {
-die unless $entry;
+die unless $entry || $submenu;
+if(!defined $entry && defined $submenu){
+logm("Met end of a submenu starting from ".
+"$submenu->{StartLine}. ".
+"Our want kern is $want_kernver");
+$submenu=undef;
+next;
+}
 my (@missing) =
 grep { !defined $entry->{$_} } 
(defined $xenhopt
@@ -432,21 +440,24 @@ sub setupboot_grub2 () {
 $entry= { Title => $1, StartLine => $., Number => $count };
 $count++;
 }
-if (m/^\s*multiboot\s*\/(xen\-[0-9][-+.0-9a-z]*\S+)/) {
+if (m/^submenu\s+[\'\"](.*)[\'\"].*\{\s*$/) {
+$submenu={ StartLine =>$.};
+}
+if (m/^\s*multiboot\s*(?:\/boot)*\/(xen\S+)/) {
 die unless $entry;
 $entry->{Hv}= $1;
 }
-if (m/^\s*multiboot\s*\/(vmlinu[xz]-(\S+))/) {
+if (m/^\s*multiboot\s*(?:\/boot)*\/(vmlinu[xz]-(\S+))/) {
 die unless $entry;
 $entry->{KernOnly}= $1;
 $entry->{KernVer}= $2;
 }
-if (m/^\s*module\s*\/(vmlinu[xz]-(\S+))/) {
+if (m/^\s*module\s*(?:\/boot)*\/(vmlinu[xz]-(\S+))/) {
 die unless $entry;
 $entry->{KernDom0}= $1;
 $entry->{KernVer}= $2;
 }
-if (m/^\s*module\s*\/(initrd\S+)/) {
+if (m/^\s*module\s*(?:\/boot)*\/(initrd\S+)/) {
 $entry->{Initrd}= $1;
 }
if (m/^\s*module\s*\/(xenpolicy\S+)/) {
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v7 0/6] Introduction of netsted HVM test job

2015-03-27 Thread longtao.pang

This patch set adds nested HVM test case for osstest.
In this test case, a Xen hypervisor (L1) runs on top of another Xen hypervisor 
(L0). 
Upon L1 hypervisor, we will then create a nested guest (L2), and test if the 
Linux guest can then be installed and run well.
About nested Xen virtualization, refer to 
http://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xen.

Test steps
0. To run osstest in standalone mode, write a config file in
   '~/.xen-osstest/config', and then create a standalone.config file to
   define 'TREE_LINUX', 'REVISION_LINUX' and 'NESTED_OS_IMAGE' which
   will be used for nested test. The directry path of 'Debian Images'
   could be difined in '~/.xen-osstest/config'. 
1. run 'build-amd64' job and then 'build-amd64-pvops', to prepare xen
   installation tarball and hvm guest kernel.
2. run 'test-amd64-amd6-nested' job, it does following:
a. invoke test step of 'ts-debain-hvm-install' to install 
   a normal HVM guest
b. invoke test step of 'ts-nested-setup' to make some
   appropriate runvars which selecthost() would recognise and
   prepare the configurations for installing L2 guest VM. 
c. invoke test step of 'ts-xen-install' to install xen on 
   the normal guest, alter it into a L1 hypervisor
d. invoke test step of 'ts-debain-hvm-install' again, but 
   take the L1 hypervisor as host, install the L2 guest on it
e. invoke test step of 'ts-guest-stop', stop L2 guest.
f. invoke test step of 'ts-guest-destroy' to destroy L1 guest.

This patch set reuses 'ts-debian-hvm-install' for both L1 installation and L2 
installation, define 'nested' and 'nested2' as L1 and L2's hostname, define 
'nested_l1 as L1's host ident.
It also reuses 'ts-xen-install' with L1's hostname 'nested' input param to 
differentiate from L0 Xen installation.
This patch series has been tested on test machines of amd64 arch, 
debian-7.6.0-amd64 as guests OS, with hvm domain0 of Linux kernel 3.18.5, in 
standalone mode.
Also, we use linux-stable tree as domain0 kernel source.

longtao.pang (6):
  parsing grub which has 'submenu' primitive
  Edit some testsupport APIs for nested test
  Changes on test step of debain hvm guest install
  Add new script to custmize nested test configuration
  Add test job for nest test case
  Compose the main recipe of nested test job

 Osstest/Debian.pm  |   21 ++-
 Osstest/TestSupport.pm |8 ++-
 make-flight|   21 +++
 sg-run-job |   11 ++
 ts-debian-hvm-install  |8 +--
 ts-nested-setup|   54 
 6 files changed, 115 insertions(+), 8 deletions(-)
 create mode 100755 ts-nested-setup


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v7 2/6] Edit some testsupport APIs for nested test

2015-03-27 Thread longtao.pang

1. Designate vif model to 'e1000' by make-flight.
2. In L2 installation context, its host (L1) IP address is not queried
from DNS, but after running "ts-nested-setup + host + nested", L1 IP is
stored in runvar.

Signed-off-by: longtao.pang 
---
Changes in v7:
1.set vif model as e1000 when make-flight
2.remove the changes about multi-reboot-time, since this is not code
issue but the network DNS setting not correct in my previous osstest
env.
---
 Osstest/TestSupport.pm |8 +++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index a467713..10c229d 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -819,6 +819,10 @@ sub selecthost ($) {
logm("Host $name is in HostGroup $ho->{Properties}{HostGroup}");
 }
 
+if ( $r{"${name}_ip"} ) {
+$setprop->("IpAddr", $r{"${name}_ip"});
+}
+
 # Next, we use the config file's general properites as defaults
 foreach my $k (keys %c) {
next unless $k =~ m/^HostProp_([A-Z].*)$/;
@@ -1548,11 +1552,13 @@ sub prepareguest_part_xencfg ($) {
 my $oncrash= $xopts->{OnCrash} || 'preserve';
 my $vcpus= guest_var($gho, 'vcpus', $xopts->{DefVcpus} || 2);
 my $xoptcfg= $xopts->{ExtraConfig};
+my $vif= guest_var($gho, 'vifmodel','');
+my $vifmodel= $vif ? "model=$vif" : '';;
 $xoptcfg='' unless defined $xoptcfg;
 my $xencfg= <{Ether}' ]
+vif = [ 'type=ioemu,${vifmodel},mac=$gho->{Ether}' ]
 #
 on_poweroff = '$onpoweroff'
 on_reboot   = '$onreboot'
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v7 4/6] Add new script to custmize nested test configuration

2015-03-27 Thread longtao.pang

1. In this script, make some appropriate runvars which selecthost would
recognise.
2. Prepare the configurations for installing L2 guest VM.
3. Create a lv disk in L0 and hot-attach it to L1, need to restart L1 to
make the block disk to be recognized by L1 system, then using this disk
to create a VG that used for installing L2.

Signed-off-by: longtao.pang 
---
Changes in v7:
Add new ts script for nested configuration according maintainer's reply.
---
 ts-nested-setup |   54 ++
 1 file changed, 54 insertions(+)
 create mode 100755 ts-nested-setup

diff --git a/ts-nested-setup b/ts-nested-setup
new file mode 100755
index 000..153c4a3
--- /dev/null
+++ b/ts-nested-setup
@@ -0,0 +1,54 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2015 Intel Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use DBI;
+use Osstest;
+use Osstest::Debian;
+use Osstest::TestSupport;
+
+tsreadconfig();
+our ($ident_l0,$nested_host) = @ARGV;
+our ($l0,$l1) = ts_get_host_guest(@ARGV);
+
+guest_check_ip($l1);
+target_cmd_root($l1, "mkdir -p /home/osstest/.ssh && cp 
/root/.ssh/authorized_keys /home/osstest/.ssh/");
+my $url = $c{WebspaceUrl}.$c{WebspaceCommon}.$l0->{Name}."_".'overlay.tar';
+target_cmd_root($l1, <{Ip});
+store_runvar('multi_reboot_time',5);
+
+my $l2_disk_mb = 2;
+my $l2_lv_name = 'nestedl2';
+my $vgname = $l0->{Name};
+$vgname .= ".$c{TestHostDomain}" if ($l0->{Suite} =~ m/lenny/);
+target_cmd_root($l0, <{Name} /dev/${vgname}/${l2_lv_name},raw,sdb,rw
+END
+target_install_packages_norec($l1, qw(lvm2 rsync genisoimage));
+target_reboot($l1);
+target_cmd_root($l1, "pvcreate /dev/xvdb && vgcreate ${l2_lv_name}_vg 
/dev/xvdb");
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v7 5/6] Add test job for nest test case

2015-03-27 Thread longtao.pang

1. This patch adds creation of the nested test job, when job creation
procedure is invoked.
2. 'NESTED_OS_IMAGE' is the name of 'Debian ISO Images', which defined
in standalone.config.
3. Set nested L1's vif model as e1000 by make-flight.

Signed-off-by: longtao.pang 
---
Changes in v7:
Set L1's vif model as e1000 in runvar by make-flight.
---
 make-flight |   21 +
 1 file changed, 21 insertions(+)

diff --git a/make-flight b/make-flight
index 8ac3a87..b8f266f 100755
--- a/make-flight
+++ b/make-flight
@@ -204,6 +204,26 @@ do_hvm_win7_x64_tests () {
 all_hostflags=$most_hostflags,hvm
 }
 
+do_hvm_debian_nested_tests () {
+  if [ $xenarch != amd64 ]; then
+return
+  fi
+  if [ $dom0arch != amd64 ]; then
+return
+  fi
+
+  job_create_test test-$xenarch$kern-$dom0arch-nested test-nested xl \
+   $xenarch $dom0arch \
+nested_image=$NESTED_OS_IMAGE \
+nested2_image=$NESTED_OS_IMAGE \
+bios=seabios \
+kernbuildjob=build-amd64-pvops \
+kernkind=pvops \
+nested_vifmodel='e1000' \
+device_model_version=qemu-xen \
+all_hostflags=$most_hostflags,hvm
+}
+
 do_hvm_debian_test_one () {
   testname=$1
   bios=$2
@@ -430,6 +450,7 @@ test_matrix_do_one () {
 done
 
   fi
+  do_hvm_debian_nested_tests
   do_passthrough_tests
 }
 
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v7 6/6] Compose the main recipe of nested test job

2015-03-27 Thread longtao.pang

Signed-off-by: longtao.pang 
---
Changes in v7:
Add 'ts-guest-stop' and 'ts-guest-destroy' scripts to shutdown L2 and L1
after finishing test.
---
 sg-run-job |   11 +++
 1 file changed, 11 insertions(+)

diff --git a/sg-run-job b/sg-run-job
index a1ff24b..be55da8 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -297,6 +297,17 @@ proc run-job/test-pair {} {
 #run-ts . remus-failover ts-remus-check src_host dst_host + debian
 }
 
+proc need-hosts/test-nested {} {return host}
+proc run-job/test-nested {} {
+run-ts . = ts-debian-hvm-install + host + nested
+run-ts . = ts-nested-setup + host + nested
+run-ts . = ts-xen-install nested_l1
+run-ts . = ts-host-reboot nested_l1
+run-ts . = ts-debian-hvm-install nested_l1 nested2
+run-ts . = ts-guest-stop nested_l1 nested2
+run-ts . = ts-guest-destroy host nested
+}
+
 proc test-guest-migr {g} {
 if {[catch { run-ts . = ts-migrate-support-check + host $g }]} return
 
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH v7 3/6] Changes on test step of debain hvm guest install

2015-03-27 Thread longtao.pang

1. Increase disk size to accomodate to nested test requirment.
2. Since 'Debain-xxx-.iso' image will be stored there, therefore needs
more disk capacity, increase root partition size in preseed generation.
3. In L1 installation context, assign more memory to it; Since it
acts as a nested hypervisor anyway.
4. Comment out CDROM entry in sources.list to make HTTP URL entry
available for L1 hvm guest.
5. Enable nestedhvm feature in ExtraConfig.

Signed-off-by: longtao.pang 
---
Changes in v7:
1. Comment out CDROM entry in sources.list to make HTTP URL entry
available for L1 hvm guest.
2. Enable nestedhvm feature in ExtraConfig.
---
 ts-debian-hvm-install |8 ++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/ts-debian-hvm-install b/ts-debian-hvm-install
index cfd5144..8e676fa 100755
--- a/ts-debian-hvm-install
+++ b/ts-debian-hvm-install
@@ -36,7 +36,7 @@ our $ho= selecthost($whhost);
 
 # guest memory size will be set based on host free memory, see below
 our $ram_mb;
-our $disk_mb= 1;
+our $disk_mb= 15000;
 
 our $guesthost= "$gn.guest.osstest";
 our $gho;
@@ -60,7 +60,7 @@ d-i partman-auto/expert_recipe string \\
 use_filesystem{ } filesystem{ vfat } \\
 mountpoint{ /boot/efi } \\
 . \\
-5000 50 5000 ext4 \\
+1 50 1 ext4 \\
 method{ format } format{ } \\
 use_filesystem{ } filesystem{ ext4 } \\
 mountpoint{ / } \\
@@ -75,6 +75,7 @@ d-i preseed/late_command string \\
 in-target mkdir -p /boot/efi/EFI/boot; \\
 in-target cp /boot/efi/EFI/debian/grubx64.efi 
/boot/efi/EFI/boot/bootx64.efi ;\\
 in-target mkdir -p /root/.ssh; \\
+in-target sed -i '/^deb *cdrom/s/^/#/' /etc/apt/sources.list; \\
 in-target sh -c "echo -e '$authkeys'> /root/.ssh/authorized_keys";
 END
 return $preseed_file;
@@ -152,6 +153,7 @@ sub prep () {
 more_prepareguest_hvm($ho,$gho, $ram_mb, $disk_mb,
   OnReboot => 'preserve',
   Bios => $r{bios},
+  ExtraConfig => 'nestedhvm=1',
   PostImageHook => sub {
 my $cmds = iso_copy_content_from_image($gho, $newiso);
 $cmds .= prepare_initrd($initrddir,$newiso,$preseed_file_path);
@@ -173,6 +175,8 @@ my $ram_minslop = 100;
 my $ram_lots = 5000;
 if ($host_freemem_mb > $ram_lots * 2 + $ram_minslop) {
 $ram_mb = $ram_lots;
+} elsif ($gn eq 'nested') {
+$ram_mb = 3072;
 } else {
 $ram_mb = 768;
 }
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH 0/6] Introduction of netsted HVM test job

2015-03-16 Thread longtao.pang

This patch set adds nested HVM test case for osstest.
In this test case, a Xen hypervisor (L1) runs on top of another Xen hypervisor 
(L0). 
Upon L1 hypervisor, we will then create a nested guest (L2), and test if the 
Linux guest can then be installed and run well.
About nested Xen virtualization, refer to 
http://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xen.

Test steps
0. To run osstest in standalone mode, write a config file in
   '~/.xen-osstest/config', and then create a standalone.config file to
   define 'TREE_LINUX', 'REVISION_LINUX' and 'NESTED_OS_IMAGE' which
   will be used for nested test. The directry path of 'Debian Images'
   could be difined in '~/.xen-osstest/config'. 
1. run 'build-amd64' job and then 'build-amd64-pvops', to prepare xen
   installation tarball and hvm guest kernel.
2. run 'test-amd64-amd6-nested' job, it does following:
a. invoke test step of 'ts-debain-hvm-install' to install 
   a normal HVM guest
b. invoke test step of 'ts-nested-setup' to make some
   appropriate runvars which selecthost would recognise and
   prepare the configurations for installing L2 guest VM. 
c. invoke test step of 'ts-xen-install' to install xen on 
   the normal guest, alter it into a L1 hypervisor
d. invoke test step of 'ts-debain-hvm-install' again, but 
   take the L1 hypervisor as host, install the L2 guest on it
e. invoke test step of 'ts-guest-stop', stop L2 guest.
f. invoke test step of 'ts-guest-destroy' to destroy L1 guest.

This patch set reuses 'ts-debian-hvm-install' for both L1 installation and L2 
installation, define 'nested' and 'nested2' as L1 and L2's hostname, define 
'nested_l1 as L1's host ident.
It also reuses 'ts-xen-install' with 'nested' input param to differentiate L1 
Xen installation from L0 Xen installation.
This patch series has been tested on test machines of amd64 arch, 
debian-7.6.0-amd64 as guests OS, with hvm domain0 of Linux kernel 3.18.5, in 
standalone mode.

longtao.pang (6):
  parsing grub which has 'submenu' primitive
  Add and expose some testsupport APIs
  Changes on test step of debain hvm guest install in hvm.
  Add new ts-nested-setup script to custmize nested test configuration 
before the testing.
  Add test job for nest test case
  Compose the main recipe of test-nested test job.

 Osstest/Debian.pm  |   52 

 Osstest/TestSupport.pm |   21 +++--
 make-flight|   20 
 sg-run-job |   11 +++
 ts-debian-hvm-install  |   14 +++---
 ts-nested-setup|   54 
++
 6 files changed, 147 insertions(+), 25 deletions(-)
 create mode 100755 ts-nested-setup


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH 3/6] Changes on test step of debain hvm guest install in hvm.

2015-03-16 Thread longtao.pang

From: "longtao.pang" 

1. Increase disk size to accomodate to nested test requirment.
2. Since 'Debain-xxx-.iso' image will be stored there, therefore needs more
disk capacity, increase root partition size in preseed generation.
3. In L1 installation context, assign more memory to it; since it
acts as a nested hypervisor anyway.
4. In hvm guest configuration file, add '#nestedhvm=1',
which will later be uncommented by guest_editconfig_cd() after
xen installed in L1, and about to boot into a nested xen environment.
---
 ts-debian-hvm-install |   14 +++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/ts-debian-hvm-install b/ts-debian-hvm-install
index 449b96c..f334e30 100755
--- a/ts-debian-hvm-install
+++ b/ts-debian-hvm-install
@@ -36,7 +36,7 @@ our $ho= selecthost($whhost);
 
 # guest memory size will be set based on host free memory, see below
 our $ram_mb;
-our $disk_mb= 1;
+our $disk_mb= 15000;
 
 our $guesthost= "$gn.guest.osstest";
 our $gho;
@@ -60,7 +60,7 @@ d-i partman-auto/expert_recipe string \\
 use_filesystem{ } filesystem{ vfat } \\
 mountpoint{ /boot/efi } \\
 . \\
-5000 50 5000 ext4 \\
+1 50 1 ext4 \\
 method{ format } format{ } \\
 use_filesystem{ } filesystem{ ext4 } \\
 mountpoint{ / } \\
@@ -152,6 +152,7 @@ sub prep () {
 more_prepareguest_hvm($ho,$gho, $ram_mb, $disk_mb,
   OnReboot => 'preserve',
   Bios => $r{bios},
+  ExtraConfig => '#nestedhvm=1',
   PostImageHook => sub {
 my $cmds = iso_copy_content_from_image($gho, $newiso);
 $cmds .= prepare_initrd($initrddir,$newiso,$preseed_file_path);
@@ -173,6 +174,8 @@ my $ram_minslop = 100;
 my $ram_lots = 5000;
 if ($host_freemem_mb > $ram_lots * 2 + $ram_minslop) {
 $ram_mb = $ram_lots;
+} elsif ($gn eq 'nested') {
+$ram_mb = 3072;
 } else {
 $ram_mb = 768;
 }
@@ -189,7 +192,12 @@ if ($stage<2) {
 guest_destroy($gho);
 }
 
-guest_editconfig_nocd($gho,$emptyiso);
+if ($gn eq 'nested') {
+guest_editconfig_cd($gho);
+} else {
+guest_editconfig_nocd($gho,$emptyiso);
+}
+
 guest_create($gho);
 guest_await_dhcp_tcp($gho,300);
 guest_check_up($gho);
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH 1/6] parsing grub which has 'submenu' primitive

2015-03-16 Thread longtao.pang

From: "longtao.pang" 

>From a hvm kernel build from Linux stable Kernel tree,
the auto generated grub2 menu will have 'submenu' primitive, upon the
'menuentry' items. Xen boot entries will be grouped into a submenu. This
patch adds capability to support such grub formats.
---
 Osstest/Debian.pm |   52 
 1 file changed, 32 insertions(+), 20 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index e3e1c90..9fdacd7 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -1,5 +1,6 @@
 # This is part of "osstest", an automated testing framework for Xen.
 # Copyright (C) 2009-2013 Citrix Inc.
+# Copyright (C) 2014-2015 Intel Inc.
 # 
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
@@ -362,26 +363,34 @@ sub setupboot_grub2 ($$$) {
 
 my $count= 0;
 my $entry;
+my $submenu;
 while (<$f>) {
 next if m/^\s*\#/ || !m/\S/;
 if (m/^\s*\}\s*$/) {
-die unless $entry;
+die unless $entry || $submenu;
+   if(!defined $entry && defined $submenu){
+ logm("Met end of a submenu starting from ".
+   "$submenu->{StartLine}. ".
+   "Our want kern is $want_kernver");
+ $submenu=undef;
+ next;
+   }
 my (@missing) =
-grep { !defined $entry->{$_} } 
-   (defined $xenhopt
-? qw(Title Hv KernDom0 KernVer)
-: qw(Title Hv KernOnly KernVer));
-   if (@missing) {
-   logm("(skipping entry at $entry->{StartLine};".
-" no @missing)");
-   } elsif (defined $want_kernver &&
-$entry->{KernVer} ne $want_kernver) {
-   logm("(skipping entry at $entry->{StartLine};".
-" kernel $entry->{KernVer}, not $want_kernver)");
-   } else {
-   # yes!
-   last;
-   }
+grep { !defined $entry->{$_} }
+(defined $xenhopt
+ ? qw(Title Hv KernDom0 KernVer)
+ : qw(Title Hv KernOnly KernVer));
+if (@missing) {
+logm("(skipping entry at $entry->{StartLine};".
+ " no @missing)");
+} elsif (defined $want_kernver &&
+ $entry->{KernVer} ne $want_kernver) {
+logm("(skipping entry at $entry->{StartLine};".
+ " kernel $entry->{KernVer}, not $want_kernver)");
+} else {
+# yes!
+last;
+}
 $entry= undef;
 next;
 }
@@ -393,21 +402,24 @@ sub setupboot_grub2 ($$$) {
 $entry= { Title => $1, StartLine => $., Number => $count };
 $count++;
 }
-if (m/^\s*multiboot\s*\/(xen\-[0-9][-+.0-9a-z]*\S+)/) {
+   if(m/^submenu\s+[\'\"](.*)[\'\"].*\{\s*$/){
+   $submenu={ StartLine =>$.};
+   }
+if (m/^\s*multiboot\s*(?:\/boot)*\/(xen\S+)/) {
 die unless $entry;
 $entry->{Hv}= $1;
 }
-if (m/^\s*multiboot\s*\/(vmlinu[xz]-(\S+))/) {
+   if (m/^\s*multiboot\s*(?:\/boot)*\/(vmlinu[xz]-(\S+))/) {
 die unless $entry;
 $entry->{KernOnly}= $1;
 $entry->{KernVer}= $2;
 }
-if (m/^\s*module\s*\/(vmlinu[xz]-(\S+))/) {
+   if (m/^\s*module\s*(?:\/boot)*\/(vmlinu[xz]-(\S+))/) {
 die unless $entry;
 $entry->{KernDom0}= $1;
 $entry->{KernVer}= $2;
 }
-if (m/^\s*module\s*\/(initrd\S+)/) {
+   if (m/^\s*module\s*(?:\/boot)*\/(initrd\S+)/) {
 $entry->{Initrd}= $1;
 }
 }
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH 4/6] Add new ts-nested-setup script to custmize nested test configuration before the testing.

2015-03-16 Thread longtao.pang

From: "longtao.pang" 

1. In this script, make some appropriate runvars which selecthost would 
recognise.
2. Prepare the configurations for installing L2 guest VM.
3. Create a lv disk in L0 and hot-attach it to L1, need to restart L1 to make
the block disk to be recognized by L1 system, then using this disk to create a 
VG that
used for installing L2.
---
 ts-nested-setup |   54 ++
 1 file changed, 54 insertions(+)
 create mode 100755 ts-nested-setup

diff --git a/ts-nested-setup b/ts-nested-setup
new file mode 100755
index 000..69dc407
--- /dev/null
+++ b/ts-nested-setup
@@ -0,0 +1,54 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2015 Intel Inc.
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+# 
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use DBI;
+use Osstest;
+use Osstest::Debian;
+use Osstest::TestSupport;
+
+tsreadconfig();
+our ($ident_l0,$nested_host) = @ARGV;
+our ($l0,$l1) = ts_get_host_guest(@ARGV);
+
+guest_check_ip($l1);
+target_cmd_root($l1, "mkdir -p /home/osstest/.ssh && cp 
/root/.ssh/authorized_keys /home/osstest/.ssh/");
+my $url = $c{WebspaceUrl}.$c{WebspaceCommon}.$l0->{Name}."_".'overlay.tar'; 
+target_cmd_root($l1, <{Ip});
+store_runvar('multi_reboot_time',5);
+
+my $l2_disk_mb = 2;
+my $l2_lv_name = 'nestedl2';
+my $vgname = $l0->{Name};
+$vgname .= ".$c{TestHostDomain}" if ($l0->{Suite} =~ m/lenny/);
+target_cmd_root($l0, <{Name} /dev/${vgname}/${l2_lv_name},raw,sdb,rw
+END
+target_install_packages_norec($l1, qw(lvm2 rsync genisoimage));
+target_reboot($l1);
+target_cmd_root($l1, "pvcreate /dev/xvdb && vgcreate ${l2_lv_name}_vg 
/dev/xvdb");
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH 6/6] Compose the main recipe of test-nested test job.

2015-03-16 Thread longtao.pang

From: "longtao.pang" 

---
 sg-run-job |   11 +++
 1 file changed, 11 insertions(+)

diff --git a/sg-run-job b/sg-run-job
index 94d091b..ababebe 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -297,6 +297,17 @@ proc run-job/test-pair {} {
 #run-ts . remus-failover ts-remus-check src_host dst_host + debian
 }
 
+proc need-hosts/test-nested {} {return host}
+proc run-job/test-nested {} {
+run-ts . = ts-debian-hvm-install + host + nested
+run-ts . = ts-nested-setup + host + nested
+run-ts . = ts-xen-install nested_l1
+run-ts . = ts-host-reboot nested_l1
+run-ts . = ts-debian-hvm-install nested_l1 nested2
+run-ts . = ts-guest-stop nested_l1 nested2
+run-ts . = ts-guest-destroy host nested
+}
+
 proc test-guest-migr {g} {
 if {[catch { run-ts . = ts-migrate-support-check + host $g }]} return
 
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH 5/6] Add test job for nest test case

2015-03-16 Thread longtao.pang

From: "longtao.pang" 

This patch adds creation of the nested test job, when job creation procedure
is invoked.
'NESTED_OS_IMAGE' is the name of 'Debian ISO Images', which defined in
 standalone.config.
---
 make-flight |   20 
 1 file changed, 20 insertions(+)

diff --git a/make-flight b/make-flight
index 63b14f5..fb35ac4 100755
--- a/make-flight
+++ b/make-flight
@@ -204,6 +204,25 @@ do_hvm_win7_x64_tests () {
 all_hostflags=$most_hostflags,hvm
 }
 
+do_hvm_debian_nested_tests () {
+  if [ $xenarch != amd64 ]; then
+return
+  fi
+  if [ $dom0arch != amd64 ]; then
+return
+  fi
+
+  job_create_test test-$xenarch$kern-$dom0arch-nested test-nested xl \
+   $xenarch $dom0arch \
+nested_image=$NESTED_OS_IMAGE \
+nested2_image=$NESTED_OS_IMAGE \
+bios=seabios \
+kernbuildjob=build-amd64-pvops \
+kernkind=pvops \
+device_model_version=qemu-xen \
+all_hostflags=$most_hostflags,hvm
+}
+
 do_hvm_debian_test_one () {
   testname=$1
   bios=$2
@@ -401,6 +420,7 @@ test_matrix_do_one () {
 done
 
   fi
+  do_hvm_debian_nested_tests
   do_passthrough_tests
 }
 
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST Nested PATCH 2/6] Add and expose some testsupport APIs

2015-03-16 Thread longtao.pang

From: "longtao.pang" 

1. Designate vif model to 'e1000', otherwise, with default device model,
the L1 eth0 interface disappear, hence xenbridge cannot work.
Maybe this limitation can be removed later after some fix it. For now, we
have to accomodate to it.
2. Since reboot L1 guest VM will take more time to boot up, we increase
multi-times for reboot-confirm-booted if test nested job, and the multi value 
is stored as a runvar in 'ts-nested-setup' script. Added another function 
'guest_editconfig_cd' and expose it, this function bascically changes guest
boot device sequence, alter its on_reboot behavior to restart and enabled 
nestedhvm feature.
3. In L2 installation context, its host (L1) IP address is not queried from DNS,
but after running "ts-nested-setup + host + nested", L1 IP is stored in runvar.
---
 Osstest/TestSupport.pm |   21 +++--
 1 file changed, 19 insertions(+), 2 deletions(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index e1ebcb7..57c7167 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -111,6 +111,7 @@ BEGIN {
   iso_gen_flags_basic
   iso_copy_content_from_image
   guest_editconfig_nocd
+  guest_editconfig_cd
   );
 %EXPORT_TAGS = ( );
 
@@ -819,6 +820,10 @@ sub selecthost ($) {
logm("Host $name is in HostGroup $ho->{Properties}{HostGroup}");
 }
 
+if ( $r{"${name}_ip"} ) {
+$setprop->("IpAddr", $r{"${name}_ip"});
+}
+
 # Next, we use the config file's general properites as defaults
 foreach my $k (keys %c) {
next unless $k =~ m/^HostProp_([A-Z].*)$/;
@@ -990,8 +995,9 @@ sub common_toolstack ($) {
 
 sub host_reboot ($) {
 my ($ho) = @_;
+my $wait_time = $r{multi_reboot_time} ? $r{multi_reboot_time}*40 : 40;
 target_reboot($ho);
-poll_loop(40,2, 'reboot-confirm-booted', sub {
+poll_loop($wait_time,2, 'reboot-confirm-booted', sub {
 my $output;
 if (!eval {
 $output= target_cmd_output($ho, <{Ether}' ]
+vif = [ 'type=ioemu,model=e1000,mac=$gho->{Ether}' ]
 #
 on_poweroff = '$onpoweroff'
 on_reboot   = '$onreboot'
@@ -2113,4 +2119,15 @@ sub guest_editconfig_nocd ($$) {
 });
 }
 
+sub guest_editconfig_cd ($) {
+my ($gho) = @_;
+guest_editconfig($gho->{Host}, $gho, sub {
+if (m/^\s*boot\s*= '\s*d\s*c\s*'/) {
+s/dc/cd/;
+}
+s/^on_reboot.*/on_reboot='restart'/;
+s/#nestedhvm/nestedhvm/;
+});
+}
+
 1;
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST PATCH 4/4] Insert nested test job name and runvars into

2014-12-10 Thread longtao.pang

From: "longtao.pang" 

This patch is used for inserting nested test job name and runvars into
standalone.db database after execute command './standalone-reset'.

---
 make-flight |   19 +++
 mfi-common  |8 
 2 files changed, 27 insertions(+)

diff --git a/make-flight b/make-flight
index 9963a46..fe64237 100755
--- a/make-flight
+++ b/make-flight
@@ -197,6 +197,24 @@ do_hvm_win7_x64_tests () {
 all_hostflags=$most_hostflags,hvm
 }
 
+do_hvm_debian_nested_tests () {
+  if [ $xenarch != amd64 ]; then
+return
+  fi
+  if [ $dom0arch != amd64 ]; then
+return
+  fi
+
+  job_create_test test-$xenarch$kern-$dom0arch-nested test-nested xl \
+   $xenarch $dom0arch \
+nested_image=debian-7.6.0-amd64-DVD-1.iso \
+   bios=seabios \
+   kernbuildjob=build-amd64-hvm \
+   kernkind=hvm \
+   device_model_version=qemu-xen \
+all_hostflags=$most_hostflags,hvm
+}
+
 do_hvm_debian_test_one () {
   testname=$1
   bios=$2
@@ -364,6 +382,7 @@ test_matrix_do_one () {
 
   fi
 
+  do_hvm_debian_nested_tests
   do_passthrough_tests
 }
 
diff --git a/mfi-common b/mfi-common
index 5c4f5d5..b65f0b5 100644
--- a/mfi-common
+++ b/mfi-common
@@ -166,6 +166,14 @@ create_build_jobs () {
 revision_qemu=$REVISION_QEMU \
 revision_qemuu=$REVISION_QEMU_UPSTREAM
 fi
+./cs-job-create $flight build-$arch-hvm build-kern   \
+arch=$arch kconfighow=xen-enable-xen-config  \
+$RUNVARS $BUILD_RUNVARS $BUILD_LINUX_RUNVARS $arch_runvars   \
+$suite_runvars   \
+host_hostflags=$build_hostflags  \
+revision_linux=v3.16 \
+
tree_linux=git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
 \
+${TREEVCS_LINUX:+treevcs_linux=}${TREEVCS_LINUX}
 
 ./cs-job-create $flight build-$arch-pvops build-kern \
 arch=$arch kconfighow=xen-enable-xen-config  \
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST PATCH 0/4] Introduction of the patches.

2014-12-10 Thread longtao.pang

We updated these patchs(version_3) again about adding Nested test job into 
OSSTest.
Nested virtualization is the function of running a hypervisor inside a virtual 
machine.
The hypervisor that runs on the real hardware is called a level 0 or L0;
The hypervisor that runs as a guest inside L0 is called level 1 or L1;
A guest that running inside the L1 hypervisor is called a level 2 or L2.

For running nested test job, we should run build-amd64 job and build-amd64-hvm 
job first.
During the testing, it will install L1 guest VM inside L0,
the L1 guest should build xen and HVM Dom0 kernel and then boot into xen kernel.
Next, begin to install L2 guest VM inside L1.
After that, make sure ping L2 is successfully


longtao.pang (4):
  Add nested testcase of preparing and installing L1 guest VM 
  Build XEN and HVM Dom0 kernel for L1 guest VM
  Add nested testcase of installing L2 guest VM
  Insert nested test job name and runvars

 Osstest/Debian.pm  |   27 +---
 Osstest/TestSupport.pm |   31 +++--
 make-flight|   19 ++
 mfi-common |8 +++
 sg-run-job |8 +++
 ts-debian-hvm-install  |   34 +++---
 ts-debian-install  |  166 +---
 ts-xen-install |  149 +++
 8 files changed, 343 insertions(+), 99 deletions(-)


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST PATCH 2/4] Build XEN and HVM Dom0 kernel for L1 guest VM

2014-12-10 Thread longtao.pang

From: "longtao.pang" 

This patch is used for building XEN and HVM Dom0 kernel for L1 guest VM,
and then reboot L1 guest into xen kernel.

---
 sg-run-job |1 +
 ts-xen-install |  149 +---
 2 files changed, 111 insertions(+), 39 deletions(-)

diff --git a/sg-run-job b/sg-run-job
index 8dcf7af..e513bd1 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -291,6 +291,7 @@ proc run-job/test-pair {} {
 proc need-hosts/test-nested {} {return host}
 proc run-job/test-nested {} {
 run-ts . = ts-debian-hvm-install + host + nested + nested_L1
+run-ts . = ts-xen-install + host + nested + nested_build
 }
 
 proc test-guest-migr {g} {
diff --git a/ts-xen-install b/ts-xen-install
index 4d34d1f..c175d6d 100755
--- a/ts-xen-install
+++ b/ts-xen-install
@@ -28,19 +28,25 @@ use Osstest::CXFabric;
 my $checkmode= 0;
 
 tsreadconfig();
-
+our $w_ho;
 our @hos;
-
-if (@ARGV and $ARGV[0] eq '--check') {
-$checkmode= 1;
-shift @ARGV;
-logm("checking builds are done...");
+our ($whhost,$gn,$nested_build) = @ARGV;
+$nested_build ||= '';
+if ($nested_build eq 'nested_build') {
+$whhost ||= 'host';
+$gn ||= 'nested';
 } else {
-if (!@ARGV) {
-   push @ARGV, 'host';
-}
-foreach my $k (@ARGV) {
-push @hos, selecthost($k);
+if (@ARGV and $ARGV[0] eq '--check') {
+$checkmode= 1;
+shift @ARGV;
+logm("checking builds are done...");
+} else {
+if (!@ARGV) {
+push @ARGV, 'host';
+}
+foreach my $k (@ARGV) {
+push @hos, selecthost($k);
+}
 }
 }
 
@@ -49,18 +55,18 @@ our $ho;
 my %distpath;
 
 sub packages () {
-target_install_packages($ho,
+target_install_packages($w_ho,
 qw(bridge-utils vncsnapshot libaio1 libpixman-1-0
libsdl1.2debian libglib2.0-0 liblzma5));
-target_install_packages($ho,
+target_install_packages($w_ho,
$ho->{Suite} =~ /squeeze/ ? "libyajl1" : 
"libyajl2");
 if ($ho->{Suite} !~ m/lenny|squeeze/) {
-target_install_packages($ho, 'libfdt1');
+target_install_packages($w_ho, 'libfdt1');
 }
 if ($r{arch} eq 'i386') {
-   target_install_packages($ho, 'libc6-xen');
+   target_install_packages($w_ho, 'libc6-xen');
 }
-target_install_packages($ho, @{toolstack()->{ExtraPackages}})
+target_install_packages($w_ho, @{toolstack()->{ExtraPackages}})
 if toolstack()->{ExtraPackages};
 }
 
@@ -69,14 +75,14 @@ sub extract () {
 push @parts, 'libvirt' if $r{toolstack} eq "libvirt";
 
 foreach my $part (@parts) {
-target_extract_jobdistpath($ho, $part, "path_${part}dist",
+target_extract_jobdistpath($w_ho, $part, "path_${part}dist",
   $r{"${part}buildjob"}, \%distpath);
 }
-target_cmd_root($ho, '/sbin/ldconfig');
+target_cmd_root($w_ho, '/sbin/ldconfig');
 }
 
 sub adjustconfig () {
-target_editfile_root($ho, "/etc/xen/xend-config.sxp",
+target_editfile_root($w_ho, "/etc/xen/xend-config.sxp",
 "xend-config.sxp", sub {
my (@domains) = (qw(localhost localhost.localdomain),
 ".".$c{DnsDomain}, ".".$c{TestHostDomain});
@@ -108,13 +114,13 @@ sub adjustconfig () {
 /etc/sysconfig/xencommons
 /etc/default/xend
 /etc/sysconfig/xend)) {
-next unless target_file_exists($ho, $try);
+next unless target_file_exists($w_ho, $try);
 $trace_config_file= $try;
 last;
 }
 die unless defined $trace_config_file;
 
-target_editfile_root($ho, $trace_config_file, sub {
+target_editfile_root($w_ho, $trace_config_file, sub {
 my $prnow;
 $prnow= sub {
 print EO "XENCONSOLED_TRACE=guest\n" or die $!;
@@ -128,7 +134,7 @@ sub adjustconfig () {
 $prnow->();
 });
 
-target_cmd_root($ho, 'mkdir -p /var/log/xen/console');
+target_cmd_root($w_ho, 'mkdir -p /var/log/xen/console');
 
 setup_cxfabric($ho);
 }
@@ -156,19 +162,19 @@ sub setupboot () {
 $xenhopt .= " $append" if defined $append;
 
 my @hooks;
-
+
 if (host_involves_pcipassthrough($ho)) {
 push @hooks, {
 EditBootOptions => sub {
 my ($ho,$hopt,$kopt) = @_;
 $$hopt .= ' iommu=on';
 my $hide= ' xen-pciback.hide='. join '',map { "($_->{Bdf})" }
-host_get_pcipassthrough_

[Xen-devel] [OSSTEST PATCH 1/4] Add nested testcase of preparing and installing L1 guest VM

2014-12-10 Thread longtao.pang

From: "longtao.pang" 

This patch is used for preparing and installing L1 guest VM inside L0 system
on testhost machine.

---
 Osstest/Debian.pm  |   27 ++-
 Osstest/TestSupport.pm |   31 ++-
 sg-run-job |5 +
 ts-debian-hvm-install  |   34 --
 4 files changed, 73 insertions(+), 24 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index c8db601..a671d20 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -1,5 +1,6 @@
 # This is part of "osstest", an automated testing framework for Xen.
 # Copyright (C) 2009-2013 Citrix Inc.
+# Copyright (C) 2014 Intel Inc.
 # 
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
@@ -34,6 +35,7 @@ BEGIN {
 @EXPORT  = qw(debian_boot_setup
   %preseed_cmds
   preseed_base
+  setupboot_grub2
   preseed_create
   preseed_hook_command preseed_hook_installscript
   di_installcmdline_core
@@ -286,15 +288,18 @@ sub setupboot_grub2 ($$$) {
 
 my $count= 0;
 my $entry;
+   my $submenu;
 while (<$f>) {
 next if m/^\s*\#/ || !m/\S/;
 if (m/^\s*\}\s*$/) {
-die unless $entry;
+die unless $entry || $submenu;
+   if(!defined $entry && defined $submenu){
+ logm("Met end of a submenu starting from 
$submenu->{StartLine}.Our want kern is $want_kernver");
+ $submenu=undef;
+ next;
+   }
 my (@missing) =
-grep { !defined $entry->{$_} } 
-   (defined $xenhopt
-? qw(Title Hv KernDom0 KernVer)
-: qw(Title Hv KernOnly KernVer));
+   grep { !defined $entry->{$_} }  (defined $xenhopt ? qw(Title Hv 
KernDom0 KernVer) : qw(Title Hv KernOnly KernVer));
if (@missing) {
logm("(skipping entry at $entry->{StartLine};".
 " no @missing)");
@@ -317,21 +322,25 @@ sub setupboot_grub2 ($$$) {
 $entry= { Title => $1, StartLine => $., Number => $count };
 $count++;
 }
-if (m/^\s*multiboot\s*\/(xen\-[0-9][-+.0-9a-z]*\S+)/) {
+   if(m/^submenu\s+[\'\"](.*)[\'\"].*\{\s*$/){
+   $submenu={ StartLine =>$.};
+   }
+if (m/^\s*multiboot\s*(?:\/boot)*\/(xen\S+)/) {
+#  if (m/^\s*multiboot\s*(?:\/boot)*\/(xen\-[0-9][-+.0-9a-z]*\S+)/) {
 die unless $entry;
 $entry->{Hv}= $1;
 }
-if (m/^\s*multiboot\s*\/(vmlinu[xz]-(\S+))/) {
+   if (m/^\s*multiboot\s*(?:\/boot)*\/(vmlinu[xz]-(\S+))/) {
 die unless $entry;
 $entry->{KernOnly}= $1;
 $entry->{KernVer}= $2;
 }
-if (m/^\s*module\s*\/(vmlinu[xz]-(\S+))/) {
+   if (m/^\s*module\s*(?:\/boot)*\/(vmlinu[xz]-(\S+))/) {
 die unless $entry;
 $entry->{KernDom0}= $1;
 $entry->{KernVer}= $2;
 }
-if (m/^\s*module\s*\/(initrd\S+)/) {
+   if (m/^\s*module\s*(?:\/boot)*\/(initrd\S+)/) {
 $entry->{Initrd}= $1;
 }
 }
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index a3b6936..1e47039 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -55,8 +55,9 @@ BEGIN {
   target_putfilecontents_stash
  target_putfilecontents_root_stash
   target_put_guest_image target_editfile
-  target_editfile_root target_file_exists
-  target_run_apt
+ target_editfile_root target_file_exists 
+ target_file_exists_root
+ target_run_apt
   target_install_packages target_install_packages_norec
   target_jobdir target_extract_jobdistpath_subdir
   target_extract_jobdistpath target_guest_lv_name
@@ -67,7 +68,7 @@ BEGIN {
   selecthost get_hostflags get_host_property
   get_host_native_linux_console
   power_state power_cycle power_cycle_time
-  serial_fetch_logs
+  serial_fetch_logs select_ether
   propname_massage
  
   get_stashed open_unique_stashfile compress_stashed
@@ -109,6 +110,7 @@ BEGIN {
   iso_gen_flags_basic

[Xen-devel] [OSSTEST PATCH 3/4] Add nested testcase of installing L2 guest VM

2014-12-10 Thread longtao.pang

From: "longtao.pang" 

This patch is used for installing L2 guest VM inside L1 guest VM.

---
 sg-run-job|2 +
 ts-debian-install |  166 +
 2 files changed, 132 insertions(+), 36 deletions(-)

diff --git a/sg-run-job b/sg-run-job
index e513bd1..85f7b22 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -292,6 +292,8 @@ proc need-hosts/test-nested {} {return host}
 proc run-job/test-nested {} {
 run-ts . = ts-debian-hvm-install + host + nested + nested_L1
 run-ts . = ts-xen-install + host + nested + nested_build
+run-ts . = ts-debian-install + host + nested + amd64 + nested_L2
+run-ts . = ts-guest-destroy + host nested
 }
 
 proc test-guest-migr {g} {
diff --git a/ts-debian-install b/ts-debian-install
index 58ea743..2ca54e8 100755
--- a/ts-debian-install
+++ b/ts-debian-install
@@ -22,41 +22,61 @@ use Osstest::TestSupport;
 
 tsreadconfig();
 
-our ($whhost,$gn) = @ARGV;
-$whhost ||= 'host';
-$gn ||= 'debian';
+our ($whhost,$gn,$arch,$nested_L2) = @ARGV;
+$nested_L2 ||= '';
 
-our $ho= selecthost($whhost);
+if($nested_L2 eq 'nested_L2') {
+my $L1_IP = &get_VM_IP($r{'nested_ether'});
+$whhost = "host=$L1_IP";
+$gn ||= 'nested';
+$arch ||= 'amd64';
+} else {
+$whhost ||= 'host';
+$gn ||= 'debian';  
+}
 
+our $ho= selecthost($whhost);
 our $ram_mb=512;
 our $swap_mb=  1000;
 our $disk_mb= 1;
-
 our $guesthost= "$gn.guest.osstest";
 our $gho;
+our $L2_MAC = select_ether($ho,"L2_ether");
 
 sub prep () {
 target_install_packages_norec($ho, qw(lvm2 xen-tools));
-
-$gho= prepareguest($ho, $gn, $guesthost, 22,
-   $swap_mb + $disk_mb + 2,
-   40);
-target_cmd_root($ho, "umount $gho->{Lvdev} ||:");
+unless($nested_L2 eq 'nested_L2') {
+   $gho= prepareguest($ho, $gn, $guesthost, 22,
+  $swap_mb + $disk_mb + 2,
+  40);
+   target_cmd_root($ho, "umount $gho->{Lvdev} ||:");
+}
 }
 
 sub ginstall () {
-my $arch= $r{"$gho->{Guest}_arch"};
+my $gsuite;
+my $kernpath;
+my $initrd;
+my $kernver;
+if($nested_L2 eq 'nested_L2'){
+my $suite= "$c{DebianSuite}";
+$gsuite= defined($suite) ? "--dist=$suite" : '';
+$kernver ||= target_cmd_output_root($ho, 'uname -r');
+$kernpath = "/boot/vmlinuz-$kernver";
+$initrd ||= "/boot/initrd.img-$kernver";
+} else {
+my $arch= $r{"$gho->{Guest}_arch"};
+$gsuite= guest_var($gho,'suite',$c{GuestDebianSuite});
+$kernpath = guest_var($gho,'kernel_path',$r{xen_kernel_path});
+$initrd = guest_var($gho,'initrd_path',$r{xen_initrd_path});
+if (!$kernpath) {
+   $kernver= guest_var($gho,'kernel_ver',$r{xen_kernel_ver});
+$kernver ||= target_cmd_output($ho, 'uname -r');
+   $kernpath = "/boot/vmlinuz-$kernver";
+   $initrd ||= "/boot/initrd.img-$kernver";
+} 
+} 
 my $archarg= defined($arch) ? "--arch $arch" : '';
-my $gsuite= guest_var($gho,'suite',$c{GuestDebianSuite});
-
-my $kernpath = guest_var($gho,'kernel_path',$r{xen_kernel_path});
-my $initrd = guest_var($gho,'initrd_path',$r{xen_initrd_path});
-if (!$kernpath) {
-   my $kernver= guest_var($gho,'kernel_ver',$r{xen_kernel_ver});
-   $kernver ||= target_cmd_output($ho, 'uname -r');
-   $kernpath = "/boot/vmlinuz-$kernver";
-   $initrd ||= "/boot/initrd.img-$kernver";
-}
 if (!$initrd) {
$initrd = $kernpath;
$initrd =~ s,/vmlinuz-,/initrd.img-, or die "$initrd ?";
@@ -76,23 +96,97 @@ sub ginstall () {
 fi
 END
 }
-target_cmd_root($ho, <{Ether} \\
---memory ${ram_mb}Mb --swap ${swap_mb}Mb \\
---dist $gsuite \\
---mirror http://$c{DebianMirrorHost}/$c{DebianMirrorSubpath} \\
---hostname $gho->{Name} \\
---lvm $gho->{Vg} --force \\
---kernel $kernpath \\
---genpass 0 --password xenroot \\
-$initrd_opt \\
-$archarg
+if($nested_L2 eq 'nested_L2') {
+target_cmd_root($ho, <http://$c{DebianMirrorHost}/$c{DebianMirrorSubpath} \\
+--hostname $guesthost \\
+--dir=/testnested \\
+--kernel=$kernpath \\
+--genpass 0 --password xenroot \\
+$initrd_opt \\
+$gsuite \\
+$archarg
+END
+} else {
+target_cmd_root($ho, <{E

[Xen-devel] [OSSTEST PATCH 0/4] Introduction of the patches.

2014-11-27 Thread longtao.pang

We updated these patchs about adding Nested test job into OSSTest.
Nested virtualization is the function of running a hypervisor inside a virtual 
machine.
The hypervisor that runs on the real hardware is called a level 0 or L0; 
The hypervisor that runs as a guest inside L0 is called level 1 or L1; 
A guest that running inside the L1 hypervisor is called a level 2 or L2.

For running nested test job, we should run build-amd64 job and build-amd64-hvm 
job first. 
During the testing, it will install L1 guest VM inside L0, 
the L1 guest should build xen and HVM Dom0 kernel and then boot into xen 
kernel. 
Next, begin to install L2 guest VM inside L1. 
After that, make sure ping L2 is successfully


root (4):
  Add nested testcase of preparing and installing L1 guest
  Building XEN and HVM Dom0 kernel for L1 guest VM
  Add nested test case of installing L2 guest VM
  Insert nested test job name and runvars into standalone database

 Osstest/Debian.pm |   25 ++-
 Osstest/TestSupport.pm|   31 +++-
 make-flight   |   19 ++
 mfi-common|8 +
 sg-run-job|8 +
 ts-debian-install |  166 +
 ts-nested-L1-debian-install-part1 |  202 
 ts-nested-L1-debian-install-part2 |  364 +
 8 files changed, 773 insertions(+), 50 deletions(-)
 create mode 100755 ts-nested-L1-debian-install-part1
 create mode 100755 ts-nested-L1-debian-install-part2


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST PATCH 4/4] Insert nested test job name and runvars

2014-11-27 Thread longtao.pang

From: "longtao.pang" 

This patch is used for inserting nested test job name and runvars into 
standalone.db database after execute command './standalone-reset'.

---
 make-flight |   19 +++
 mfi-common  |8 
 2 files changed, 27 insertions(+)

diff --git a/make-flight b/make-flight
index 9963a46..fe64237 100755
--- a/make-flight
+++ b/make-flight
@@ -197,6 +197,24 @@ do_hvm_win7_x64_tests () {
 all_hostflags=$most_hostflags,hvm
 }
 
+do_hvm_debian_nested_tests () {
+  if [ $xenarch != amd64 ]; then
+return
+  fi
+  if [ $dom0arch != amd64 ]; then
+return
+  fi
+
+  job_create_test test-$xenarch$kern-$dom0arch-nested test-nested xl \
+   $xenarch $dom0arch \
+nested_image=debian-7.6.0-amd64-DVD-1.iso \
+   bios=seabios \
+   kernbuildjob=build-amd64-hvm \
+   kernkind=hvm \
+   device_model_version=qemu-xen \
+all_hostflags=$most_hostflags,hvm
+}
+
 do_hvm_debian_test_one () {
   testname=$1
   bios=$2
@@ -364,6 +382,7 @@ test_matrix_do_one () {
 
   fi
 
+  do_hvm_debian_nested_tests
   do_passthrough_tests
 }
 
diff --git a/mfi-common b/mfi-common
index 5c4f5d5..0cc0101 100644
--- a/mfi-common
+++ b/mfi-common
@@ -166,6 +166,14 @@ create_build_jobs () {
 revision_qemu=$REVISION_QEMU \
 revision_qemuu=$REVISION_QEMU_UPSTREAM
 fi
+./cs-job-create $flight build-$arch-hvm build-kern   \
+arch=$arch kconfighow=xen-enable-xen-config  \
+$RUNVARS $BUILD_RUNVARS $BUILD_LINUX_RUNVARS $arch_runvars   \
+$suite_runvars   \
+host_hostflags=$build_hostflags  \
+revision_linux=HEAD  \
+
tree_linux=git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
 \
+${TREEVCS_LINUX:+treevcs_linux=}${TREEVCS_LINUX}
 
 ./cs-job-create $flight build-$arch-pvops build-kern \
 arch=$arch kconfighow=xen-enable-xen-config  \
-- 
1.7.10.4


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [OSSTEST PATCH 1/4] Add nested testcase of preparing and installing L1 guest

2014-11-27 Thread longtao.pang

From: "longtao.pang" 

This patch is used for preparing and installing L1 guest VM inside L0 system 
on testhost machine.

---
 Osstest/Debian.pm |   25 +++--
 Osstest/TestSupport.pm|   31 +-
 sg-run-job|5 +
 ts-nested-L1-debian-install-part1 |  202 +
 4 files changed, 249 insertions(+), 14 deletions(-)
 create mode 100755 ts-nested-L1-debian-install-part1

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 8f80eb4..68da2cb 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -1,5 +1,6 @@
 # This is part of "osstest", an automated testing framework for Xen.
 # Copyright (C) 2009-2013 Citrix Inc.
+# Copyright (C) 2014 Intel Inc.
 # 
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as published by
@@ -286,15 +287,18 @@ sub setupboot_grub2 ($$$) {
 
 my $count= 0;
 my $entry;
+   my $submenu;
 while (<$f>) {
 next if m/^\s*\#/ || !m/\S/;
 if (m/^\s*\}\s*$/) {
-die unless $entry;
+die unless $entry || $submenu;
+   if(!defined $entry && defined $submenu){
+ logm("Met end of a submenu starting from 
$submenu->{StartLine}.Our want kern is $want_kernver");
+ $submenu=undef;
+ next;
+   }
 my (@missing) =
-grep { !defined $entry->{$_} } 
-   (defined $xenhopt
-? qw(Title Hv KernDom0 KernVer)
-: qw(Title Hv KernOnly KernVer));
+   grep { !defined $entry->{$_} }  (defined $xenhopt ? qw(Title Hv 
KernDom0 KernVer) : qw(Title Hv KernOnly KernVer));
if (@missing) {
logm("(skipping entry at $entry->{StartLine};".
 " no @missing)");
@@ -317,21 +321,24 @@ sub setupboot_grub2 ($$$) {
 $entry= { Title => $1, StartLine => $., Number => $count };
 $count++;
 }
-if (m/^\s*multiboot\s*\/(xen\-[0-9][-+.0-9a-z]*\S+)/) {
+   if(m/^submenu\s+[\'\"](.*)[\'\"].*\{\s*$/){
+   $submenu={ StartLine =>$.};
+   }
+   if (m/^\s*multiboot\s*(?:\/boot)*\/(xen\-[0-9][-+.0-9a-z]*\S+)/) {
 die unless $entry;
 $entry->{Hv}= $1;
 }
-if (m/^\s*multiboot\s*\/(vmlinu[xz]-(\S+))/) {
+   if (m/^\s*multiboot\s*(?:\/boot)*\/(vmlinu[xz]-(\S+))/) {
 die unless $entry;
 $entry->{KernOnly}= $1;
 $entry->{KernVer}= $2;
 }
-if (m/^\s*module\s*\/(vmlinu[xz]-(\S+))/) {
+   if (m/^\s*module\s*(?:\/boot)*\/(vmlinu[xz]-(\S+))/) {
 die unless $entry;
 $entry->{KernDom0}= $1;
 $entry->{KernVer}= $2;
 }
-if (m/^\s*module\s*\/(initrd\S+)/) {
+   if (m/^\s*module\s*(?:\/boot)*\/(initrd\S+)/) {
 $entry->{Initrd}= $1;
 }
 }
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 45ceee9..21955b8 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -55,8 +55,9 @@ BEGIN {
   target_putfilecontents_stash
  target_putfilecontents_root_stash
   target_put_guest_image target_editfile
-  target_editfile_root target_file_exists
-  target_run_apt
+ target_editfile_root target_file_exists 
+ target_file_exists_root
+ target_run_apt
   target_install_packages target_install_packages_norec
   target_jobdir target_extract_jobdistpath_subdir
   target_extract_jobdistpath target_guest_lv_name
@@ -67,7 +68,7 @@ BEGIN {
   selecthost get_hostflags get_host_property
   get_host_native_linux_console
   power_state power_cycle power_cycle_time
-  serial_fetch_logs
+  serial_fetch_logs select_ether
   propname_massage
  
   get_stashed open_unique_stashfile compress_stashed
@@ -109,6 +110,7 @@ BEGIN {
   iso_gen_flags_basic
   iso_copy_content_from_image
   guest_editconfig_nocd
+ guest_editconfig_cd
   );
 %EXPORT_TAGS = ( );
 
@@ -481,6 +483,14 @@ sub target_file_exists ($$) {
 die "$rfile $out ?";
 }
 
+sub target_file_exists_root ($$) {
+my ($ho,$rfile) = @_;
+my $out= target_cmd_output

[Xen-devel] [OSSTEST PATCH 2/4] Building XEN and HVM Dom0 kernel for L1 guest VM

2014-11-27 Thread longtao.pang

From: "longtao.pang" 

This patch is used for building XEN and HVM Dom0 kernel for L1 guest VM, 
and then reboot L1 guest into xen kernel.

---
 sg-run-job|1 +
 ts-nested-L1-debian-install-part2 |  364 +
 2 files changed, 365 insertions(+)
 create mode 100755 ts-nested-L1-debian-install-part2

diff --git a/sg-run-job b/sg-run-job
index cd8b468..a4c0de1 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -291,6 +291,7 @@ proc run-job/test-pair {} {
 proc need-hosts/test-nested {} {return host}
 proc run-job/test-nested {} {
 run-ts . = ts-nested-L1-debian-install-part1
+run-ts . = ts-nested-L1-debian-install-part2
 }
 
 proc test-guest-migr {g} {
diff --git a/ts-nested-L1-debian-install-part2 
b/ts-nested-L1-debian-install-part2
new file mode 100755
index 000..2f52edc
--- /dev/null
+++ b/ts-nested-L1-debian-install-part2
@@ -0,0 +1,364 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013 Citrix Inc.
+# Copyright (C) 2014 Intel Inc.
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+# 
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use DBI;
+use Osstest;
+use Osstest::Debian;
+use Osstest::TestSupport;
+
+tsreadconfig();
+
+our ($whhost,$gn)= @ARGV;
+$whhost ||= 'host';
+$gn ||= 'nested';
+
+our $ho= selecthost($whhost);
+our $gho;
+our $nested_L2;
+
+my %distpath;
+sub guest_packages () {
+target_install_packages($gho,
+qw(bridge-utils vncsnapshot libaio1 libpixman-1-0
+   libsdl1.2debian libglib2.0-0 liblzma5 vim));
+target_install_packages($gho,
+   $ho->{Suite} =~ /squeeze/ ? "libyajl1" : 
"libyajl2");   #we always let L1 same as L0
+if ($ho->{Suite} !~ m/lenny|squeeze/) {
+target_install_packages($gho, 'libfdt1');
+}
+if ($r{arch} eq 'i386') {
+   target_install_packages($gho, 'libc6-xen');
+}
+target_install_packages($gho, @{toolstack()->{ExtraPackages}})
+if toolstack()->{ExtraPackages};
+}
+
+sub guest_extract () {
+my @parts = ('', 'kern', 'xen');
+push @parts, 'libvirt' if $r{toolstack} eq "libvirt";
+
+foreach my $part (@parts) {
+target_extract_jobdistpath($gho, $part, "path_${part}dist",
+  $r{"${part}buildjob"}, \%distpath);
+}
+target_cmd_root($gho, '/sbin/ldconfig');
+}
+
+sub guest_adjustconfig () {
+target_editfile_root($gho, "/etc/xen/xend-config.sxp",
+"xend-config.sxp", sub {
+   my (@domains) = (qw(localhost localhost.localdomain),
+".".$c{DnsDomain}, ".".$c{TestHostDomain});
+   logm("relocation domains: @domains");
+   foreach (@domains) {
+   s/\./\\$&/g;
+   s/^/^/g;
+   s/$/\$/g;
+   s/^\^(\\\.)/.\*$1/;
+   }
+   $_= join ' ', @domains;
+   s/[\'\\]/\\$&/g;
+   my $extra= "(xend-relocation-hosts-allow '$_')";
+   logm("relocation setting: $extra");
+   $extra .= "\n";
+while () {
+   s/^\s*\(xend-relocation-hosts-allow/#$&/;
+   print EO or die $!;
+   if (m/^\#\(xend-relocation-hosts-allow/) {
+   print EO $extra or die $!;
+   $extra= '';
+   }
+   }
+   print EO $extra or die $!;
+}) if toolstack()->{Name} eq "xend";
+
+my $trace_config_file;
+foreach my $try (qw(/etc/default/xencommons
+/etc/sysconfig/xencommons
+/etc/default/xend
+/etc/sysconfig/xend)) {
+next unless target_file_exists_root($gho, $try);
+$trace_config_file= $try;
+last;
+}
+die unless defined $trace_config_file;
+
+target_editfile_root($gho, $trace_config_file, sub {
+my $prnow;
+$prnow= sub {
+print EO "XENCONSOLED_TRACE=guest\n" or die $!;
+$prnow= sub { };
+};
+while () {
+print EO or die $! unl

[Xen-devel] [OSSTEST PATCH 3/4] Add nested test case of installing L2 guest VM

2014-11-27 Thread longtao.pang

From: "longtao.pang" 

This patch is used for installing L2 guest VM inside L1 guest VM.

---
 sg-run-job|2 +
 ts-debian-install |  166 +
 2 files changed, 132 insertions(+), 36 deletions(-)

diff --git a/sg-run-job b/sg-run-job
index a4c0de1..7c0c6ca 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -292,6 +292,8 @@ proc need-hosts/test-nested {} {return host}
 proc run-job/test-nested {} {
 run-ts . = ts-nested-L1-debian-install-part1
 run-ts . = ts-nested-L1-debian-install-part2
+run-ts . = ts-debian-install + host + nested + amd64 + nested_L2
+run-ts . = ts-guest-destroy + host nested
 }
 
 proc test-guest-migr {g} {
diff --git a/ts-debian-install b/ts-debian-install
index 58ea743..2ca54e8 100755
--- a/ts-debian-install
+++ b/ts-debian-install
@@ -22,41 +22,61 @@ use Osstest::TestSupport;
 
 tsreadconfig();
 
-our ($whhost,$gn) = @ARGV;
-$whhost ||= 'host';
-$gn ||= 'debian';
+our ($whhost,$gn,$arch,$nested_L2) = @ARGV;
+$nested_L2 ||= '';
 
-our $ho= selecthost($whhost);
+if($nested_L2 eq 'nested_L2') {
+my $L1_IP = &get_VM_IP($r{'nested_ether'});
+$whhost = "host=$L1_IP";
+$gn ||= 'nested';
+$arch ||= 'amd64';
+} else {
+$whhost ||= 'host';
+$gn ||= 'debian';  
+}
 
+our $ho= selecthost($whhost);
 our $ram_mb=512;
 our $swap_mb=  1000;
 our $disk_mb= 1;
-
 our $guesthost= "$gn.guest.osstest";
 our $gho;
+our $L2_MAC = select_ether($ho,"L2_ether");
 
 sub prep () {
 target_install_packages_norec($ho, qw(lvm2 xen-tools));
-
-$gho= prepareguest($ho, $gn, $guesthost, 22,
-   $swap_mb + $disk_mb + 2,
-   40);
-target_cmd_root($ho, "umount $gho->{Lvdev} ||:");
+unless($nested_L2 eq 'nested_L2') {
+   $gho= prepareguest($ho, $gn, $guesthost, 22,
+  $swap_mb + $disk_mb + 2,
+  40);
+   target_cmd_root($ho, "umount $gho->{Lvdev} ||:");
+}
 }
 
 sub ginstall () {
-my $arch= $r{"$gho->{Guest}_arch"};
+my $gsuite;
+my $kernpath;
+my $initrd;
+my $kernver;
+if($nested_L2 eq 'nested_L2'){
+my $suite= "$c{DebianSuite}";
+$gsuite= defined($suite) ? "--dist=$suite" : '';
+$kernver ||= target_cmd_output_root($ho, 'uname -r');
+$kernpath = "/boot/vmlinuz-$kernver";
+$initrd ||= "/boot/initrd.img-$kernver";
+} else {
+my $arch= $r{"$gho->{Guest}_arch"};
+$gsuite= guest_var($gho,'suite',$c{GuestDebianSuite});
+$kernpath = guest_var($gho,'kernel_path',$r{xen_kernel_path});
+$initrd = guest_var($gho,'initrd_path',$r{xen_initrd_path});
+if (!$kernpath) {
+   $kernver= guest_var($gho,'kernel_ver',$r{xen_kernel_ver});
+$kernver ||= target_cmd_output($ho, 'uname -r');
+   $kernpath = "/boot/vmlinuz-$kernver";
+   $initrd ||= "/boot/initrd.img-$kernver";
+} 
+} 
 my $archarg= defined($arch) ? "--arch $arch" : '';
-my $gsuite= guest_var($gho,'suite',$c{GuestDebianSuite});
-
-my $kernpath = guest_var($gho,'kernel_path',$r{xen_kernel_path});
-my $initrd = guest_var($gho,'initrd_path',$r{xen_initrd_path});
-if (!$kernpath) {
-   my $kernver= guest_var($gho,'kernel_ver',$r{xen_kernel_ver});
-   $kernver ||= target_cmd_output($ho, 'uname -r');
-   $kernpath = "/boot/vmlinuz-$kernver";
-   $initrd ||= "/boot/initrd.img-$kernver";
-}
 if (!$initrd) {
$initrd = $kernpath;
$initrd =~ s,/vmlinuz-,/initrd.img-, or die "$initrd ?";
@@ -76,23 +96,97 @@ sub ginstall () {
 fi
 END
 }
-target_cmd_root($ho, <{Ether} \\
---memory ${ram_mb}Mb --swap ${swap_mb}Mb \\
---dist $gsuite \\
---mirror http://$c{DebianMirrorHost}/$c{DebianMirrorSubpath} \\
---hostname $gho->{Name} \\
---lvm $gho->{Vg} --force \\
---kernel $kernpath \\
---genpass 0 --password xenroot \\
-$initrd_opt \\
-$archarg
+if($nested_L2 eq 'nested_L2') {
+target_cmd_root($ho, <http://$c{DebianMirrorHost}/$c{DebianMirrorSubpath} \\
+--hostname $guesthost \\
+--dir=/testnested \\
+--kernel=$kernpath \\
+--genpass 0 --password xenroot \\
+$initrd_opt \\
+$gsuite \\
+$archarg
+END
+} else {
+target_cmd_root($ho, <{Ether} \\
+

60 matches

Mail list logo