[OE-core] [wic][PATCH] directdisk*.wks: add serial console support

[Ed Bartosh]

Added serial console to kernel command line to to make it
easier to boot wic images on devices without display.

Tested on MinnowBoard MAX.

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 scripts/lib/wic/canned-wks/directdisk-gpt.wks  | 2 +-
 scripts/lib/wic/canned-wks/directdisk-multi-rootfs.wks | 2 +-
 scripts/lib/wic/canned-wks/directdisk.wks  | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/wic/canned-wks/directdisk-gpt.wks 
b/scripts/lib/wic/canned-wks/directdisk-gpt.wks
index ea01cf3..8d7d8de 100644
--- a/scripts/lib/wic/canned-wks/directdisk-gpt.wks
+++ b/scripts/lib/wic/canned-wks/directdisk-gpt.wks
@@ -6,5 +6,5 @@
 part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 
1024
 part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 
1024 --use-uuid
 
-bootloader  --ptable gpt --timeout=0  --append="rootwait rootfstype=ext4 
video=vesafb vga=0x318 console=tty0"
+bootloader  --ptable gpt --timeout=0  --append="rootwait rootfstype=ext4 
video=vesafb vga=0x318 console=tty0 console=ttyS0,115200n8"
 
diff --git a/scripts/lib/wic/canned-wks/directdisk-multi-rootfs.wks 
b/scripts/lib/wic/canned-wks/directdisk-multi-rootfs.wks
index 8a81f8f..f61d941 100644
--- a/scripts/lib/wic/canned-wks/directdisk-multi-rootfs.wks
+++ b/scripts/lib/wic/canned-wks/directdisk-multi-rootfs.wks
@@ -19,5 +19,5 @@ part /boot --source bootimg-pcbios --ondisk sda --label boot 
--active --align 10
 part / --source rootfs --rootfs-dir=rootfs1 --ondisk sda --fstype=ext4 --label 
platform --align 1024
 part /rescue --source rootfs --rootfs-dir=rootfs2 --ondisk sda --fstype=ext4 
--label secondary --align 1024
 
-bootloader  --timeout=0  --append="rootwait rootfstype=ext4 video=vesafb 
vga=0x318 console=tty0"
+bootloader  --timeout=0  --append="rootwait rootfstype=ext4 video=vesafb 
vga=0x318 console=tty0 console=ttyS0,115200n8"
 
diff --git a/scripts/lib/wic/canned-wks/directdisk.wks 
b/scripts/lib/wic/canned-wks/directdisk.wks
index 6db74a7..8c8e06b 100644
--- a/scripts/lib/wic/canned-wks/directdisk.wks
+++ b/scripts/lib/wic/canned-wks/directdisk.wks
@@ -4,5 +4,5 @@
 
 include common.wks.inc
 
-bootloader  --timeout=0  --append="rootwait rootfstype=ext4 video=vesafb 
vga=0x318 console=tty0"
+bootloader  --timeout=0  --append="rootwait rootfstype=ext4 video=vesafb 
vga=0x318 console=tty0 console=ttyS0,115200n8"
 
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [wic][PATCH v2] wic: remove partition images

[Ed Bartosh]

Preserving images for every partition doubles disk space
consumed by an image build. As those images are not used,
so it's better to remove them after assembling final image.

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 scripts/lib/wic/utils/partitionedfs.py | 9 -
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/wic/utils/partitionedfs.py 
b/scripts/lib/wic/utils/partitionedfs.py
index 46b5d34..cafb933 100644
--- a/scripts/lib/wic/utils/partitionedfs.py
+++ b/scripts/lib/wic/utils/partitionedfs.py
@@ -43,6 +43,7 @@ class Image():
 def __init__(self, native_sysroot=None):
 self.disks = {}
 self.partitions = []
+self.partimages = []
 # Size of a sector used in calculations
 self.sector_size = SECTOR_SIZE
 self._partitions_layed_out = False
@@ -336,6 +337,10 @@ class Image():
 disk['disk'].cleanup()
 except:
 pass
+# remove partition images
+for image in self.partimages:
+if os.path.isfile(image):
+os.remove(image)
 
 def assemble(self, image_file):
 msger.debug("Installing partitions")
@@ -351,7 +356,9 @@ class Image():
 (source, part['num'], part['start'],
  part['start'] + part['size'] - 1, part['size']))
 
-os.rename(source, image_file + '.p%d' % part['num'])
+partimage = image_file + '.p%d' % part['num']
+os.rename(source, partimage)
+self.partimages.append(partimage)
 
 def create(self):
 for dev in self.disks:
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [wic][PATCH] wic: add beaglebone.wks

[Ed Bartosh]

On Wed, Sep 21, 2016 at 06:34:33PM +0200, Maciej Borzęcki wrote:
> On Wed, Sep 21, 2016 at 5:18 PM, Ed Bartosh <ed.bart...@linux.intel.com> 
> wrote:
> > Added wks to make an image for Beaglebone board.
> >
> > Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
> > ---
> >  scripts/lib/wic/canned-wks/beaglebone.wks | 7 +++
> >  1 file changed, 7 insertions(+)
> >  create mode 100644 scripts/lib/wic/canned-wks/beaglebone.wks
> >
> > diff --git a/scripts/lib/wic/canned-wks/beaglebone.wks 
> > b/scripts/lib/wic/canned-wks/beaglebone.wks
> > new file mode 100644
> > index 000..753418e
> > --- /dev/null
> > +++ b/scripts/lib/wic/canned-wks/beaglebone.wks
> > @@ -0,0 +1,7 @@
> > +# short-description: Create Beaglebone disk image
> > +# long-description: Creates a partitioned disk image for Beaglebone board
> > +# which can be directly copied to SD card.
> > +
> > +part /boot --ondisk mmcblk0 --label boot --fstype vfat --source 
> > bootimg-partition --align 1024 --active
> > +part / --ondisk mmcblk0 --label root --fstype ext4 --source rootfs 
> > --align 1024
> > +
> 
> Should be basically the same as sdimage-bootpart.wks as I used BBB
> back when I posted it along with some wic fixes. It also used to be
> possible to build a usable RaspberryPI image with the same kickstart
> file, though I have not tried that lately.

Thank you for pointing this out to me. I've just built wic image using
sdimage-bootpart.wks and it worked just fine on my BBB.

I didn't need any fixes though. Have your fixes been merged? Can you
point me out to them?

--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [wic][PATCH] wic: add beaglebone.wks

[Ed Bartosh]

Please, ignore this patch. It's not needed as fully functional image
can be built using existing wks file sdimage-bootpart.wks

On Wed, Sep 21, 2016 at 06:18:12PM +0300, Ed Bartosh wrote:
> Added wks to make an image for Beaglebone board.
> 
> Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
> ---
>  scripts/lib/wic/canned-wks/beaglebone.wks | 7 +++
>  1 file changed, 7 insertions(+)
>  create mode 100644 scripts/lib/wic/canned-wks/beaglebone.wks
> 
> diff --git a/scripts/lib/wic/canned-wks/beaglebone.wks 
> b/scripts/lib/wic/canned-wks/beaglebone.wks
> new file mode 100644
> index 000..753418e
> --- /dev/null
> +++ b/scripts/lib/wic/canned-wks/beaglebone.wks
> @@ -0,0 +1,7 @@
> +# short-description: Create Beaglebone disk image
> +# long-description: Creates a partitioned disk image for Beaglebone board
> +# which can be directly copied to SD card.
> +
> +part /boot --ondisk mmcblk0 --label boot --fstype vfat --source 
> bootimg-partition --align 1024 --active
> +part / --ondisk mmcblk0 --label root --fstype ext4 --source rootfs 
> --align 1024
> +
> -- 
> 2.1.4
> 

-- 
--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [wic][PATCH] wic: remove partition images

[Ed Bartosh]

Preserving images for every partition doubles disk space
consumed by a build. As those images are not used it's
better to remove them after assembling final image.

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 scripts/lib/wic/utils/partitionedfs.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/lib/wic/utils/partitionedfs.py 
b/scripts/lib/wic/utils/partitionedfs.py
index 46b5d34..3dbf9a6 100644
--- a/scripts/lib/wic/utils/partitionedfs.py
+++ b/scripts/lib/wic/utils/partitionedfs.py
@@ -351,7 +351,7 @@ class Image():
 (source, part['num'], part['start'],
  part['start'] + part['size'] - 1, part['size']))
 
-os.rename(source, image_file + '.p%d' % part['num'])
+os.unlink(source)
 
 def create(self):
 for dev in self.disks:
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [wic][PATCH] image_types: add parted-native to do_image_wic depends

[Ed Bartosh]

As parted is always used by wic it makes sense to make do_image_wic
dependent on parted-native:do_populate_sysroot. This should help
to avoid adding it to all wic image recipes.

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 meta-selftest/recipes-test/images/wic-image-minimal.bb | 2 +-
 meta/classes/image_types.bbclass   | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta-selftest/recipes-test/images/wic-image-minimal.bb 
b/meta-selftest/recipes-test/images/wic-image-minimal.bb
index 754689f..58bf5a5 100644
--- a/meta-selftest/recipes-test/images/wic-image-minimal.bb
+++ b/meta-selftest/recipes-test/images/wic-image-minimal.bb
@@ -7,7 +7,7 @@ IMAGE_INSTALL = "packagegroup-core-boot 
${ROOTFS_PKGMANAGE_BOOTSTRAP}"
 IMAGE_FSTYPES = "wic"
 RM_OLD_IMAGE = "1"
 
-DEPENDS = "syslinux syslinux-native parted-native dosfstools-native 
mtools-native gptfdisk-native"
+DEPENDS = "syslinux syslinux-native dosfstools-native mtools-native 
gptfdisk-native"
 
 LIC_FILES_CHKSUM = 
"file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
 
diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index 5ef6f60..0bb6252 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -218,6 +218,7 @@ IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES"
 USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' 
'.join('wic.%s' % c for c in '${CONVERSIONTYPES}'.split()), '1', '', d)}"
 WKS_FILE_CHECKSUM = "${@'${WKS_FULL_PATH}:%s' % 
os.path.exists('${WKS_FULL_PATH}') if '${USING_WIC}' else ''}"
 do_image_wic[file-checksums] += "${WKS_FILE_CHECKSUM}"
+do_image_wic[depends] += "parted-native:do_populate_sysroot"
 
 python do_write_wks_template () {
 """Write out expanded template contents to WKS_FULL_PATH."""
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [wic][PATCH] wic: selftest: add test for mksystemd-bootdisk

[Ed Bartosh]

Test creation of mksystemd-bootdisk image.

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 meta/lib/oeqa/selftest/wic.py | 7 +++
 1 file changed, 7 insertions(+)

diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index e550785..6012abc 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -284,3 +284,10 @@ class Wic(oeSelfTest):
 self.assertEqual(0, status)
 self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image)))
 self.assertEqual(1, len(glob(self.resultdir + "%s-*direct.bmap" % 
image)))
+
+def test_mksystemd_bootdisk(self):
+"""Test creation of mksystemd-bootdisk image"""
+image = "mksystemd-bootdisk"
+self.assertEqual(0, runCmd("wic create %s -e core-image-minimal" \
+   % image).status)
+self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image)))
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [wic][PATCH] image_types.bbclass: remove redundant dependency

[Ed Bartosh]

Removed parted-native dependency from do_image_wic as it's
already mentioned in IMAGE_DEPENDS_wic variable.

Thanks to Christopher Larson for pointing out to this.

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 meta/classes/image_types.bbclass | 1 -
 1 file changed, 1 deletion(-)

diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index 0bb6252..5ef6f60 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -218,7 +218,6 @@ IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES"
 USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' 
'.join('wic.%s' % c for c in '${CONVERSIONTYPES}'.split()), '1', '', d)}"
 WKS_FILE_CHECKSUM = "${@'${WKS_FULL_PATH}:%s' % 
os.path.exists('${WKS_FULL_PATH}') if '${USING_WIC}' else ''}"
 do_image_wic[file-checksums] += "${WKS_FILE_CHECKSUM}"
-do_image_wic[depends] += "parted-native:do_populate_sysroot"
 
 python do_write_wks_template () {
 """Write out expanded template contents to WKS_FULL_PATH."""
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] scripts: add new script

[Ed Bartosh]

Added 'native' convenience shell script to run native tools.
Example of usage:
  > bitbake bmap-tools-native
  > native bmaptool --version

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 scripts/native | 48 
 1 file changed, 48 insertions(+)
 create mode 100755 scripts/native

diff --git a/scripts/native b/scripts/native
new file mode 100755
index 000..5e904b0
--- /dev/null
+++ b/scripts/native
@@ -0,0 +1,48 @@
+#!/bin/sh
+#
+# Copyright (c) 2016,  Intel Corporation.
+# All Rights Reserved
+#
+# This program is free software;  you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY;  without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See
+# the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program;  if not, see <http://www.gnu.org/licenses/>
+#
+
+#
+# This script is for running tools from native oe sysroot
+#
+
+if [ $# -lt 1 -o "$1" == '--help' -o "$1" == '-h' ]; then
+echo "Usage: $0  [parameters]"
+exit 1
+fi
+
+SYSROOT_SETUP_SCRIPT=`which oe-find-native-sysroot 2> /dev/null`
+if [ -z "$SYSROOT_SETUP_SCRIPT" ]; then
+echo "Error: Unable to find oe-find-native-sysroot script"
+exit 1
+fi
+. $SYSROOT_SETUP_SCRIPT
+
+OLDPATH=$PATH
+
+# look for a tool only in native sysroot
+PATH=$OECORE_NATIVE_SYSROOT/usr/bin:$OECORE_NATIVE_SYSROOT/bin:$OECORE_NATIVE_SYSROOT/usr/sbin:$OECORE_NATIVE_SYSROOT/sbin
+tool=`/usr/bin/which $1 2>/dev/null`
+
+if [ -n "$tool" ] ; then
+# add old path to allow usage of host tools
+PATH=$PATH:$OLD_PATH $@
+else
+echo "Error: Unable to find '$1' in native sysroot"
+exit 1
+fi
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] runqemu: explicitly set image format

[Ed Bartosh]

QEMU produces a warning if drive format is not specified:
  WARNING: Image format was not specified for
  'tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic'
   and probing guessed raw.
   Automatically detecting the format is dangerous for raw images,
   write operations on block 0 will be restricted.
   Specify the 'raw' format explicitly to remove the restrictions.

Set image format to 'vmdk', 'qcow2' or 'vdi' for correspondent image
types. Set it to 'raw' for the rest of image types.

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 scripts/runqemu | 11 +++
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/scripts/runqemu b/scripts/runqemu
index 45bcad7..09b231b 100755
--- a/scripts/runqemu
+++ b/scripts/runqemu
@@ -817,11 +817,13 @@ class BaseConfig(object):
 else:
 self.setup_tap()
 
+rootfs_format = self.fstype if self.fstype in ('vmdk', 'qcow2', 'vdi') 
else 'raw'
+
 qb_rootfs_opt = self.get('QB_ROOTFS_OPT')
 if qb_rootfs_opt:
 self.rootfs_options = qb_rootfs_opt.replace('@ROOTFS@', 
self.rootfs)
 else:
-self.rootfs_options = '-drive file=%s,if=virtio,format=raw' % 
self.rootfs
+self.rootfs_options = '-drive file=%s,if=virtio,format=%s' % 
(self.rootfs, rootfs_format)
 
 if self.fstype in ('cpio.gz', 'cpio'):
 self.kernel_cmdline = 'root=/dev/ram0 rw debugshell'
@@ -835,14 +837,15 @@ class BaseConfig(object):
 cmd2 = "grep -q 'root=/dev/hd' %s" % self.rootfs
 if subprocess.call(cmd1, shell=True) == 0:
 logger.info('Using scsi drive')
-vm_drive = '-drive if=none,id=hd,file=%s -device 
virtio-scsi-pci,id=scsi -device scsi-hd,drive=hd' % self.rootfs
+vm_drive = '-drive if=none,id=hd,file=%s,format=%s 
-device virtio-scsi-pci,id=scsi -device scsi-hd,drive=hd' \
+   % (self.rootfs, rootfs_format)
 elif subprocess.call(cmd2, shell=True) == 0:
 logger.info('Using scsi drive')
-vm_drive = self.rootfs
+vm_drive = "%s,format=%s" % (self.rootfs, 
rootfs_format)
 else:
 logger.warn("Can't detect drive type %s" % self.rootfs)
 logger.warn('Tring to use virtio block drive')
-vm_drive = '-drive if=virtio,file=%s' % self.rootfs
+vm_drive = '-drive if=virtio,file=%s,format=%s' % 
(self.rootfs, rootfs_format)
 self.rootfs_options = '%s -no-reboot' % vm_drive
 self.kernel_cmdline = 'root=%s rw highres=off' % 
(self.get('QB_KERNEL_ROOT'))
 
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [wic][PATCH] mkefidisk.wks: use partition UUID and GPT partition table

[Ed Bartosh]

This is a preparation to use mkefidisk as a default wks for
genericx86* BSPs. This change enables usage of partition UUID
instead of device name to specify root partition in kernel
command line. It should make images to boot on devices with
boot device names that differ from what's mentioned in wks file.

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 scripts/lib/wic/canned-wks/mkefidisk.wks | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/wic/canned-wks/mkefidisk.wks 
b/scripts/lib/wic/canned-wks/mkefidisk.wks
index 696e94e..73886ab 100644
--- a/scripts/lib/wic/canned-wks/mkefidisk.wks
+++ b/scripts/lib/wic/canned-wks/mkefidisk.wks
@@ -4,8 +4,8 @@
 
 part /boot --source bootimg-efi --sourceparams="loader=grub-efi" --ondisk sda 
--label msdos --active --align 1024
 
-part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024
+part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 
1024 --use-uuid
 
 part swap --ondisk sda --size 44 --label swap1 --fstype=swap
 
-bootloader  --timeout=10  --append="rootwait rootfstype=ext4 
console=ttyPCH0,115200 console=tty0 vmalloc=256MB snd-hda-intel.enable_msi=0"
+bootloader --ptable gpt --timeout=10 --append="rootwait rootfstype=ext4 
console=ttyPCH0,115200 console=tty0 vmalloc=256MB snd-hda-intel.enable_msi=0"
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v2] scripts: add new script 'native'

[Ed Bartosh]

On Sun, Oct 02, 2016 at 04:30:58PM +0200, Ulf Magnusson wrote:
> On Sun, Oct 2, 2016 at 4:15 PM, Jérémy Rosen  wrote:
> > "native" might be a bit too generic...
> >
> > could I suggest oe_native-run or bitbake-native ?
> >
> > something that says it's yocto-galaxy specific and can be autocompleted with
> > a reasonable prefix...
> >
> >
> > Jérémy Rosen
> 
> I think overly namespacing things that aren't likely to clash in
> practice gets annoying. I would've gone for "run-native" (more
> descriptive and English-like, less generic), but how about something
> simple and easy to type like "oe-native" or "oe-run-native" if a
> prefix is added? That's consistent with the naming scheme used for
> many other scripts in scripts/ too.
> 

I'm ok to rename it. I agree that 'native' may look too generic, but
I don't know about any existing tool with this name.

Anyway, let me know which name I should rename it to. Feel free to send a
patch too. I personally like 'oe-native' more than other proposals.

--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] wic: selftest: add test for sdimage-bootpart

[Ed Bartosh]

Test image creation using mksystemd-bootdisk.wks

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 meta/lib/oeqa/selftest/wic.py | 8 
 1 file changed, 8 insertions(+)

diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index 6012abc..b05300c 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -291,3 +291,11 @@ class Wic(oeSelfTest):
 self.assertEqual(0, runCmd("wic create %s -e core-image-minimal" \
% image).status)
 self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image)))
+
+def test_sdimage_bootpart(self):
+"""Test creation of sdimage-bootpart image"""
+image = "sdimage-bootpart"
+self.write_config('IMAGE_BOOT_FILES = "bzImage"\n')
+self.assertEqual(0, runCmd("wic create %s -e core-image-minimal" \
+   % image).status)
+self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image)))
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v2] scripts: add new script 'native'

[Ed Bartosh]

Added 'native' convenience shell script to run native tools.
Example of usage:
  > bitbake bmap-tools-native
  > native bmaptool --version

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 scripts/native | 48 
 1 file changed, 48 insertions(+)
 create mode 100755 scripts/native

diff --git a/scripts/native b/scripts/native
new file mode 100755
index 000..496e34f
--- /dev/null
+++ b/scripts/native
@@ -0,0 +1,48 @@
+#!/bin/sh
+#
+# Copyright (c) 2016,  Intel Corporation.
+# All Rights Reserved
+#
+# This program is free software;  you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY;  without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See
+# the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program;  if not, see <http://www.gnu.org/licenses/>
+#
+
+#
+# This script is for running tools from native oe sysroot
+#
+
+if [ $# -lt 1 -o "$1" = '--help' -o "$1" = '-h' ] ; then
+echo "Usage: $0  [parameters]"
+exit 1
+fi
+
+SYSROOT_SETUP_SCRIPT=`which oe-find-native-sysroot 2> /dev/null`
+if [ -z "$SYSROOT_SETUP_SCRIPT" ]; then
+echo "Error: Unable to find oe-find-native-sysroot script"
+exit 1
+fi
+. $SYSROOT_SETUP_SCRIPT
+
+OLDPATH=$PATH
+
+# look for a tool only in native sysroot
+PATH=$OECORE_NATIVE_SYSROOT/usr/bin:$OECORE_NATIVE_SYSROOT/bin:$OECORE_NATIVE_SYSROOT/usr/sbin:$OECORE_NATIVE_SYSROOT/sbin
+tool=`/usr/bin/which $1 2>/dev/null`
+
+if [ -n "$tool" ] ; then
+# add old path to allow usage of host tools
+PATH=$PATH:$OLD_PATH $@
+else
+echo "Error: Unable to find '$1' in native sysroot"
+exit 1
+fi
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v2] uninative: rebuild uninative for gcc 4.8 and 4.9

[Ed Bartosh]

Some c++ libraries fail to build if uninative is built
with gcc 5.x and host gcc version is either 4.8 or 4.9.

The issue should be solved by making separate uninative sstate
directory structure sstate-cache/universal- for host gcc
versions 4.8 and 4.9. This causes rebuilds of uninative if host gcc
is either 4.8 or 4.9 and it doesn't match gcc version used to build
uninative.

[YOCTO #10441]

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 meta/classes/uninative.bbclass | 16 +++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/meta/classes/uninative.bbclass b/meta/classes/uninative.bbclass
index 89cec07..45f9262 100644
--- a/meta/classes/uninative.bbclass
+++ b/meta/classes/uninative.bbclass
@@ -95,11 +95,25 @@ python uninative_event_enable() {
 enable_uninative(d)
 }
 
+def gcc_version(d):
+compiler = d.getVar("BUILD_CC", True)
+retval, output = oe.utils.getstatusoutput("%s --version" % compiler)
+if retval:
+bb.fatal("Error running %s --version: %s" % (compiler, output))
+
+import re
+match = re.match(".* (\d\.\d)\.\d .*", output.split('\n')[0])
+if not match:
+bb.fatal("Can't get compiler version from %s --version output" % 
compiler)
+
+version = match.group(1)
+return "-%s" % version if version in ("4.8", "4.9") else ""
+
 def enable_uninative(d):
 loader = d.getVar("UNINATIVE_LOADER", True)
 if os.path.exists(loader):
 bb.debug(2, "Enabling uninative")
-d.setVar("NATIVELSBSTRING", "universal")
+d.setVar("NATIVELSBSTRING", "universal%s" % gcc_version(d))
 d.appendVar("SSTATEPOSTUNPACKFUNCS", " uninative_changeinterp")
 d.prependVar("PATH", 
"${STAGING_DIR}-uninative/${BUILD_ARCH}-linux${bindir_native}:")
 
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] uninative: rebuild uninative for gcc 4.8 and 4.9

[Ed Bartosh]

Some c++ libraries fail to build if uninative is built
with gcc 5.x and host gcc version is either 4.8 or 4.9.

The issue should be solved by making separate uninative sstate
directory structure sstate-cache/universal- for host gcc
versions 4.8 and 4.9. This causes rebuilds of uninative if host gcc
is either 4.8 or 4.9 and it doesn't match gcc version used to build
uninative.

[YOCTO #10441]

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 meta/classes/uninative.bbclass | 20 +++-
 1 file changed, 19 insertions(+), 1 deletion(-)

diff --git a/meta/classes/uninative.bbclass b/meta/classes/uninative.bbclass
index 89cec07..35c1496 100644
--- a/meta/classes/uninative.bbclass
+++ b/meta/classes/uninative.bbclass
@@ -95,11 +95,29 @@ python uninative_event_enable() {
 enable_uninative(d)
 }
 
+def gcc_version(d):
+compiler = d.getVar("BUILD_CC", True)
+retval, output = oe.utils.getstatusoutput("%s --version" % compiler)
+if retval:
+bb.fatal("Error running %s --version: %s" % (compiler, output))
+
+import re
+match = re.match(".* (\d\.\d)\.\d .*", output.split('\n')[0])
+if not match:
+bb.fatal("Can't get compiler version from %s --version output" % 
compiler)
+
+version = match.group(1)
+return "-%s" % version if version in ("4.8", "4.9") else ""
+
 def enable_uninative(d):
 loader = d.getVar("UNINATIVE_LOADER", True)
 if os.path.exists(loader):
 bb.debug(2, "Enabling uninative")
-d.setVar("NATIVELSBSTRING", "universal")
+gcc_ver = d.getVar("GCC_VERSION", True)
+if not gcc_ver:
+gcc_ver = gcc_version(d)
+d.setVar("GCC_VERSION", gcc_ver)
+d.setVar("NATIVELSBSTRING", "universal%s" % gcc_ver)
 d.appendVar("SSTATEPOSTUNPACKFUNCS", " uninative_changeinterp")
 d.prependVar("PATH", 
"${STAGING_DIR}-uninative/${BUILD_ARCH}-linux${bindir_native}:")
 
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] mkefidisk.sh: add deprecation warning to the output

[Ed Bartosh]

On Thu, Nov 03, 2016 at 02:42:39PM -0700, Khem Raj wrote:
> 
> > On Oct 31, 2016, at 3:46 AM, Ed Bartosh <ed.bart...@linux.intel.com> wrote:
> > 
> > mkefidisk.sh will soon be deprecated in favor of .wic images.
> > 
> > Added deprecation warning to the script to inform users that
> > this script will soon be removed from the codebase.
> > 
> > Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
> > ---
> > scripts/contrib/mkefidisk.sh | 5 +
> > 1 file changed, 5 insertions(+)
> > 
> > diff --git a/scripts/contrib/mkefidisk.sh b/scripts/contrib/mkefidisk.sh
> > index d8db3c0..a175895 100755
> > --- a/scripts/contrib/mkefidisk.sh
> > +++ b/scripts/contrib/mkefidisk.sh
> > @@ -20,6 +20,11 @@
> > 
> > LANG=C
> > 
> > +echo
> > +echo "WARNING: This script is deprecated and will be removed soon."
> > +echo "Please consider using wic EFI images instead."
> > +echo
> > +
> 
> is .wic image dd’able directly ?
Yes, they are.

> We should also document, the wic steps in wiki pages e.g. 
> http://wiki.minnowboard.org/Yocto_Project
It's already documented in README.hardware: 
https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/README.hardware

And in Yocto manual:
http://www.yoctoproject.org/docs/2.2/mega-manual/mega-manual.html#building-an-image-for-hardware

--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] mkefidisk.sh: add deprecation warning to the output

[Ed Bartosh]

On Thu, Nov 03, 2016 at 07:23:24PM -0700, Christopher Larson wrote:
> On Thu, Nov 3, 2016 at 2:42 PM, Khem Raj <raj.k...@gmail.com> wrote:
> 
> > > On Oct 31, 2016, at 3:46 AM, Ed Bartosh <ed.bart...@linux.intel.com>
> > wrote:
> > >
> > > mkefidisk.sh will soon be deprecated in favor of .wic images.
> > >
> > > Added deprecation warning to the script to inform users that
> > > this script will soon be removed from the codebase.
> > >
> > > Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
> > > ---
> > > scripts/contrib/mkefidisk.sh | 5 +
> > > 1 file changed, 5 insertions(+)
> > >
> > > diff --git a/scripts/contrib/mkefidisk.sh b/scripts/contrib/mkefidisk.sh
> > > index d8db3c0..a175895 100755
> > > --- a/scripts/contrib/mkefidisk.sh
> > > +++ b/scripts/contrib/mkefidisk.sh
> > > @@ -20,6 +20,11 @@
> > >
> > > LANG=C
> > >
> > > +echo
> > > +echo "WARNING: This script is deprecated and will be removed soon."
> > > +echo "Please consider using wic EFI images instead."
> > > +echo
> > > +
> >
> > is .wic image dd’able directly ?
> >
> 
> Yes, that’s the point :) Though I prefer bmaptool w/ IMAGE_FSTYPES +=
> “wic.bmap” personally, don’t have to rely on the file keeping its
> sparseness when scp’ing it around and whatnot.

Recommending bmaptool as a preferred way of flashing is in the plans :)

--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 2/5] wic: use partition size when creating empty partition files

[Ed Bartosh]

On Tue, Nov 08, 2016 at 04:56:08PM +0100, Maciej Borzecki wrote:
> It seems that prepare_empty_partition_ext() and
> prepare_empty_partition_btrfs() got broken in commit
> c8669749e37fe865c197c98d5671d9de176ff4dd, thus one could observe the
> following backtrace:
> 
> Backtrace:
>   File "/poky/scripts/lib/wic/plugins/imager/direct_plugin.py", line 
> 93, in do_create
> creator.create()
>   File "/poky/scripts/lib/wic/imager/baseimager.py", line 159, in create
> self._create()
>   File "/poky/scripts/lib/wic/imager/direct.py", line 290, in _create
> self.bootimg_dir, self.kernel_dir, self.native_sysroot)
>   File "/poky/scripts/lib/wic/partition.py", line 146, in prepare
> method(rootfs, oe_builddir, native_sysroot)
>   File "/poky/scripts/lib/wic/partition.py", line 325, in 
> prepare_empty_partition_ext
> os.ftruncate(sparse.fileno(), rootfs_size * 1024)
> NameError: name 'rootfs_size' is not defined
> 
> Signed-off-by: Maciej Borzecki 
> ---
>  scripts/lib/wic/partition.py | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
> index 
> 4b8d769437120adadb5dba2f3919d4eb96141292..8adc698240c8e3bd9f4118663a5d7a167e0bb4a4
>  100644
> --- a/scripts/lib/wic/partition.py
> +++ b/scripts/lib/wic/partition.py
> @@ -308,7 +308,7 @@ class Partition():
>  Prepare an empty ext2/3/4 partition.
>  """
>  with open(rootfs, 'w') as sparse:
> -os.ftruncate(sparse.fileno(), rootfs_size * 1024)
> +os.ftruncate(sparse.fileno(), self.size * 1024)
>  
>  extra_imagecmd = "-i 8192"
>  
> @@ -326,7 +326,7 @@ class Partition():
>  Prepare an empty btrfs partition.
>  """
>  with open(rootfs, 'w') as sparse:
> -os.ftruncate(sparse.fileno(), rootfs_size * 1024)
> +os.ftruncate(sparse.fileno(), self.size * 1024)
>  
>  label_str = ""
>  if self.label:

Thank you for the fix. Sorry for breaking this. I thought I
double-checked and all tests were passing :(


--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 5/5] wic: add --fixed-size wks option

[Ed Bartosh]

On Tue, Nov 08, 2016 at 04:56:11PM +0100, Maciej Borzecki wrote:
> Added new option --fixed-size to wks. The option can be used to indicate
> the exact size of a partition. The option cannot be added together with
> --size, in which case an error will be raised. Other options that
> influence automatic partition size (--extra-space, --overhead-factor),
> if specifiec along with --fixed-size, will raise an error.
> 
> If it partition data is larger than the amount of space specified with
> --fixed-size option wic will raise an error.
> 
> Signed-off-by: Maciej Borzecki 
> ---
>  scripts/lib/wic/help.py| 14 --
>  scripts/lib/wic/imager/direct.py   |  2 +-
>  scripts/lib/wic/ksparser.py| 41 +++--
>  scripts/lib/wic/partition.py   | 83 
> --
>  scripts/lib/wic/utils/partitionedfs.py |  2 +-
>  5 files changed, 100 insertions(+), 42 deletions(-)
> 
> diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
> index 
> e5347ec4b7c900c68fc64351a5293e75de0672b3..daa11bf489c135627ddfe4cef968e48f8e3ad1d8
>  100644
> --- a/scripts/lib/wic/help.py
> +++ b/scripts/lib/wic/help.py
> @@ -646,6 +646,12 @@ DESCRIPTION
>   not specified, the size is in MB.
>   You do not need this option if you use --source.
>  
> + --fixed-size: Exact partition size. Value format is the same
> +   as for --size option. This option cannot be
> +   specified along with --size. If partition data
> +   is larger than --fixed-size and error will be
> +   raised when assembling disk image.
> +
>   --source: This option is a wic-specific option that names the
> source of the data that will populate the
> partition.  The most common value for this option
> @@ -719,13 +725,15 @@ DESCRIPTION
>  space after the space filled by the content
>  of the partition. The final size can go
>  beyond the size specified by --size.
> -By default, 10MB.
> +By default, 10MB. This option cannot be used
> +with --fixed-size option.
>  
>   --overhead-factor: This option is specific to wic. The
>  size of the partition is multiplied by
>  this factor. It has to be greater than or
> -equal to 1.
> -The default value is 1.3.
> +equal to 1. The default value is 1.3.
> +This option cannot be used with --fixed-size
> +option.
>  
>   --part-type: This option is specific to wic. It specifies partition
>type GUID for GPT partitions.
> diff --git a/scripts/lib/wic/imager/direct.py 
> b/scripts/lib/wic/imager/direct.py
> index 
> 2bedef08d6450096c786def6f75a9ee53fcd4b3b..c01a1ea538428e36a75ac5b31a822e01901bea6a
>  100644
> --- a/scripts/lib/wic/imager/direct.py
> +++ b/scripts/lib/wic/imager/direct.py
> @@ -290,7 +290,7 @@ class DirectImageCreator(BaseImageCreator):
>   self.bootimg_dir, self.kernel_dir, 
> self.native_sysroot)
>  
>  
> -self.__image.add_partition(int(part.size),
> +self.__image.add_partition(part.get_size(),
> part.disk,
> part.mountpoint,
> part.source_file,
> diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
> index 
> 0894e2b199a299fbbed272f2e1c95e9d692e3ab1..62c490274aa92bf82aac304d9323250e3b728d0c
>  100644
> --- a/scripts/lib/wic/ksparser.py
> +++ b/scripts/lib/wic/ksparser.py
> @@ -113,6 +113,9 @@ def systemidtype(arg):
>  class KickStart():
>  Kickstart parser implementation."""
>  
> +DEFAULT_EXTRA_SPACE = 10*1024
> +DEFAULT_OVERHEAD_FACTOR = 1.3
> +
>  def __init__(self, confpath):
>  
>  self.partitions = []
> @@ -127,16 +130,24 @@ class KickStart():
>  part.add_argument('mountpoint', nargs='?')
>  part.add_argument('--active', action='store_true')
>  part.add_argument('--align', type=int)
> -part.add_argument("--extra-space", type=sizetype, default=10*1024)
> +part.add_argument("--extra-space", type=sizetype)
>  part.add_argument('--fsoptions', dest='fsopts')
>  part.add_argument('--fstype')
>  part.add_argument('--label')
>  part.add_argument('--no-table', action='store_true')
>  part.add_argument('--ondisk', '--ondrive', dest='disk')
> -part.add_argument("--overhead-factor", type=overheadtype, 
> default=1.3)
> +part.add_argument("--overhead-factor", type=overheadtype)
>  

Re: [OE-core] [PATCH 0/5] wic: bugfixes & --fixed-size support

[Ed Bartosh]

On Tue, Nov 08, 2016 at 04:56:06PM +0100, Maciej Borzecki wrote:
> The patch series is a follow-up after a previous attempt of adding
> --reserved-size option to wic[1].
> 
> The series introduces a number of fixes in patches 1 - 4.
> 
> The last patch in the series introduces --fixed-size option as discussed in 
> [1].
> The patch also introduces minor refactoring to code responsible for computing
> partition size.
> 
> Aside from new option, another user-visible change is how the size rootfs
> partitions with vfat is calculated. In previous code, vfat rootfs partition 
> size
> did not account for --extra-space nor --overhead-factor. Now, all rootfs
> partitions (except for squashfs) are subject to the same rules of partition
> sizing.
> 
Thank you for the patchset!

+1 (if my comments are taken into account)

It would be great if you add tests for this functionality to oe-selftest
wic suite.

> http://lists.openembedded.org/pipermail/openembedded-core/2016-October/127634.html
> 
> Maciej Borzecki (5):
>   wic: make sure that partition size is always an integer in internal
> processing
>   wic: use partition size when creating empty partition files
>   wic: check that filesystem is specified for a rootfs partition
>   wic: fix function comment typos
>   wic: add --fixed-size wks option
> 
>  scripts/lib/wic/help.py| 14 +++--
>  scripts/lib/wic/imager/direct.py   |  2 +-
>  scripts/lib/wic/ksparser.py| 41 +--
>  scripts/lib/wic/partition.py   | 93 
> +-
>  scripts/lib/wic/utils/partitionedfs.py |  6 +--
>  5 files changed, 109 insertions(+), 47 deletions(-)
> 
> -- 
> 2.5.0
> 

-- 
--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH 3/5] wic: check that filesystem is specified for a rootfs partition

[Ed Bartosh]

On Tue, Nov 08, 2016 at 04:56:09PM +0100, Maciej Borzecki wrote:
> Signed-off-by: Maciej Borzecki 
> ---
>  scripts/lib/wic/partition.py | 4 
>  1 file changed, 4 insertions(+)
> 
> diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
> index 
> 8adc698240c8e3bd9f4118663a5d7a167e0bb4a4..24e657592738dc7c5cdff78e3740d7c373021e9d
>  100644
> --- a/scripts/lib/wic/partition.py
> +++ b/scripts/lib/wic/partition.py
> @@ -184,6 +184,10 @@ class Partition():
>  if os.path.isfile(rootfs):
>  os.remove(rootfs)
>  
> +if not self.fstype:
> +msger.error("File system for partition %s not specified in 
> kickstart, " \
> +"use --fstype option" % (self.mountpoint))
> +
Would it make sense to make --fstype mandatory in ksparser?

>  for prefix in ("ext", "btrfs", "vfat", "squashfs"):
>  if self.fstype.startswith(prefix):
>  method = getattr(self, "prepare_rootfs_" + prefix)
> -- 
> 2.5.0
> 

-- 
--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] mkefidisk.sh: add deprecation warning to the output

[Ed Bartosh]

On Tue, Nov 08, 2016 at 01:05:23PM -0800, John Hawley wrote:
> On 11/08/2016 12:44 PM, Randy Witt wrote:
>  We should also document, the wic steps in wiki pages e.g.
>  http://wiki.minnowboard.org/Yocto_Project
> >>> It's already documented in README.hardware:
> >>> https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/README.hardware
> >>>
> >>> And in Yocto manual:
> >>> http://www.yoctoproject.org/docs/2.2/mega-manual/mega-manual.html#building-an-image-for-hardware
> >>>
> >>
> >> Thanks, now can you also nudge the minnowboard.org to do the same ?
> >>
> > 
> > Pinging you John. :)
> 
> So two obvious questions in reading the documentation off the YP site:
> 
> 1) Does WIC actually handle the EFI partition for booting off of UEFI
> systems? (guessing yes, but wanted to double check)
Yes, it does.

> 2) I assume it only creates MBR based images, seeing as the suggestion
> is to DD the image down, how does GPT based images work since you can't
> (exactly) DD a GPT image down and have it work "correctly"?
>
Please, elaborate on "correctly". Do you mean that kernel will complain
that backup GPT header is not at the end of disk? This is not handled by
wic obviously. However, even without this image is bootable and
functional.

> Sorry, I'm sure that's covered somewhere else, just want to double check
> this will be ok before I update the documentation on MB


--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v2] uninative: rebuild uninative for gcc 4.8 and 4.9

[Ed Bartosh]

Some c++ libraries fail to build if uninative is built
with gcc 5.x and host gcc version is either 4.8 or 4.9.

The issue should be solved by making separate uninative sstate
directory structure sstate-cache/universal- for host gcc
versions 4.8 and 4.9. This causes rebuilds of uninative if host gcc
is either 4.8 or 4.9 and it doesn't match gcc version used to build
uninative.

[YOCTO #10441]

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 meta/classes/uninative.bbclass | 16 +++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/meta/classes/uninative.bbclass b/meta/classes/uninative.bbclass
index 89cec07..3862b31 100644
--- a/meta/classes/uninative.bbclass
+++ b/meta/classes/uninative.bbclass
@@ -95,11 +95,25 @@ python uninative_event_enable() {
 enable_uninative(d)
 }
 
+def gcc_version(d):
+compiler = d.getVar("BUILD_CC", True)
+retval, output = oe.utils.getstatusoutput("%s --version" % compiler)
+if retval:
+bb.fatal("Error running %s --version: %s" % (compiler, output))
+
+import re
+match = re.match(".* (\d\.\d)\.\d.*", output.split('\n')[0])
+if not match:
+bb.fatal("Can't get compiler version from %s --version output" % 
compiler)
+
+version = match.group(1)
+return "-%s" % version if version in ("4.8", "4.9") else ""
+
 def enable_uninative(d):
 loader = d.getVar("UNINATIVE_LOADER", True)
 if os.path.exists(loader):
 bb.debug(2, "Enabling uninative")
-d.setVar("NATIVELSBSTRING", "universal")
+d.setVar("NATIVELSBSTRING", "universal%s" % gcc_version(d))
 d.appendVar("SSTATEPOSTUNPACKFUNCS", " uninative_changeinterp")
 d.prependVar("PATH", 
"${STAGING_DIR}-uninative/${BUILD_ARCH}-linux${bindir_native}:")
 
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] wic: add --reserved-size wks option

[Ed Bartosh]

On Tue, Oct 18, 2016 at 01:07:48PM +0200, Maciej Borzęcki wrote:
> On Tue, Oct 18, 2016 at 12:17 PM, Ed Bartosh <ed.bart...@linux.intel.com> 
> wrote:
> > On Tue, Oct 18, 2016 at 12:24:55PM +0200, Maciej Borzęcki wrote:
> >> On Tue, Oct 18, 2016 at 11:27 AM, Ed Bartosh <ed.bart...@linux.intel.com> 
> >> wrote:
> >> > On Tue, Oct 18, 2016 at 11:24:14AM +0200, Maciej Borzęcki wrote:
> >> >> On Tue, Oct 18, 2016 at 10:38 AM, Ed Bartosh 
> >> >> <ed.bart...@linux.intel.com> wrote:
> >> >> > On Tue, Oct 18, 2016 at 10:37:22AM +0200, Maciej Borzęcki wrote:
> >> >> >> On Tue, Oct 18, 2016 at 9:31 AM, Ed Bartosh 
> >> >> >> <ed.bart...@linux.intel.com> wrote:
> >> >> >> > On Mon, Oct 17, 2016 at 04:46:00PM +0200, Maciej Borzęcki wrote:
> >> >> >> >> On Mon, Oct 17, 2016 at 3:22 PM, Ed Bartosh 
> >> >> >> >> <ed.bart...@linux.intel.com> wrote:
> >> >> >> >> > Hi Maciej,
> >> >> >> >> >
> >> >> >> >> > There is already --size and --extra-space options.
> >> >> >> >> > Can we get the same or similar result by just using them? Do we 
> >> >> >> >> > really
> >> >> >> >> > need new option for similar purpose?
> >> >> >> >>
> >> >> >> >> --reserved-size serves a different purpose, it establishes an 
> >> >> >> >> upper
> >> >> >> >> bound on the size of a partition during layout. Unlike
> >> >> >> >> --size/--extra-space does not depend on the size of the filesystem
> >> >> >> >> image.
> >> >> >> >>
> >> >> >> >> For instance, assume I'm creating an image for SD card/eMMC with a
> >> >> >> >> fixed partition layout (something simple: boot partition, primary 
> >> >> >> >> &
> >> >> >> >> secondary rootfs partitions, some data partition). Because future
> >> >> >> >> system updates are delivered as filesystem image, I want to make 
> >> >> >> >> sure
> >> >> >> >> that there is exactly xxx MBs for my current and future rootfs 
> >> >> >> >> images
> >> >> >> >> (regardless of current image size). Neither --size nor 
> >> >> >> >> --extra-space
> >> >> >> >> can do that. I could use, say `--size 200 --overhead-factor 1`, 
> >> >> >> >> but
> >> >> >> >> this will needlessly create a 200MB rootfs image and if I happen 
> >> >> >> >> to
> >> >> >> >> cross the 200MB boundary I will not get an error.
> >> >> >> >>
> >> >> >> >> I had a private patch that added --fixed-size to enforce --size, 
> >> >> >> >> but
> >> >> >> >> that would still end up creating filesystem image to fill the 
> >> >> >> >> whole
> >> >> >> >> space.
> >> >> >> > I didn't get the difference between enforcing partition size and 
> >> >> >> > below
> >> >> >> > implementation. Can you elaborate a bit?
> >> >> >>
> >> >> >> `--fixed-size` was something that I had added to my fork back in 
> >> >> >> 2014,
> >> >> >> even before `--overhead-factor` came in. The problem is that 
> >> >> >> depending
> >> >> >> on a project you might want to have more control over how partitions
> >> >> >> are laid out, or even need to have a fixed layout. Adding
> >> >> >> `--fixed-size` would had a similar effect to what `--overhead-factor
> >> >> >> 1` does right now. Combined with `--size` would ensure that rootfs is
> >> >> >> say, 200MB large. The downside was that wic would actually create a
> >> >> >> 200MB rootfs, something that is not really necessary. In fact, I only
> >> >> >> wanted to have 200MB gap so that I have some spare space for future
> >> >> >> updates (where update is just a rootfs image you dd to the 
> >> >> >> partition).
>

Re: [OE-core] [PATCH] wic: add --reserved-size wks option

[Ed Bartosh]

On Tue, Oct 18, 2016 at 12:24:55PM +0200, Maciej Borzęcki wrote:
> On Tue, Oct 18, 2016 at 11:27 AM, Ed Bartosh <ed.bart...@linux.intel.com> 
> wrote:
> > On Tue, Oct 18, 2016 at 11:24:14AM +0200, Maciej Borzęcki wrote:
> >> On Tue, Oct 18, 2016 at 10:38 AM, Ed Bartosh <ed.bart...@linux.intel.com> 
> >> wrote:
> >> > On Tue, Oct 18, 2016 at 10:37:22AM +0200, Maciej Borzęcki wrote:
> >> >> On Tue, Oct 18, 2016 at 9:31 AM, Ed Bartosh 
> >> >> <ed.bart...@linux.intel.com> wrote:
> >> >> > On Mon, Oct 17, 2016 at 04:46:00PM +0200, Maciej Borzęcki wrote:
> >> >> >> On Mon, Oct 17, 2016 at 3:22 PM, Ed Bartosh 
> >> >> >> <ed.bart...@linux.intel.com> wrote:
> >> >> >> > Hi Maciej,
> >> >> >> >
> >> >> >> > There is already --size and --extra-space options.
> >> >> >> > Can we get the same or similar result by just using them? Do we 
> >> >> >> > really
> >> >> >> > need new option for similar purpose?
> >> >> >>
> >> >> >> --reserved-size serves a different purpose, it establishes an upper
> >> >> >> bound on the size of a partition during layout. Unlike
> >> >> >> --size/--extra-space does not depend on the size of the filesystem
> >> >> >> image.
> >> >> >>
> >> >> >> For instance, assume I'm creating an image for SD card/eMMC with a
> >> >> >> fixed partition layout (something simple: boot partition, primary &
> >> >> >> secondary rootfs partitions, some data partition). Because future
> >> >> >> system updates are delivered as filesystem image, I want to make sure
> >> >> >> that there is exactly xxx MBs for my current and future rootfs images
> >> >> >> (regardless of current image size). Neither --size nor --extra-space
> >> >> >> can do that. I could use, say `--size 200 --overhead-factor 1`, but
> >> >> >> this will needlessly create a 200MB rootfs image and if I happen to
> >> >> >> cross the 200MB boundary I will not get an error.
> >> >> >>
> >> >> >> I had a private patch that added --fixed-size to enforce --size, but
> >> >> >> that would still end up creating filesystem image to fill the whole
> >> >> >> space.
> >> >> > I didn't get the difference between enforcing partition size and below
> >> >> > implementation. Can you elaborate a bit?
> >> >>
> >> >> `--fixed-size` was something that I had added to my fork back in 2014,
> >> >> even before `--overhead-factor` came in. The problem is that depending
> >> >> on a project you might want to have more control over how partitions
> >> >> are laid out, or even need to have a fixed layout. Adding
> >> >> `--fixed-size` would had a similar effect to what `--overhead-factor
> >> >> 1` does right now. Combined with `--size` would ensure that rootfs is
> >> >> say, 200MB large. The downside was that wic would actually create a
> >> >> 200MB rootfs, something that is not really necessary. In fact, I only
> >> >> wanted to have 200MB gap so that I have some spare space for future
> >> >> updates (where update is just a rootfs image you dd to the partition).
> >> >>
> >> > Thanks for the explanations. Now I got it - reserved size is not a part
> >> > of partition, it's a gap between partitions.
> >>
> >> I might have not been clear enough when explaining. It's not a gap,
> >> it's just a container of size --reserved-size listed in MBR/GPT.
> >> There's probably a filesystem inside but not necessarily.
> >> Graphically it looks as like this:
> >>
> >>  --reserved-size
> >>   |--|
> >>   v  v
> >> +-+--+-+
> >> |. stuff .|xx|. stuff .|
> >> +-+--+-+
> >>   ^ ^^
> >>   |-||
> >>--size--extra-space
> >>
> >>
&g

Re: [OE-core] [PATCH] wic: add --reserved-size wks option

[Ed Bartosh]

On Mon, Oct 17, 2016 at 04:46:00PM +0200, Maciej Borzęcki wrote:
> On Mon, Oct 17, 2016 at 3:22 PM, Ed Bartosh <ed.bart...@linux.intel.com> 
> wrote:
> > Hi Maciej,
> >
> > There is already --size and --extra-space options.
> > Can we get the same or similar result by just using them? Do we really
> > need new option for similar purpose?
> 
> --reserved-size serves a different purpose, it establishes an upper
> bound on the size of a partition during layout. Unlike
> --size/--extra-space does not depend on the size of the filesystem
> image.
> 
> For instance, assume I'm creating an image for SD card/eMMC with a
> fixed partition layout (something simple: boot partition, primary &
> secondary rootfs partitions, some data partition). Because future
> system updates are delivered as filesystem image, I want to make sure
> that there is exactly xxx MBs for my current and future rootfs images
> (regardless of current image size). Neither --size nor --extra-space
> can do that. I could use, say `--size 200 --overhead-factor 1`, but
> this will needlessly create a 200MB rootfs image and if I happen to
> cross the 200MB boundary I will not get an error.
> 
> I had a private patch that added --fixed-size to enforce --size, but
> that would still end up creating filesystem image to fill the whole
> space.
I didn't get the difference between enforcing partition size and below
implementation. Can you elaborate a bit?

> Another workaround we used until now was to create a filesystem image
> manually and use fsimage source plugin. To get a fixed layout with
> rootfs image that may grow/shrink between builds, you still need to
> fiddle with --align (for instance, set it to 215040 to have the
> partition start at 210MB offset).
> 

Thanks for the explanations. I'm almost convinced.

See my comments below.

> >
> >
> > On Mon, Oct 17, 2016 at 03:06:18PM +0200, Maciej Borzecki wrote:
> >> Added new option --reserved-size to wks. The option can be used to
> >> indicate how much space should be reserved for a partition. This is
> >> useful if the disk will be a subject to full filesystem image updates
> >> and puts an upper limit of the size of filesystem images.
> >>
> >> The actual filesystem image may be smaller than the reserved space, thus
> >> leaving some room for growth. If it is larger, an error will be raised.
> >>
> >> Signed-off-by: Maciej Borzecki <maciej.borze...@rndity.com>
> >> ---
> >>  scripts/lib/wic/help.py| 11 ++
> >>  scripts/lib/wic/imager/direct.py   |  3 ++-
> >>  scripts/lib/wic/ksparser.py|  1 +
> >>  scripts/lib/wic/partition.py   |  1 +
> >>  scripts/lib/wic/utils/partitionedfs.py | 37 
> >> +-
> >>  5 files changed, 43 insertions(+), 10 deletions(-)
> >>
> >> diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
> >> index 
> >> e5347ec4b7c900c68fc64351a5293e75de0672b3..1a5c7020ba0cdc5ef2e477a2b14360e09098a896
> >>  100644
> >> --- a/scripts/lib/wic/help.py
> >> +++ b/scripts/lib/wic/help.py
> >> @@ -646,6 +646,17 @@ DESCRIPTION
> >>   not specified, the size is in MB.
> >>   You do not need this option if you use --source.
> >>
> >> + --reserved-size: This option specifies that there should be
> >> +  at least that many bytes reserved for
According to the code if partition size > reserved size wic will throw
an error. This somehow conflicts with 'at least that many bytes' part of the
description in my opinion.

As far as I uderstood your code this parameter specifies hard upper
limit for partition size, right?

> >> +  the partition during layout. This is useful
> >> +  when the target disk will be a subject
> >> +  to full system image updates in the future.
> >> +  Specifying --reserved-size ensures that
> >> +  there is extra space in the partition allowing
> >> +  for future growth of the file system stored
> >> +  inside. Value format is the same as for
> >> +  --size option.
> >> +
> >>   --source: This option is a wic-specific option that names the
> >> source of the data that will populate the
> >> partition.  The most common value for this option
> >> diff --git a/scripts/

Re: [OE-core] [PATCH] wic: add --reserved-size wks option

[Ed Bartosh]

On Tue, Oct 18, 2016 at 11:24:14AM +0200, Maciej Borzęcki wrote:
> On Tue, Oct 18, 2016 at 10:38 AM, Ed Bartosh <ed.bart...@linux.intel.com> 
> wrote:
> > On Tue, Oct 18, 2016 at 10:37:22AM +0200, Maciej Borzęcki wrote:
> >> On Tue, Oct 18, 2016 at 9:31 AM, Ed Bartosh <ed.bart...@linux.intel.com> 
> >> wrote:
> >> > On Mon, Oct 17, 2016 at 04:46:00PM +0200, Maciej Borzęcki wrote:
> >> >> On Mon, Oct 17, 2016 at 3:22 PM, Ed Bartosh 
> >> >> <ed.bart...@linux.intel.com> wrote:
> >> >> > Hi Maciej,
> >> >> >
> >> >> > There is already --size and --extra-space options.
> >> >> > Can we get the same or similar result by just using them? Do we really
> >> >> > need new option for similar purpose?
> >> >>
> >> >> --reserved-size serves a different purpose, it establishes an upper
> >> >> bound on the size of a partition during layout. Unlike
> >> >> --size/--extra-space does not depend on the size of the filesystem
> >> >> image.
> >> >>
> >> >> For instance, assume I'm creating an image for SD card/eMMC with a
> >> >> fixed partition layout (something simple: boot partition, primary &
> >> >> secondary rootfs partitions, some data partition). Because future
> >> >> system updates are delivered as filesystem image, I want to make sure
> >> >> that there is exactly xxx MBs for my current and future rootfs images
> >> >> (regardless of current image size). Neither --size nor --extra-space
> >> >> can do that. I could use, say `--size 200 --overhead-factor 1`, but
> >> >> this will needlessly create a 200MB rootfs image and if I happen to
> >> >> cross the 200MB boundary I will not get an error.
> >> >>
> >> >> I had a private patch that added --fixed-size to enforce --size, but
> >> >> that would still end up creating filesystem image to fill the whole
> >> >> space.
> >> > I didn't get the difference between enforcing partition size and below
> >> > implementation. Can you elaborate a bit?
> >>
> >> `--fixed-size` was something that I had added to my fork back in 2014,
> >> even before `--overhead-factor` came in. The problem is that depending
> >> on a project you might want to have more control over how partitions
> >> are laid out, or even need to have a fixed layout. Adding
> >> `--fixed-size` would had a similar effect to what `--overhead-factor
> >> 1` does right now. Combined with `--size` would ensure that rootfs is
> >> say, 200MB large. The downside was that wic would actually create a
> >> 200MB rootfs, something that is not really necessary. In fact, I only
> >> wanted to have 200MB gap so that I have some spare space for future
> >> updates (where update is just a rootfs image you dd to the partition).
> >>
> > Thanks for the explanations. Now I got it - reserved size is not a part
> > of partition, it's a gap between partitions.
> 
> I might have not been clear enough when explaining. It's not a gap,
> it's just a container of size --reserved-size listed in MBR/GPT.
> There's probably a filesystem inside but not necessarily.
> Graphically it looks as like this:
> 
>  --reserved-size
>   |--|
>   v  v
> +-+--+-+
> |. stuff .|xx|. stuff .|
> +-+--+-+
>   ^ ^^
>   |-||
>--size--extra-space
> 
> 
Ah, I'm wrong again. It's a partition size limit, but it's not necessary
formatted, right? It's only formatted if size == reserved_size.

> >
> > What's the advantage of creating unusable gap over creating partition of
> > the same size that can be used?
> 
> Just convenience.
>
What's the convenience of having extra space on partition that can't be
used for data over having it formatted and used?

> > Even if that space is not needed it doesn't harm to have it, does it?
> 
> I have not seen any negative side effects.
>
I do. If user needs that reserved space it's impossible to get it without
reformatting partition. The free space exists, but can't be used.

--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] wic: add --reserved-size wks option

[Ed Bartosh]

On Tue, Oct 18, 2016 at 10:37:22AM +0200, Maciej Borzęcki wrote:
> On Tue, Oct 18, 2016 at 9:31 AM, Ed Bartosh <ed.bart...@linux.intel.com> 
> wrote:
> > On Mon, Oct 17, 2016 at 04:46:00PM +0200, Maciej Borzęcki wrote:
> >> On Mon, Oct 17, 2016 at 3:22 PM, Ed Bartosh <ed.bart...@linux.intel.com> 
> >> wrote:
> >> > Hi Maciej,
> >> >
> >> > There is already --size and --extra-space options.
> >> > Can we get the same or similar result by just using them? Do we really
> >> > need new option for similar purpose?
> >>
> >> --reserved-size serves a different purpose, it establishes an upper
> >> bound on the size of a partition during layout. Unlike
> >> --size/--extra-space does not depend on the size of the filesystem
> >> image.
> >>
> >> For instance, assume I'm creating an image for SD card/eMMC with a
> >> fixed partition layout (something simple: boot partition, primary &
> >> secondary rootfs partitions, some data partition). Because future
> >> system updates are delivered as filesystem image, I want to make sure
> >> that there is exactly xxx MBs for my current and future rootfs images
> >> (regardless of current image size). Neither --size nor --extra-space
> >> can do that. I could use, say `--size 200 --overhead-factor 1`, but
> >> this will needlessly create a 200MB rootfs image and if I happen to
> >> cross the 200MB boundary I will not get an error.
> >>
> >> I had a private patch that added --fixed-size to enforce --size, but
> >> that would still end up creating filesystem image to fill the whole
> >> space.
> > I didn't get the difference between enforcing partition size and below
> > implementation. Can you elaborate a bit?
> 
> `--fixed-size` was something that I had added to my fork back in 2014,
> even before `--overhead-factor` came in. The problem is that depending
> on a project you might want to have more control over how partitions
> are laid out, or even need to have a fixed layout. Adding
> `--fixed-size` would had a similar effect to what `--overhead-factor
> 1` does right now. Combined with `--size` would ensure that rootfs is
> say, 200MB large. The downside was that wic would actually create a
> 200MB rootfs, something that is not really necessary. In fact, I only
> wanted to have 200MB gap so that I have some spare space for future
> updates (where update is just a rootfs image you dd to the partition).
>
Thanks for the explanations. Now I got it - reserved size is not a part
of partition, it's a gap between partitions.

What's the advantage of creating unusable gap over creating partition of
the same size that can be used? Even if that space is not needed it
doesn't harm to have it, does it?

> >> >>
> >> >> + --reserved-size: This option specifies that there should be
> >> >> +  at least that many bytes reserved for
> > According to the code if partition size > reserved size wic will throw
> > an error. This somehow conflicts with 'at least that many bytes' part of the
> > description in my opinion.
> >
> > As far as I uderstood your code this parameter specifies hard upper
> > limit for partition size, right?
> 
> Thanks, the wording is a bit off. I should rephrase it to 'that many
> bytes' or just state that it's an upper limit.
>
Upper limit is a wrong term here. I'm sorry for confusing you.
I proposed it because I misunderstood that it's not a partition size but a 
reserved
space after partition. Anyway, I think above description should be
modified to explain the option in a clear way. Using ascii picture of
the partition layout would probably help to show what this option means.

> >> >>  msger.debug("Assigned %s to %s%d, sectors range %d-%d size 
> >> >> %d "
> >> >> -"sectors (%d bytes)." \
> >> >> +"sectors (%d bytes), reserved %d bytes." \
> >> >>  % (part['mountpoint'], part['disk_name'], 
> >> >> part['num'],
> >> >> -   part['start'], part['start'] + 
> >> >> part['size'] - 1,
> >> >> -   part['size'], part['size'] * 
> >> >> self.sector_size))
> >> >> +   part['start'], disk['offset'] - 1,
> >> >> +   part['size'], part['size'] * 
> >> >> self.sector_size,
> >> >> +  

Re: [OE-core] [PATCH] wic: add --reserved-size wks option

[Ed Bartosh]

On Tue, Oct 18, 2016 at 03:59:23PM +0200, Maciej Borzęcki wrote:
> >> >> >> > What's the advantage of creating unusable gap over creating 
> >> >> >> > partition of
> >> >> >> > the same size that can be used?
> >> >> >>
> >> >> >> Just convenience.
> >> >> >>
> >> >> > What's the convenience of having extra space on partition that can't 
> >> >> > be
> >> >> > used for data over having it formatted and used?
> >> >> >
> >> >> >> > Even if that space is not needed it doesn't harm to have it, does 
> >> >> >> > it?
> >> >> >>
> >> >> >> I have not seen any negative side effects.
> >> >> >>
> >> >> > I do. If user needs that reserved space it's impossible to get it 
> >> >> > without
> >> >> > reformatting partition. The free space exists, but can't be used.
> >> >>
> >> >> That's not the point and is not aligned with use case I'm trying to 
> >> >> solve.
> >> >>
> >> >> My case is rather simple, I'm creating an image for SD card that is
> >> >> deployed in the field. In that particular case, there's a primary and
> >> >> a secondary (aka. active and inactive) rootfs partitions that are
> >> >> switched whenever a system update comes in. The update is a file
> >> >> system image that is copied over to the inactive partition, followed
> >> >> by a system reboot.
> >> >>
> >> >> What I need is the ability to set a certain size of a partition (say
> >> >> 100MB), regardless of current rootfs size (which may be, say 70MB).
> >> >> The remaining unused space sets an upper limit on how much the rootfs
> >> >> may grow in the future (in this example case, it's 30MB). RIght now
> >> >> the best I can do is to describe a partition like this: `part /
> >> >> --source rootfs --size 100MB --overhead-factor 1`, hoping that if
> >> >> rootfs grows beyond 100MB I will somehow still be able to catch that
> >> >> and that the future images remain size compatible.
> >> >>
> >> >> The resulting filesystem inside the partition is larger than what
> >> >> IMAGE_CMD (ex. IMAGE_CMD_ext4) would give me, because of explicit
> >> >> --size in kickstart. I would prefer to have something comparable in
> >> >> size just to avoid later surprises, what implies using defaults.
> >> >> However, using defaults, means that I cannot control the layout
> >> >> because it will likely change each time rootfs size gets changed.
> >> >> There is no `--fixed-size` or other option to enforce specific size.
> >> >>
> >> >> Summing up, a simple use case that cannot be currently solved using wic.
> >> >>
> >> >> BTW. actually we're missing an ability to enforce --size (i.e.
> >> >> --fixed-size?) and a method passing an explicit partition offset
> >> >> inside the disk image (something useful for `--source rawcopy
> >> >> --no-table` partitions, currently solved with `--align`).
> >> >>
> >> > I undertood the problem and I agree that wic doesn't provide a solution.
> >> >
> >> > However, instead of making unformatted space I'd propose to format it,
> >> > i.e. to have --max-size option that would confict with --size and
> >> > specify upper size limit for the partition. All partition will be
> >> > formatted and available for data. This is identical to --fixed-size 
> >> > option
> >> > you've described. This approach would solve the problem you're
> >> > addressing and it would also make additional space usable.
> >> >
> >> > I'd also suggest to rename --size to --min-size and make --size 
> >> > deprecated.
> >> >
> >> > Does this make sense to you?
> >>
> >> No strong opinions here, just that deprecating --size might current
> >> users uneasy.
> >>
> > By deprecting --size I didn't mean removing it completely. We can just
> > print a warning suggesting usage of other options.
> >
> >> Perhaps --max-size could be a boolean switch? We could just name it
> >> --fixed-size (bool, defaults to False), with semantics that if
> >> --fixed-size is provided, the partition will have size --size,
> >> occurrence of --overhead-factor or --extra-space will raise an error.
> >>
> > That would work too, but it looks a bit confusing to me to have 2 different
> > types of size-related options.
> 
> Ok, but now we would have --min-size (previously known as --size) and
> --size (or --max-size?). That's still 2 size related options plus a
> deprecation warning.
> 

I agree, it doesn't look good. Moreover --max-size doesn't make it
clear that partition will be this size. It rather suggests that partition
can be this size or less.

So, we have --size which is actually minimum partition size, we have
couple of options to extend partition size (--overhead-factor and
--extra-space), but we don't have ability to set upper limit for partition
size.

OK, let's agree on using --fixed-size(int)
Using --fixed-size with any of --size or --overhead-factor or --extra-space
options should raise ks parser error. If rootfs size is bigger than
--fixed-size wic should raise an error too.

Any other opinions?

--
Regards,
Ed
-- 
___

Re: [OE-core] [wic][PATCH] systemd-bootdisk.wks: use PARTUUID

[Ed Bartosh]

On Mon, Oct 24, 2016 at 10:34:40AM -0700, Saul Wold wrote:
> On Mon, 2016-10-24 at 16:54 +0300, Ed Bartosh wrote:
> > Root device name in systemd-bootdisk.wks is 'sda'. This can cause
> > images, produced using this wks to refuse booting if real device
> > name is not 'sda'. For example, when booting MinnowBoard MAX from
> > MicroSD card the boot process stucks with this message on the boot
> > console output: Waiting for root device /dev/sda2...
> > This happens because real device name of MicroSD card on this device
> > is mmcblk1.
> > 
> > Used --use-uuid option for root partition. This should make
> > wic to put partiion UUID instead of device name into kernel command
> > line.
> > 
> > [YOCTO #10485]
> > 
> > Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
> > ---
> >  scripts/lib/wic/canned-wks/systemd-bootdisk.wks | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/scripts/lib/wic/canned-wks/systemd-bootdisk.wks
> > b/scripts/lib/wic/canned-wks/systemd-bootdisk.wks
> > index b900023..4bd9d6a 100644
> > --- a/scripts/lib/wic/canned-wks/systemd-bootdisk.wks
> > +++ b/scripts/lib/wic/canned-wks/systemd-bootdisk.wks
> > @@ -4,7 +4,7 @@
> >  
> >  part /boot --source bootimg-efi --sourceparams="loader=systemd-boot" 
> > --ondisk sda --label msdos --active --align 1024
> >  
> > -part / --source rootfs --ondisk sda --fstype=ext4 --label platform
> > --align 1024
> > +part / --source rootfs --ondisk sda --fstype=ext4 --label platform
> > --align 1024 --use-uuid
> >  
>  part swap --ondisk sda --size 44 --label swap1 --fstype=swap
> >  
> I understand the --use-uuid for the partition labeling, but how does it
> affect everything marked as ondisk sda?  Or is ondisk ignored with use-
> uuid?

--use-uuid uses PARTUUID to specify root device in kernel command line
instead of /dev/sdaX. In this sense yes, sda is ignored.

However, disk name(sda) is also used to group partitions to one disk image.
Theoretically wic can produce several images if more than one disk name
is used in .wks file.

For swap position sda is still used, which can make swap partitions not
functional on the devices with different disk names. This needs to be fixed
by using PARTUUID in fstab for swap partitions. For this to work in poky images
busybox swapon should support PARTUUID syntax, which I'm not entirely sure it 
does.

--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [wic][PATCH] systemd-bootdisk.wks: use PARTUUID

[Ed Bartosh]

Root device name in systemd-bootdisk.wks is 'sda'. This can cause
images, produced using this wks to refuse booting if real device
name is not 'sda'. For example, when booting MinnowBoard MAX from
MicroSD card the boot process stucks with this message on the boot
console output: Waiting for root device /dev/sda2...
This happens because real device name of MicroSD card on this device
is mmcblk1.

Used --use-uuid option for root partition. This should make
wic to put partiion UUID instead of device name into kernel command
line.

[YOCTO #10485]

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 scripts/lib/wic/canned-wks/systemd-bootdisk.wks | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/lib/wic/canned-wks/systemd-bootdisk.wks 
b/scripts/lib/wic/canned-wks/systemd-bootdisk.wks
index b900023..4bd9d6a 100644
--- a/scripts/lib/wic/canned-wks/systemd-bootdisk.wks
+++ b/scripts/lib/wic/canned-wks/systemd-bootdisk.wks
@@ -4,7 +4,7 @@
 
 part /boot --source bootimg-efi --sourceparams="loader=systemd-boot" --ondisk 
sda --label msdos --active --align 1024
 
-part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024
+part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 
1024 --use-uuid
 
 part swap --ondisk sda --size 44 --label swap1 --fstype=swap
 
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v1] wic: Add --exclude-path option to rootfs source plugin.

[Ed Bartosh]

On Fri, Nov 25, 2016 at 01:07:34PM +0100, Kristian Amlie wrote:
> On 25/11/16 11:33, Patrick Ohly wrote:
> > On Fri, 2016-11-25 at 11:15 +0100, Kristian Amlie wrote:
> >> +if os.stat(real_rootfs_dir).st_dev ==
> >> os.stat(cr_workdir).st_dev:
> >> +# Optimization if both directories are on the same
> >> file system:
> >> +# copy using hardlinks.
> >> +cp_args = "-al"
> >> +else:
> >> +cp_args = "-a"
> >> +exec_cmd("cp %s %s %s" % (cp_args, real_rootfs_dir,
> >> new_rootfs))
> > 
> > Not a full review (I'll leave that to Ed), just one thing which caught
> > my eye: when the rootfs contains xattrs, they get lost here.
> > 
> > Use oe.path.copyhardlinktree() instead, it also does the hardlinking
> > trick.
> 
> Thanks, that's a good tip! I'll include that in the next patchset.
> 

Thank you for so fast implementation!

Sorry for not answering on original e-mail. I've lost it somehow.

My comments so far:

What's the reason of insisting that path must be absolute?
May be it's just me, but I find it a bit scaring to use absolute path in .wks
The patch is relative to the rootfs directory from my point of view.

It also looks quite strange in the code to insist on absolute path
+if not os.path.isabs(path):
+msger.error("Must be absolute: --exclude-path=%s" %

and then immediately making it relative:
+
+while os.path.isabs(path):
+path = path[1:]


I know, this is just a matter of taste, but I'd not use brackets around
head, tail here. They're redundant and the code looks better without
them from my point of view.
+(head, tail) = os.path.split(remaining)

This causes rmtree to throw NotADirectoryError on files:
+for entry in os.listdir(full_path):
+shutil.rmtree(os.path.join(full_path, entry))

--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v1] wic: Add --exclude-path option to rootfs source plugin.

[Ed Bartosh]

On Fri, Nov 25, 2016 at 01:35:53PM +0100, Kristian Amlie wrote:
> On 25/11/16 13:28, Maciej Borzęcki wrote:
> > On Fri, Nov 25, 2016 at 11:15 AM, Kristian Amlie
> >> +# Disallow '..', because doing so could be quite 
> >> disastrous
> >> +# (we will delete the directory).
> >> +remaining = path
> >> +while True:
> >> +(head, tail) = os.path.split(remaining)
> >> +if tail == '..':
> >> +msger.error("'..' not allowed: --exclude-path=%s" 
> >> % orig_path)
> >> +elif head == "":
> >> +break
> >> +remaining = head
> > 
> > Why not do this instead?
> > 
> > if '..' in path:
> > msger.error("'..' not allowed: --exclude-path=%s" % orig_path)
> > 
would "'/..' in path" or something similar work?

--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] Contents of non-rootfs partitions

[Ed Bartosh]

On Tue, Nov 22, 2016 at 12:54:52PM +0100, Kristian Amlie wrote:
> On 22/11/16 12:10, Patrick Ohly wrote:
> >> ...
> > 
> > All of these introduce some special mechanism. Let me propose something
> > that might integrate better with the existing tooling:
> > 
> > The "rootfs" directory gets redefined as representing the entire virtual
> > file system. When creating a disk image, it gets split up into different
> > partitions based on the image configuration.
> > 
> > For example, the /home or /data directories in the rootfs could hold the
> > content that in some image configurations goes into separate partitions.
> > 
> > The advantage of this approach is that the tooling for staging content
> > for image creation does not need to be changed. The same staged content
> > then can be used to create different images, potentially even using
> > different partition layouts.
> 
> That's a very good idea. I think it beats all of my suggestions!
> 
> > To implement this approach with wic, wic needs to be taught how to
> > exclude directories from the main rootfs. Ideally, the mkfs.* tools
> > should also support that without having to make an intermediate copy of
> > the files for a certain partition, but initially wic could create
> > temporary directory trees.
> 
> Yes, some work would be needed here, but ultimately it would be contained 
> within wic and related tools, which is a good thing.
>

I support the idea. Let's discuss the details of implementation and
create a bug in bugzilla to track the development 

This can be done by extending existing rootfs plugin. It should be able
to do 2 things:

- populate content of one rootfs directory to the partition. We can
  extend syntax of --rootfs-dir parameter to specify optional directory path to 
use

- exclude rootfs directories when populating partitions. I'd propose to
  introduce --exclude-dirs wks parser option to handle this.

Example of wks file with proposed new options:
part / --source rootfs --rootfs-dir=core-image-minimal   --ondisk sda 
--fstype=ext4 --label root --align 1024 --exclude-dirs data --exclude-dirs home
part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk sda 
--fstype=ext4 --label data --align 1024
part /home --source rootfs --rootfs-dir=core-image-minimal:/data --ondisk sda 
--fstype=ext4 --label data --align 1024

Does this make sense?

Any other ideas?

--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v4 5/7] wic: selftest: do not repeat core-image-minimal

[Ed Bartosh]

On Wed, Nov 23, 2016 at 12:39:26PM +0100, Maciej Borzęcki wrote:
> On Wed, Nov 23, 2016 at 12:23 PM, Ed Bartosh <ed.bart...@linux.intel.com> 
> wrote:
> > On Wed, Nov 23, 2016 at 08:46:31AM +0100, Maciej Borzecki wrote:
> >> Replace repeated core-image-minimal with Wic class field.
> >>
> >> Signed-off-by: Maciej Borzecki <maciej.borze...@rndity.com>
> >> ---
> >>  meta/lib/oeqa/selftest/wic.py | 111 
> >> +++---
> >>  1 file changed, 73 insertions(+), 38 deletions(-)
> >>
> >> diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
> >> index 
> >> 2db14445956bc5adcf1e755844bbdb69edcb468f..37ed2c6de5a7f22f982f921476fa392304995b2e
> >>  100644
> >> --- a/meta/lib/oeqa/selftest/wic.py
> >> +++ b/meta/lib/oeqa/selftest/wic.py
> >> @@ -57,6 +57,8 @@ class Wic(oeSelfTest):
> >>  resultdir = "/var/tmp/wic/build/"
> >>  image_is_ready = False
> >>
> >> +OE_IMAGE = "core-image-minimal"
> >> +
> >>  def setUpLocal(self):
> >>  """This code is executed before each test method."""
> >>  arch = get_bb_var('HOST_ARCH', 'core-image-minimal')
> >> @@ -75,7 +77,7 @@ class Wic(oeSelfTest):
> >>  tools += ' syslinux syslinux-native'
> >>  bitbake(tools)
> >>
> >> -bitbake('core-image-minimal')
> >> +bitbake(self.OE_IMAGE)
> >>  Wic.image_is_ready = True
> > I'd agree with Ross here - it looks less readable.
> > How it would look if we decide to add test case for another image?
> >
> 
> The usual way would be to:
> 
> class WicSatoImage(Wic):
> OE_IMAGE = "core-image-sato"
> 
> But I think you make a fair point. The patch introduces unnecessary
> complexity, and is not really needed right now. I'll drop this patch if next
> version will be necessary.
> 
> Can you take a look at the rest of the patchset as well?
>
I already did. The patchset looks good to me. Thank you for your work!
+1

--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] Contents of non-rootfs partitions

[Ed Bartosh]

On Wed, Nov 23, 2016 at 02:08:28PM +0100, Kristian Amlie wrote:
> On 23/11/16 13:08, Ed Bartosh wrote:
> > On Tue, Nov 22, 2016 at 12:54:52PM +0100, Kristian Amlie wrote:
> >> On 22/11/16 12:10, Patrick Ohly wrote:
> >>>> ...
> >>>
> >>> All of these introduce some special mechanism. Let me propose something
> >>> that might integrate better with the existing tooling:
> >>>
> >>> The "rootfs" directory gets redefined as representing the entire virtual
> >>> file system. When creating a disk image, it gets split up into different
> >>> partitions based on the image configuration.
> >>>
> >>> For example, the /home or /data directories in the rootfs could hold the
> >>> content that in some image configurations goes into separate partitions.
> >>>
> >>> The advantage of this approach is that the tooling for staging content
> >>> for image creation does not need to be changed. The same staged content
> >>> then can be used to create different images, potentially even using
> >>> different partition layouts.
> >>
> >> That's a very good idea. I think it beats all of my suggestions!
> >>
> >>> To implement this approach with wic, wic needs to be taught how to
> >>> exclude directories from the main rootfs. Ideally, the mkfs.* tools
> >>> should also support that without having to make an intermediate copy of
> >>> the files for a certain partition, but initially wic could create
> >>> temporary directory trees.
> >>
> >> Yes, some work would be needed here, but ultimately it would be contained 
> >> within wic and related tools, which is a good thing.
> >>
> > 
> > I support the idea. Let's discuss the details of implementation and
> > create a bug in bugzilla to track the development 
> 
> Do you want me to create the ticket? (it'll be my first, so apologies in
> advance if I omit something important)
> 
Yes, please create an issue in bugzilla and assign it to me. Don't
worry. If you miss something other people will point to in in the
comments :)

--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] Contents of non-rootfs partitions

[Ed Bartosh]

On Wed, Nov 23, 2016 at 02:08:28PM +0100, Kristian Amlie wrote:
> On 23/11/16 13:08, Ed Bartosh wrote:
> > On Tue, Nov 22, 2016 at 12:54:52PM +0100, Kristian Amlie wrote:
> >> On 22/11/16 12:10, Patrick Ohly wrote:
> >>>> ...
> >>>
> >>> All of these introduce some special mechanism. Let me propose something
> >>> that might integrate better with the existing tooling:
> >>>
> >>> The "rootfs" directory gets redefined as representing the entire virtual
> >>> file system. When creating a disk image, it gets split up into different
> >>> partitions based on the image configuration.
> >>>
> >>> For example, the /home or /data directories in the rootfs could hold the
> >>> content that in some image configurations goes into separate partitions.
> >>>
> >>> The advantage of this approach is that the tooling for staging content
> >>> for image creation does not need to be changed. The same staged content
> >>> then can be used to create different images, potentially even using
> >>> different partition layouts.
> >>
> >> That's a very good idea. I think it beats all of my suggestions!
> >>
> >>> To implement this approach with wic, wic needs to be taught how to
> >>> exclude directories from the main rootfs. Ideally, the mkfs.* tools
> >>> should also support that without having to make an intermediate copy of
> >>> the files for a certain partition, but initially wic could create
> >>> temporary directory trees.
> >>
> >> Yes, some work would be needed here, but ultimately it would be contained 
> >> within wic and related tools, which is a good thing.
> >>
> > 
> > I support the idea. Let's discuss the details of implementation and
> > create a bug in bugzilla to track the development 
> 
> Do you want me to create the ticket? (it'll be my first, so apologies in
> advance if I omit something important)
> 
> > This can be done by extending existing rootfs plugin. It should be able
> > to do 2 things:
> > 
> > - populate content of one rootfs directory to the partition. We can
> >   extend syntax of --rootfs-dir parameter to specify optional directory 
> > path to use
> > 
> > - exclude rootfs directories when populating partitions. I'd propose to
> >   introduce --exclude-dirs wks parser option to handle this.
> > 
> > Example of wks file with proposed new options:
> > part / --source rootfs --rootfs-dir=core-image-minimal   --ondisk 
> > sda --fstype=ext4 --label root --align 1024 --exclude-dirs data 
> > --exclude-dirs home
> > part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk 
> > sda --fstype=ext4 --label data --align 1024
> > part /home --source rootfs --rootfs-dir=core-image-minimal:/data --ondisk 
> > sda --fstype=ext4 --label data --align 1024
> > 
> > Does this make sense?
> 
> Looks good. The only thing I would question is that, in the interest of
> reducing redundancy, maybe we should omit --exclude-dirs and have wic
> figure this out by combining all the entries, since "--exclude-dirs
> " and the corresponding "part " will almost always come in
> pairs. Possibly we could mark the "/" partition with one single
> --no-overlapping-dirs to force wic to make this consideration. Or do you
> think that's too magical?
>
Tt's quite implicit from my point of view. However, if people like it we
can implement it this way.

> (I haven't checked how feasible this is in the code btw)

I think it would be much easier to implement --exclude-dirs.

BTW, it will also allow to exclude directories from any partition, not just from
root, e.g. to use --exclude dirs this way:
part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk sda 
--fstype=ext4 --label data --align 1024 --exclude-dirs cache --exclude-dirs tmp

--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v4 7/7] wic: selftest: add tests for --fixed-size partition flags

[Ed Bartosh]

Hi Maciej,

Thank you for the patchset!
The changes and the tests look good to me.
I have little suggestions, but most of them just a matter of taste, so
feel free to ignore them.

On Wed, Nov 23, 2016 at 08:46:33AM +0100, Maciej Borzecki wrote:
> wic has a new flag for setting a fixed parition size --fixed-size. Add
> tests that verify if partition is indeed sized properly and that errors
> are signaled when there is not enough space to fit partition data.
> 
> Signed-off-by: Maciej Borzecki 
> ---
>  meta/lib/oeqa/selftest/wic.py | 69 
> +++
>  1 file changed, 69 insertions(+)
> 
> diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
> index 
> ad783043b92130a023fd70120becec479c6253a7..45f68df1e74828e11401f57dd732a88a50dd1f00
>  100644
> --- a/meta/lib/oeqa/selftest/wic.py
> +++ b/meta/lib/oeqa/selftest/wic.py
> @@ -29,6 +29,7 @@ import unittest
>  from glob import glob
>  from shutil import rmtree
>  from functools import wraps
> +from tempfile import NamedTemporaryFile
>  
>  from oeqa.selftest.base import oeSelfTest
>  from oeqa.utils.commands import runCmd, bitbake, get_bb_var, runqemu
> @@ -61,6 +62,8 @@ class Wic(oeSelfTest):
>  
>  def setUpLocal(self):
>  """This code is executed before each test method."""
> +self.native_sysroot = get_bb_var('STAGING_DIR_NATIVE', 
> 'core-image-minimal')
> +
>  arch = get_bb_var('HOST_ARCH', 'core-image-minimal')
>  is_x86 = arch in ['i586', 'i686', 'x86_64']
>  if is_x86:
> @@ -378,3 +381,69 @@ class Wic(oeSelfTest):
>  self.assertEqual(0, runCmd("wic create %(wks)s -e %(image)s" \
> % wic_cmd_vars).status)
>  self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image)))
> +
> +def _make_fixed_size_wks(self, size):
> +"""
> +Create a wks of an image with a single partition. Size of the 
> partition is set
> +using --fixed-size flag. Returns a tuple: (path to wks file, wks 
> image name)
> +"""
> +with NamedTemporaryFile("w", suffix=".wks", delete=False) as tf:
> +wkspath = tf.name
> +tf.write("part " \
> + "--source rootfs --ondisk hda --align 4 --fixed-size %d 
> "
> + "--fstype=ext4\n" % size)
> +wksname = os.path.splitext(os.path.basename(wkspath))[0]
> +
> +return (wkspath, wksname)
Would 'return wkspath, wksname' be a bit more readable?

> +
> +def test_fixed_size(self):
> +"""
> +Test creation of a simple image with partition size controlled 
> through
> +--fixed-size flag
> +"""
> +wkspath, wksname = self._make_fixed_size_wks(200)
> +
> +wic_cmd_vars = {
> +'wks': wkspath,
> +'image': self.OE_IMAGE,
> +}
> +self.assertEqual(0, runCmd("wic create %(wks)s -e %(image)s" \
> +   % wic_cmd_vars).status)
> +os.remove(wkspath)
> +wicout = glob(self.resultdir + "%s-*direct" % wksname)
> +self.assertEqual(1, len(wicout))
> +
> +wicimg = wicout[0]
> +
> +# verify partition size with wic
> +res = runCmd("parted -m %s unit mib p 2>/dev/null" % wicimg,
> + ignore_status=True,
> + native_sysroot=self.native_sysroot)
> +self.assertEqual(0, res.status)
> +
> +# parse parted output which looks like this:
> +# BYT;\n
> +# 
> /var/tmp/wic/build/tmpfwvjjkf_-201611101222-hda.direct:200MiB:file:512:512:msdos::;\n
> +# 1:0.00MiB:200MiB:200MiB:ext4::;\n
> +partlns = res.output.splitlines()[2:]
> +
> +self.assertEqual(1, len(partlns))
> +self.assertEqual("1:0.00MiB:200MiB:200MiB:ext4::;", partlns[0])
> +
> +def test_fixed_size_error(self):
> +"""
> +Test creation of a simple image with partition size controlled 
> through
> +--fixed-size flag. The size of partition is intentionally set to 1MiB
> +in order to trigger an error in wic.
> +"""
> +wkspath, wksname = self._make_fixed_size_wks(1)
> +
> +wic_cmd_vars = {
> +'wks': wkspath,
> +'image': self.OE_IMAGE,
> +}
> +self.assertEqual(1, runCmd("wic create %(wks)s -e %(image)s" \
> +   % wic_cmd_vars, 
> ignore_status=True).status)
> +os.remove(wkspath)
> +wicout = glob(self.resultdir + "%s-*direct" % wksname)
> +self.assertEqual(0, len(wicout))

--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v4 5/7] wic: selftest: do not repeat core-image-minimal

[Ed Bartosh]

On Wed, Nov 23, 2016 at 08:46:31AM +0100, Maciej Borzecki wrote:
> Replace repeated core-image-minimal with Wic class field.
> 
> Signed-off-by: Maciej Borzecki 
> ---
>  meta/lib/oeqa/selftest/wic.py | 111 
> +++---
>  1 file changed, 73 insertions(+), 38 deletions(-)
> 
> diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
> index 
> 2db14445956bc5adcf1e755844bbdb69edcb468f..37ed2c6de5a7f22f982f921476fa392304995b2e
>  100644
> --- a/meta/lib/oeqa/selftest/wic.py
> +++ b/meta/lib/oeqa/selftest/wic.py
> @@ -57,6 +57,8 @@ class Wic(oeSelfTest):
>  resultdir = "/var/tmp/wic/build/"
>  image_is_ready = False
>  
> +OE_IMAGE = "core-image-minimal"
> +
>  def setUpLocal(self):
>  """This code is executed before each test method."""
>  arch = get_bb_var('HOST_ARCH', 'core-image-minimal')
> @@ -75,7 +77,7 @@ class Wic(oeSelfTest):
>  tools += ' syslinux syslinux-native'
>  bitbake(tools)
>  
> -bitbake('core-image-minimal')
> +bitbake(self.OE_IMAGE)
>  Wic.image_is_ready = True
I'd agree with Ross here - it looks less readable.
How it would look if we decide to add test case for another image?

>  
>  rmtree(self.resultdir, ignore_errors=True)
> @@ -100,14 +102,14 @@ class Wic(oeSelfTest):
>  def test_build_image_name(self):
>  """Test wic create directdisk --image-name core-image-minimal"""
>  self.assertEqual(0, runCmd("wic create directdisk "
> -   "--image-name core-image-minimal").status)
> +   "--image-name %s" % self.OE_IMAGE).status)
>  self.assertEqual(1, len(glob(self.resultdir + 
> "directdisk-*.direct")))
>  
>  @testcase(1212)
>  @onlyForArch('i586', 'i686', 'x86_64')
>  def test_build_artifacts(self):
>  """Test wic create directdisk providing all artifacts."""
> -bbvars = dict((var.lower(), get_bb_var(var, 'core-image-minimal')) \
> +bbvars = dict((var.lower(), get_bb_var(var, self.OE_IMAGE)) \
>  for var in ('STAGING_DATADIR', 'DEPLOY_DIR_IMAGE',
>  'STAGING_DIR_NATIVE', 'IMAGE_ROOTFS'))
>  status = runCmd("wic create directdisk "
> @@ -123,7 +125,7 @@ class Wic(oeSelfTest):
>  def test_gpt_image(self):
>  """Test creation of core-image-minimal with gpt table and UUID 
> boot"""
>  self.assertEqual(0, runCmd("wic create directdisk-gpt "
> -   "--image-name core-image-minimal").status)
> +   "--image-name %s" % self.OE_IMAGE).status)
>  self.assertEqual(1, len(glob(self.resultdir + 
> "directdisk-*.direct")))
>  
>  @testcase(1213)
> @@ -157,8 +159,8 @@ class Wic(oeSelfTest):
>  def test_compress_gzip(self):
>  """Test compressing an image with gzip"""
>  self.assertEqual(0, runCmd("wic create directdisk "
> -   "--image-name core-image-minimal "
> -   "-c gzip").status)
> +   "--image-name %s "
> +   "-c gzip" % self.OE_IMAGE).status)
>  self.assertEqual(1, len(glob(self.resultdir + \
>   "directdisk-*.direct.gz")))
>  
> @@ -167,8 +169,8 @@ class Wic(oeSelfTest):
>  def test_compress_bzip2(self):
>  """Test compressing an image with bzip2"""
>  self.assertEqual(0, runCmd("wic create directdisk "
> -   "--image-name core-image-minimal "
> -   "-c bzip2").status)
> +   "--image-name %s "
> +   "-c bzip2" % self.OE_IMAGE).status)
>  self.assertEqual(1, len(glob(self.resultdir + \
>   "directdisk-*.direct.bz2")))
>  
> @@ -177,8 +179,8 @@ class Wic(oeSelfTest):
>  def test_compress_xz(self):
>  """Test compressing an image with xz"""
>  self.assertEqual(0, runCmd("wic create directdisk "
> -   "--image-name core-image-minimal "
> -   "-c xz").status)
> +   "--image-name %s "
> +   "-c xz" % self.OE_IMAGE).status)
>  self.assertEqual(1, len(glob(self.resultdir + \
>   "directdisk-*.direct.xz")))
>  
> @@ -187,26 +189,31 @@ class Wic(oeSelfTest):
>  def test_wrong_compressor(self):
>  """Test how wic breaks if wrong compressor is provided"""
>  self.assertEqual(2, runCmd("wic create directdisk "
> -   "--image-name core-image-minimal "
> -   "-c wrong", 

Re: [OE-core] Contents of non-rootfs partitions

[Ed Bartosh]

On Wed, Nov 23, 2016 at 02:51:20PM +0100, Maciej Borzęcki wrote:
> On Wed, Nov 23, 2016 at 2:22 PM, Ed Bartosh <ed.bart...@linux.intel.com> 
> wrote:
> > On Wed, Nov 23, 2016 at 02:08:28PM +0100, Kristian Amlie wrote:
> >> On 23/11/16 13:08, Ed Bartosh wrote:
> >> > On Tue, Nov 22, 2016 at 12:54:52PM +0100, Kristian Amlie wrote:
> >> >> On 22/11/16 12:10, Patrick Ohly wrote:
> >> >>>> ...
> >> >>>
> >> >>> All of these introduce some special mechanism. Let me propose something
> >> >>> that might integrate better with the existing tooling:
> >> >>>
> >> >>> The "rootfs" directory gets redefined as representing the entire 
> >> >>> virtual
> >> >>> file system. When creating a disk image, it gets split up into 
> >> >>> different
> >> >>> partitions based on the image configuration.
> >> >>>
> >> >>> For example, the /home or /data directories in the rootfs could hold 
> >> >>> the
> >> >>> content that in some image configurations goes into separate 
> >> >>> partitions.
> >> >>>
> >> >>> The advantage of this approach is that the tooling for staging content
> >> >>> for image creation does not need to be changed. The same staged content
> >> >>> then can be used to create different images, potentially even using
> >> >>> different partition layouts.
> >> >>
> >> >> That's a very good idea. I think it beats all of my suggestions!
> >> >>
> >> >>> To implement this approach with wic, wic needs to be taught how to
> >> >>> exclude directories from the main rootfs. Ideally, the mkfs.* tools
> >> >>> should also support that without having to make an intermediate copy of
> >> >>> the files for a certain partition, but initially wic could create
> >> >>> temporary directory trees.
> >> >>
> >> >> Yes, some work would be needed here, but ultimately it would be 
> >> >> contained within wic and related tools, which is a good thing.
> >> >>
> >> >
> >> > I support the idea. Let's discuss the details of implementation and
> >> > create a bug in bugzilla to track the development
> >>
> >> Do you want me to create the ticket? (it'll be my first, so apologies in
> >> advance if I omit something important)
> >>
> >> > This can be done by extending existing rootfs plugin. It should be able
> >> > to do 2 things:
> >> >
> >> > - populate content of one rootfs directory to the partition. We can
> >> >   extend syntax of --rootfs-dir parameter to specify optional directory 
> >> > path to use
> >> >
> >> > - exclude rootfs directories when populating partitions. I'd propose to
> >> >   introduce --exclude-dirs wks parser option to handle this.
> >> >
> >> > Example of wks file with proposed new options:
> >> > part / --source rootfs --rootfs-dir=core-image-minimal   
> >> > --ondisk sda --fstype=ext4 --label root --align 1024 --exclude-dirs data 
> >> > --exclude-dirs home
> >> > part /data --source rootfs --rootfs-dir=core-image-minimal:/home 
> >> > --ondisk sda --fstype=ext4 --label data --align 1024
> >> > part /home --source rootfs --rootfs-dir=core-image-minimal:/data 
> >> > --ondisk sda --fstype=ext4 --label data --align 1024
> >> >
> >> > Does this make sense?
> >>
> >> Looks good. The only thing I would question is that, in the interest of
> >> reducing redundancy, maybe we should omit --exclude-dirs and have wic
> >> figure this out by combining all the entries, since "--exclude-dirs
> >> " and the corresponding "part " will almost always come in
> >> pairs. Possibly we could mark the "/" partition with one single
> >> --no-overlapping-dirs to force wic to make this consideration. Or do you
> >> think that's too magical?
> >>
> > Tt's quite implicit from my point of view. However, if people like it we
> > can implement it this way.
> >
> >> (I haven't checked how feasible this is in the code btw)
> >
> > I think it would be much easier to implement --exclude-dirs.
> >
> > BTW, it will also allow to exclude directories from any partition, not just 
> > from
> > root, e.g. to use --exclude dirs this way:
> > part /data --source rootfs --rootfs-dir=core-image-minimal:/home --ondisk 
> > sda --fstype=ext4 --label data --align 1024 --exclude-dirs cache 
> > --exclude-dirs tmp
> 
> Just to be sure, --exclude-dirs means that the contents of the
> directory are skipped, not the directory itself?
>
Thanks for pointing out to this. I didn't think about it, but after you
pointed out to it I think it makes sense to keep the directory and only skip the
content.

Another way to do it is to skip directory and content and create mount points
mentioned in wks file when generating /etc/fstab. However, in this case
directory permissions and ownership will be lost.

--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] Contents of non-rootfs partitions

[Ed Bartosh]

On Thu, Nov 24, 2016 at 08:38:46AM +0100, Kristian Amlie wrote:
> On 24/11/16 07:15, Ulrich Ölmann wrote:
> > Hi,
> > 
> > On Wed, Nov 23, 2016 at 04:56:56PM +0100, Patrick Ohly wrote:
> >> On Wed, 2016-11-23 at 15:22 +0200, Ed Bartosh wrote:
> >>> On Wed, Nov 23, 2016 at 02:08:28PM +0100, Kristian Amlie wrote:
> >>>> On 23/11/16 13:08, Ed Bartosh wrote:
> >>>>> On Tue, Nov 22, 2016 at 12:54:52PM +0100, Kristian Amlie wrote:
> >>>>> [...]
> >>>>> This can be done by extending existing rootfs plugin. It should be able
> >>>>> to do 2 things:
> >>>>>
> >>>>> - populate content of one rootfs directory to the partition. We can
> >>>>>   extend syntax of --rootfs-dir parameter to specify optional directory 
> >>>>> path to use
> >>>>>
> >>>>> - exclude rootfs directories when populating partitions. I'd propose to
> >>>>>   introduce --exclude-dirs wks parser option to handle this.
> >>>>>
> >>>>> Example of wks file with proposed new options:
> >>>>> part / --source rootfs --rootfs-dir=core-image-minimal   
> >>>>> --ondisk sda --fstype=ext4 --label root --align 1024 --exclude-dirs 
> >>>>> data --exclude-dirs home
> >>>>> part /data --source rootfs --rootfs-dir=core-image-minimal:/home 
> >>>>> --ondisk sda --fstype=ext4 --label data --align 1024
> >>>>> part /home --source rootfs --rootfs-dir=core-image-minimal:/data 
> >>>>> --ondisk sda --fstype=ext4 --label data --align 1024
> >>>>>
> >>>>> Does this make sense?
> >>>>
> >>>> Looks good. The only thing I would question is that, in the interest of
> >>>> reducing redundancy, maybe we should omit --exclude-dirs and have wic
> >>>> figure this out by combining all the entries, since "--exclude-dirs
> >>>> " and the corresponding "part " will almost always come in
> >>>> pairs. Possibly we could mark the "/" partition with one single
> >>>> --no-overlapping-dirs to force wic to make this consideration. Or do you
> >>>> think that's too magical?
> >>>>
> >>> Tt's quite implicit from my point of view. However, if people like it we
> >>> can implement it this way.
> >>
> >> I prefer the explicit --exclude-dirs. It's less surprising and perhaps
> >> there are usages for having the same content in different partitions
> >> (redundancy, factory reset, etc.).
> >>
> >> Excluding only the directory content but not the actual directory is
> >> indeed a good point. I'm a bit undecided. When excluding only the
> >> directory content, there's no way of building a rootfs without that
> >> mount point, if that's desired. OTOH, when excluding also the directory,
> >> the data would have to be staged under a different path in the rootfs
> >> and the mount point would have to be a separate, empty directory.
> >>
> >> I'm leaning towards excluding the directory content and keeping the
> >> directory.
> > 
> > what about having both possibilities by leaning against the syntax that 
> > rsync
> > uses to specify if a whole source directory or only it's contents shall be
> > synced to some destination site (see [1])?
> > 
> > In analogy to this to exclude only the contents of the directory named 
> > 'data'
> > you would use
> > 
> >   --exclude-dirs data/
> > 
> > but to additionally exclude the dir itself as well it would read
> > 
> >   --exclude-dirs data
> 
> This is creative, but ultimately too unintuitive IMHO. Rsync is the only
> tool which uses this syntax AFAIK, and it's a constant source of
> confusion, especially when mixed with cp or similar commands.
> 

Would this way be less intuitive?
--exclude-path data/*
--exclude-path data

We can go even further with it allowing any level of directories:
--exclude-path data/tmp/*
--exclude-path data/db/tmp
...

--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v1] wic: Add --exclude-path option to rootfs source plugin.

[Ed Bartosh]

On Mon, Nov 28, 2016 at 12:01:09PM +0100, Kristian Amlie wrote:
> On 28/11/16 11:52, Ed Bartosh wrote:
> > On Mon, Nov 28, 2016 at 08:15:26AM +0100, Kristian Amlie wrote:
> >>>
> >>> What's the reason of insisting that path must be absolute?
> >>> May be it's just me, but I find it a bit scaring to use absolute path in 
> >>> .wks
> >>> The patch is relative to the rootfs directory from my point of view.
> >>>
> >>> It also looks quite strange in the code to insist on absolute path
> >>> +if not os.path.isabs(path):
> >>> +msger.error("Must be absolute: --exclude-path=%s" %
> >>>
> >>> and then immediately making it relative:
> >>> +
> >>> +while os.path.isabs(path):
> >>> +path = path[1:]
> >>
> >> Not really any strong reason. I just thought it was a logical thing to
> >> do from a user perspective: When you're making an image you're thinking
> >> about paths in the final image, and the path after "part" is absolute,
> >> so I thought this one should be too.
> >>
> >> The fact that it's made relative in the code is just an implementation
> >> detail to make join() work correctly.
> >>
> >> I'm fine either way, so just let me know which you prefer.
> >>
> > I'd prefer relative path as I think about --exclude-path as a path in
> > the source rootfs directory that we want to exclude from copying to
> > the target partition. However, I agree that that it can be also treated
> > as a path to the target directory.
> 
> Ok!
> 
> I will get a new patchset up soon.
> 

Great!

I'd appreciate if you also do the following before sending v2:
 - check that your code doesn't regress pylint checks
 - add test case for your code to meta/lib/oeqa/selftest/wic.py
 - check that your code doesn't break oe-selftest --coverage -r wic and
   doesn't regress test coverage

--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] buildhistory-diff: report directory renames

[Ed Bartosh]

The script detects directory renaming if two different
directories with the same set of files are added and removed.

[YOCTO #10691]

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 meta/lib/oe/buildhistory_analysis.py | 34 +++---
 1 file changed, 31 insertions(+), 3 deletions(-)

diff --git a/meta/lib/oe/buildhistory_analysis.py 
b/meta/lib/oe/buildhistory_analysis.py
index b6c0265..19b3bc4 100644
--- a/meta/lib/oe/buildhistory_analysis.py
+++ b/meta/lib/oe/buildhistory_analysis.py
@@ -69,7 +69,22 @@ class ChangeRecord:
 pkglist.append(k)
 return pkglist
 
+def detect_renamed_dirs(aitems, bitems):
+adirs = set(map(os.path.dirname, aitems))
+bdirs = set(map(os.path.dirname, bitems))
+files_ab = [(name, sorted(os.path.basename(item) for item in 
aitems if os.path.dirname(item) == name)) \
+for name in adirs - bdirs]
+files_ba = [(name, sorted(os.path.basename(item) for item in 
bitems if os.path.dirname(item) == name)) \
+for name in bdirs - adirs]
+renamed_dirs = [(dir1, dir2) for dir1, files1 in files_ab for 
dir2, files2 in files_ba if files1 == files2]
+# remove files that belong to renamed dirs from aitems and bitems
+for dir1, dir2 in renamed_dirs:
+aitems = [item for item in aitems if os.path.dirname(item) not 
in (dir1, dir2)]
+bitems = [item for item in bitems if os.path.dirname(item) not 
in (dir1, dir2)]
+return renamed_dirs, aitems, bitems
+
 if self.fieldname in list_fields or self.fieldname in 
list_order_fields:
+renamed_dirs = []
 if self.fieldname in ['RPROVIDES', 'RDEPENDS', 'RRECOMMENDS', 
'RSUGGESTS', 'RREPLACES', 'RCONFLICTS']:
 (depvera, depverb) = compare_pkg_lists(self.oldvalue, 
self.newvalue)
 aitems = pkglist_combine(depvera)
@@ -77,16 +92,29 @@ class ChangeRecord:
 else:
 aitems = self.oldvalue.split()
 bitems = self.newvalue.split()
+if self.fieldname == 'FILELIST':
+renamed_dirs, aitems, bitems = detect_renamed_dirs(aitems, 
bitems)
+
 removed = list(set(aitems) - set(bitems))
 added = list(set(bitems) - set(aitems))
 
+lines = []
+if renamed_dirs:
+for dfrom, dto in renamed_dirs:
+lines.append('directory renamed %s -> %s' % (dfrom, dto))
 if removed or added:
 if removed and not bitems:
-out = '%s: removed all items "%s"' % (self.fieldname, ' 
'.join(removed))
+lines.append('removed all items "%s"' % ' '.join(removed))
 else:
-out = '%s:%s%s' % (self.fieldname, ' removed "%s"' % ' 
'.join(removed) if removed else '', ' added "%s"' % ' '.join(added) if added 
else '')
+if removed:
+lines.append('removed "%s"' % ' '.join(removed))
+if added:
+lines.append('added "%s"' % ' '.join(added))
 else:
-out = '%s changed order' % self.fieldname
+lines.append('changed order')
+
+out = '%s: %s' % (self.fieldname, ', '.join(lines))
+
 elif self.fieldname in numeric_fields:
 aval = int(self.oldvalue or 0)
 bval = int(self.newvalue or 0)
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] Contents of non-rootfs partitions

[Ed Bartosh]

On Thu, Nov 24, 2016 at 03:43:18PM +0100, Kristian Amlie wrote:
> On 24/11/16 14:23, Ed Bartosh wrote:
> > On Thu, Nov 24, 2016 at 08:38:46AM +0100, Kristian Amlie wrote:
> >> On 24/11/16 07:15, Ulrich Ölmann wrote:
> >>> Hi,
> >>>
> >>> On Wed, Nov 23, 2016 at 04:56:56PM +0100, Patrick Ohly wrote:
> >>>> On Wed, 2016-11-23 at 15:22 +0200, Ed Bartosh wrote:
> >>>>> On Wed, Nov 23, 2016 at 02:08:28PM +0100, Kristian Amlie wrote:
> >>>>>> On 23/11/16 13:08, Ed Bartosh wrote:
> >>>>>>> On Tue, Nov 22, 2016 at 12:54:52PM +0100, Kristian Amlie wrote:
> >>>>>>> [...]
> >>>>>>> This can be done by extending existing rootfs plugin. It should be 
> >>>>>>> able
> >>>>>>> to do 2 things:
> >>>>>>>
> >>>>>>> - populate content of one rootfs directory to the partition. We can
> >>>>>>>   extend syntax of --rootfs-dir parameter to specify optional 
> >>>>>>> directory path to use
> >>>>>>>
> >>>>>>> - exclude rootfs directories when populating partitions. I'd propose 
> >>>>>>> to
> >>>>>>>   introduce --exclude-dirs wks parser option to handle this.
> >>>>>>>
> >>>>>>> Example of wks file with proposed new options:
> >>>>>>> part / --source rootfs --rootfs-dir=core-image-minimal   
> >>>>>>> --ondisk sda --fstype=ext4 --label root --align 1024 --exclude-dirs 
> >>>>>>> data --exclude-dirs home
> >>>>>>> part /data --source rootfs --rootfs-dir=core-image-minimal:/home 
> >>>>>>> --ondisk sda --fstype=ext4 --label data --align 1024
> >>>>>>> part /home --source rootfs --rootfs-dir=core-image-minimal:/data 
> >>>>>>> --ondisk sda --fstype=ext4 --label data --align 1024
> >>>>>>>
> >>>>>>> Does this make sense?
> >>>>>>
> >>>>>> Looks good. The only thing I would question is that, in the interest of
> >>>>>> reducing redundancy, maybe we should omit --exclude-dirs and have wic
> >>>>>> figure this out by combining all the entries, since "--exclude-dirs
> >>>>>> " and the corresponding "part " will almost always come in
> >>>>>> pairs. Possibly we could mark the "/" partition with one single
> >>>>>> --no-overlapping-dirs to force wic to make this consideration. Or do 
> >>>>>> you
> >>>>>> think that's too magical?
> >>>>>>
> >>>>> Tt's quite implicit from my point of view. However, if people like it we
> >>>>> can implement it this way.
> >>>>
> >>>> I prefer the explicit --exclude-dirs. It's less surprising and perhaps
> >>>> there are usages for having the same content in different partitions
> >>>> (redundancy, factory reset, etc.).
> >>>>
> >>>> Excluding only the directory content but not the actual directory is
> >>>> indeed a good point. I'm a bit undecided. When excluding only the
> >>>> directory content, there's no way of building a rootfs without that
> >>>> mount point, if that's desired. OTOH, when excluding also the directory,
> >>>> the data would have to be staged under a different path in the rootfs
> >>>> and the mount point would have to be a separate, empty directory.
> >>>>
> >>>> I'm leaning towards excluding the directory content and keeping the
> >>>> directory.
> >>>
> >>> what about having both possibilities by leaning against the syntax that 
> >>> rsync
> >>> uses to specify if a whole source directory or only it's contents shall be
> >>> synced to some destination site (see [1])?
> >>>
> >>> In analogy to this to exclude only the contents of the directory named 
> >>> 'data'
> >>> you would use
> >>>
> >>>   --exclude-dirs data/
> >>>
> >>> but to additionally exclude the dir itself as well it would read
> >>>
> >>>   --exclude-dirs data
> >>
> >> This is creative, but ultimately too unintuitive IMHO. Rsync is the only
> >> tool which uses this syntax AFAIK, and it's a constant source of
> >> confusion, especially when mixed with cp or similar commands.
> >>
> > 
> > Would this way be less intuitive?
> > --exclude-path data/*
> > --exclude-path data
> > 
> > We can go even further with it allowing any level of directories:
> > --exclude-path data/tmp/*
> > --exclude-path data/db/tmp
> > ...
> 
> I agree, this is pretty unambiguous and easy to understand.
> 
> But this raises the question: Should we go all the way and support
> wildcards? Which might make it a bit complicated. Maybe support only
> pure '*' for now?

As it shouldn't be hard to implement I'd go for it.

--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [wic][PATCH] qemux86*.conf: set wic-related parameters

[Ed Bartosh]

Set directdisk.wks as default wks to use for qemux86 machines.
Set requried dependeincies to build directdisk image.

This should simplify building wic images for qemux86* machines.
It should be enough to add wic to the list of IMAGE_FSTYPES to get
the images built.

[YOCTO #10637, YOCTO #8719]

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 meta/conf/machine/qemux86-64.conf | 3 +++
 meta/conf/machine/qemux86.conf| 3 +++
 2 files changed, 6 insertions(+)

diff --git a/meta/conf/machine/qemux86-64.conf 
b/meta/conf/machine/qemux86-64.conf
index f2b2aeb..7559724 100644
--- a/meta/conf/machine/qemux86-64.conf
+++ b/meta/conf/machine/qemux86-64.conf
@@ -28,3 +28,6 @@ XSERVER = "xserver-xorg \
 MACHINE_FEATURES += "x86"
 
 MACHINE_ESSENTIAL_EXTRA_RDEPENDS += "v86d"
+
+WKS_FILE = "directdisk.wks"
+do_image_wic[depends] += "syslinux:do_build 
syslinux-native:do_populate_sysroot mtools-native:do_populate_sysroot 
dosfstools-native:do_populate_sysroot"
diff --git a/meta/conf/machine/qemux86.conf b/meta/conf/machine/qemux86.conf
index 09555ad..7785999 100644
--- a/meta/conf/machine/qemux86.conf
+++ b/meta/conf/machine/qemux86.conf
@@ -27,3 +27,6 @@ XSERVER = "xserver-xorg \
 MACHINE_FEATURES += "x86"
 
 MACHINE_ESSENTIAL_EXTRA_RDEPENDS += "v86d"
+
+WKS_FILE = "directdisk.wks"
+do_image_wic[depends] += "syslinux:do_build 
syslinux-native:do_populate_sysroot mtools-native:do_populate_sysroot 
dosfstools-native:do_populate_sysroot"
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v1] wic: Add --exclude-path option to rootfs source plugin.

[Ed Bartosh]

On Mon, Nov 28, 2016 at 08:15:26AM +0100, Kristian Amlie wrote:
> > 
> > What's the reason of insisting that path must be absolute?
> > May be it's just me, but I find it a bit scaring to use absolute path in 
> > .wks
> > The patch is relative to the rootfs directory from my point of view.
> > 
> > It also looks quite strange in the code to insist on absolute path
> > +if not os.path.isabs(path):
> > +msger.error("Must be absolute: --exclude-path=%s" %
> > 
> > and then immediately making it relative:
> > +
> > +while os.path.isabs(path):
> > +path = path[1:]
> 
> Not really any strong reason. I just thought it was a logical thing to
> do from a user perspective: When you're making an image you're thinking
> about paths in the final image, and the path after "part" is absolute,
> so I thought this one should be too.
> 
> The fact that it's made relative in the code is just an implementation
> detail to make join() work correctly.
> 
> I'm fine either way, so just let me know which you prefer.
> 
I'd prefer relative path as I think about --exclude-path as a path in
the source rootfs directory that we want to exclude from copying to
the target partition. However, I agree that that it can be also treated
as a path to the target directory.

--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v1] wic: Add --exclude-path option to rootfs source plugin.

[Ed Bartosh]

On Mon, Nov 28, 2016 at 08:07:05AM +0100, Kristian Amlie wrote:
> On 25/11/16 17:33, Ed Bartosh wrote:
> > On Fri, Nov 25, 2016 at 01:35:53PM +0100, Kristian Amlie wrote:
> >> On 25/11/16 13:28, Maciej Borzęcki wrote:
> >>> On Fri, Nov 25, 2016 at 11:15 AM, Kristian Amlie
> >>>> +# Disallow '..', because doing so could be quite 
> >>>> disastrous
> >>>> +# (we will delete the directory).
> >>>> +remaining = path
> >>>> +while True:
> >>>> +(head, tail) = os.path.split(remaining)
> >>>> +if tail == '..':
> >>>> +msger.error("'..' not allowed: 
> >>>> --exclude-path=%s" % orig_path)
> >>>> +elif head == "":
> >>>> +break
> >>>> +remaining = head
> >>>
> >>> Why not do this instead?
> >>>
> >>> if '..' in path:
> >>> msger.error("'..' not allowed: --exclude-path=%s" % orig_path)
> >>>
> > would "'/..' in path" or something similar work?
> 
> '/..my-file' is a valid file name, so no, it wouldn't work. I realize
> that this would be uncommon, and it makes the check more complex, so I'm
> ok with making it simpler if you want me to, but we have to trade
> simplicity for correctness in that case.
>
OK, makes sense.

Would something like this work for you?

> mkdir -p /tmp/rootfs/bla/bla/bla/bla
> python -c "import os;print 
> os.path.realpath('/tmp/rootfs/bla/../bla/../bla/../bla/../').startswith('/tmp/rootfs/')"
False
> python -c "import os;print 
> os.path.realpath('/tmp/rootfs/bla/../bla/../bla/bla/../').startswith('/tmp/rootfs/')"
True

--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [wic][PATCH] wic: call os.ftruncate instead of running truncate

[Ed Bartosh]

Replaced running of truncate utility with the standard library
call os.ftruncate

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 scripts/lib/wic/imager/direct.py |  5 +++--
 scripts/lib/wic/partition.py | 15 ++-
 2 files changed, 13 insertions(+), 7 deletions(-)

diff --git a/scripts/lib/wic/imager/direct.py b/scripts/lib/wic/imager/direct.py
index edf5e5d..2bedef0 100644
--- a/scripts/lib/wic/imager/direct.py
+++ b/scripts/lib/wic/imager/direct.py
@@ -56,8 +56,9 @@ class DiskImage():
 if self.created:
 return
 # create sparse disk image
-cmd = "truncate %s -s %s" % (self.device, self.size)
-exec_cmd(cmd)
+with open(self.device, 'w') as sparse:
+os.ftruncate(sparse.fileno(), self.size)
+
 self.created = True
 
 class DirectImageCreator(BaseImageCreator):
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 90f65a1..89c33ab 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -217,7 +217,8 @@ class Partition():
 msger.debug("Added %d extra blocks to %s to get to %d total blocks" % \
 (extra_blocks, self.mountpoint, rootfs_size))
 
-exec_cmd("truncate %s -s %d" % (rootfs, rootfs_size * 1024))
+with open(rootfs, 'w') as sparse:
+os.ftruncate(sparse.fileno(), rootfs_size * 1024)
 
 extra_imagecmd = "-i 8192"
 
@@ -250,7 +251,8 @@ class Partition():
 msger.debug("Added %d extra blocks to %s to get to %d total blocks" % \
 (extra_blocks, self.mountpoint, rootfs_size))
 
-exec_cmd("truncate %s -s %d" % (rootfs, rootfs_size * 1024))
+with open(rootfs, 'w') as sparse:
+os.ftruncate(sparse.fileno(), rootfs_size * 1024)
 
 label_str = ""
 if self.label:
@@ -305,7 +307,8 @@ class Partition():
 """
 Prepare an empty ext2/3/4 partition.
 """
-exec_cmd("truncate %s -s %d" % (rootfs, self.size * 1024))
+with open(rootfs, 'w') as sparse:
+os.ftruncate(sparse.fileno(), rootfs_size * 1024)
 
 extra_imagecmd = "-i 8192"
 
@@ -322,7 +325,8 @@ class Partition():
 """
 Prepare an empty btrfs partition.
 """
-exec_cmd("truncate %s -s %d" % (rootfs, self.size * 1024))
+with open(rootfs, 'w') as sparse:
+os.ftruncate(sparse.fileno(), rootfs_size * 1024)
 
 label_str = ""
 if self.label:
@@ -383,7 +387,8 @@ class Partition():
 """
 path = "%s/fs.%s" % (cr_workdir, self.fstype)
 
-exec_cmd("truncate %s -s %d" % (path, self.size * 1024))
+with open(path, 'w') as sparse:
+os.ftruncate(sparse.fileno(), self.size * 1024)
 
 import uuid
 label_str = ""
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] wic: Add dosfstools-native mtools-native to dependencies

[Ed Bartosh]

Hi Khem,

On Tue, Oct 25, 2016 at 06:50:17PM -0700, Khem Raj wrote:
> these tools are needed for building wic images for some
> boards e.g. overo/gumstix
NAK

Usage of native tools depends on .wks content: modules, partition types,
etc. Your change will cause unnecessary build activity for some
machines/wks files where those tools are not used.

I'd suggest to put these dependencies to board-dependent meta.
Please, look how it's done for genericx86, beaglebone and edgerouter:
https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta-yocto-bsp/conf/machine/include/genericx86-common.inc
https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta-yocto-bsp/conf/machine/beaglebone.conf
https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta-yocto-bsp/conf/machine/edgerouter.conf

> Signed-off-by: Khem Raj 
> ---
>  meta/classes/image_types.bbclass | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/classes/image_types.bbclass 
> b/meta/classes/image_types.bbclass
> index a7fffbc..be7e38c 100644
> --- a/meta/classes/image_types.bbclass
> +++ b/meta/classes/image_types.bbclass
> @@ -298,7 +298,7 @@ IMAGE_DEPENDS_elf = "virtual/kernel mkelfimage-native"
>  IMAGE_DEPENDS_ubi = "mtd-utils-native"
>  IMAGE_DEPENDS_ubifs = "mtd-utils-native"
>  IMAGE_DEPENDS_multiubi = "mtd-utils-native"
> -IMAGE_DEPENDS_wic = "parted-native"
> +IMAGE_DEPENDS_wic = "parted-native dosfstools-native mtools-native"
>  
>  # This variable is available to request which values are suitable for 
> IMAGE_FSTYPES
>  IMAGE_TYPES = " \
> -- 
> 2.10.1
> 
> -- 
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] mkefidisk.sh: add deprecation warning to the output

[Ed Bartosh]

mkefidisk.sh will soon be deprecated in favor of .wic images.

Added deprecation warning to the script to inform users that
this script will soon be removed from the codebase.

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 scripts/contrib/mkefidisk.sh | 5 +
 1 file changed, 5 insertions(+)

diff --git a/scripts/contrib/mkefidisk.sh b/scripts/contrib/mkefidisk.sh
index d8db3c0..a175895 100755
--- a/scripts/contrib/mkefidisk.sh
+++ b/scripts/contrib/mkefidisk.sh
@@ -20,6 +20,11 @@
 
 LANG=C
 
+echo
+echo "WARNING: This script is deprecated and will be removed soon."
+echo "Please consider using wic EFI images instead."
+echo
+
 # Set to 1 to enable additional output
 DEBUG=0
 OUT="/dev/null"
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] image_types.bbclass: add dependency do_image_wic -> do_bootimg

[Ed Bartosh]

To produce certain types of images wic uses do_bootimg results
to assemble final image. For example, it copies BOOT/EFI directory
produced by do_bootimg to boot partition for every EFI image.

The tricky part of this is that do_bootimg task is not always run,
so we can't always make do_image_wic depend on do_bootimg. We only
need to do it if do_bootimg present in task graph.

Thank to Cristopher Larson for this fix.

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 meta/classes/image_types.bbclass | 5 +
 1 file changed, 5 insertions(+)

diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index 5ef6f60..a7fffbc 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -219,6 +219,11 @@ USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 
'wic ' + ' '.join('wic.%s
 WKS_FILE_CHECKSUM = "${@'${WKS_FULL_PATH}:%s' % 
os.path.exists('${WKS_FULL_PATH}') if '${USING_WIC}' else ''}"
 do_image_wic[file-checksums] += "${WKS_FILE_CHECKSUM}"
 
+python () {
+if d.getVar('USING_WIC', True) and 'do_bootimg' in d:
+bb.build.addtask('do_image_wic', '', 'do_bootimg', d)
+}
+
 python do_write_wks_template () {
 """Write out expanded template contents to WKS_FULL_PATH."""
 import re
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [wic][PATCH] canned-wks: use GPT partition table

[Ed Bartosh]

According to UEFI specification all EFI platforms must support
GUID Partition Table(GPT) disk layout. Here is a list of advantages
of using GPT disk layout over the legacy MBR partitioning:

 - Logical Block Addresses (LBAs) are 64 bits (rather than 32 bits).
 - Supports many partitions (rather than just four primary partitions).
 - Provides both a primary and backup partition table for redundancy.
 - Uses version number and size fields for future expansion.
 - Uses CRC32 fields for improved data integrity.
 - Defines a GUID for uniquely identifying each partition.
 - Uses a GUID and attributes to define partition content type.
 - Each partition contains a 36 character human readable name.

Used GPT partitioning in all EFI kickstart files.
Tested result images on NUC, MinnowBoard MAX and MinnowBoard Turbot.

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 scripts/lib/wic/canned-wks/mkefidisk.wks| 2 +-
 scripts/lib/wic/canned-wks/mkgummidisk.wks  | 2 +-
 scripts/lib/wic/canned-wks/systemd-bootdisk.wks | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/wic/canned-wks/mkefidisk.wks 
b/scripts/lib/wic/canned-wks/mkefidisk.wks
index 7c287f7..9f534fe 100644
--- a/scripts/lib/wic/canned-wks/mkefidisk.wks
+++ b/scripts/lib/wic/canned-wks/mkefidisk.wks
@@ -8,4 +8,4 @@ part / --source rootfs --ondisk sda --fstype=ext4 --label 
platform --align 1024
 
 part swap --ondisk sda --size 44 --label swap1 --fstype=swap
 
-bootloader --timeout=5 --append="rootfstype=ext4 console=ttyS0,115200 
console=tty0"
+bootloader --ptable gpt --timeout=5 --append="rootfstype=ext4 
console=ttyS0,115200 console=tty0"
diff --git a/scripts/lib/wic/canned-wks/mkgummidisk.wks 
b/scripts/lib/wic/canned-wks/mkgummidisk.wks
index 616d3ce..f3ae090 100644
--- a/scripts/lib/wic/canned-wks/mkgummidisk.wks
+++ b/scripts/lib/wic/canned-wks/mkgummidisk.wks
@@ -8,4 +8,4 @@ part / --source rootfs --ondisk sda --fstype=ext4 --label 
platform --align 1024
 
 part swap --ondisk sda --size 44 --label swap1 --fstype=swap
 
-bootloader  --timeout=5  --append="rootwait rootfstype=ext4 
console=ttyS0,115200 console=tty0"
+bootloader --ptable gpt --timeout=5  --append="rootwait rootfstype=ext4 
console=ttyS0,115200 console=tty0"
diff --git a/scripts/lib/wic/canned-wks/systemd-bootdisk.wks 
b/scripts/lib/wic/canned-wks/systemd-bootdisk.wks
index a49d130..b900023 100644
--- a/scripts/lib/wic/canned-wks/systemd-bootdisk.wks
+++ b/scripts/lib/wic/canned-wks/systemd-bootdisk.wks
@@ -8,4 +8,4 @@ part / --source rootfs --ondisk sda --fstype=ext4 --label 
platform --align 1024
 
 part swap --ondisk sda --size 44 --label swap1 --fstype=swap
 
-bootloader --timeout=5 --append="rootwait rootfstype=ext4 console=ttyS0,115200 
console=tty0"
+bootloader --ptable gpt --timeout=5 --append="rootwait rootfstype=ext4 
console=ttyS0,115200 console=tty0"
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [wic][PATCH] systemd-bootdisk.wks: update kernel command line

[Ed Bartosh]

Used ttyS0 console.
Removed usage of ttyPCH0 (FRI2 leftover)
Decreased bootloader timeout to 5 seconds
Removed 'vmalloc=256MB snd-hda-intel.enable_msi=0' as it's not
needed for any of reference BSPs.

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 scripts/lib/wic/canned-wks/systemd-bootdisk.wks | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/lib/wic/canned-wks/systemd-bootdisk.wks 
b/scripts/lib/wic/canned-wks/systemd-bootdisk.wks
index d80189b..a49d130 100644
--- a/scripts/lib/wic/canned-wks/systemd-bootdisk.wks
+++ b/scripts/lib/wic/canned-wks/systemd-bootdisk.wks
@@ -8,4 +8,4 @@ part / --source rootfs --ondisk sda --fstype=ext4 --label 
platform --align 1024
 
 part swap --ondisk sda --size 44 --label swap1 --fstype=swap
 
-bootloader  --timeout=10  --append="rootwait rootfstype=ext4 
console=ttyPCH0,115200 console=tty0 vmalloc=256MB snd-hda-intel.enable_msi=0"
+bootloader --timeout=5 --append="rootwait rootfstype=ext4 console=ttyS0,115200 
console=tty0"
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] wic: add --reserved-size wks option

[Ed Bartosh]

Hi Maciej,

There is already --size and --extra-space options.
Can we get the same or similar result by just using them? Do we really
need new option for similar purpose?


On Mon, Oct 17, 2016 at 03:06:18PM +0200, Maciej Borzecki wrote:
> Added new option --reserved-size to wks. The option can be used to
> indicate how much space should be reserved for a partition. This is
> useful if the disk will be a subject to full filesystem image updates
> and puts an upper limit of the size of filesystem images.
> 
> The actual filesystem image may be smaller than the reserved space, thus
> leaving some room for growth. If it is larger, an error will be raised.
> 
> Signed-off-by: Maciej Borzecki 
> ---
>  scripts/lib/wic/help.py| 11 ++
>  scripts/lib/wic/imager/direct.py   |  3 ++-
>  scripts/lib/wic/ksparser.py|  1 +
>  scripts/lib/wic/partition.py   |  1 +
>  scripts/lib/wic/utils/partitionedfs.py | 37 
> +-
>  5 files changed, 43 insertions(+), 10 deletions(-)
> 
> diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
> index 
> e5347ec4b7c900c68fc64351a5293e75de0672b3..1a5c7020ba0cdc5ef2e477a2b14360e09098a896
>  100644
> --- a/scripts/lib/wic/help.py
> +++ b/scripts/lib/wic/help.py
> @@ -646,6 +646,17 @@ DESCRIPTION
>   not specified, the size is in MB.
>   You do not need this option if you use --source.
>  
> + --reserved-size: This option specifies that there should be
> +  at least that many bytes reserved for
> +  the partition during layout. This is useful
> +  when the target disk will be a subject
> +  to full system image updates in the future.
> +  Specifying --reserved-size ensures that
> +  there is extra space in the partition allowing
> +  for future growth of the file system stored
> +  inside. Value format is the same as for
> +  --size option.
> +
>   --source: This option is a wic-specific option that names the
> source of the data that will populate the
> partition.  The most common value for this option
> diff --git a/scripts/lib/wic/imager/direct.py 
> b/scripts/lib/wic/imager/direct.py
> index 
> edf5e5d2214f8e78b6c2a98d7f6cd45fcc0065c4..02e293b9d744b760fcdf17610505dafef3e164ad
>  100644
> --- a/scripts/lib/wic/imager/direct.py
> +++ b/scripts/lib/wic/imager/direct.py
> @@ -301,7 +301,8 @@ class DirectImageCreator(BaseImageCreator):
> no_table=part.no_table,
> part_type=part.part_type,
> uuid=part.uuid,
> -   system_id=part.system_id)
> +   system_id=part.system_id,
> +   reserved_size=part.reserved_size)
>  
>  if fstab_path:
>  shutil.move(fstab_path + ".orig", fstab_path)
> diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
> index 
> 0894e2b199a299fbbed272f2e1c95e9d692e3ab1..4118bffdf4337f2d2d393d7e096632cd7aa37402
>  100644
> --- a/scripts/lib/wic/ksparser.py
> +++ b/scripts/lib/wic/ksparser.py
> @@ -137,6 +137,7 @@ class KickStart():
>  part.add_argument('--part-type')
>  part.add_argument('--rootfs-dir')
>  part.add_argument('--size', type=sizetype, default=0)
> +part.add_argument('--reserved-size', type=sizetype, default=0)
>  part.add_argument('--source')
>  part.add_argument('--sourceparams')
>  part.add_argument('--system-id', type=systemidtype)
> diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
> index 
> 90f65a1e3976a5460cd1b265b238168cce22781f..162a3a289de891ccf81437876c1f7a6f3c797b3b
>  100644
> --- a/scripts/lib/wic/partition.py
> +++ b/scripts/lib/wic/partition.py
> @@ -54,6 +54,7 @@ class Partition():
>  self.part_type = args.part_type
>  self.rootfs_dir = args.rootfs_dir
>  self.size = args.size
> +self.reserved_size = args.reserved_size
>  self.source = args.source
>  self.sourceparams = args.sourceparams
>  self.system_id = args.system_id
> diff --git a/scripts/lib/wic/utils/partitionedfs.py 
> b/scripts/lib/wic/utils/partitionedfs.py
> index 
> cb03009fc7e3c97305079629ded7d2ff01eba4c4..5d3b1588231459dedf0142f807114736f0bb28ea
>  100644
> --- a/scripts/lib/wic/utils/partitionedfs.py
> +++ b/scripts/lib/wic/utils/partitionedfs.py
> @@ -91,7 +91,7 @@ class Image():
>  
>  def add_partition(self, size, disk_name, mountpoint, source_file=None, 
> fstype=None,
>label=None, fsopts=None, boot=False, align=None, 
> no_table=False,
> - 

[OE-core] [wic][PATCH] wic: rename command line option -p -> -s

[Ed Bartosh]

Short variant of wic command line option --skip-build-check
is incorretly named -p. It's named -s in wic help and Yocto
documentation.

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 scripts/wic | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/wic b/scripts/wic
index fe2c33f..1ad1666 100755
--- a/scripts/wic
+++ b/scripts/wic
@@ -107,7 +107,7 @@ def wic_create_subcommand(args, usage_str):
 parser.add_option("-n", "--native-sysroot", dest="native_sysroot",
   help="path to the native sysroot containing the tools "
"to use to build the image")
-parser.add_option("-p", "--skip-build-check", dest="build_check",
+parser.add_option("-s", "--skip-build-check", dest="build_check",
   action="store_false", default=True, help="skip the build 
check")
 parser.add_option("-f", "--build-rootfs", action="store_true", help="build 
rootfs")
 parser.add_option("-c", "--compress-with", choices=("gzip", "bzip2", "xz"),
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] oe-run-native: standardize usage output

[Ed Bartosh]

Made usage output of oe-run-native to look similar to the
output of other oe scripts.

[YOCTO #10751]

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 scripts/oe-run-native | 11 +--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/scripts/oe-run-native b/scripts/oe-run-native
index 496e34f..7e26ea3 100755
--- a/scripts/oe-run-native
+++ b/scripts/oe-run-native
@@ -22,8 +22,15 @@
 #
 
 if [ $# -lt 1 -o "$1" = '--help' -o "$1" = '-h' ] ; then
-echo "Usage: $0  [parameters]"
-exit 1
+echo 'oe-run-native: error: the following arguments are required: '
+echo 'Usage: oe-run-native tool [parameters]'
+echo ''
+echo 'OpenEmbedded run-native - runs native tools'
+echo ''
+echo 'arguments:'
+echo '  toolNative tool to run'
+echo ''
+exit 2
 fi
 
 SYSROOT_SETUP_SCRIPT=`which oe-find-native-sysroot 2> /dev/null`
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] oe-trim-schemas: create usage output

[Ed Bartosh]

Created usage output for oe-trim-schemas script.

[YOCTO #10751]

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 scripts/oe-trim-schemas | 9 +
 1 file changed, 9 insertions(+)

diff --git a/scripts/oe-trim-schemas b/scripts/oe-trim-schemas
index 66a1b8d..7c199ef 100755
--- a/scripts/oe-trim-schemas
+++ b/scripts/oe-trim-schemas
@@ -18,6 +18,15 @@ def children (elem, name=None):
 l = [e for e in l if e.tag == name]
 return l
 
+if len(sys.argv) < 2 or sys.argv[1] in ('-h', '--help'):
+print('oe-trim-schemas: error: the following arguments are required: 
schema\n'
+  'Usage: oe-trim-schemas schema\n\n'
+  'OpenEmbedded trim schemas - remove unneeded schema locale 
translations\n'
+  'from gconf schema files\n\n'
+  'arguments:\n'
+  '  schemagconf schema file to trim\n')
+sys.exit(2)
+
 xml = etree.parse(sys.argv[1])
 
 for schema in child(xml.getroot(), "schemalist").getchildren():
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] oe-setup-builddir: create usage output

[Ed Bartosh]

Created usage output for oe-setup-builddir script.

[YOCTO #10751]

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 scripts/oe-setup-builddir | 8 
 1 file changed, 8 insertions(+)

diff --git a/scripts/oe-setup-builddir b/scripts/oe-setup-builddir
index e53f73c..ef49551 100755
--- a/scripts/oe-setup-builddir
+++ b/scripts/oe-setup-builddir
@@ -23,6 +23,14 @@ if [ -z "$BUILDDIR" ]; then
 exit 1
 fi
 
+if [ "$1" = '--help' -o "$1" = '-h' ]; then
+echo 'Usage: oe-setup-builddir'
+echo ''
+echo "OpenEmbedded setup-builddir - setup build directory $BUILDDIR"
+echo ''
+exit 2
+fi
+
 mkdir -p "$BUILDDIR/conf"
 
 if [ ! -d "$BUILDDIR" ]; then
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] oepydevshell-internal.py: standardize usage output

[Ed Bartosh]

Made usage output of oepydevshell-internal.py to look
similar to the output of other oe scripts.

[YOCTO #10751]

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 scripts/oepydevshell-internal.py | 15 +++
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/scripts/oepydevshell-internal.py b/scripts/oepydevshell-internal.py
index a22bec3..04621ae 100755
--- a/scripts/oepydevshell-internal.py
+++ b/scripts/oepydevshell-internal.py
@@ -22,9 +22,16 @@ def cbreaknoecho(fd):
 old[3] = old[3] &~ termios.ECHO &~ termios.ICANON
 termios.tcsetattr(fd, termios.TCSADRAIN, old)
 
-if len(sys.argv) != 3:
-print("Incorrect parameters")
-sys.exit(1)
+if len(sys.argv) != 3 or sys.argv[1] in ('-h', '--help'):
+print('oepydevshell-internal.py: error: the following arguments are 
required: pty, pid\n'
+  'Usage: oepydevshell-internal.py pty pid\n\n'
+  'OpenEmbedded oepydevshell-internal.py - internal script called from 
meta/classes/devshell.bbclass\n\n'
+  'arguments:\n'
+  '  pty   pty device name\n'
+  '  pid   parent process id\n\n'
+  'options:\n'
+  '  -h, --helpshow this help message and exit\n')
+sys.exit(2)
 
 pty = open(sys.argv[1], "w+b", 0)
 parent = int(sys.argv[2])
@@ -38,7 +45,7 @@ readline.parse_and_bind("tab: complete")
 try:
 readline.read_history_file(histfile)
 except IOError:
-pass 
+pass
 
 try:
 
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] oe-find-native-sysroot: create usage output

[Ed Bartosh]

Created usage output for oe-find-native-sysroot script.

[YOCTO #10751]

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 scripts/oe-find-native-sysroot | 14 ++
 1 file changed, 14 insertions(+)

diff --git a/scripts/oe-find-native-sysroot b/scripts/oe-find-native-sysroot
index 81d62b8..13a5c46 100755
--- a/scripts/oe-find-native-sysroot
+++ b/scripts/oe-find-native-sysroot
@@ -30,6 +30,20 @@
 # with this program; if not, write to the Free Software Foundation, Inc.,
 # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
+if [ "$1" = '--help' -o "$1" = '-h' ] ; then
+echo 'Usage: oe-find-native-sysroot [-h|--help]'
+echo ''
+echo 'OpenEmbedded find-native-sysroot - helper script to set'
+echo 'environment variables OECORE_NATIVE_SYSROOT and PSEUDO'
+echo 'to the path of the native sysroot directory and pseudo'
+echo 'executable binary'
+echo ''
+echo 'options:'
+echo '  -h, --help  show this help message and exit'
+echo ''
+exit 2
+fi
+
 if [ "x$OECORE_NATIVE_SYSROOT" = "x" ]; then
 BITBAKE=`which bitbake 2> /dev/null`
 if [ "x$BITBAKE" != "x" ]; then
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] oe-setup-rpmrepo: standardize usage output

[Ed Bartosh]

Made usage output of oe-setup-rpmrepo to look similar to the
output of other oe scripts.

[YOCTO #10751]

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 scripts/oe-setup-rpmrepo | 28 +---
 1 file changed, 17 insertions(+), 11 deletions(-)

diff --git a/scripts/oe-setup-rpmrepo b/scripts/oe-setup-rpmrepo
index 917b98b..7dcded8 100755
--- a/scripts/oe-setup-rpmrepo
+++ b/scripts/oe-setup-rpmrepo
@@ -23,16 +23,6 @@
 # Instead, use OE_TMPDIR for passing this in externally.
 TMPDIR="$OE_TMPDIR"
 
-function usage() {
-   echo "Usage: $0 "
-   echo "  : default is $TMPDIR/deploy/rpm"
-}
-
-if [ $# -gt 1 ]; then
-   usage
-   exit 1
-fi
-
 setup_tmpdir() {
 if [ -z "$TMPDIR" ]; then
 # Try to get TMPDIR from bitbake
@@ -53,6 +43,23 @@ setup_tmpdir() {
 fi
 }
 
+setup_tmpdir
+
+function usage() {
+echo 'Usage: oe-setup-rpmrepo rpm-dir'
+echo ''
+echo 'OpenEmbedded setup-rpmrepo - setup rpm repository'
+echo ''
+echo 'arguments:'
+echo "  rpm-dir   rpm repo directory, default is 
$TMPDIR/deploy/rpm"
+echo ''
+}
+
+if [ $# -gt 1 -o "$1" = '--help' -o "$1" = '-h' ]; then
+usage
+exit 2
+fi
+
 setup_sysroot() {
# Toolchain installs set up $OECORE_NATIVE_SYSROOT in their
# environment script. If that variable isn't set, we're
@@ -68,7 +75,6 @@ setup_sysroot() {
fi 
 }
 
-setup_tmpdir
 setup_sysroot
 
 
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] oe-git-proxy: create usage output

[Ed Bartosh]

Created usage output for oe-git-proxy script.

[YOCTO #10751]

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 scripts/oe-git-proxy | 21 +
 1 file changed, 21 insertions(+)

diff --git a/scripts/oe-git-proxy b/scripts/oe-git-proxy
index 0078e95..7a43fe6 100755
--- a/scripts/oe-git-proxy
+++ b/scripts/oe-git-proxy
@@ -18,6 +18,27 @@
 # AUTHORS
 # Darren Hart <dvh...@linux.intel.com>
 
+if [ $# -lt 2 -o "$1" = '--help' -o "$1" = '-h' ] ; then
+echo 'oe-git-proxy: error: the following arguments are required: host port'
+echo 'Usage: oe-git-proxy host port'
+echo ''
+echo 'OpenEmbedded git-proxy - a simple tool to be used via 
GIT_PROXY_COMMAND.'
+echo 'It uses socat to make SOCKS or HTTPS proxy connections.'
+echo 'It uses ALL_PROXY to determine the proxy server, protocol, and port.'
+echo 'It uses NO_PROXY to skip using the proxy for a comma delimited list'
+echo 'of hosts, host globs (*.example.com), IPs, or CIDR masks 
(192.168.1.0/24).'
+echo 'It is known to work with both bash and dash shells.runs native tools'
+echo ''
+echo 'arguments:'
+echo '  hostproxy host to use'
+echo '  portproxy port to use'
+echo ''
+echo 'options:'
+echo '  -h, --help  show this help message and exit'
+echo ''
+exit 2
+fi
+
 # Locate the netcat binary
 SOCAT=$(which socat 2>/dev/null)
 if [ $? -ne 0 ]; then
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v3] uninative: rebuild uninative for gcc 4.8 and 4.9

[Ed Bartosh]

Some c++ libraries fail to build if uninative is built
with gcc 5.x and host gcc version is either 4.8 or 4.9.

The issue should be solved by making separate uninative sstate
directory structure sstate-cache/universal- for host gcc
versions 4.8 and 4.9. This causes rebuilds of uninative if host gcc
is either 4.8 or 4.9 and it doesn't match gcc version used to build
uninative.

[YOCTO #10441]

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass |  5 +++--
 meta/classes/uninative.bbclass|  2 +-
 meta/lib/oe/utils.py  | 14 ++
 3 files changed, 18 insertions(+), 3 deletions(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass 
b/meta/classes/populate_sdk_ext.bbclass
index 3c3a73c..1affa9d 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -374,8 +374,9 @@ python copy_buildsystem () {
 
 sstate_out = baseoutpath + '/sstate-cache'
 bb.utils.remove(sstate_out, True)
-# uninative.bbclass sets NATIVELSBSTRING to 'universal'
-fixedlsbstring = 'universal'
+
+# uninative.bbclass sets NATIVELSBSTRING to 'universal%s' % 
oe.utils.host_gcc_version(d)
+fixedlsbstring = "universal%s" % oe.utils.host_gcc_version(d)
 
 sdk_include_toolchain = (d.getVar('SDK_INCLUDE_TOOLCHAIN', True) == '1')
 sdk_ext_type = d.getVar('SDK_EXT_TYPE', True)
diff --git a/meta/classes/uninative.bbclass b/meta/classes/uninative.bbclass
index 9242320..11cbf9b 100644
--- a/meta/classes/uninative.bbclass
+++ b/meta/classes/uninative.bbclass
@@ -88,7 +88,7 @@ def enable_uninative(d):
 loader = d.getVar("UNINATIVE_LOADER", True)
 if os.path.exists(loader):
 bb.debug(2, "Enabling uninative")
-d.setVar("NATIVELSBSTRING", "universal")
+d.setVar("NATIVELSBSTRING", "universal%s" % 
oe.utils.host_gcc_version(d))
 d.appendVar("SSTATEPOSTUNPACKFUNCS", " uninative_changeinterp")
 d.prependVar("PATH", 
"${STAGING_DIR}-uninative/${BUILD_ARCH}-linux${bindir_native}:")
 
diff --git a/meta/lib/oe/utils.py b/meta/lib/oe/utils.py
index d6545b1..2b095f1 100644
--- a/meta/lib/oe/utils.py
+++ b/meta/lib/oe/utils.py
@@ -230,6 +230,20 @@ def format_pkg_list(pkg_dict, ret_format=None):
 
 return '\n'.join(output)
 
+def host_gcc_version(d):
+compiler = d.getVar("BUILD_CC", True)
+retval, output = getstatusoutput("%s --version" % compiler)
+if retval:
+bb.fatal("Error running %s --version: %s" % (compiler, output))
+
+import re
+match = re.match(".* (\d\.\d)\.\d.*", output.split('\n')[0])
+if not match:
+bb.fatal("Can't get compiler version from %s --version output" % 
compiler)
+
+version = match.group(1)
+return "-%s" % version if version in ("4.8", "4.9") else ""
+
 #
 # Python 2.7 doesn't have threaded pools (just multiprocessing)
 # so implement a version here
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] selftest/wic: extending test coverage for WIC script options

[Ed Bartosh]

On Tue, Dec 13, 2016 at 06:24:57PM -0600, Jair Gonzalez wrote:
> Hi Ed,
> 
> Thank you for your response and suggestions. Below are my comments.
> 

> > > +@testcase(1557)
> > > +def test_listed_images_help(self):
> > > +"""Test wic listed images help"""
> > > +output = runCmd('wic list images').output
> > > +imageDetails = [line.split() for line in output.split('\n')]
> > > +imageList = [row[0] for row in imageDetails]
> > How about replacing two last lines with this?
> > imagelist = [line.split()[0] for line in output.split('\n')]
> I agree. What about this?
> imagelist = [line.split()[0] for line in output.splitlines()]

This is event better, thanks!

> > > +
> > > + "--image-name=core-image-minimal").status)
> > Is '=' mandatory here?
> On wic's help it appears as mandatory, but on practice, it can be used both
> ways. I decided to use both ways along the module to test both usages and
> increase coverage, but not to dedicate specific test cases to each
> combination.

Makes sense to me.

> > >  def test_compress_gzip(self):
> > >  """Test compressing an image with gzip"""
> > >  self.assertEqual(0, runCmd("wic create directdisk "
> > > -   "--image-name core-image-minimal "
> > > +   "-e core-image-minimal "
> > --image-name is more readable than -e from my point of view.
> Similarly to the '=' to define long option names' arguments, I used both
> forms of each option along the module to increase coverage.
OK

> > > +def test_debug_skip_build_check_and_build_rootfs(self):
> > > +"""Test wic debug, skip-build-check and build_rootfs"""
> > > +self.assertEqual(0, runCmd("wic create directdisk "
> > > +   "--image-name=core-image-minimal "
> > > +   "-D -s -f").status)
> > > +self.assertEqual(1, len(glob(self.resultdir +
> "directdisk-*.direct")))
> > > +self.assertEqual(0, runCmd("wic create directdisk "
> > > +   "--image-name=core-image-minimal "
> > > +   "--debug "
> > > +   "--skip-build-check "
> > > +   "--build-rootfs").status)
> > > +self.assertEqual(1, len(glob(self.resultdir +
> > > + "directdisk-*.direct")))
> > > +
> > I'd split this to two test cases as they're testing two different options.
> Actually, those are three different options (with their short and long
> versions). I did this to not add too many test cases, but as you mention,
> probably it's better to separate them by option to make it clearer.
Agreed.

> core-image-minimal").status)
> > > -self.assertEqual(1, len(glob(self.resultdir + "HYBRID_ISO_IMG-
> > *.direct")))
> > > -self.assertEqual(1, len(glob(self.resultdir +
> "HYBRID_ISO_IMG-*.iso")))
> > > +   "--image-name core-image-minimal"
> > > +   ).status)
> > This is less readable. Is this only to fit the line into 80 chars?
> > If so, let's not do it. Lines up to 100 chars long are more readable than
> this I
> > believe.
> I changed it to conform to PEP8 and increase readability on editors adjusted
> to 80 chars. However, if you consider it's better to leave it on 100 chars,
> I could work within that.

Let's agree on max 100 chars/line if it improves readability. Otherwise
80 is ok.

> > > +self.assertEqual(0, runCmd("wic create directdisk "
> > > +   "--image-name=%s "
> > > +   "--vars %s"
> > > +   % (image, imgenvdir)).status)
> > > +
> > Do we really want to test short and long variant of options?
> > If so, we should do it for all options.
> Within the module, all short and long variant of options are tested. Not all
> combinations of long variants with '=' and without it are tested, though.

OK, makes sense to me.

> > > +@testcase(1562)
> > > +def test_alternate_output_dir(self):
> > > +"""Test alternate output directory"""
> > > +self.assertEqual(0, runCmd("wic create directdisk "
> > > +   "-e core-image-minimal "
> > > +   "-o %s"
> > > +   % self.alternate_resultdir).status)
> > > +self.assertEqual(1, len(glob(self.alternate_resultdir +
> > > + "build/directdisk-*.direct")))
> > > +self.assertEqual(0, runCmd("wic create mkefidisk -e "
> > > +   "core-image-minimal "
> > > +   "--outdir %s"
> > > +   % self.alternate_resultdir).status)
> > > +self.assertEqual(1, len(glob(self.alternate_resultdir +
> > > + 

Re: [OE-core] [PATCH] wic: obey the rootfs size from the metadata

[Ed Bartosh]

Hi Christopher,

Thank you for the patch!

+1

On Thu, Dec 15, 2016 at 12:42:39PM -0700, Christopher Larson wrote:
> From: Christopher Larson 
> 
> When no --size is specified for the rootfs in the .wks, we want to obey the
> rootfs size from the metadata, otherwise the defined IMAGE_ROOTFS_EXTRA_SPACE
> and IMAGE_OVERHEAD_FACTOR will not be obeyed. In some cases, this can result
> in image construction failure, if the size determined by du was insufficient
> to hold the files without the aforementioned extra space.
> 
> This fallback from --size to ROOTFS_SIZE was already implemented when
> --rootfs-dir is specified in the .wks, but it did not occur otherwise, neither
> when --rootfs-dir= was passed to `wic create` nor when IMAGE_ROOTFS was used.
> This made a certain amount of sense, as this fallback logic happened at such
> a level that it wasn't able to identify which partitions were rootfs
> partitions otherwise. Rather than doing it at that level, we can do it in
> prepare_rootfs(), which is run by the rootfs source plugins.
> 
> Note that IMAGE_OVERHEAD_FACTOR and a --overhead-factor in the .wks will now
> both be applied when --size isn't specified in the .wks. A warning is added
> about this, though a user won't see it unless wic fails or they examine the
> do_image_wic log.
> 
> Fixes [YOCTO #10815]
> 
> Signed-off-by: Christopher Larson 
> ---
>  scripts/lib/wic/partition.py | 13 -
>  1 file changed, 12 insertions(+), 1 deletion(-)
> 
> diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
> index ac4c836..b191cde 100644
> --- a/scripts/lib/wic/partition.py
> +++ b/scripts/lib/wic/partition.py
> @@ -28,7 +28,7 @@ import os
>  import tempfile
>  
>  from wic.utils.oe.misc import msger, parse_sourceparams
> -from wic.utils.oe.misc import exec_cmd, exec_native_cmd
> +from wic.utils.oe.misc import exec_cmd, exec_native_cmd, get_bitbake_var
>  from wic.plugin import pluginmgr
>  
>  partition_methods = {
> @@ -194,6 +194,17 @@ class Partition():
>  msger.error("File system for partition %s not specified in 
> kickstart, " \
>  "use --fstype option" % (self.mountpoint))
>  
> +# Get rootfs size from bitbake variable if it's not set in .ks file
> +if not self.size:
> +# Bitbake variable ROOTFS_SIZE is calculated in
> +# Image._get_rootfs_size method from meta/lib/oe/image.py
> +# using IMAGE_ROOTFS_SIZE, IMAGE_ROOTFS_ALIGNMENT,
> +# IMAGE_OVERHEAD_FACTOR and IMAGE_ROOTFS_EXTRA_SPACE
> +rsize_bb = get_bitbake_var('ROOTFS_SIZE')
> +if rsize_bb:
> +msger.warning('overhead-factor was specified, but size was 
> not, so bitbake variables will be used for the size. In this case both 
> IMAGE_OVERHEAD_FACTOR and --overhead-factor will be applied')
> +self.size = int(round(float(rsize_bb)))
> +
>  for prefix in ("ext", "btrfs", "vfat", "squashfs"):
>  if self.fstype.startswith(prefix):
>  method = getattr(self, "prepare_rootfs_" + prefix)
> -- 
> 2.8.0
> 

-- 
--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v2 3/3] selftest/wic: Add tests for --exclude-dir option.

[Ed Bartosh]

On Wed, Dec 14, 2016 at 05:28:54PM +0100, Kristian Amlie wrote:
> Based partially on an earlier patch by Maciej Borzecki.
> 
> Signed-off-by: Kristian Amlie 
> ---
>  meta/lib/oeqa/selftest/wic.py | 106 
> +-
>  1 file changed, 105 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
> index faac11e..b2b0fe8 100644
> --- a/meta/lib/oeqa/selftest/wic.py
> +++ b/meta/lib/oeqa/selftest/wic.py
> @@ -49,7 +49,8 @@ class Wic(oeSelfTest):
>  # setUpClass being unavailable.
>  if not Wic.image_is_ready:
>  bitbake('syslinux syslinux-native parted-native gptfdisk-native '
> -'dosfstools-native mtools-native bmap-tools-native')
> +'dosfstools-native mtools-native bmap-tools-native '
> +'e2tools-native')
>  bitbake('core-image-minimal')
>  Wic.image_is_ready = True
>  
> @@ -299,3 +300,106 @@ class Wic(oeSelfTest):
>  self.assertEqual(0, runCmd("wic create %s -e core-image-minimal" \
> % image).status)
>  self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image)))
> +
> +def test_exclude_path(self):
> +"""Test --exclude-path wks option."""
> +
> +# For using 'e2ls'.
> +old_path = os.environ['PATH']
> +os.environ['PATH'] = get_bb_var('PATH', 'core-image-minimal')
> +
> +wks_file = 'temp.wks'
> +ks = open(wks_file, 'w')
I'd use more pythonic 'with open(wks_file, 'w') as wks:' here.

> +rootfs_dir = get_bb_var('IMAGE_ROOTFS', 'core-image-minimal')
> +ks.write("""part / --source rootfs --ondisk mmcblk0 --fstype=ext4 
> --exclude-path usr
> +part /usr --source rootfs --ondisk mmcblk0 --fstype=ext4 --rootfs-dir %s/usr
> +part /etc --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path bin/ 
> --rootfs-dir %s/usr"""
> + % (rootfs_dir, rootfs_dir))
> +ks.close()
> +self.assertEqual(0, runCmd("wic create %s -e core-image-minimal" \
> +   % wks_file).status)
> +
> +os.remove(wks_file)
> +wicout = glob(self.resultdir + "%s-*direct" % 'temp')
> +self.assertEqual(1, len(wicout))
> +
> +wicimg = wicout[0]
> +
> +# verify partition size with wic
> +res = runCmd("parted -m %s unit b p 2>/dev/null" % wicimg)
> +self.assertEqual(0, res.status)
> +
> +# parse parted output which looks like this:
> +# BYT;\n
> +# 
> /var/tmp/wic/build/tmpfwvjjkf_-201611101222-hda.direct:200MiB:file:512:512:msdos::;\n
> +# 1:0.00MiB:200MiB:200MiB:ext4::;\n
> +partlns = res.output.splitlines()[2:]
> +
> +self.assertEqual(3, len(partlns))
> +
> +for part in [1, 2, 3]:
> +part_file = os.path.join(self.resultdir, "selftest_img.part%d" % 
> part)
> +partln = partlns[part-1].split(":")
> +self.assertEqual(7, len(partln))
> +start = int(partln[1].rstrip("B")) / 512
> +length = int(partln[3].rstrip("B")) / 512
> +self.assertEqual(0, runCmd("dd if=%s of=%s skip=%d count=%d" %
> +   (wicimg, part_file, start, 
> length)).status)
> +
> +# Test partition 1, should contain the normal root directories, 
> except
> +# /usr.
> +res = runCmd("e2ls %s" % os.path.join(self.resultdir, 
> "selftest_img.part1"))
> +self.assertEqual(0, res.status)
> +files = res.output.split()
> +self.assertIn("etc", files)
> +self.assertNotIn("usr", files)
> +
> +# Partition 2, should contain common directories for /usr, not root
> +# directories.
> +res = runCmd("e2ls %s" % os.path.join(self.resultdir, 
> "selftest_img.part2"))
> +self.assertEqual(0, res.status)
> +files = res.output.split()
> +self.assertNotIn("etc", files)
> +self.assertNotIn("usr", files)
> +self.assertIn("share", files)
> +
> +# Partition 3, should contain the same as partition 2, including the 
> bin
> +# directory, but not the files inside it.
> +res = runCmd("e2ls %s" % os.path.join(self.resultdir, 
> "selftest_img.part3"))
> +self.assertEqual(0, res.status)
> +files = res.output.split()
> +self.assertNotIn("etc", files)
> +self.assertNotIn("usr", files)
> +self.assertIn("share", files)
> +self.assertIn("bin", files)
> +res = runCmd("e2ls %s:bin" % os.path.join(self.resultdir, 
> "selftest_img.part3"))
> +self.assertEqual(0, res.status)
> +self.assertEqual("No files found!", res.output.strip())
> +
> +for part in [1, 2, 3]:
> +part_file = os.path.join(self.resultdir, "selftest_img.part%d" % 
> part)
> +os.remove(part_file)
> +
> +  

Re: [OE-core] [PATCH] edgerouter.conf: enable generation of wic.bmap

[Ed Bartosh]

On Sat, Dec 10, 2016 at 11:40:38AM -0700, Christopher Larson wrote:
> On Fri, Dec 9, 2016 at 10:46 AM, Ed Bartosh <ed.bart...@linux.intel.com>
> wrote:
> 
> > Enabled generation of block map(bmap) files for wic images
> > built for edgerouter machine. This should simplify flashing
> > images with bmaptool.
> >
> > [YOCTO #10621]
> >
> > Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
> >
> 
> I think this is more of a distro decision than machine, whether they want
> to support use of bmaptool for their users/customers. It’d be nice if it
> was easier to control compression from the distro, too. For example, if we
> want to distribute wic.bz2+wic.bmap in our installers, it’d be nice to be
> able to switch to that for all machines that use wic. I’m not sure how best
> to pull that off, however. Something to think about for the future.

Thanks for pointing out to it. I did it this way for two reasons:
- IMAGE_FSTYPES was already modified in edgerouter.conf
- I wanted to enable generation of wic.bmap for subset of machines, i.e.
  only for hardware reference BSPs mentioned in README.hardware

If there is a better to do it I'd be happy to update my patch.

--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] populate_sdk_ext: get NATIVELSBSTRING variable

[Ed Bartosh]

Setting fixedlsbstring to 'universal' can break populate_sdk_ext
task as NATIVELSBSTRING can be set to universal- in
some cases.

Getting NATIVELSBSTRING value using getVar should fix this.

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass 
b/meta/classes/populate_sdk_ext.bbclass
index 3c3a73c..6ef48c0 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -374,8 +374,8 @@ python copy_buildsystem () {
 
 sstate_out = baseoutpath + '/sstate-cache'
 bb.utils.remove(sstate_out, True)
-# uninative.bbclass sets NATIVELSBSTRING to 'universal'
-fixedlsbstring = 'universal'
+
+fixedlsbstring = d.getVar('NATIVELSBSTRING', True)
 
 sdk_include_toolchain = (d.getVar('SDK_INCLUDE_TOOLCHAIN', True) == '1')
 sdk_ext_type = d.getVar('SDK_EXT_TYPE', True)
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v5 0/6] wic: bugfixes & --fixed-size support, tests, oe-selftest: minor fixes

[Ed Bartosh]

On Tue, Dec 13, 2016 at 09:07:30AM +0100, Maciej Borzęcki wrote:
> On Thu, Nov 24, 2016 at 8:08 AM, Maciej Borzecki
>  wrote:
> > v5 of a patch series previously posted here [1].
> >
> > Changes since v4:
> >
> > * dropped `wic: selftest: do not repeat core-image-minimal` & rebased
> >   dependant patches
> >
> > * minor formatting fix in `wic: selftest: add tests for --fixed-size
> >   partition flags`
> >
> > [1]. 
> > http://lists.openembedded.org/pipermail/openembedded-core/2016-November/129103.html
> >
> > Maciej Borzecki (6):
> >   oe-selftest: enforce en_US.UTF-8 locale
> >   oeqa/utils/commands.py: allow use of binaries from native sysroot
> >   wic: add --fixed-size wks option
> >   wic: selftest: avoid COMPATIBLE_HOST issues
> >   wic: selftest: do not assume bzImage kernel image
> >   wic: selftest: add tests for --fixed-size partition flags
> >
> >  meta/lib/oeqa/selftest/wic.py  | 123 
> > +++--
> >  meta/lib/oeqa/utils/commands.py|   9 ++-
> >  scripts/lib/wic/help.py|  14 +++-
> >  scripts/lib/wic/imager/direct.py   |   2 +-
> >  scripts/lib/wic/ksparser.py|  41 +--
> >  scripts/lib/wic/partition.py   |  88 ++-
> >  scripts/lib/wic/utils/partitionedfs.py |   2 +-
> >  scripts/oe-selftest|   3 +
> >  8 files changed, 234 insertions(+), 48 deletions(-)
> 
> Is there any additional action needed from me at this point? A rebase
> or a resend perhaps?
> 

Rebase would be nice as at lest one patch from this patchset is already 
accepted.

--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] selftest/wic: extending test coverage for WIC script options

[Ed Bartosh]

Hi Jair,

Thank you for the patch! My comments are below.

On Tue, Dec 13, 2016 at 09:53:27AM -0600, Jair Gonzalez wrote:
> The previous WIC script selftest didn't cover all of its command
> line options. The following test cases were added to complete
> covering them:
> 
> 1552 Test wic --version
> 1553 Test wic help create
> 1554 Test wic help list
> 1555 Test wic list images
> 1556 Test wic list source-plugins
> 1557 Test wic listed images help
> 1558 Test wic debug, skip-build-check and build_rootfs
> 1559 Test image vars directory selection
> 1562 Test alternate output directory

> In addition, the following test cases were assigned an ID number on
> Testopia:
> 
> 1560 Test creation of systemd-bootdisk image
> 1561 Test creation of sdimage-bootpart image
> 
> Finally, part of the test methods were rearranged to group them by
> functionality, and some cleanup was made to improve the code's
> compliance with PEP8 style guide.

I'd suggest to split this patch to at least 3 patches:
- new testcases (fix for YOCTO 10594)
- assigning id numbers
- removing WKS_FILE = "wic-image-minimal" from config
- code cleanup

> Fixes [YOCTO 10594]
> 
> Signed-off-by: Jair Gonzalez 
> 
> ---
>  meta/lib/oeqa/selftest/wic.py | 246 
> +-
>  1 file changed, 174 insertions(+), 72 deletions(-)
> 
> diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
> index e652fad..46bfb94 100644
> --- a/meta/lib/oeqa/selftest/wic.py
> +++ b/meta/lib/oeqa/selftest/wic.py
> @@ -37,13 +37,13 @@ class Wic(oeSelfTest):
>  """Wic test class."""
>  
>  resultdir = "/var/tmp/wic/build/"
> +alternate_resultdir = "/var/tmp/wic/build/alt/"
>  image_is_ready = False
>  
>  def setUpLocal(self):
>  """This code is executed before each test method."""
>  self.write_config('IMAGE_FSTYPES += " hddimg"\n'
> -  'MACHINE_FEATURES_append = " efi"\n'
> -  'WKS_FILE = "wic-image-minimal"\n')
I like the change, but it should be also in a separate patch.

> +  'MACHINE_FEATURES_append = " efi"\n')
>  
>  # Do this here instead of in setUpClass as the base setUp does some
>  # clean up which can result in the native tools built earlier in
> @@ -56,10 +56,16 @@ class Wic(oeSelfTest):
>  
>  rmtree(self.resultdir, ignore_errors=True)
>  
> +@testcase(1552)
> +def test_version(self):
> +"""Test wic --version"""
> +self.assertEqual(0, runCmd('wic --version').status)
> +
>  @testcase(1208)
>  def test_help(self):
> -"""Test wic --help"""
> +"""Test wic --help and wic -h"""
>  self.assertEqual(0, runCmd('wic --help').status)
> +self.assertEqual(0, runCmd('wic -h').status)
>  @testcase(1209)
>  def test_createhelp(self):
> @@ -71,19 +77,74 @@ class Wic(oeSelfTest):
>  """Test wic list --help"""
>  self.assertEqual(0, runCmd('wic list --help').status)
>  
> +@testcase(1553)
> +def test_help_create(self):
> +"""Test wic help create"""
> +self.assertEqual(0, runCmd('wic help create').status)
> +
> +@testcase(1554)
> +def test_help_list(self):
> +"""Test wic help list"""
> +self.assertEqual(0, runCmd('wic help list').status)
> +
> +@testcase(1215)
> +def test_help_overview(self):
> +"""Test wic help overview"""
> +self.assertEqual(0, runCmd('wic help overview').status)
> +
> +@testcase(1216)
> +def test_help_plugins(self):
> +"""Test wic help plugins"""
> +self.assertEqual(0, runCmd('wic help plugins').status)
> +
> +@testcase(1217)
> +def test_help_kickstart(self):
> +"""Test wic help kickstart"""
> +self.assertEqual(0, runCmd('wic help kickstart').status)
> +
> +@testcase(1555)
> +def test_list_images(self):
> +"""Test wic list images"""
> +self.assertEqual(0, runCmd('wic list images').status)
> +
> +@testcase(1556)
> +def test_list_source_plugins(self):
> +"""Test wic list source-plugins"""
> +self.assertEqual(0, runCmd('wic list source-plugins').status)
> +
> +@testcase(1557)
> +def test_listed_images_help(self):
> +"""Test wic listed images help"""
> +output = runCmd('wic list images').output
> +imageDetails = [line.split() for line in output.split('\n')]
> +imageList = [row[0] for row in imageDetails]
How about replacing two last lines with this?
imagelist = [line.split()[0] for line in output.split('\n')]

> +for image in imageList:
> +self.assertEqual(0, runCmd('wic list %s help' % image).status)
> +
> +@testcase(1213)
> +def test_unsupported_subcommand(self):
> +"""Test unsupported subcommand"""
> +self.assertEqual(1, runCmd('wic unsupported',
> +   

Re: [OE-core] [wic][PATCH] meta-yocto-bsp: add kickstart file for MPC8315

[Ed Bartosh]

Sorry, wrong commit subject line.
Please, ignore.

On Fri, Dec 16, 2016 at 07:05:14PM +0200, Ed Bartosh wrote:
> Added kickstart file to produce partitioned image for
> MPC8315 reference hardware.
> 
> [YOCTO #8719]
> 
> Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
> ---
>  scripts/lib/wic/canned-wks/mpc8315e-rdb.wks | 6 ++
>  1 file changed, 6 insertions(+)
>  create mode 100644 scripts/lib/wic/canned-wks/mpc8315e-rdb.wks
> 
> diff --git a/scripts/lib/wic/canned-wks/mpc8315e-rdb.wks 
> b/scripts/lib/wic/canned-wks/mpc8315e-rdb.wks
> new file mode 100644
> index 000..af587b2
> --- /dev/null
> +++ b/scripts/lib/wic/canned-wks/mpc8315e-rdb.wks
> @@ -0,0 +1,6 @@
> +# short-description: Create SD card image with a boot partition
> +# long-description: Creates a partitioned SD card image. Boot files
> +# are located in the first partition.
> +
> +part /boot --source bootimg-partition --ondisk sdb --fstype=ext3 --label boot
> +part / --source rootfs --ondisk sdb --fstype=ext3 --label root
> -- 
> 2.1.4
> 

-- 
--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [wic][PATCH] meta-yocto-bsp: add kickstart file for MPC8315

[Ed Bartosh]

Added kickstart file to produce partitioned image for
MPC8315 reference hardware.

[YOCTO #8719]

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 scripts/lib/wic/canned-wks/mpc8315e-rdb.wks | 6 ++
 1 file changed, 6 insertions(+)
 create mode 100644 scripts/lib/wic/canned-wks/mpc8315e-rdb.wks

diff --git a/scripts/lib/wic/canned-wks/mpc8315e-rdb.wks 
b/scripts/lib/wic/canned-wks/mpc8315e-rdb.wks
new file mode 100644
index 000..af587b2
--- /dev/null
+++ b/scripts/lib/wic/canned-wks/mpc8315e-rdb.wks
@@ -0,0 +1,6 @@
+# short-description: Create SD card image with a boot partition
+# long-description: Creates a partitioned SD card image. Boot files
+# are located in the first partition.
+
+part /boot --source bootimg-partition --ondisk sdb --fstype=ext3 --label boot
+part / --source rootfs --ondisk sdb --fstype=ext3 --label root
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [wic][PATCH] wic: look for wks files in /recipes-images

[Ed Bartosh]

Currently wic looks for wks files in
/scripts/lib/wic/canned-wks/ directories.
This path is too nested and doesn't look consistent with the
naming scheme of layer directories.

Added 'recipes-images' directory to the list of paths
to look for wks files.

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 scripts/lib/wic/engine.py | 7 +--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index 5b10463..8353edb 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -52,6 +52,7 @@ def verify_build_env():
 
 CANNED_IMAGE_DIR = "lib/wic/canned-wks" # relative to scripts
 SCRIPTS_CANNED_IMAGE_DIR = "scripts/" + CANNED_IMAGE_DIR
+RECIPES_IMAGES_DIR = "recipes-images"
 
 def build_canned_image_list(path):
 layers_path = misc.get_bitbake_var("BBLAYERS")
@@ -59,8 +60,10 @@ def build_canned_image_list(path):
 
 if layers_path is not None:
 for layer_path in layers_path.split():
-cpath = os.path.join(layer_path, SCRIPTS_CANNED_IMAGE_DIR)
-canned_wks_layer_dirs.append(cpath)
+for wks_path in (RECIPES_IMAGES_DIR, SCRIPTS_CANNED_IMAGE_DIR):
+cpath = os.path.join(layer_path, wks_path)
+if os.path.isdir(cpath):
+canned_wks_layer_dirs.append(cpath)
 
 cpath = os.path.join(path, CANNED_IMAGE_DIR)
 canned_wks_layer_dirs.append(cpath)
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] edgerouter.conf: enable generation of wic.bmap

[Ed Bartosh]

Enabled generation of block map(bmap) files for wic images
built for edgerouter machine. This should simplify flashing
images with bmaptool.

[YOCTO #10621]

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 meta-yocto-bsp/conf/machine/edgerouter.conf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta-yocto-bsp/conf/machine/edgerouter.conf 
b/meta-yocto-bsp/conf/machine/edgerouter.conf
index 720a947..c6193d9 100644
--- a/meta-yocto-bsp/conf/machine/edgerouter.conf
+++ b/meta-yocto-bsp/conf/machine/edgerouter.conf
@@ -18,7 +18,7 @@ USE_VT ?= "0"
 
 MACHINE_EXTRA_RRECOMMENDS = " kernel-modules"
 
-IMAGE_FSTYPES ?= "jffs2 tar.bz2 wic"
+IMAGE_FSTYPES ?= "jffs2 tar.bz2 wic wic.bmap"
 JFFS2_ERASEBLOCK = "0x1"
 
 WKS_FILE ?= "sdimage-bootpart.wks"
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] oe-buildenv-internal: show usage output

[Ed Bartosh]

Show usage text if script is not sourced.
Tested in bash, zsh and dash.

[YOCTO #10751]

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 scripts/oe-buildenv-internal | 12 
 1 file changed, 12 insertions(+)

diff --git a/scripts/oe-buildenv-internal b/scripts/oe-buildenv-internal
index 9fae3b4..c890552 100755
--- a/scripts/oe-buildenv-internal
+++ b/scripts/oe-buildenv-internal
@@ -18,6 +18,18 @@
 # along with this program; if not, write to the Free Software
 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 
+if ! $(return >/dev/null 2>&1) ; then
+echo 'oe-buildenv-internal: error: this script must be sourced'
+echo ''
+echo 'Usage: . $OEROOT/scripts/oe-buildenv-internal &&'
+echo ''
+echo 'OpenEmbedded oe-buildenv-internal - an internal script that is'
+echo 'used in oe-init-build-env and oe-init-build-env-memres to'
+echo 'initialize oe build environment'
+echo ''
+exit 2
+fi
+
 # It is assumed OEROOT is already defined when this is called
 if [ -z "$OEROOT" ]; then
 echo >&2 "Error: OEROOT is not defined!"
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] canned-wks: remove mpc8315e-rdb.wks

[Ed Bartosh]

This file has been moved to meta-yocto-bsp/wic/

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 scripts/lib/wic/canned-wks/mpc8315e-rdb.wks | 6 --
 1 file changed, 6 deletions(-)
 delete mode 100644 scripts/lib/wic/canned-wks/mpc8315e-rdb.wks

diff --git a/scripts/lib/wic/canned-wks/mpc8315e-rdb.wks 
b/scripts/lib/wic/canned-wks/mpc8315e-rdb.wks
deleted file mode 100644
index af587b2..000
--- a/scripts/lib/wic/canned-wks/mpc8315e-rdb.wks
+++ /dev/null
@@ -1,6 +0,0 @@
-# short-description: Create SD card image with a boot partition
-# long-description: Creates a partitioned SD card image. Boot files
-# are located in the first partition.
-
-part /boot --source bootimg-partition --ondisk sdb --fstype=ext3 --label boot
-part / --source rootfs --ondisk sdb --fstype=ext3 --label root
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 2/2] wic: _exec_cmd: produce error if exit code is not 0

[Ed Bartosh]

Current code doesn't always show error output of the
external command and even ignores non-zero exit code.

Moved checking of exit code value to the lowest level
possible: to _exec_cmd. This should make wic to always
check exit code of the external command and issue
an error if it's not 0.

[YOCTO #10816]

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 scripts/lib/wic/utils/oe/misc.py | 14 --
 1 file changed, 4 insertions(+), 10 deletions(-)

diff --git a/scripts/lib/wic/utils/oe/misc.py b/scripts/lib/wic/utils/oe/misc.py
index fe188c9..fd71935 100644
--- a/scripts/lib/wic/utils/oe/misc.py
+++ b/scripts/lib/wic/utils/oe/misc.py
@@ -65,6 +65,10 @@ def _exec_cmd(cmd_and_args, as_shell=False, catch=3):
 else:
 ret, out = runner.runtool(args, catch)
 out = out.strip()
+if ret != 0:
+msger.error("_exec_cmd: %s returned '%s' instead of 0\noutput: %s" % \
+(cmd_and_args, ret, out))
+
 msger.debug("_exec_cmd: output for %s (rc = %d): %s" % \
 (cmd_and_args, ret, out))
 
@@ -79,10 +83,6 @@ def exec_cmd(cmd_and_args, as_shell=False, catch=3):
 """
 ret, out = _exec_cmd(cmd_and_args, as_shell, catch)
 
-if ret != 0:
-msger.error("exec_cmd: %s returned '%s' instead of 0" % \
-(cmd_and_args, ret))
-
 return out
 
 def exec_native_cmd(cmd_and_args, native_sysroot, catch=3, pseudo=""):
@@ -126,12 +126,6 @@ def exec_native_cmd(cmd_and_args, native_sysroot, catch=3, 
pseudo=""):
 msg += "Wic failed to find a recipe to build native %s. Please "\
"file a bug against wic.\n" % prog
 msger.error(msg)
-if out:
-msger.debug('"%s" output: %s' % (args[0], out))
-
-if ret != 0:
-msger.error("exec_cmd: '%s' returned '%s' instead of 0" % \
-(cmd_and_args, ret))
 
 return ret, out
 
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 1/2] direct.py: fix getting image name

[Ed Bartosh]

part.rootfs_dir was used as an image name in the code.
However, when multi-rootfs feature is used this attribute
points to the name of the rootfs, e.g. if --rootfs command line
is rootfs1=core-image-minimal partf.rootfs_dir is 'rootfs1'.

The code also fails when image name is not provided in wic
commandline. For example, when wic is called with
--rootfs-dir= part.rootfs_dir will contain path and
wic will crash trying to call bitbake -e  to get
value of ROOTFS_SIZE variable.

Fixed the code by getting image name properly and checking
if it's not a path.

[YOCTO #10815]

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 scripts/lib/wic/imager/direct.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/wic/imager/direct.py b/scripts/lib/wic/imager/direct.py
index 11ec15e..52828c1 100644
--- a/scripts/lib/wic/imager/direct.py
+++ b/scripts/lib/wic/imager/direct.py
@@ -270,8 +270,8 @@ class DirectImageCreator(BaseImageCreator):
 # get rootfs size from bitbake variable if it's not set in .ks file
 if not part.size:
 # and if rootfs name is specified for the partition
-image_name = part.rootfs_dir
-if image_name:
+image_name = self.rootfs_dir.get(part.rootfs_dir)
+if image_name and os.path.sep not in image_name:
 # Bitbake variable ROOTFS_SIZE is calculated in
 # Image._get_rootfs_size method from meta/lib/oe/image.py
 # using IMAGE_ROOTFS_SIZE, IMAGE_ROOTFS_ALIGNMENT,
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v1] wic: Add --exclude-path option to rootfs source plugin.

[Ed Bartosh]

On Wed, Nov 30, 2016 at 02:30:12PM +0100, Kristian Amlie wrote:
> On 28/11/16 12:18, Ed Bartosh wrote:
> > I'd appreciate if you also do the following before sending v2:
> >  - check that your code doesn't regress pylint checks
> 
> Is there an official process here? Sorry, it's not very well documented.
No official process yet, I'm sorry. I can put it on wiki if it helps.

> Just running pylint3 manually on the scripts/lib/wic folder gives
> identical results, however, running on rootfs.py alone, the rating
> increases:
> 
>   Your code has been rated at 5.67/10 (previous run: 2.26/10, +3.41)
Looks good to me. Thanks.
Running pylint on the files you've changed before and after the changes should 
be enough.

> >  - add test case for your code to meta/lib/oeqa/selftest/wic.py
> >  - check that your code doesn't break oe-selftest --coverage -r wic and
> >doesn't regress test coverage
> 
> In the interest of saving me some build time, am I correct in assuming
> that I need the "qemux86-64" MACHINE target for this to work? It appears
> to be dependent on syslinux which has COMPATIBLE_HOSTS set to non-arm
> only. Sorry, I'm a bit new to this framework.
> 

You're right. oe-selftest requires qemux86-64 target.

Even if you usually work in different environment it makes sense to run
oe-selftest to ensure that your changes don't breake it.
This is because our maintainers use oe-selftest results as one of the
acceptance criterias. It means that your changes would be be rejected anyway
if they break oe-selftest, so it's better to check it in advance.

--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] buildhistory-diff: report directory renames

[Ed Bartosh]

On Tue, Nov 29, 2016 at 03:51:21PM +0200, Alexander Kanavin wrote:
> On 11/28/2016 07:28 PM, Ed Bartosh wrote:
> >The script detects directory renaming if two different
> >directories with the same set of files are added and removed.
> 
> What happens if the sets of files are slightly different? (e.g. a
> couple of files are added or removed). Can we get some wiggle space
> here?
> 
In this case script will behave like it does currently. It will show all
added and removed files.

--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [wic][PATCH] selftest: wic: fix test_qemu

[Ed Bartosh]

Setting WKS_FILE variable in qemux86-64 made wic test to
use wrong wks file to produce an image and resulted in
test_qemu failure.

Used conditional assignment in qemux86-64 and explicitly
set WKS_FILE in wic testing suite to make the suite to use
wic-image-minimal.wsk. This should fix test_qemu failure.

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 meta/conf/machine/qemux86-64.conf | 2 +-
 meta/lib/oeqa/selftest/wic.py | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/meta/conf/machine/qemux86-64.conf 
b/meta/conf/machine/qemux86-64.conf
index 7559724..3117ca1 100644
--- a/meta/conf/machine/qemux86-64.conf
+++ b/meta/conf/machine/qemux86-64.conf
@@ -29,5 +29,5 @@ MACHINE_FEATURES += "x86"
 
 MACHINE_ESSENTIAL_EXTRA_RDEPENDS += "v86d"
 
-WKS_FILE = "directdisk.wks"
+WKS_FILE ?= "directdisk.wks"
 do_image_wic[depends] += "syslinux:do_build 
syslinux-native:do_populate_sysroot mtools-native:do_populate_sysroot 
dosfstools-native:do_populate_sysroot"
diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index faac11e..e652fad 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -42,7 +42,8 @@ class Wic(oeSelfTest):
 def setUpLocal(self):
 """This code is executed before each test method."""
 self.write_config('IMAGE_FSTYPES += " hddimg"\n'
-  'MACHINE_FEATURES_append = " efi"\n')
+  'MACHINE_FEATURES_append = " efi"\n'
+  'WKS_FILE = "wic-image-minimal"\n')
 
 # Do this here instead of in setUpClass as the base setUp does some
 # clean up which can result in the native tools built earlier in
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v1] wic: Add --exclude-path option to rootfs source plugin.

[Ed Bartosh]

On Fri, Dec 02, 2016 at 03:36:47PM +0100, Kristian Amlie wrote:
> On 30/11/16 16:29, Ed Bartosh wrote:
> >> ...
> > 
> > You're right. oe-selftest requires qemux86-64 target.
> > 
> > Even if you usually work in different environment it makes sense to run
> > oe-selftest to ensure that your changes don't breake it.
> > This is because our maintainers use oe-selftest results as one of the
> > acceptance criterias. It means that your changes would be be rejected anyway
> > if they break oe-selftest, so it's better to check it in advance.
> 
> This is taking some time due to unrelated issues, but in the meantime:
> 
> What are the thoughts on having a similar --exclude-path mechanism for
> the standard rootfs building? IOW, the do_image_ext4 and friends?

That's a great idea! If source rootfs direcotry contains only requred
content then we don't even need to change wic code. It's much better
then implementing the same functionality in wic and in oe code.

> Our usecase is that we use wic to build a complete image, using
> --exclude-path to group things into partitions. And then we want to
> build a regular ext4 image and use this as an update to the device, and
> in this case we want the same content as wic generates, but in a raw
> rootfs file instead.

--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] oe-selftest: Improve BUILDDIR environment handling

[Ed Bartosh]

On Wed, Jan 04, 2017 at 11:48:53PM +, Richard Purdie wrote:
> Its possible something (like bitbake/tinfoil2) may mess around with the
> environment and using the enviroment as a global variable store isn't
> particularly nice anyway.
> 
> This patch changes the BUILDDIR usages so that the environment isn't used
> as a global store and a global variable is used instead. Whilst that
> is still not perfect, it does avoid the current double and triple backtraces
> we're seeing where tinfoil2/bitbake has trampled the enviroment leading
> to failures of failures making debugging even harder.
> 
> Signed-off-by: Richard Purdie 
> ---
>  scripts/oe-selftest | 19 +--
>  1 file changed, 13 insertions(+), 6 deletions(-)
> 
> diff --git a/scripts/oe-selftest b/scripts/oe-selftest
> index bfcea66..e166521 100755
> --- a/scripts/oe-selftest
> +++ b/scripts/oe-selftest
> @@ -111,9 +111,13 @@ def get_args_parser():
>  help='Submit test results to a repository')
>  return parser
>  
> +builddir = None
> +
>  
>  def preflight_check():
>  
> +global builddir
> +
>  log.info("Checking that everything is in order before running the tests")
>  
>  if not os.environ.get("BUILDDIR"):
> @@ -135,7 +139,7 @@ def preflight_check():
>  return True
>  
>  def add_include():
> -builddir = os.environ.get("BUILDDIR")
> +global builddir
You don't need to use 'global' here. It's only mandatory if
you change variable value:
https://docs.python.org/3/reference/simple_stmts.html#the-global-statement

Would it be more readable to use name in upper case: BUILDDIR?

>  if "#include added by oe-selftest.py" \
>  not in ftools.read_file(os.path.join(builddir, "conf/local.conf")):
>  log.info("Adding: \"include selftest.inc\" in local.conf")
> @@ -149,7 +153,7 @@ def add_include():
>  "\n#include added by oe-selftest.py\ninclude 
> bblayers.inc")
>  
>  def remove_include():
> -builddir = os.environ.get("BUILDDIR")
> +global builddir
>  if builddir is None:
>  return
>  if "#include added by oe-selftest.py" \
> @@ -165,18 +169,21 @@ def remove_include():
>  "\n#include added by oe-selftest.py\ninclude 
> bblayers.inc")
>  
>  def remove_inc_files():
> +global builddir
> +if builddir is None:
> +return
>  try:
> -os.remove(os.path.join(os.environ.get("BUILDDIR"), 
> "conf/selftest.inc"))
> +os.remove(os.path.join(builddir, "conf/selftest.inc"))
>  for root, _, files in os.walk(get_test_layer()):
>  for f in files:
>  if f == 'test_recipe.inc':
>  os.remove(os.path.join(root, f))
> -except (AttributeError, OSError,) as e:# AttributeError may happen 
> if BUILDDIR is not set
> +except OSError as e:
>  pass
>  
>  for incl_file in ['conf/bblayers.inc', 'conf/machine.inc']:
>  try:
> -os.remove(os.path.join(os.environ.get("BUILDDIR"), incl_file))
> +os.remove(os.path.join(builddir, incl_file))
>  except:
>  pass
>  
> @@ -394,7 +401,7 @@ def coverage_setup(coverage_source, coverage_include, 
> coverage_omit):
>  """ Set up the coverage measurement for the testcases to be run """
>  import datetime
>  import subprocess
> -builddir = os.environ.get("BUILDDIR")
> +global builddir
>  pokydir = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
>  curcommit= subprocess.check_output(["git", "--git-dir", 
> os.path.join(pokydir, ".git"), "rev-parse", "HEAD"]).decode('utf-8')
>  coveragerc = "%s/.coveragerc" % builddir
> -- 
> 2.7.4
> 
> -- 
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

-- 
--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] oe-selftest: Improve BUILDDIR environment handling

[Ed Bartosh]

On Thu, Jan 05, 2017 at 10:10:50AM +, Richard Purdie wrote:
> On Thu, 2017-01-05 at 11:40 +0200, Ed Bartosh wrote:
> > On Wed, Jan 04, 2017 at 11:48:53PM +, Richard Purdie wrote:
> > > 
> > > Its possible something (like bitbake/tinfoil2) may mess around with
> > > the
> > > environment and using the enviroment as a global variable store
> > > isn't
> > > particularly nice anyway.
> > > 
> > > This patch changes the BUILDDIR usages so that the environment
> > > isn't used
> > > as a global store and a global variable is used instead. Whilst
> > > that
> > > is still not perfect, it does avoid the current double and triple
> > > backtraces
> > > we're seeing where tinfoil2/bitbake has trampled the enviroment
> > > leading
> > > to failures of failures making debugging even harder.
> > > 
> > > Signed-off-by: Richard Purdie <richard.pur...@linuxfoundation.org>
> > > ---
> > >  scripts/oe-selftest | 19 +--
> > >  1 file changed, 13 insertions(+), 6 deletions(-)
> > > 
> > > diff --git a/scripts/oe-selftest b/scripts/oe-selftest
> > > index bfcea66..e166521 100755
> > > --- a/scripts/oe-selftest
> > > +++ b/scripts/oe-selftest
> > > @@ -111,9 +111,13 @@ def get_args_parser():
> > >  help='Submit test results to a
> > > repository')
> > >  return parser
> > >  
> > > +builddir = None
> > > +
> > >  
> > >  def preflight_check():
> > >  
> > > +global builddir
> > > +
> > >  log.info("Checking that everything is in order before running
> > > the tests")
> > >  
> > >  if not os.environ.get("BUILDDIR"):
> > > @@ -135,7 +139,7 @@ def preflight_check():
> > >  return True
> > >  
> > >  def add_include():
> > > -builddir = os.environ.get("BUILDDIR")
> > > +global builddir
> > You don't need to use 'global' here. It's only mandatory if
> > you change variable value:
> > https://docs.python.org/3/reference/simple_stmts.html#the-global-stat
> > ement
> 
> In this case I did it purely for readability to make it clear how we
> were expecting it to work and to match the other uses. It doesn't hurt
> anything afaik.
> 
> > Would it be more readable to use name in upper case: BUILDDIR?
> 
> Not sure, that isn't something we've used as a style anywhere else so
> it doesn't really match any other style. Is that a python convention?
No, it's not. I thought it would be more readable as uppercase name
is the same as environment variable name. And it would be different
from local variable names, so you wouldn't need to use 'global'.

--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v4] wic/isoimage-isohybrid: remove do_stage_partition()

[Ed Bartosh]

Hi Ioan-Adrian,

Thank you for the patch!

It looks good to me.
+1

On Sat, Jan 07, 2017 at 08:03:01PM +0200, Ioan-Adrian Ratiu wrote:
> The purpouse of this function was to check dependencies for building a
> hybrid iso and build them using bitbake if not found. Calling bitbake in
> this context means this wic plugin itself cannot be instrumented inside
> bitbake recipes which is undesirable, the benefits of this are clear:
> there is no need to maintain outside scripts to generate an iso using wic
> and the isohybrid building logic can be further abstracted away into an
> isohybrid.bbclass in the future which can be easily inherited or something
> similar.
> 
> So remove the function and add all dependencies to NATIVE_RECIPES so that
> wic can print useful errors when they're not built.
> 
> To automate building the isohybrid image dependencies, add the following
> somewhere in your image build inheritence hierarcy (or maybe create a
> bbclass in the future to do these sort of things automatically):
> 
> DEPENDS += "syslinux syslinux-native cdrtools-native e2fsprogs-native \
> parted-native dosfstools-native mtools-native grub-efi-native"
> 
> Signed-off-by: Ioan-Adrian Ratiu 
> ---
>  .../lib/wic/plugins/source/isoimage-isohybrid.py   | 49 
> --
>  scripts/lib/wic/utils/oe/misc.py   |  4 ++
>  2 files changed, 4 insertions(+), 49 deletions(-)
> 
> diff --git a/scripts/lib/wic/plugins/source/isoimage-isohybrid.py 
> b/scripts/lib/wic/plugins/source/isoimage-isohybrid.py
> index 3858fd439b..15fd858d22 100644
> --- a/scripts/lib/wic/plugins/source/isoimage-isohybrid.py
> +++ b/scripts/lib/wic/plugins/source/isoimage-isohybrid.py
> @@ -194,55 +194,6 @@ class IsoImagePlugin(SourcePlugin):
>  return initrd
>  
>  @classmethod
> -def do_stage_partition(cls, part, source_params, creator, cr_workdir,
> -   oe_builddir, bootimg_dir, kernel_dir,
> -   native_sysroot):
> -"""
> -Special content staging called before do_prepare_partition().
> -It cheks if all necessary tools are available, if not
> -tries to instal them.
> -"""
> -# Make sure parted is available in native sysroot
> -if not os.path.isfile("%s/usr/sbin/parted" % native_sysroot):
> -msger.info("Building parted-native...\n")
> -exec_cmd("bitbake parted-native")
> -
> -# Make sure mkfs.ext2/3/4 is available in native sysroot
> -if not os.path.isfile("%s/sbin/mkfs.ext2" % native_sysroot):
> -msger.info("Building e2fsprogs-native...\n")
> -exec_cmd("bitbake e2fsprogs-native")
> -
> -# Make sure syslinux is available in sysroot and in native sysroot
> -syslinux_dir = get_bitbake_var("STAGING_DATADIR")
> -if not syslinux_dir:
> -msger.error("Couldn't find STAGING_DATADIR, exiting.\n")
> -if not os.path.exists("%s/syslinux" % syslinux_dir):
> -msger.info("Building syslinux...\n")
> -exec_cmd("bitbake syslinux")
> -if not os.path.exists("%s/syslinux" % syslinux_dir):
> -msger.error("Please build syslinux first\n")
> -
> -# Make sure syslinux is available in native sysroot
> -if not os.path.exists("%s/usr/bin/syslinux" % native_sysroot):
> -msger.info("Building syslinux-native...\n")
> -exec_cmd("bitbake syslinux-native")
> -
> -#Make sure mkisofs is available in native sysroot
> -if not os.path.isfile("%s/usr/bin/mkisofs" % native_sysroot):
> -msger.info("Building cdrtools-native...\n")
> -exec_cmd("bitbake cdrtools-native")
> -
> -# Make sure mkfs.vfat is available in native sysroot
> -if not os.path.isfile("%s/sbin/mkfs.vfat" % native_sysroot):
> -msger.info("Building dosfstools-native...\n")
> -exec_cmd("bitbake dosfstools-native")
> -
> -# Make sure mtools is available in native sysroot
> -if not os.path.isfile("%s/usr/bin/mcopy" % native_sysroot):
> -msger.info("Building mtools-native...\n")
> -exec_cmd("bitbake mtools-native")
> -
> -@classmethod
>  def do_configure_partition(cls, part, source_params, creator, cr_workdir,
> oe_builddir, bootimg_dir, kernel_dir,
> native_sysroot):
> diff --git a/scripts/lib/wic/utils/oe/misc.py 
> b/scripts/lib/wic/utils/oe/misc.py
> index fe188c9d26..489393a24e 100644
> --- a/scripts/lib/wic/utils/oe/misc.py
> +++ b/scripts/lib/wic/utils/oe/misc.py
> @@ -35,8 +35,11 @@ from wic.utils import runner
>  
>  # executable -> recipe pairs for exec_native_cmd
>  NATIVE_RECIPES = {"bmaptool": "bmap-tools",
> +  "grub-mkimage": "grub-efi",
> +  "isohybrid": "syslinux",
>"mcopy": "mtools",
>

[OE-core] [PATCH 2/2] populate_sdk_ext: whitelist do_package tasks

[Ed Bartosh]

With enabled SSTATE_MIRRORS sstate code expects mirrors to
contain entries for all tasks, which is not the case for ext
installer as it uses reduced sstate cache.

Added do_package tasks to BB_SETSCENE_ENFORCE_WHITELIST to prevent
installer failing with ERROR: Sstate artifact unavailable

[YOCTO #10832]

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>

diff --git a/meta/classes/populate_sdk_ext.bbclass 
b/meta/classes/populate_sdk_ext.bbclass
index fc9cc22..d1b3fb4 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -305,7 +305,7 @@ python copy_buildsystem () {
 f.write('SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "warn"\n\n')
 
 # Set up whitelist for run on install
-f.write('BB_SETSCENE_ENFORCE_WHITELIST = "%:* *:do_shared_workdir 
*:do_rm_work"\n\n')
+f.write('BB_SETSCENE_ENFORCE_WHITELIST = "%:* *:do_shared_workdir 
*:do_rm_work *:do_package"\n\n')
 
 # Hide the config information from bitbake output (since it's 
fixed within the SDK)
 f.write('BUILDCFG_HEADER = ""\n\n')
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH 1/2] populate_sdk_ext: fix working with uninative sstate

[Ed Bartosh]

Mapped uninative sstate directories to make ext SDK installer to
use them when it's run on systems with gcc version different from
gcc version used to build installer.

[YOCTO #10832]

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>

diff --git a/meta/classes/populate_sdk_ext.bbclass 
b/meta/classes/populate_sdk_ext.bbclass
index 0812eea..fc9cc22 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -308,7 +308,10 @@ python copy_buildsystem () {
 f.write('BB_SETSCENE_ENFORCE_WHITELIST = "%:* *:do_shared_workdir 
*:do_rm_work"\n\n')
 
 # Hide the config information from bitbake output (since it's 
fixed within the SDK)
-f.write('BUILDCFG_HEADER = ""\n')
+f.write('BUILDCFG_HEADER = ""\n\n')
+
+# Map gcc-dependent uninative sstate cache for installer usage
+f.write('SSTATE_MIRRORS = "file://universal/(.*) 
file://universal-4.9/\\1\\nfile://universal-4.9/(.*) 
file://universal-4.8/\\1"\n\n')
 
 # Allow additional config through sdk-extra.conf
 fn = bb.cookerdata.findConfigFile('sdk-extra.conf', d)
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v2] wic/isoimage-isohybrid: check for grub-mkimage

[Ed Bartosh]

On Wed, Jan 04, 2017 at 09:04:43PM +0200, Ioan-Adrian Ratiu wrote:
> The isohybrid plugin uses grub-mkimage but doesn't make sure it gets built.
> Add a check to avoid the following error:
> 
> Error: A native program grub-mkimage required to build the image was not 
> found (see details above).
> 
> Wic failed to find a recipe to build native grub-mkimage. Please file a bug 
> against wic.
> 
> Signed-off-by: Ioan-Adrian Ratiu 
> ---
>  scripts/lib/wic/plugins/source/isoimage-isohybrid.py | 5 +
>  1 file changed, 5 insertions(+)
> 
> diff --git a/scripts/lib/wic/plugins/source/isoimage-isohybrid.py 
> b/scripts/lib/wic/plugins/source/isoimage-isohybrid.py
> index 849fd8bea3..d616316d61 100644
> --- a/scripts/lib/wic/plugins/source/isoimage-isohybrid.py
> +++ b/scripts/lib/wic/plugins/source/isoimage-isohybrid.py
> @@ -242,6 +242,11 @@ class IsoImagePlugin(SourcePlugin):
>  msger.info("Building mtools-native...\n")
>  exec_cmd("bitbake mtools-native")
>  
> +# Make sure grub-mkimage is available in native sysroot
> +if not os.path.isfile("%s/usr/bin/grub-mkimage" % native_sysroot):
> +msger.info("Building grub-native...\n")
> +exec_cmd("bitbake grub-native")
> +
>  @classmethod
>  def do_configure_partition(cls, part, source_params, creator, cr_workdir,
> oe_builddir, bootimg_dir, kernel_dir,

Unfortunately this approach will not work if wic is run from bitbake.

I'd suggest to simply add grub-native to NATIVE_RECIPES dictionary in
scripts/lib/wic/utils/oe/misc.py
It will make error message more informative and useful.

--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v2] wic/isoimage-isohybrid: check for grub-mkimage

[Ed Bartosh]

On Wed, Jan 04, 2017 at 10:12:55PM +0200, Ed Bartosh wrote:
> On Wed, Jan 04, 2017 at 09:04:43PM +0200, Ioan-Adrian Ratiu wrote:
> > The isohybrid plugin uses grub-mkimage but doesn't make sure it gets built.
> > Add a check to avoid the following error:
> > 
> > Error: A native program grub-mkimage required to build the image was not 
> > found (see details above).
> > 
> > Wic failed to find a recipe to build native grub-mkimage. Please file a bug 
> > against wic.
> > 
> > Signed-off-by: Ioan-Adrian Ratiu <adrian.ra...@ni.com>
> > ---
> >  scripts/lib/wic/plugins/source/isoimage-isohybrid.py | 5 +
> >  1 file changed, 5 insertions(+)
> > 
> > diff --git a/scripts/lib/wic/plugins/source/isoimage-isohybrid.py 
> > b/scripts/lib/wic/plugins/source/isoimage-isohybrid.py
> > index 849fd8bea3..d616316d61 100644
> > --- a/scripts/lib/wic/plugins/source/isoimage-isohybrid.py
> > +++ b/scripts/lib/wic/plugins/source/isoimage-isohybrid.py
> > @@ -242,6 +242,11 @@ class IsoImagePlugin(SourcePlugin):
> >  msger.info("Building mtools-native...\n")
> >  exec_cmd("bitbake mtools-native")
> >  
> > +# Make sure grub-mkimage is available in native sysroot
> > +if not os.path.isfile("%s/usr/bin/grub-mkimage" % native_sysroot):
> > +msger.info("Building grub-native...\n")
> > +exec_cmd("bitbake grub-native")
> > +
> >  @classmethod
> >  def do_configure_partition(cls, part, source_params, creator, 
> > cr_workdir,
> > oe_builddir, bootimg_dir, kernel_dir,
> 
> Unfortunately this approach will not work if wic is run from bitbake.
> 
> I'd suggest to simply add grub-native to NATIVE_RECIPES dictionary in
> scripts/lib/wic/utils/oe/misc.py
> It will make error message more informative and useful.
> 
The same should probably be done with the rest of native tools called from
this module. It's better to avoid building them implicitly.

--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v6 0/5] wic: bugfixes & --fixed-size support, tests, oeqa: enhancements

[Ed Bartosh]

On Tue, Dec 20, 2016 at 07:27:04PM +0100, Maciej Borzęcki wrote:
> On Tue, Dec 20, 2016 at 3:04 PM, Ed Bartosh <ed.bart...@linux.intel.com> 
> wrote:
> > On Mon, Dec 19, 2016 at 12:20:56PM +0100, Maciej Borzecki wrote:
> >> v6 of a patch series previously posted here [1].
> >>
> >> Changes since v5:
> >>
> >> * `oe-selftest: enforce en_US.UTF-8 locale` was merged to master, hence 
> >> it's no
> >>   longer part of this series
> >>
> >> * rebased on top of master (d0d260b0b85790ceb136dd6b0445e8e33d038f5e)
> >>
> >
> > The patchset looks good to me. Thank you.
> >
> > Couple of questions:
> > - did you test if Lars' change
> >   
> > https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=db08ffee0ad1451f3bf710f4d1b623938ba9aefb
> >   works with your patchset?
> 
> Yes. The series was rebased on top of
> d0d260b0b85790ceb136dd6b0445e8e33d038f5e, which includes Lars' patch.

The series causes conflicts on the latest master. Can you check if it's
really the case or it's just something wrong with my setup?

git-pw apply 3904
Applying series: wic: bugfixes & --fixed-size support, tests,
oe-selftest: minor fixes (rev 3)
Applying: oe-selftest: enforce en_US.UTF-8 locale
Using index info to reconstruct a base tree...
M   scripts/oe-selftest
Falling back to patching base and 3-way merge...
No changes -- Patch already applied.
Applying: oeqa/utils/commands.py: allow use of binaries from native
sysroot
Applying: wic: add --fixed-size wks option
Applying: wic: selftest: avoid COMPATIBLE_HOST issues
Using index info to reconstruct a base tree...
M   meta/lib/oeqa/selftest/wic.py
Falling back to patching base and 3-way merge...
Auto-merging meta/lib/oeqa/selftest/wic.py
CONFLICT (content): Merge conflict in meta/lib/oeqa/selftest/wic.py
Failed to merge in the changes.
Patch failed at 0004 wic: selftest: avoid COMPATIBLE_HOST issues
The copy of the patch that failed is found in:
   /home/ed/git/yocto/patchtest/openembedded-core/.git/rebase-apply/patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".


> The tests passed on all MACHINES I've used (though I'm not sure if
> scenario addressed in that patch is covered by relevant test cases).
>

I've asked Lars to add test case for that functionality.

> I believe I have also hit a weird bug in BitbakeVars._parse_line() while
> testing with this patch and meta-mender. I Hope you can confirm it.
>

Yes, I confirm this. Thanks for poining out to it. I'll fix it today.

> From a quick investigation I have determined that _parse_line() may
> interpret arbitrary lines that are of 'key=value' format as legitimate
> bitbake variables. In my case, IMAGE_CMD_sdimg defined in
> mender-sdimg.bbclass includes the following piece:
> 
> IMAGE_CMD_sdimg() {
> ...
> REMAINING_SIZE=$(expr ${MENDER_STORAGE_TOTAL_SIZE_MB} - \
> ${MENDER_BOOT_PART_SIZE_MB} - \
> ${MENDER_DATA_PART_SIZE_MB} - \
> ${MENDER_PARTITIONING_OVERHEAD_MB} - \
> $boot_env_size_mb)
> ROOTFS_SIZE=$(expr $REMAINING_SIZE / 2)
> ...
> }
> 
> This will obviously be seen in `bitbake -e output`. Looking at code
> here:
> https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/scripts/lib/wic/utils/oe/misc.py?id=db08ffee0ad1451f3bf710f4d1b623938ba9aefb#n151
> the line will be split at '=', then whitespace will be stripped from
> both key and value. So a line (taken from bitbake -e):
> 
> ROOTFS_SIZE=\$(expr \$REMAINING_SIZE / 2)
> 
> Will define a variable ROOTFS_SIZE with value '\\$(expr \\$REMAINING_SIZE / 
> 2)'.
> 
> It's a pure coincidence that Lars' patch queries ROOTFS_SIZE which also
> appears in bitbake -e output under exactly the same name.
> 
> IMO, when parsing env output, we should expect variable name to start
> right at the beginning of a line, IOW skip the line if key[0].isspace() == 
> True
> 
> >
> > - Regarding oe-selftest fixes, wich machine did you run oe-selftest on?
> >   It looks to me that a lot of test cases can be run on your machine
> >   if we use another wks.
> 
> I've run it on beaglebone, vexpress-qemu, qemux86 and qemuarm. Agree
> with your observation though. I think that at least these TCs could be
> made machine agnostic:
> 
> - test_build_artifacts
> - test_build_image_name
> - test_compress_*
> - test_gpt_image
> - test_wic_image_type
> - test_wrong_compressor
> - test_rootfs_*
> 
> 

Yes, that's exactly my point. We should make generic tests work on any
machine.

Would you be wil

[OE-core] [wic][PATCH] wic: fix parsing of 'bitbake -e' output

[Ed Bartosh]

Current parsing code can wrongly interpret arbitrary lines
that are of 'key=value' format as legitimate bitbake variables.

Implemented more strict parsing of key=value pairs using
regular expressions.

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 scripts/lib/wic/utils/oe/misc.py | 12 +---
 1 file changed, 5 insertions(+), 7 deletions(-)

diff --git a/scripts/lib/wic/utils/oe/misc.py b/scripts/lib/wic/utils/oe/misc.py
index fe188c9..1dbbe92 100644
--- a/scripts/lib/wic/utils/oe/misc.py
+++ b/scripts/lib/wic/utils/oe/misc.py
@@ -27,6 +27,7 @@
 """Miscellaneous functions."""
 
 import os
+import re
 from collections import defaultdict
 from distutils import spawn
 
@@ -155,14 +156,11 @@ class BitbakeVars(defaultdict):
 """
 if "=" not in line:
 return
-try:
-key, val = line.split("=")
-except ValueError:
+match = re.match("^(\w+)=(.+)", line)
+if not match:
 return
-key = key.strip()
-val = val.strip()
-if key.replace('_', '').isalnum():
-self[image][key] = val.strip('"')
+key, val = match.groups()
+self[image][key] = val.strip('"')
 
 def get_var(self, var, image=None):
 """
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v2 0/4] selftest/wic: extending test coverage for WIC script options

[Ed Bartosh]

On Tue, Dec 20, 2016 at 04:04:06PM -0600, Jair Gonzalez wrote:
> > -Original Message-
> > From: Ed Bartosh [mailto:ed.bart...@linux.intel.com]
> > Sent: Tuesday, December 20, 2016 7:07 AM
> > To: Jair Gonzalez <jair.de.jesus.gonzalez.plascen...@linux.intel.com>
> > Cc: openembedded-core@lists.openembedded.org
> > Subject: Re: [OE-core] [PATCH v2 0/4] selftest/wic: extending test
> coverage
> > for WIC script options
> > 
> > On Mon, Dec 19, 2016 at 03:07:30PM -0600, Jair Gonzalez wrote:
> > > Changed in V2:
> > >
> > > The original patch was splitted and updated according to the
> discussion on:
> > > http://lists.openembedded.org/pipermail/openembedded-core/2016-
> > Decembe
> > > r/130131.html
> > > NOTE: The WKS_FILE entry deletion was ommitted from this patch set as
> > > it was introduced on the first patch by mistake.
> > >
> > > The following changes since commit
> > 573c646d4cc62dcd0c230381df4940bdf314d495:
> > >
> > >   bitbake: BBHandler: use with instead of open/close (2016-12-16
> > > 10:23:24 +)
> > >
> > > are available in the git repository at:
> > >
> > >   git://git.yoctoproject.org/poky-contrib jairglez/wictest
> > >
> > > http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=jairglez/wict
> > > est
> > >
> > > Jair Gonzalez (4):
> > >   selftest/wic: adding Testopia ID numbers to test cases missing it
> > >   selftest/wic: code cleanup
> > >   selftest/wic: reorganizing test methods by functionality
> > >   selftest/wic: extending test coverage for WIC script options
> > >
> > >  meta/lib/oeqa/selftest/wic.py | 370
> > > +++---
> > >  1 file changed, 238 insertions(+), 132 deletions(-)
> > >
> > 
> > Thank you for the updated patchset!
> > It looks good to me.
> > 
> > BTW, did you measure its impact on the test run time?
> > 
> > --
> > Regards,
> > Ed
> 
> Hi Ed,
> 
> Thanks. Regarding the impact, I tested it on my local machine after cleaning
> the cache, sstate-cache, downloads and tmp directories, and the difference
> between them was about 130 seconds after I applied my commits:
> Before:
> Ran 28 tests in 4446.939s
> After:
> Ran 39 tests in 4578.604s
> Difference:
> 131.665s
> 
> I also tried with a source repository with existing cache and the difference
> was about 70 seconds:
> Before:
> Ran 28 tests in 272.170s
> After:
> Ran 39 tests in 339.637s
> Difference:
> 67.467s
> 

Hi Jair,

Thanks. The difference is acceptable from my point of view.

--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v2] wic: look for wks files in /wic

[Ed Bartosh]

On Mon, Dec 19, 2016 at 03:31:29PM -0800, Khem Raj wrote:
> On Mon, Dec 19, 2016 at 4:41 AM, Ed Bartosh <ed.bart...@linux.intel.com> 
> wrote:
> > Currently wic looks for wks files in
> > /scripts/lib/wic/canned-wks/ directories.
> > This path is too nested and doesn't look consistent with the
> > naming scheme of layer directories.
> >
> > Added /wic directory to the list of paths
> > to look for wks files.
> 
> Does it also affect documentation ?
> 

Yes, in a way. There is only indirect mentioning of
scripts/lib/wic/canned-wks/ in the documentation. I'm going to add new
path there or even directly mention both paths in dev manual.

> >
> > Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
> > ---
> >  scripts/lib/wic/engine.py | 7 +--
> >  1 file changed, 5 insertions(+), 2 deletions(-)
> >
> > diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
> > index 5b10463..2adef2f 100644
> > --- a/scripts/lib/wic/engine.py
> > +++ b/scripts/lib/wic/engine.py
> > @@ -52,6 +52,7 @@ def verify_build_env():
> >
> >  CANNED_IMAGE_DIR = "lib/wic/canned-wks" # relative to scripts
> >  SCRIPTS_CANNED_IMAGE_DIR = "scripts/" + CANNED_IMAGE_DIR
> > +WIC_DIR = "wic"
> >
> >  def build_canned_image_list(path):
> >  layers_path = misc.get_bitbake_var("BBLAYERS")
> > @@ -59,8 +60,10 @@ def build_canned_image_list(path):
> >
> >  if layers_path is not None:
> >  for layer_path in layers_path.split():
> > -cpath = os.path.join(layer_path, SCRIPTS_CANNED_IMAGE_DIR)
> > -canned_wks_layer_dirs.append(cpath)
> > +for wks_path in (WIC_DIR, SCRIPTS_CANNED_IMAGE_DIR):
> > +cpath = os.path.join(layer_path, wks_path)
> > +if os.path.isdir(cpath):
> > +canned_wks_layer_dirs.append(cpath)
> >
> >  cpath = os.path.join(path, CANNED_IMAGE_DIR)
> >  canned_wks_layer_dirs.append(cpath)
> > --
> > 2.1.4
> >
> > --
> > ___
> > Openembedded-core mailing list
> > Openembedded-core@lists.openembedded.org
> > http://lists.openembedded.org/mailman/listinfo/openembedded-core

-- 
--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v2 0/4] selftest/wic: extending test coverage for WIC script options

[Ed Bartosh]

On Mon, Dec 19, 2016 at 03:07:30PM -0600, Jair Gonzalez wrote:
> Changed in V2:
> 
> The original patch was splitted and updated according to the discussion on:
> http://lists.openembedded.org/pipermail/openembedded-core/2016-December/130131.html
> NOTE: The WKS_FILE entry deletion was ommitted from this patch set as it
> was introduced on the first patch by mistake.
> 
> The following changes since commit 573c646d4cc62dcd0c230381df4940bdf314d495:
> 
>   bitbake: BBHandler: use with instead of open/close (2016-12-16 10:23:24 
> +)
> 
> are available in the git repository at:
> 
>   git://git.yoctoproject.org/poky-contrib jairglez/wictest
>   http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=jairglez/wictest
> 
> Jair Gonzalez (4):
>   selftest/wic: adding Testopia ID numbers to test cases missing it
>   selftest/wic: code cleanup
>   selftest/wic: reorganizing test methods by functionality
>   selftest/wic: extending test coverage for WIC script options
> 
>  meta/lib/oeqa/selftest/wic.py | 370 
> +++---
>  1 file changed, 238 insertions(+), 132 deletions(-)
> 

Thank you for the updated patchset!
It looks good to me.

BTW, did you measure its impact on the test run time?

--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v6 0/5] wic: bugfixes & --fixed-size support, tests, oeqa: enhancements

[Ed Bartosh]

On Mon, Dec 19, 2016 at 12:20:56PM +0100, Maciej Borzecki wrote:
> v6 of a patch series previously posted here [1].
> 
> Changes since v5:
> 
> * `oe-selftest: enforce en_US.UTF-8 locale` was merged to master, hence it's 
> no
>   longer part of this series
> 
> * rebased on top of master (d0d260b0b85790ceb136dd6b0445e8e33d038f5e)
> 

The patchset looks good to me. Thank you.

Couple of questions:
- did you test if Lars' change
  
https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=db08ffee0ad1451f3bf710f4d1b623938ba9aefb
  works with your patchset?

- Regarding oe-selftest fixes, wich machine did you run oe-selftest on?
  It looks to me that a lot of test cases can be run on your machine
  if we use another wks.

> [1]. 
> http://lists.openembedded.org/pipermail/openembedded-core/2016-November/129187.html
> 
> Maciej Borzecki (5):
>   oeqa/utils/commands.py: allow use of binaries from native sysroot
>   wic: add --fixed-size wks option
>   wic: selftest: avoid COMPATIBLE_HOST issues
>   wic: selftest: do not assume bzImage kernel image
>   wic: selftest: add tests for --fixed-size partition flags
> 
>  meta/lib/oeqa/selftest/wic.py  | 117 
> +++--
>  meta/lib/oeqa/utils/commands.py|   9 ++-
>  scripts/lib/wic/help.py|  14 +++-
>  scripts/lib/wic/imager/direct.py   |   2 +-
>  scripts/lib/wic/ksparser.py|  41 ++--
>  scripts/lib/wic/partition.py   |  88 +++--
>  scripts/lib/wic/utils/partitionedfs.py |   2 +-
>  7 files changed, 224 insertions(+), 49 deletions(-)

--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v2 3/3] selftest/wic: Add tests for --exclude-dir option.

[Ed Bartosh]

On Mon, Dec 19, 2016 at 10:09:22AM +0100, Kristian Amlie wrote:
> 
> >> +def test_exclude_path(self):
> >> +"""Test --exclude-path wks option."""
> >> +
> >> +# For using 'e2ls'.
> >> +old_path = os.environ['PATH']
> >> +os.environ['PATH'] = get_bb_var('PATH', 'core-image-minimal')
> >> +
> >> +wks_file = 'temp.wks'
> >> +ks = open(wks_file, 'w')
> > 
> > I'd use more pythonic 'with open(wks_file, 'w') as wks:' here.
> 
> Done for all three blocks!
> 
Thank you!

The patchset looks good to me.

--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] wic: obey the rootfs size from the metadata

[Ed Bartosh]

On Fri, Dec 16, 2016 at 04:47:28PM +0200, Ed Bartosh wrote:
> Hi Christopher,
> 
> Thank you for the patch!
> 
> +1
> 

Would you be willing to add test case(s) for this functionality to wic
selftest module?

> On Thu, Dec 15, 2016 at 12:42:39PM -0700, Christopher Larson wrote:
> > From: Christopher Larson <chris_lar...@mentor.com>
> > 
> > When no --size is specified for the rootfs in the .wks, we want to obey the
> > rootfs size from the metadata, otherwise the defined 
> > IMAGE_ROOTFS_EXTRA_SPACE
> > and IMAGE_OVERHEAD_FACTOR will not be obeyed. In some cases, this can result
> > in image construction failure, if the size determined by du was insufficient
> > to hold the files without the aforementioned extra space.
> > 
> > This fallback from --size to ROOTFS_SIZE was already implemented when
> > --rootfs-dir is specified in the .wks, but it did not occur otherwise, 
> > neither
> > when --rootfs-dir= was passed to `wic create` nor when IMAGE_ROOTFS was 
> > used.
> > This made a certain amount of sense, as this fallback logic happened at such
> > a level that it wasn't able to identify which partitions were rootfs
> > partitions otherwise. Rather than doing it at that level, we can do it in
> > prepare_rootfs(), which is run by the rootfs source plugins.
> > 
> > Note that IMAGE_OVERHEAD_FACTOR and a --overhead-factor in the .wks will now
> > both be applied when --size isn't specified in the .wks. A warning is added
> > about this, though a user won't see it unless wic fails or they examine the
> > do_image_wic log.
> > 
> > Fixes [YOCTO #10815]
> > 
> > Signed-off-by: Christopher Larson <chris_lar...@mentor.com>
> > ---
> >  scripts/lib/wic/partition.py | 13 -
> >  1 file changed, 12 insertions(+), 1 deletion(-)
> > 
> > diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
> > index ac4c836..b191cde 100644
> > --- a/scripts/lib/wic/partition.py
> > +++ b/scripts/lib/wic/partition.py
> > @@ -28,7 +28,7 @@ import os
> >  import tempfile
> >  
> >  from wic.utils.oe.misc import msger, parse_sourceparams
> > -from wic.utils.oe.misc import exec_cmd, exec_native_cmd
> > +from wic.utils.oe.misc import exec_cmd, exec_native_cmd, get_bitbake_var
> >  from wic.plugin import pluginmgr
> >  
> >  partition_methods = {
> > @@ -194,6 +194,17 @@ class Partition():
> >  msger.error("File system for partition %s not specified in 
> > kickstart, " \
> >  "use --fstype option" % (self.mountpoint))
> >  
> > +# Get rootfs size from bitbake variable if it's not set in .ks file
> > +if not self.size:
> > +# Bitbake variable ROOTFS_SIZE is calculated in
> > +# Image._get_rootfs_size method from meta/lib/oe/image.py
> > +# using IMAGE_ROOTFS_SIZE, IMAGE_ROOTFS_ALIGNMENT,
> > +# IMAGE_OVERHEAD_FACTOR and IMAGE_ROOTFS_EXTRA_SPACE
> > +rsize_bb = get_bitbake_var('ROOTFS_SIZE')
> > +if rsize_bb:
> > +msger.warning('overhead-factor was specified, but size was 
> > not, so bitbake variables will be used for the size. In this case both 
> > IMAGE_OVERHEAD_FACTOR and --overhead-factor will be applied')
> > +self.size = int(round(float(rsize_bb)))
> > +
> >  for prefix in ("ext", "btrfs", "vfat", "squashfs"):
> >  if self.fstype.startswith(prefix):
> >  method = getattr(self, "prepare_rootfs_" + prefix)
> > -- 
> > 2.8.0
> > 
> 
> -- 
> --
> Regards,
> Ed
> -- 
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

-- 
--
Regards,
Ed
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] oe-selftest: import git module only when needed

[Ed Bartosh]

git module is not included into standard Python
library and therefore causes import errors on the systems
where PythonGit is not installed.

As git module only used in the code implementing --repository
functionality it's better to import git only in the scope
that requires it.

[YOCTO #10821]

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 meta/lib/oeqa/utils/metadata.py | 2 +-
 scripts/oe-selftest | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oeqa/utils/metadata.py b/meta/lib/oeqa/utils/metadata.py
index ecbe763..5d8bf84 100644
--- a/meta/lib/oeqa/utils/metadata.py
+++ b/meta/lib/oeqa/utils/metadata.py
@@ -5,7 +5,6 @@
 # Functions to get metadata from the testing host used
 # for analytics of test results.
 
-from git import Repo, InvalidGitRepositoryError, NoSuchPathError
 from collections import OrderedDict
 from collections.abc import MutableMapping
 from xml.dom.minidom import parseString
@@ -46,6 +45,7 @@ def metadata_from_data_store(d):
 
 def get_layers(layers):
 """ Returns layer name, branch, and revision as OrderedDict. """
+from git import Repo, InvalidGitRepositoryError, NoSuchPathError
 
 layer_dict = OrderedDict()
 for layer in layers.split():
diff --git a/scripts/oe-selftest b/scripts/oe-selftest
index f4b861f..bfcea66 100755
--- a/scripts/oe-selftest
+++ b/scripts/oe-selftest
@@ -36,7 +36,6 @@ import re
 import fnmatch
 import collections
 import imp
-import git
 
 sys.path.insert(0, os.path.dirname(os.path.realpath(__file__)) + '/lib')
 import scriptpath
@@ -577,6 +576,7 @@ def main():
 log.info("Finished")
 
 if args.repository:
+import git
 # Commit tests results to repository
 metadata = metadata_from_bb()
 git_dir = os.path.join(os.getcwd(), 'selftest')
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v2] wic: look for wks files in /wic

[Ed Bartosh]

Currently wic looks for wks files in
/scripts/lib/wic/canned-wks/ directories.
This path is too nested and doesn't look consistent with the
naming scheme of layer directories.

Added /wic directory to the list of paths
to look for wks files.

Signed-off-by: Ed Bartosh <ed.bart...@linux.intel.com>
---
 scripts/lib/wic/engine.py | 7 +--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index 5b10463..2adef2f 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -52,6 +52,7 @@ def verify_build_env():
 
 CANNED_IMAGE_DIR = "lib/wic/canned-wks" # relative to scripts
 SCRIPTS_CANNED_IMAGE_DIR = "scripts/" + CANNED_IMAGE_DIR
+WIC_DIR = "wic"
 
 def build_canned_image_list(path):
 layers_path = misc.get_bitbake_var("BBLAYERS")
@@ -59,8 +60,10 @@ def build_canned_image_list(path):
 
 if layers_path is not None:
 for layer_path in layers_path.split():
-cpath = os.path.join(layer_path, SCRIPTS_CANNED_IMAGE_DIR)
-canned_wks_layer_dirs.append(cpath)
+for wks_path in (WIC_DIR, SCRIPTS_CANNED_IMAGE_DIR):
+cpath = os.path.join(layer_path, wks_path)
+if os.path.isdir(cpath):
+canned_wks_layer_dirs.append(cpath)
 
 cpath = os.path.join(path, CANNED_IMAGE_DIR)
 canned_wks_layer_dirs.append(cpath)
-- 
2.1.4

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core