[OE-core] [PATCH] Use the built-in options for removing pack tools
From: Ed Tanous
For distros that want to use the ENABLE_LIB_ONLY option, the rm call
will fail, because ENABLE_HPACK_TOOLS (set implicitly as part of
ENABLE_LIB_ONLY) removes those two binaries from the build, so they then
can't be removed again. This commit sets ENABLE_HPACK_TOOLS=OFF, which not
only allows for the option to be overridden in other meta layers, also
allows a simplified use of ENABLE_LIB_ONLY in meta layers that don't
want to ship the binaries.
Signed-off-by: Ed Tanous
---
meta-networking/recipes-support/nghttp2/nghttp2_1.43.0.bb | 6 +-
1 file changed, 1 insertion(+), 5 deletions(-)
diff --git a/meta-networking/recipes-support/nghttp2/nghttp2_1.43.0.bb
b/meta-networking/recipes-support/nghttp2/nghttp2_1.43.0.bb
index 959cccf35..08b855262 100644
--- a/meta-networking/recipes-support/nghttp2/nghttp2_1.43.0.bb
+++ b/meta-networking/recipes-support/nghttp2/nghttp2_1.43.0.bb
@@ -18,11 +18,7 @@ PACKAGECONFIG[manpages] = ""
# examples are never installed, and don't need to be built in the
# first place
-EXTRA_OECMAKE = "-DENABLE_EXAMPLES=OFF -DENABLE_APP=ON"
-
-do_install_append() {
- rm ${D}${bindir}/deflatehd ${D}${bindir}/inflatehd
-}
+EXTRA_OECMAKE = "-DENABLE_EXAMPLES=OFF -DENABLE_APP=ON
-DENABLE_HPACK_TOOLS=OFF"
PACKAGES =+ "lib${PN} ${PN}-client ${PN}-proxy ${PN}-server"
--
2.32.0.272.g935e593368-goog
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#153031):
https://lists.openembedded.org/g/openembedded-core/message/153031
Mute This Topic: https://lists.openembedded.org/mt/83587116/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCHv2 0/5] Enable wic in eSDK
On Thu, Jan 11, 2018 at 10:55:18PM +0800, rebecca.swee.fun.ch...@intel.com wrote: > Hi all, > > As the subject called out: this patch series enable wic in eSDK. > The details of what I have done are documented within the commit message. > Basically wic requires an OE build environment, but we are using a > different environment setup script in eSDK. Hence, I have added some > code for wic to explicitly export bitbake variables within eSDK. I > have also make wic to use the shared code in scriptpath for oe lib > and bitbake path addition to sys.path. > > I have run the changes on wic oe-selftest and the tests are passing. > What's next: I think it would better to have some test cases > for wic within eSDK if this series are merged. > Would it make sense to cover this new functionality by tests? > > The following changes since commit 364f8bcfcbd04e722490f363ad36a15fb7066ba7: > > linux-firmware: Bump revision to 65b1c68c (2018-01-11 10:26:07 +) > > are available in the Git repository at: > > git://push.yoctoproject.org/poky-contrib rebeccas/wic-dev > > Chang Rebecca Swee Fun (5): > scripts/wic: use scriptpath module to find bitbake path and oe lib > path > scripts/wic: append bitbake executable file path in eSDK environment > scripts/wic: fix error of import wic module in eSDK environment > scripts/wic: explicitly set BUILDDIR within eSDK > classes/populate_sdk_ext: support wic in eSDK > > meta/classes/populate_sdk_ext.bbclass | 2 +- > scripts/wic | 23 ++- > 2 files changed, 19 insertions(+), 6 deletions(-) > > -- > 2.15.0 > > -- > ___ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-core -- -- Regards, Ed -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH v2] wic: if we can't get from ioctl, try from os.stat()
On Tue, Jan 09, 2018 at 04:35:24PM +0300, Dogukan Ergun wrote:
> Under some conditions, ioctl FIGETBSZ can't return real value.
> We can try to use fallback via os.stat() to get block size.
>
Thank you for the patch!
+1
> Source of patch:
> https://github.com/intel/bmap-tools/commit/17365f4fe9089df7ee9800a2a0ced177ec4798a4
>
> Signed-off-by: Dogukan Ergun
> ---
> scripts/lib/wic/filemap.py | 10 +-
> 1 file changed, 9 insertions(+), 1 deletion(-)
>
> diff --git a/scripts/lib/wic/filemap.py b/scripts/lib/wic/filemap.py
> index 77e32b9..a72fa09 100644
> --- a/scripts/lib/wic/filemap.py
> +++ b/scripts/lib/wic/filemap.py
> @@ -37,7 +37,15 @@ def get_block_size(file_obj):
> # Get the block size of the host file-system for the image file by
> calling
> # the FIGETBSZ ioctl (number 2).
> binary_data = fcntl.ioctl(file_obj, 2, struct.pack('I', 0))
> -return struct.unpack('I', binary_data)[0]
> +bsize = struct.unpack('I', binary_data)[0]
> +if not bsize:
> +import os
> +stat = os.fstat(file_obj.fileno())
> +if hasattr(stat, 'st_blksize'):
> +bsize = stat.st_blksize
> +else:
> +raise IOError("Unable to determine block size")
> +return bsize
>
> class ErrorNotSupp(Exception):
> """
> --
> 2.7.4
>
> --
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
--
--
Regards,
Ed
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/1] wic: argparse now used for help functionality.
On Tue, Dec 12, 2017 at 03:54:04AM -0800, anelliot wrote:
> The wic help output formally consisted of manually created strings mixed with
> argparse,
> which wasunformatted and unusable. This fix cleans up the help messages,
> rewrites help
> functionality to use argparse, and adds functionality to show information for
> plugins (similar to canned images).
>
> Fixes [YOCTO #12205]
>
Thank you for the patch!
Can you rebase it on top of recent master? It looks like it conflicts
with the latest changes.
It would be great to cover this functionality with the tests. There are
some related test cases in meta/lib/oeqa/selftest/cases/wic.py. Please,
make sure they pass and add new cases if needed.
> Signed-off-by: anelliot
> ---
> scripts/lib/wic/engine.py | 87 +++-
> scripts/lib/wic/help.py | 1072
> +
> scripts/wic | 169 +++
> 3 files changed, 243 insertions(+), 1085 deletions(-)
>
> diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
> index edcfab3..5d1e1d8 100644
> --- a/scripts/lib/wic/engine.py
> +++ b/scripts/lib/wic/engine.py
> @@ -58,6 +58,8 @@ def verify_build_env():
>
> CANNED_IMAGE_DIR = "lib/wic/canned-wks" # relative to scripts
> SCRIPTS_CANNED_IMAGE_DIR = "scripts/" + CANNED_IMAGE_DIR
> +SOURCE_PLUGIN_DIR = "lib/wic/plugins/source"
> +SCRIPTS_SOURCE_PLUGIN_DIR = "scripts/" + SOURCE_PLUGIN_DIR
> WIC_DIR = "wic"
>
> def build_canned_image_list(path):
> @@ -76,6 +78,23 @@ def build_canned_image_list(path):
>
> return canned_wks_layer_dirs
>
> +def build_source_plugin_list(path):
> +layers_path = get_bitbake_var("BBLAYERS")
> +canned_wks_layer_dirs = []
> +
> +if layers_path is not None:
> +for layer_path in layers_path.split():
> +for wks_path in (WIC_DIR, SCRIPTS_SOURCE_PLUGIN_DIR):
> +cpath = os.path.join(layer_path, wks_path)
> +if os.path.isdir(cpath):
> +canned_wks_layer_dirs.append(cpath)
> +
> +cpath = os.path.join(path, SOURCE_PLUGIN_DIR)
> +canned_wks_layer_dirs.append(cpath)
> +
> +return canned_wks_layer_dirs
> +
> +
> def find_canned_image(scripts_path, wks_file):
> """
> Find a .wks file with the given name in the canned files dir.
> @@ -95,6 +114,24 @@ def find_canned_image(scripts_path, wks_file):
> return None
>
>
> +def find_source_plugin(scripts_path, plugin_file):
> +"""
> +Find a .py file with the given name in the source plugin dir.
> +
> +Return False if not found
> +"""
> +layers_source_plugin_dir = build_source_plugin_list(scripts_path)
> +
> +for source_plugin_dir in layers_source_plugin_dir:
> +for root, dirs, files in os.walk(source_plugin_dir):
> +for fname in files:
> +if fname.endswith("~") or fname.endswith("#"):
> +continue
> +if fname.endswith(".py") and plugin_file + ".py" == fname:
> +fullpath = os.path.join(source_plugin_dir, fname)
> +return fullpath
> +return None
> +
> def list_canned_images(scripts_path):
> """
> List the .wks files in the canned image dir, minus the extension.
> @@ -119,7 +156,7 @@ def list_canned_images(scripts_path):
> print(" %s\t\t%s" % (basename.ljust(30), desc))
>
>
> -def list_canned_image_help(scripts_path, fullpath):
> +def list_canned_image_help(fullpath):
> """
> List the help and params in the specified canned image.
> """
> @@ -139,6 +176,7 @@ def list_canned_image_help(scripts_path, fullpath):
> if idx != -1:
> print(line[idx + len("#:"):].rstrip())
> else:
> +print()
> break
>
>
> @@ -152,6 +190,29 @@ def list_source_plugins():
> print(" %s" % plugin)
>
>
> +def list_source_plugins_help(fullpath):
> +"""
> +List the help and params in the specified canned image.
> +"""
> +found = False
> +with open(fullpath) as plugin:
> +for line in plugin:
> +if not found:
> +idx = line.find("DESCRIPTION")
> +if idx != -1:
> +print(line[idx + len("DESCRIPTION"):].strip())
> +found = True
> +continue
> +if not line.strip():
> +break
> +idx = line.find("#")
> +auth = line.find("AUTHORS")
> +if idx != -1 and auth == -1:
> +print(line[idx + len("#:"):].rstrip())
> +else:
> +break
> +
> +
> def wic_create(wks_file, rootfs_dir, bootimg_dir, kernel_dir,
> native_sysroot, options):
> """
> @@ -204,7 +265,7 @@ def wic_create(wks_file, rootfs_dir, bootimg_dir,
> kernel_dir,
> logger.info("The image(s) were created using OE kickstart file:\n %s",
> wks_file)
>
>
> -def wic_list(args,
Re: [OE-core] [PATCH] wic: support filesystem label for rawcopy
On Tue, Nov 28, 2017 at 01:56:11PM +0100, Martin Hundebøll wrote:
> The '--label' argument should work for '--source rawcopy' as it does for
> '--source rootfs', so add a method in RawCopyPlugin to update the label
> on the temporary filesystem images.
>
> Signed-off-by: Martin Hundebøll
Thank you for the patch!
+1
> ---
> scripts/lib/wic/plugins/source/rawcopy.py | 22 ++
> 1 file changed, 22 insertions(+)
>
> diff --git a/scripts/lib/wic/plugins/source/rawcopy.py
> b/scripts/lib/wic/plugins/source/rawcopy.py
> index 424ed26ed6..e86398ac8f 100644
> --- a/scripts/lib/wic/plugins/source/rawcopy.py
> +++ b/scripts/lib/wic/plugins/source/rawcopy.py
> @@ -32,6 +32,25 @@ class RawCopyPlugin(SourcePlugin):
>
> name = 'rawcopy'
>
> +@staticmethod
> +def do_image_label(fstype, dst, label):
> +if fstype.startswith('ext'):
> +cmd = 'tune2fs -L %s %s' % (label, dst)
> +elif fstype in ('msdos', 'vfat'):
> +cmd = 'dosfslabel %s %s' % (dst, label)
> +elif fstype == 'btrfs':
> +cmd = 'btrfs filesystem label %s %s' % (dst, label)
> +elif fstype == 'swap':
> +cmd = 'mkswap -L %s %s' % (label, dst)
> +elif fstype == 'squashfs':
> +raise WicError("It's not possible to update a squashfs "
> + "filesystem label '%s'" % (label))
> +else:
> +raise WicError("Cannot update filesystem label: "
> + "Unknown fstype: '%s'" % (fstype))
> +
> +exec_cmd(cmd)
> +
> @classmethod
> def do_prepare_partition(cls, part, source_params, cr, cr_workdir,
> oe_builddir, bootimg_dir, kernel_dir,
> @@ -66,4 +85,7 @@ class RawCopyPlugin(SourcePlugin):
> if filesize > part.size:
> part.size = filesize
>
> +if part.label:
> +RawCopyPlugin.do_image_label(part.fstype, dst, part.label)
> +
> part.source_file = dst
> --
> 2.15.0
>
> --
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
--
--
Regards,
Ed
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCHv2 0/4] wic: Further enhance UUID / fstab support
Hi Tom, Thank you for the great patchset! +1 On Fri, Nov 17, 2017 at 11:08:16AM -0500, Tom Rini wrote: > Hey all, > > So, per Ed's feedback on my first series, I went and spent some time > trying to figure out how to have wic know what the UUID would be when > updating the fstab. It turns out the easiest answer here is to have WIC > make the UUID. Per Otavio's concern last time, I also make sure that > the filesystem UUID can be passed in, for reproducibility. One thing to > keep in mind here is that FAT filesystem UUIDs are a bit funny to deal > with as mkfs.vfat / mkdosfs / etc want to be given a 32bit hexadecimal > value. But when we talk mount, it must be split and it must be in > uppercase. To make the rest of the code easier I'm encoding the '0x' > portion into part.fsuuid rather than doing "-i 0x%s" in a bunch of > places. > > While preparing all of this, I found a few minor things such as we did > not test for squashfs and --use-uuid (not supported) and an incorrect > comment around the btrfs support. > > Since v1, I've added a testcase into wic.Wic.test_qemu for a UUID mount > and the UUID that we've given. I think this is cleaner than adding a > python function to make a wks file just for this task. > > -- > Tom -- -- Regards, Ed -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] wic: fallback to dd, if sparse_copy does not work
On Thu, Nov 16, 2017 at 02:49:43PM +0300, Dogukan Ergun wrote:
> Docker's aufs filesystem doesn't support file ioctl operations like FIGETBSZ
> or FIEMAP.
> Sparse_copy operation will fail if those ioctls are not supported.
> If sparse_copy fails while generating wic images, fallback to dd for copying
> filesystems on final image.
>
You can make sparse_copy to use SEEK_HOLE / SEEK_DATA instead of FIEMAP.
This can be done by using 'api' parameter. If AUFS supports it this
there is no need to fall back to using dd.
> Signed-off-by: Dogukan Ergun
> ---
> scripts/lib/wic/plugins/imager/direct.py | 10 --
> 1 file changed, 8 insertions(+), 2 deletions(-)
>
> diff --git a/scripts/lib/wic/plugins/imager/direct.py
> b/scripts/lib/wic/plugins/imager/direct.py
> index da1c061..70f93ee 100644
> --- a/scripts/lib/wic/plugins/imager/direct.py
> +++ b/scripts/lib/wic/plugins/imager/direct.py
> @@ -577,8 +577,14 @@ class PartitionedImage():
> for part in self.partitions:
> source = part.source_file
> if source:
> -# install source_file contents into a partition
> -sparse_copy(source, self.path, seek=part.start *
> self.sector_size)
> +try:
> +# install source_file contents into a partition
> +sparse_copy(source, self.path, seek=part.start *
> self.sector_size)
> +except:
sparse_copy raises ErrorNotSupp if some ioctl is not supported. Please,
catch this particular exception instead of any exception.
> +# Sparse_copy failed, fallback to dd method
> +dd_cmd = "dd if=%s of=%s bs=%d seek=%d count=%d
> conv=notrunc" % \
> +(source, self.path, self.sector_size, part.start,
> part.size_sec)
> +exec_cmd(dd_cmd)
>
> logger.debug("Installed %s in partition %d, sectors %d-%d, "
> "size %d sectors", source, part.num, part.start,
> --
> 2.7.4
>
> --
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
--
--
Regards,
Ed
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/2] wic: When using --use-uuid make sure that we update the fstab with PARTUUID
On Wed, Nov 08, 2017 at 10:00:54AM -0500, Tom Rini wrote: > On Wed, Nov 08, 2017 at 01:20:15PM +0200, Ed Bartosh wrote: > > On Tue, Nov 07, 2017 at 09:54:57AM -0500, Tom Rini wrote: > > > On Tue, Nov 07, 2017 at 10:11:35AM +0200, Ed Bartosh wrote: > > > > On Mon, Nov 06, 2017 at 07:44:23AM -0500, Tom Rini wrote: > > > > > On Mon, Nov 06, 2017 at 09:36:20AM -0200, Otavio Salvador wrote: > > > > > > On Mon, Nov 6, 2017 at 8:08 AM, Ed Bartosh > > > > > > wrote: > > > > > > > On Fri, Nov 03, 2017 at 08:51:50AM -0400, Tom Rini wrote: > > > > > > >> On Fri, Oct 20, 2017 at 09:15:05AM -0400, Tom Rini wrote: > > > > > > >> > On Tue, Oct 10, 2017 at 05:01:49PM -0400, Tom Rini wrote: > > > > > > >> > > On Thu, Sep 21, 2017 at 01:46:16PM -0400, Tom Rini wrote: > > > > > > >> > > > When we have been told to use the UUID we should also > > > > > > >> > > > update the fstab > > > > > > >> > > > to make use of PARTUUID instead of hard-coding the device > > > > > > >> > > > in question. > > > > > > >> > > > This will make the resulting image much more portable. > > > > > > >> > > > > > > > > > >> > > > Signed-off-by: Tom Rini > > > > > > >> > > > --- > > > > > > >> > > > scripts/lib/wic/plugins/imager/direct.py | 9 ++--- > > > > > > >> > > > 1 file changed, 6 insertions(+), 3 deletions(-) > > > > > > >> > > > > > > > > >> > > Where we did we end up with this? Ed pointed out that you > > > > > > >> > > can tell wic > > > > > > >> > > to use a specific UUID, so reproducible images are not a > > > > > > >> > > problem. And > > > > > > >> > > making images that are readily portable is why other distros > > > > > > >> > > use > > > > > > >> > > UUID/LABEL and not device names as much as possible. I > > > > > > >> > > personally enjoy > > > > > > >> > > being able to put an image on uSD for minnow and have it > > > > > > >> > > work :) > > > > > > >> > > Thanks! > > > > > > >> > > > > > > > >> > ping? > > > > > > >> > > > > > > >> I was just reminded about the real problems this solves (swap > > > > > > >> isn't > > > > > > >> /dev/sda3, boot is being excessively slow), so, ping? > > > > > > >> > > > > > > > > > > > > > > I'm generally ok with the patchset. The only thing I'm thinking > > > > > > > of is if busybox mount > > > > > > > supports PARTUUID syntax in fstab. Can you check this, please? > > > > > > > > > > > > It does. > > > > > > > > > > It doesn't, you need util-linux for a mount that figures these out > > > > > (or, > > > > > it did when I wrote the patch set). > > > > > > Let me correct myself here. I think what threw me off for a moment > > > (since I hadn't picked these patches up in a while) was that busybox > > > mount might handle PARTUUID, but busybox swapon/off doesn't. But that's > > > not quite the point, either. > > > > > > > Just checked it out with core-image-minimal: > > > > root@qemux86-64:~# grep UUID /etc/fstab > > PARTUUID=452e020d-01/boot vfatdefaults0 0 > > PARTUUID=452e020d-03/mntext4defaults0 0 > > root@qemux86-64:~# mount |grep mnt > > root@qemux86-64:~# mount /mnt > > mount: mounting PARTUUID=452e020d-03 on /mnt failed: No such file or > > directory > > > > > > It could be that busybox can be configured to do this, but at least current > > oe-core busybox doesn't understand PARTUUID syntax in fstab. > > More clarity required then. :) busybox mount does support LABEL and > UUID (filesystem UUID, that is) but not PARTUUID. The kernel doesn't > support LABEL nor UUID (that is to say, root=UUID=xxx works vi
Re: [OE-core] [rocko][PATCH] wic: add 'part-name' argument for naming GPT partitions
On Wed, Nov 08, 2017 at 12:04:09PM +0100, Nicolas Dechesne wrote:
> From: Artur Mądrzak
>
> The WIC's 'part' can now give a name for GPT partition in WKS file.
> It's similar to '--label', but is naming partintions instead file systems.
> It's required by some bootloaders to partitions have specified names.
>
> Backport from master, without it WIC cannot be used on Qualcomm based
> machines.
>
+1
Thanks for the patch!
> Signed-off-by: Artur Mądrzak
> Signed-off-by: Nicolas Dechesne
> Signed-off-by: Richard Purdie
> (cherry picked from commit 9b60e3466ed7cff0cea10815851eb1304002eb52)
> ---
> scripts/lib/wic/help.py | 2 ++
> scripts/lib/wic/ksparser.py | 1 +
> scripts/lib/wic/partition.py | 1 +
> scripts/lib/wic/plugins/imager/direct.py | 11 +++
> 4 files changed, 15 insertions(+)
>
> diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
> index bd9c62e2e8..2ac45e052e 100644
> --- a/scripts/lib/wic/help.py
> +++ b/scripts/lib/wic/help.py
> @@ -970,6 +970,8 @@ DESCRIPTION
> This option cannot be used with --fixed-size
> option.
>
> + --part-name: This option is specific to wic. It specifies name for
> GPT partitions.
> +
> --part-type: This option is specific to wic. It specifies partition
>type GUID for GPT partitions.
>List of partition type GUIDS can be found here:
> diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
> index 99b66eebc5..7850e81d2f 100644
> --- a/scripts/lib/wic/ksparser.py
> +++ b/scripts/lib/wic/ksparser.py
> @@ -144,6 +144,7 @@ class KickStart():
> part.add_argument('--no-table', action='store_true')
> part.add_argument('--ondisk', '--ondrive', dest='disk',
> default='sda')
> part.add_argument("--overhead-factor", type=overheadtype)
> +part.add_argument('--part-name')
> part.add_argument('--part-type')
> part.add_argument('--rootfs-dir')
>
> diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
> index b623bb9e6d..66e61ba70c 100644
> --- a/scripts/lib/wic/partition.py
> +++ b/scripts/lib/wic/partition.py
> @@ -51,6 +51,7 @@ class Partition():
> self.no_table = args.no_table
> self.num = None
> self.overhead_factor = args.overhead_factor
> +self.part_name = args.part_name
> self.part_type = args.part_type
> self.rootfs_dir = args.rootfs_dir
> self.size = args.size
> diff --git a/scripts/lib/wic/plugins/imager/direct.py
> b/scripts/lib/wic/plugins/imager/direct.py
> index 60317eed22..bdb8385620 100644
> --- a/scripts/lib/wic/plugins/imager/direct.py
> +++ b/scripts/lib/wic/plugins/imager/direct.py
> @@ -366,6 +366,10 @@ class PartitionedImage():
> for num in range(len(self.partitions)):
> part = self.partitions[num]
>
> +if self.ptable_format == 'msdos' and part.part_name:
> +raise WicError("setting custom partition name is not " \
> + "implemented for msdos partitions")
> +
> if self.ptable_format == 'msdos' and part.part_type:
> # The --part-type can also be implemented for MBR partitions,
> # in which case it would map to the 1-byte "partition type"
> @@ -519,6 +523,13 @@ class PartitionedImage():
> self._create_partition(self.path, part.type,
> parted_fs_type, part.start, part.size_sec)
>
> +if part.part_name:
> +logger.debug("partition %d: set name to %s",
> + part.num, part.part_name)
> +exec_native_cmd("sgdisk --change-name=%d:%s %s" % \
> + (part.num, part.part_name,
> + self.path), self.native_sysroot)
> +
> if part.part_type:
> logger.debug("partition %d: set type UID to %s",
> part.num, part.part_type)
> --
> 2.15.0
>
> --
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
--
--
Regards,
Ed
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/2] wic: When using --use-uuid make sure that we update the fstab with PARTUUID
On Tue, Nov 07, 2017 at 09:54:57AM -0500, Tom Rini wrote: > On Tue, Nov 07, 2017 at 10:11:35AM +0200, Ed Bartosh wrote: > > On Mon, Nov 06, 2017 at 07:44:23AM -0500, Tom Rini wrote: > > > On Mon, Nov 06, 2017 at 09:36:20AM -0200, Otavio Salvador wrote: > > > > On Mon, Nov 6, 2017 at 8:08 AM, Ed Bartosh > > > > wrote: > > > > > On Fri, Nov 03, 2017 at 08:51:50AM -0400, Tom Rini wrote: > > > > >> On Fri, Oct 20, 2017 at 09:15:05AM -0400, Tom Rini wrote: > > > > >> > On Tue, Oct 10, 2017 at 05:01:49PM -0400, Tom Rini wrote: > > > > >> > > On Thu, Sep 21, 2017 at 01:46:16PM -0400, Tom Rini wrote: > > > > >> > > > When we have been told to use the UUID we should also update > > > > >> > > > the fstab > > > > >> > > > to make use of PARTUUID instead of hard-coding the device in > > > > >> > > > question. > > > > >> > > > This will make the resulting image much more portable. > > > > >> > > > > > > > >> > > > Signed-off-by: Tom Rini > > > > >> > > > --- > > > > >> > > > scripts/lib/wic/plugins/imager/direct.py | 9 ++--- > > > > >> > > > 1 file changed, 6 insertions(+), 3 deletions(-) > > > > >> > > > > > > >> > > Where we did we end up with this? Ed pointed out that you can > > > > >> > > tell wic > > > > >> > > to use a specific UUID, so reproducible images are not a > > > > >> > > problem. And > > > > >> > > making images that are readily portable is why other distros use > > > > >> > > UUID/LABEL and not device names as much as possible. I > > > > >> > > personally enjoy > > > > >> > > being able to put an image on uSD for minnow and have it work :) > > > > >> > > Thanks! > > > > >> > > > > > >> > ping? > > > > >> > > > > >> I was just reminded about the real problems this solves (swap isn't > > > > >> /dev/sda3, boot is being excessively slow), so, ping? > > > > >> > > > > > > > > > > I'm generally ok with the patchset. The only thing I'm thinking of is > > > > > if busybox mount > > > > > supports PARTUUID syntax in fstab. Can you check this, please? > > > > > > > > It does. > > > > > > It doesn't, you need util-linux for a mount that figures these out (or, > > > it did when I wrote the patch set). > > Let me correct myself here. I think what threw me off for a moment > (since I hadn't picked these patches up in a while) was that busybox > mount might handle PARTUUID, but busybox swapon/off doesn't. But that's > not quite the point, either. > Just checked it out with core-image-minimal: root@qemux86-64:~# grep UUID /etc/fstab PARTUUID=452e020d-01/boot vfatdefaults0 0 PARTUUID=452e020d-03/mntext4defaults0 0 root@qemux86-64:~# mount |grep mnt root@qemux86-64:~# mount /mnt mount: mounting PARTUUID=452e020d-03 on /mnt failed: No such file or directory It could be that busybox can be configured to do this, but at least current oe-core busybox doesn't understand PARTUUID syntax in fstab. > > This practically means that this patch makes it impossible to use > > --use-uuid for busybox-powered images. This is quite a big regression > > from my point of view. We need to make this feature optional then. > > How does it change anything at all for that case? Today there's cases > where you would want to in some cases use PARTUUID, via --use-uuid, but > you can't. --use-uuid is only valid for the root device as the kernel > handles parsing PARTUUID and our fstab only ever has /dev/root listed, > and all other entires in the table, even when we say --use-uuid, do not > use the UUID but instead use the device name. Today, we only pass > --use-uuid in, in that case. > > What 1/2 allows for is passing in --use-uuid for any partition and > having it be enforced. There's no canned wks file changes made at all. > > What 2/2 does is update the systemd canned wks (where you will not be > able to have busybox swapon/off, you're already forced into util-linux > for utils) to make use of --use-uuid. I would strongly encourage this > to be applied as well as it makes s
Re: [OE-core] [PATCH 1/2] wic: When using --use-uuid make sure that we update the fstab with PARTUUID
On Mon, Nov 06, 2017 at 07:44:17AM -0500, Tom Rini wrote: > > Can you rebase your patchset on top of current master? > > Yes, OK. > > > It would be also nice to get this functionality covered by tests. > > OK, but what part of oe-selftest, and how exactly? Can one easily check > the resulting fstab? Sorry for not mentioning it. You can look at wic oe-selftest test cases in meta/lib/oeqa/selftest/cases/wic.py To check image fstab you can boot image in qemu (see test cases starting with 'test_qemu' for the reference). This way you can also check if partition uids match fstab entries. P.S. Another idea is to extending 'wic cp' or implement 'wic cat'. All ingredients are already in the wic code, so it shouldn't be too hard. -- Regards, Ed -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/2] wic: When using --use-uuid make sure that we update the fstab with PARTUUID
On Mon, Nov 06, 2017 at 07:44:23AM -0500, Tom Rini wrote: > On Mon, Nov 06, 2017 at 09:36:20AM -0200, Otavio Salvador wrote: > > On Mon, Nov 6, 2017 at 8:08 AM, Ed Bartosh > > wrote: > > > On Fri, Nov 03, 2017 at 08:51:50AM -0400, Tom Rini wrote: > > >> On Fri, Oct 20, 2017 at 09:15:05AM -0400, Tom Rini wrote: > > >> > On Tue, Oct 10, 2017 at 05:01:49PM -0400, Tom Rini wrote: > > >> > > On Thu, Sep 21, 2017 at 01:46:16PM -0400, Tom Rini wrote: > > >> > > > When we have been told to use the UUID we should also update the > > >> > > > fstab > > >> > > > to make use of PARTUUID instead of hard-coding the device in > > >> > > > question. > > >> > > > This will make the resulting image much more portable. > > >> > > > > > >> > > > Signed-off-by: Tom Rini > > >> > > > --- > > >> > > > scripts/lib/wic/plugins/imager/direct.py | 9 ++--- > > >> > > > 1 file changed, 6 insertions(+), 3 deletions(-) > > >> > > > > >> > > Where we did we end up with this? Ed pointed out that you can tell > > >> > > wic > > >> > > to use a specific UUID, so reproducible images are not a problem. > > >> > > And > > >> > > making images that are readily portable is why other distros use > > >> > > UUID/LABEL and not device names as much as possible. I personally > > >> > > enjoy > > >> > > being able to put an image on uSD for minnow and have it work :) > > >> > > Thanks! > > >> > > > >> > ping? > > >> > > >> I was just reminded about the real problems this solves (swap isn't > > >> /dev/sda3, boot is being excessively slow), so, ping? > > >> > > > > > > I'm generally ok with the patchset. The only thing I'm thinking of is if > > > busybox mount > > > supports PARTUUID syntax in fstab. Can you check this, please? > > > > It does. > > It doesn't, you need util-linux for a mount that figures these out (or, > it did when I wrote the patch set). This practically means that this patch makes it impossible to use --use-uuid for busybox-powered images. This is quite a big regression from my point of view. We need to make this feature optional then. > That's why 2/2 only updates the > systemd canned wks as that _will_ have full mount. But please note that > you can already make a wks file that doesn't work with busybox mount by > using the --use-uuid flag for non-root partitions :) > What's the point of using --use-uuid for non-root partitions? I thought it only makes sense for root as it triggers update of kernel command line. > > > I'd like to see Otavio's confirmation that --uuid option solves > > > his reproducible builds concern. > > > > I am not against it if the no-fstab-update consider this too. Using > > uuid when making a fstab makes more sense but the possibility to not > > touch the fstab must be kept. > > Right. I'm only updating the systemd wks file to use this. Nothing > forces this option to be globally used. -- Regards, Ed -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/2] wic: When using --use-uuid make sure that we update the fstab with PARTUUID
On Fri, Nov 03, 2017 at 08:51:50AM -0400, Tom Rini wrote: > On Fri, Oct 20, 2017 at 09:15:05AM -0400, Tom Rini wrote: > > On Tue, Oct 10, 2017 at 05:01:49PM -0400, Tom Rini wrote: > > > On Thu, Sep 21, 2017 at 01:46:16PM -0400, Tom Rini wrote: > > > > When we have been told to use the UUID we should also update the fstab > > > > to make use of PARTUUID instead of hard-coding the device in question. > > > > This will make the resulting image much more portable. > > > > > > > > Signed-off-by: Tom Rini > > > > --- > > > > scripts/lib/wic/plugins/imager/direct.py | 9 ++--- > > > > 1 file changed, 6 insertions(+), 3 deletions(-) > > > > > > Where we did we end up with this? Ed pointed out that you can tell wic > > > to use a specific UUID, so reproducible images are not a problem. And > > > making images that are readily portable is why other distros use > > > UUID/LABEL and not device names as much as possible. I personally enjoy > > > being able to put an image on uSD for minnow and have it work :) > > > Thanks! > > > > ping? > > I was just reminded about the real problems this solves (swap isn't > /dev/sda3, boot is being excessively slow), so, ping? > I'm generally ok with the patchset. The only thing I'm thinking of is if busybox mount supports PARTUUID syntax in fstab. Can you check this, please? I'd like to see Otavio's confirmation that --uuid option solves his reproducible builds concern. Can you rebase your patchset on top of current master? It would be also nice to get this functionality covered by tests. -- Regards, Ed -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] wic: add 'part-name' argument for naming GPT partitions
On Tue, Oct 31, 2017 at 04:08:58PM +0100, Nicolas Dechesne wrote:
> On Tue, Oct 31, 2017 at 2:33 PM, Ed Bartosh
> wrote:
> >
> > On Mon, Oct 30, 2017 at 08:53:09PM +0100, Artur Mądrzak wrote:
> > > The WIC's 'part' can now give a name for GPT partition in WKS file.
> > > It's similar to '--label', but is naming partitions instead file systems.
> > > It's required by some bootloaders to partitions have specified names.
> > >
> >
> > Thank you for the patch!
> >
> > +1
>
> Yes, +1 from me as well. The context of this patch is for the Qualcomm
> SoC based board that have *interesting* bootloaders requirements, one
> of them being to look for partition by their names.
>
> Patchwork complained about the patch, so Artur, please address these
> comments and resend the patch.
>
> Ed: do you think this can be backported to previous release branches?
> The lack of --part-name on previous releases make WIC unusable for us.
> The backport is not trivial since many changes have happened, but
> Artur has a equivalent patch for morty at least.
>
Yes, I think it can be backported. It shouldn't be hard to do especially
if there is a patch for morty. Just go ahead and send it for review. It
would be a good first step in this direction.
>
> >
> >
> > > Signed-off-by: Artur Mądrzak
> > > ---
> > > scripts/lib/wic/help.py | 2 ++
> > > scripts/lib/wic/ksparser.py | 1 +
> > > scripts/lib/wic/partition.py | 1 +
> > > scripts/lib/wic/plugins/imager/direct.py | 11 +++
> > > 4 files changed, 15 insertions(+)
> > >
> > > diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
> > > index bd9c62e2e8..2ac45e052e 100644
> > > --- a/scripts/lib/wic/help.py
> > > +++ b/scripts/lib/wic/help.py
> > > @@ -970,6 +970,8 @@ DESCRIPTION
> > > This option cannot be used with --fixed-size
> > > option.
> > >
> > > + --part-name: This option is specific to wic. It specifies name
> > > for
> > > GPT partitions.
> > > +
> > > --part-type: This option is specific to wic. It specifies
> > > partition
> > >type GUID for GPT partitions.
> > >List of partition type GUIDS can be found here:
> > > diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
> > > index 99b66eebc5..7850e81d2f 100644
> > > --- a/scripts/lib/wic/ksparser.py
> > > +++ b/scripts/lib/wic/ksparser.py
> > > @@ -144,6 +144,7 @@ class KickStart():
> > > part.add_argument('--no-table', action='store_true')
> > > part.add_argument('--ondisk', '--ondrive', dest='disk',
> > > default='sda')
> > > part.add_argument("--overhead-factor", type=overheadtype)
> > > +part.add_argument('--part-name')
> > > part.add_argument('--part-type')
> > > part.add_argument('--rootfs-dir')
> > >
> > > diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
> > > index b623bb9e6d..66e61ba70c 100644
> > > --- a/scripts/lib/wic/partition.py
> > > +++ b/scripts/lib/wic/partition.py
> > > @@ -51,6 +51,7 @@ class Partition():
> > > self.no_table = args.no_table
> > > self.num = None
> > > self.overhead_factor = args.overhead_factor
> > > +self.part_name = args.part_name
> > > self.part_type = args.part_type
> > > self.rootfs_dir = args.rootfs_dir
> > > self.size = args.size
> > > diff --git a/scripts/lib/wic/plugins/imager/direct.py
> > > b/scripts/lib/wic/plugins/imager/direct.py
> > > index 60317eed22..bdb8385620 100644
> > > --- a/scripts/lib/wic/plugins/imager/direct.py
> > > +++ b/scripts/lib/wic/plugins/imager/direct.py
> > > @@ -366,6 +366,10 @@ class PartitionedImage():
> > > for num in range(len(self.partitions)):
> > > part = self.partitions[num]
> > >
> > > +if self.ptable_format == 'msdos' and part.part_name:
> > > +raise WicError("setting custom partition name is not " \
> > > + "implemented for msdos partitions")
> > > +
> > &g
Re: [OE-core] [PATCH] wic: add 'part-name' argument for naming GPT partitions
On Mon, Oct 30, 2017 at 08:53:09PM +0100, Artur Mądrzak wrote:
> The WIC's 'part' can now give a name for GPT partition in WKS file.
> It's similar to '--label', but is naming partitions instead file systems.
> It's required by some bootloaders to partitions have specified names.
>
Thank you for the patch!
+1
> Signed-off-by: Artur Mądrzak
> ---
> scripts/lib/wic/help.py | 2 ++
> scripts/lib/wic/ksparser.py | 1 +
> scripts/lib/wic/partition.py | 1 +
> scripts/lib/wic/plugins/imager/direct.py | 11 +++
> 4 files changed, 15 insertions(+)
>
> diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
> index bd9c62e2e8..2ac45e052e 100644
> --- a/scripts/lib/wic/help.py
> +++ b/scripts/lib/wic/help.py
> @@ -970,6 +970,8 @@ DESCRIPTION
> This option cannot be used with --fixed-size
> option.
>
> + --part-name: This option is specific to wic. It specifies name for
> GPT partitions.
> +
> --part-type: This option is specific to wic. It specifies
> partition
>type GUID for GPT partitions.
>List of partition type GUIDS can be found here:
> diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
> index 99b66eebc5..7850e81d2f 100644
> --- a/scripts/lib/wic/ksparser.py
> +++ b/scripts/lib/wic/ksparser.py
> @@ -144,6 +144,7 @@ class KickStart():
> part.add_argument('--no-table', action='store_true')
> part.add_argument('--ondisk', '--ondrive', dest='disk',
> default='sda')
> part.add_argument("--overhead-factor", type=overheadtype)
> +part.add_argument('--part-name')
> part.add_argument('--part-type')
> part.add_argument('--rootfs-dir')
>
> diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
> index b623bb9e6d..66e61ba70c 100644
> --- a/scripts/lib/wic/partition.py
> +++ b/scripts/lib/wic/partition.py
> @@ -51,6 +51,7 @@ class Partition():
> self.no_table = args.no_table
> self.num = None
> self.overhead_factor = args.overhead_factor
> +self.part_name = args.part_name
> self.part_type = args.part_type
> self.rootfs_dir = args.rootfs_dir
> self.size = args.size
> diff --git a/scripts/lib/wic/plugins/imager/direct.py
> b/scripts/lib/wic/plugins/imager/direct.py
> index 60317eed22..bdb8385620 100644
> --- a/scripts/lib/wic/plugins/imager/direct.py
> +++ b/scripts/lib/wic/plugins/imager/direct.py
> @@ -366,6 +366,10 @@ class PartitionedImage():
> for num in range(len(self.partitions)):
> part = self.partitions[num]
>
> +if self.ptable_format == 'msdos' and part.part_name:
> +raise WicError("setting custom partition name is not " \
> + "implemented for msdos partitions")
> +
> if self.ptable_format == 'msdos' and part.part_type:
> # The --part-type can also be implemented for MBR
> partitions,
> # in which case it would map to the 1-byte "partition type"
> @@ -519,6 +523,13 @@ class PartitionedImage():
> self._create_partition(self.path, part.type,
> parted_fs_type, part.start,
> part.size_sec)
>
> +if part.part_name:
> +logger.debug("partition %d: set name to %s",
> + part.num, part.part_name)
> +exec_native_cmd("sgdisk --change-name=%d:%s %s" % \
> + (part.num, part.part_name,
> + self.path), self.native_sysroot)
> +
> if part.part_type:
> logger.debug("partition %d: set type UID to %s",
> part.num, part.part_type)
> --
> 2.13.6
>
> --
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
--
--
Regards,
Ed
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCHv2] image_types: Fix bmaptool support for RSS
On Fri, Sep 29, 2017 at 10:28:19AM -0400, Tom Rini wrote:
> With RSS we need to ensure that when making a bmap image that the
> python3 that we created is found via /usr/bin/env rather than the host
> python3. Otherwise we're relying on the build host to have bmaptool
> support installed.
>
Thank you for the patch.
+1
> Signed-off-by: Tom Rini
> ---
> Changes in v2:
> - Switch to checking for '.bmap' not 'bmap', per Ed Bartosh.
> ---
> meta/classes/image_types.bbclass | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/meta/classes/image_types.bbclass
> b/meta/classes/image_types.bbclass
> index 61dca626311e..b373b217145f 100644
> --- a/meta/classes/image_types.bbclass
> +++ b/meta/classes/image_types.bbclass
> @@ -311,3 +311,6 @@ DEPLOYABLE_IMAGE_TYPES ?= "hddimg iso"
> # The IMAGE_TYPES_MASKED variable will be used to mask out from the
> IMAGE_FSTYPES,
> # images that will not be built at do_rootfs time: vmdk, vdi, qcow2,
> hdddirect, hddimg, iso, etc.
> IMAGE_TYPES_MASKED ?= ""
> +
> +# bmap requires python3 to be in the PATH
> +EXTRANATIVEPATH += "${@'python3-native' if
> d.getVar('IMAGE_FSTYPES').find('.bmap') else ''}"
> --
> 1.9.1
>
> --
> _______
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
--
--
Regards,
Ed
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] image_types: Fix bmaptool support for RSS
On Fri, Sep 29, 2017 at 08:36:52AM -0400, Tom Rini wrote:
> On Fri, Sep 29, 2017 at 02:16:21PM +0300, Ed Bartosh wrote:
> > On Wed, Sep 27, 2017 at 01:00:09PM -0400, Tom Rini wrote:
> > > With RSS we need to ensure that when making a bmap image that the
> > > python3 that we created is found via /usr/bin/env rather than the host
> > > python3. Otherwise we're relying on the build host to have bmaptool
> > > support installed.
> > >
> > > Signed-off-by: Tom Rini
> > > ---
> > > I played around with a few ways to try and fix this problem, and this
> > > was the best I could come up with. Once this is in we need this in pyro
> > > as well as it's broken there too.
> > > ---
> > > meta/classes/image_types.bbclass | 3 +++
> > > 1 file changed, 3 insertions(+)
> > >
> > > diff --git a/meta/classes/image_types.bbclass
> > > b/meta/classes/image_types.bbclass
> > > index 61dca626311e..3ff960e3a889 100644
> > > --- a/meta/classes/image_types.bbclass
> > > +++ b/meta/classes/image_types.bbclass
> > > @@ -311,3 +311,6 @@ DEPLOYABLE_IMAGE_TYPES ?= "hddimg iso"
> > > # The IMAGE_TYPES_MASKED variable will be used to mask out from the
> > > IMAGE_FSTYPES,
> > > # images that will not be built at do_rootfs time: vmdk, vdi, qcow2,
> > > hdddirect, hddimg, iso, etc.
> > > IMAGE_TYPES_MASKED ?= ""
> > > +
> > > +# bmap requires python3 to be in the PATH
> > > +EXTRANATIVEPATH += "${@'python3-native' if
> > > d.getVar('IMAGE_FSTYPES').find('bmap') else ''}"
> >
> > +1
> >
> > Would ...find('.bmap') be a bit more safe?
>
> Could be. Do I need to v2 or can someone just fix up while applying?
Please send v2.
--
Regards,
Ed
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] wic: Generate startup.nsh for EFI cases if none found
On Fri, Sep 29, 2017 at 08:35:42AM -0400, Tom Rini wrote: > On Fri, Sep 29, 2017 at 02:27:57PM +0300, Ed Bartosh wrote: > > On Thu, Sep 28, 2017 at 01:44:29PM -0400, Tom Rini wrote: > > > On Thu, Sep 28, 2017 at 06:47:07PM +0300, Ed Bartosh wrote: > > > > On Wed, Sep 20, 2017 at 12:03:27PM -0400, Tom Rini wrote: > > > > > In the case of non-wic images there is logic today to generate a > > > > > startup.nsh file that will be executed by EFI to run the loader that > > > > > the > > > > > image contains. In the WIC case is currently depends on that file > > > > > being > > > > > generated elsewhere and placed in DEPLOY_DIR_IMAGE and only used if > > > > > present there. > > > > > > > > What's wrong with this approach? > > > > > > No one ever provides a startup.nsh and everyone that wants one creates > > > the same one line trivial example. The end result is that no WIC images > > > are Just Bootable on UEFI systems unless you first go and spell that out > > > as the desired booting device. This isn't an awesome workflow which is > > > why the non-WIC cases make the required startup.nsh :) > > > > Would it be better if EFI providers create this file? > > > > I still believe that wic should't hack the filesystem content unless > > it's really unavoidable. So far I know only one exception: updating > > /etc/fstab. And even that is not always needed (see --no-update-fstab > > patchset for further details.) > > Well, it depends on your view of who is supposed to do what. Today, in > wic BootimgEFIPlugin mirrors the efi_populate() function of > systemd-boot/grub-efi.bbclass. I'd call this unnecessary duplication. This content should be prepared by EFI provider class or recipe and taken by wic as is, without modification. I did some work on this: http://lists.openembedded.org/pipermail/openembedded-core/2017-May/136656.html If this or similar approach is accepted wic wouldn't need most of boot plugings. Boot partition can be prepared out of bootfs directory using rootfs plugin. > That's where startup.nsh is made because > it needs to know the name of the EFI application (also technically the > path, but EFI\BOOT\ is spec mandatated I believe). So we can't > easily make the deploy functions create startup.nsh without duplicating > logic from the bbclasss. > > > > > I'd be happy to make wic to do only partitioning and assembling the > > > > image and avoiding to modify image content as much as possible. > > > > That would make wic design much more clear and let us to remove > > > > a lot of duplication between wic and meta/classes code. > > > > > > > > Regarding boot partition content, I think preparing it from bootfs > > > > directory the same way as it's done for root partition is the way to go. > > > > You can find more details about it here: > > > > https://bugzilla.yoctoproject.org/show_bug.cgi?id=10073 > > > > > > I don't conceptually see a problem with going that route. But today WIC > > > images aren't nearly as useful as they could be, with a rather tiny > > > change. > > > > If we agree that wic should avoid modifying content then the obvious way > > to solve this is to provide required content (startup.nsh in this case) > > either by EFI related recipes or core classes. > > Maybe I have to change my mind after thinking harder :) Where's the > logic that creates the boot partition now? > Now it's in several places: in meta/classes, efi recipes and wic. I think wic should be removed from this list. It's not a wic job to prepare boot content. > > > My patch is also a regression-fix, I believe, in that at some point in > > > the past, when Christopher's patch went in, things were laid out such > > > that startup.nsh was often/always generated by another class and placed > > > where WIC would find it and copy it in. At some point that was > > > broken/changed, and no one noticed / was interested enough to fix it. > > > > If this functionality is covered by wic test suite this wouldn't > > happen. > > Once we agree on what the fix looks like, I'll see if I can figure out > how to add in another test. :) -- Regards, Ed -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] wic: Generate startup.nsh for EFI cases if none found
On Thu, Sep 28, 2017 at 01:44:29PM -0400, Tom Rini wrote: > On Thu, Sep 28, 2017 at 06:47:07PM +0300, Ed Bartosh wrote: > > On Wed, Sep 20, 2017 at 12:03:27PM -0400, Tom Rini wrote: > > > In the case of non-wic images there is logic today to generate a > > > startup.nsh file that will be executed by EFI to run the loader that the > > > image contains. In the WIC case is currently depends on that file being > > > generated elsewhere and placed in DEPLOY_DIR_IMAGE and only used if > > > present there. > > > > What's wrong with this approach? > > No one ever provides a startup.nsh and everyone that wants one creates > the same one line trivial example. The end result is that no WIC images > are Just Bootable on UEFI systems unless you first go and spell that out > as the desired booting device. This isn't an awesome workflow which is > why the non-WIC cases make the required startup.nsh :) Would it be better if EFI providers create this file? I still believe that wic should't hack the filesystem content unless it's really unavoidable. So far I know only one exception: updating /etc/fstab. And even that is not always needed (see --no-update-fstab patchset for further details.) > > > I'd be happy to make wic to do only partitioning and assembling the > > image and avoiding to modify image content as much as possible. > > That would make wic design much more clear and let us to remove > > a lot of duplication between wic and meta/classes code. > > > > Regarding boot partition content, I think preparing it from bootfs > > directory the same way as it's done for root partition is the way to go. > > You can find more details about it here: > > https://bugzilla.yoctoproject.org/show_bug.cgi?id=10073 > > I don't conceptually see a problem with going that route. But today WIC > images aren't nearly as useful as they could be, with a rather tiny > change. > If we agree that wic should avoid modifying content then the obvious way to solve this is to provide required content (startup.nsh in this case) either by EFI related recipes or core classes. > My patch is also a regression-fix, I believe, in that at some point in > the past, when Christopher's patch went in, things were laid out such > that startup.nsh was often/always generated by another class and placed > where WIC would find it and copy it in. At some point that was > broken/changed, and no one noticed / was interested enough to fix it. > If this functionality is covered by wic test suite this wouldn't happen. -- Regards, Ed -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] image_types: Fix bmaptool support for RSS
On Wed, Sep 27, 2017 at 01:00:09PM -0400, Tom Rini wrote:
> With RSS we need to ensure that when making a bmap image that the
> python3 that we created is found via /usr/bin/env rather than the host
> python3. Otherwise we're relying on the build host to have bmaptool
> support installed.
>
> Signed-off-by: Tom Rini
> ---
> I played around with a few ways to try and fix this problem, and this
> was the best I could come up with. Once this is in we need this in pyro
> as well as it's broken there too.
> ---
> meta/classes/image_types.bbclass | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/meta/classes/image_types.bbclass
> b/meta/classes/image_types.bbclass
> index 61dca626311e..3ff960e3a889 100644
> --- a/meta/classes/image_types.bbclass
> +++ b/meta/classes/image_types.bbclass
> @@ -311,3 +311,6 @@ DEPLOYABLE_IMAGE_TYPES ?= "hddimg iso"
> # The IMAGE_TYPES_MASKED variable will be used to mask out from the
> IMAGE_FSTYPES,
> # images that will not be built at do_rootfs time: vmdk, vdi, qcow2,
> hdddirect, hddimg, iso, etc.
> IMAGE_TYPES_MASKED ?= ""
> +
> +# bmap requires python3 to be in the PATH
> +EXTRANATIVEPATH += "${@'python3-native' if
> d.getVar('IMAGE_FSTYPES').find('bmap') else ''}"
+1
Would ...find('.bmap') be a bit more safe?
--
Regards,
Ed
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] wic: Generate startup.nsh for EFI cases if none found
On Wed, Sep 20, 2017 at 12:03:27PM -0400, Tom Rini wrote:
> In the case of non-wic images there is logic today to generate a
> startup.nsh file that will be executed by EFI to run the loader that the
> image contains. In the WIC case is currently depends on that file being
> generated elsewhere and placed in DEPLOY_DIR_IMAGE and only used if
> present there.
What's wrong with this approach?
I'd be happy to make wic to do only partitioning and assembling the
image and avoiding to modify image content as much as possible.
That would make wic design much more clear and let us to remove
a lot of duplication between wic and meta/classes code.
Regarding boot partition content, I think preparing it from bootfs
directory the same way as it's done for root partition is the way to go.
You can find more details about it here:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=10073
> This extends the logic we have today in wic to save the
> name of the loader that's being placed and if no startup.nsh is provided
> already generate the default kind that grub-efi/systemd-boot.bbclass
> generate.
>
> Cc: Ed Bartosh
> Cc: Christopher Larson
> Signed-off-by: Tom Rini
> ---
> scripts/lib/wic/plugins/source/bootimg-efi.py | 10 --
> 1 file changed, 8 insertions(+), 2 deletions(-)
>
> diff --git a/scripts/lib/wic/plugins/source/bootimg-efi.py
> b/scripts/lib/wic/plugins/source/bootimg-efi.py
> index 4c4f36a32f56..f4643fc233b2 100644
> --- a/scripts/lib/wic/plugins/source/bootimg-efi.py
> +++ b/scripts/lib/wic/plugins/source/bootimg-efi.py
> @@ -204,13 +204,15 @@ class BootimgEFIPlugin(SourcePlugin):
> shutil.copyfile("%s/hdd/boot/EFI/BOOT/grub.cfg" % cr_workdir,
> "%s/grub.cfg" % cr_workdir)
> for mod in [x for x in os.listdir(kernel_dir) if
> x.startswith("grub-efi-")]:
> -cp_cmd = "cp %s/%s %s/EFI/BOOT/%s" % (kernel_dir, mod,
> hdddir, mod[9:])
> +loader = mod[9:]
> +cp_cmd = "cp %s/%s %s/EFI/BOOT/%s" % (kernel_dir, mod,
> hdddir, loader)
> exec_cmd(cp_cmd, True)
> shutil.move("%s/grub.cfg" % cr_workdir,
> "%s/hdd/boot/EFI/BOOT/grub.cfg" % cr_workdir)
> elif source_params['loader'] == 'systemd-boot':
> for mod in [x for x in os.listdir(kernel_dir) if
> x.startswith("systemd-")]:
> -cp_cmd = "cp %s/%s %s/EFI/BOOT/%s" % (kernel_dir, mod,
> hdddir, mod[8:])
> +loader = mod[8:]
> +cp_cmd = "cp %s/%s %s/EFI/BOOT/%s" % (kernel_dir, mod,
> hdddir, loader)
> exec_cmd(cp_cmd, True)
> else:
> raise WicError("unrecognized bootimg-efi loader: %s" %
> @@ -222,6 +224,10 @@ class BootimgEFIPlugin(SourcePlugin):
> if os.path.exists(startup):
> cp_cmd = "cp %s %s/" % (startup, hdddir)
> exec_cmd(cp_cmd, True)
> +else:
> +cfg = open("%s/startup.nsh" % hdddir, "w")
> +cfg.write("fs0:EFI\\BOOT\\%s\n" % loader)
> +cfg.close()
>
> du_cmd = "du -bks %s" % hdddir
> out = exec_cmd(du_cmd)
> --
> 1.9.1
>
--
--
Regards,
Ed
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] image_types: Fix bmaptool support for RSS
On Wed, Sep 27, 2017 at 01:00:09PM -0400, Tom Rini wrote:
> With RSS we need to ensure that when making a bmap image that the
> python3 that we created is found via /usr/bin/env rather than the host
> python3. Otherwise we're relying on the build host to have bmaptool
> support installed.
>
Can you suggest how to reproduce the issue you're trying to fix with
this patch?
Here is what I did:
1. Checked if bmaptools are available for host python3:
$ python3 -c 'import bmaptools'
Traceback (most recent call last):
File "", line 1, in
ImportError: No module named 'bmaptools'
2. Added cpio.bmap to my build configuration:
$ grep ^IMAGE_FSTYPES ./conf/local.conf
IMAGE_FSTYPES = "cpio.bmap"
3. Built core-image-minimal:
[build (master)]$ rm -rf tmp/
[build (master)]$ bitbake core-image-minimal
...
NOTE: Tasks Summary: Attempted 2395 tasks of which 2186 didn't need to
be rerun and all succeeded.
4. Checked if bmap file has been generated:
[build (master)]$ ls ./tmp/deploy/images/qemux86-64/ |grep cpio
core-image-minimal-qemux86-64-20170928135949.rootfs.cpio.bmap
core-image-minimal-qemux86-64.cpio.bmap
Hm, cpio image got removed for some reason. Looks like a bug, but not
related to bmaptool. (I remember you're fixing that code some time ago).
Should I use another MACHINE or anything else that can trigger build
failure?
PS. I used latest poky master. commit id:
fcdfe09d9cc4e2977bd602dfd18f729cd9f6d73d
> Signed-off-by: Tom Rini
> ---
> I played around with a few ways to try and fix this problem, and this
> was the best I could come up with. Once this is in we need this in pyro
> as well as it's broken there too.
> ---
> meta/classes/image_types.bbclass | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/meta/classes/image_types.bbclass
> b/meta/classes/image_types.bbclass
> index 61dca626311e..3ff960e3a889 100644
> --- a/meta/classes/image_types.bbclass
> +++ b/meta/classes/image_types.bbclass
> @@ -311,3 +311,6 @@ DEPLOYABLE_IMAGE_TYPES ?= "hddimg iso"
> # The IMAGE_TYPES_MASKED variable will be used to mask out from the
> IMAGE_FSTYPES,
> # images that will not be built at do_rootfs time: vmdk, vdi, qcow2,
> hdddirect, hddimg, iso, etc.
> IMAGE_TYPES_MASKED ?= ""
> +
> +# bmap requires python3 to be in the PATH
> +EXTRANATIVEPATH += "${@'python3-native' if
> d.getVar('IMAGE_FSTYPES').find('bmap') else ''}"
> --
> 1.9.1
>
> --
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
--
Regards,
Ed
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/2] wic: When using --use-uuid make sure that we update the fstab with PARTUUID
On Thu, Sep 21, 2017 at 05:21:44PM -0400, Tom Rini wrote: > On Thu, Sep 21, 2017 at 04:20:30PM -0500, Mark Hatle wrote: > > On 9/21/17 4:15 PM, Otavio Salvador wrote: > > > On Thu, Sep 21, 2017 at 2:46 PM, Tom Rini wrote: > > >> When we have been told to use the UUID we should also update the fstab > > >> to make use of PARTUUID instead of hard-coding the device in question. > > >> This will make the resulting image much more portable. > > >> > > >> Signed-off-by: Tom Rini > > > > > > Adding more features which changes the fstab goes to the opposite > > > direction of reproducible builds; is it desirable? > > > > There should be a corresponding way to specify the UUID for a generated > > partition. This would allow someone to pre-allocate the UUID and use it for > > their generated image, and thus the reproducible build. > > WIC does not today, but the underlying tools do, allow you to specify > the UUID to use, yes. > It does at least partly: $ wic help kickstart |grep -A4 '\-\-uuid:' --uuid: This option is specific to wic. It specifies partition UUID. It's useful if preconfigured partition UUID is added to kernel command line in bootloader configuration before running wic. In this case .wks file can be generated or modified to set preconfigured parition UUID using this option. -- Regards, Ed -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 3/3] wic: apply --extra-space + --overhead to squashfs
On Tue, Sep 12, 2017 at 03:23:40PM +0200, Enrico Scholz wrote: > Ed Bartosh writes: > > > I agree. --size is less suitable for your needs than extra space and > > overhead factor. I still don't like the idea of using them to reserve > > non-formatted space. > > Btw, exactly this is done for extX already. > Are you sure? I don't see where we produce unformatted space on extX partitions. Can you point me out? -- Regards, Ed -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 3/3] wic: apply --extra-space + --overhead to squashfs
On Tue, Sep 12, 2017 at 02:18:41PM +0200, Enrico Scholz wrote: > Ed Bartosh writes: > > >> >> >> The --extra-space and --overhead option did not had an effect to > >> >> >> squashfs > >> >> >> partitions. Although squashfs is read-only, it can be useful to > >> >> >> allocate > >> >> >> more space for the on-disk partition to avoid repartitioning of the > >> >> >> whole > >> >> >> disk when a new (and larger) squashfs image is written on later > >> >> >> updates. > >> >> > > >> >> > Is it possible to just use --size or --fixed-size in .wks to allocate > >> >> > partition of certain size? > >> >> > >> >> --fixed-size works with this patch (tested); --size should work. > >> > > >> > Sorry for not being clear. Why do we need this if we can use --size to > >> > explicity specify partition size? > >> > >> --size and --fixed-size did not had an effect for squashfs with the old > >> code. > >> > > > > I'd propose to fix this instead of applying extra space and overhead > > factor to the filesystems that don't support this by design. > > > > The idea is to make size and fixed-size parameters to set partition size > > and use extra-space and overhead-factor to extend filesystem size as > > they're currently used. > > For what is this overhead to be used? In most cases, to allow future > updates. And this is required for squashfs too. > It depends on the usage scenario. Additional space may be required for other purposes too. It may me needed for the system to be able to boot and perform what it was built for. > Only difference is, that updates for squashfs are to be applied on > partition level and these for other file systems on file system level. > But the need for extra space exists in both cases. > > > >> I want/need it to allow updates of the running system without complete > >> repartitioning. E.g. at wic creation time, the squashfs has a certain > >> size. Sometime in the future, there are needed e.g. 5 MiB more because > >> a new package was added or so. > >> > > > > Yep, I understood what you want. I think it's better achieved by setting > > partition size with --size option than artificially apply extra-space and > > overhead factor to the filesystem that doesn't support this. > > It really does not matter for me whether the filesystem itself can be > extended or whether I need a larger partition to apply future updates. > It matters for me. I think I explained why. > I just need a partition which provides an absolute or relative amount of > additional space. > > > >> I need to reserve space in the partition so that I can 'dd' the new > >> image without a complete repartitioning. > >> > >> --extra-space/--overhead has the meaning which I want (e.g. to say > >> "reserve +20% for changes in the future"). > > > > So far extra-space and overhead factor were used to allocate additional > > space on the file system. This is different from the way you want to > > use them. You're not allocating space on squashfs, you're allocating > > unformatted space on the partition. It's better to use --size for this. > > No; --size is not what I want. --size (or --fixed-size) assume that I > know the absolute size. > > Of course, I can get this size empirically. But it is highly inflexible > and requires different wks files for different image types. > > I want to reserve an extra percentage for future updates. And these > percentages can be in the range of 1 - 5 MB (for small images with > only test tools) or several hundred MB (for large image types with > e.g. desktop environments). > > --size or --fixed-size for these image types would be in completely > different ranges while --extra-space/--overhead fits to all. > I agree. --size is less suitable for your needs than extra space and overhead factor. I still don't like the idea of using them to reserve non-formatted space. Any other ideas how to do it? -- Regards, Ed -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 3/3] wic: apply --extra-space + --overhead to squashfs
On Tue, Sep 12, 2017 at 11:44:02AM +0200, Enrico Scholz wrote: > Ed Bartosh writes: > > >> >> The --extra-space and --overhead option did not had an effect to > >> >> squashfs > >> >> partitions. Although squashfs is read-only, it can be useful to > >> >> allocate > >> >> more space for the on-disk partition to avoid repartitioning of the > >> >> whole > >> >> disk when a new (and larger) squashfs image is written on later updates. > >> > > >> > Is it possible to just use --size or --fixed-size in .wks to allocate > >> > partition of certain size? > >> > >> --fixed-size works with this patch (tested); --size should work. > > > > Sorry for not being clear. Why do we need this if we can use --size to > > explicity specify partition size? > > --size and --fixed-size did not had an effect for squashfs with the old > code. > I'd propose to fix this instead of applying extra space and overhead factor to the filesystems that don't support this by design. The idea is to make size and fixed-size parameters to set partition size and use extra-space and overhead-factor to extend filesystem size as they're currently used. > > > --extra-space and --overhead have the same meaning as > > IMAGE_ROOTFS_EXTRA_SPACE and IMAGE_OVERHEAD_SIZE variables. From my > > point of view their purpose is to allocate additional space on the > > file system. This doesn't make sense for squashfs. That's the reason > > those options are not used for squashfs. Using them to implicitly make > > bigger partition is a questionable and possible confusing approach. > > I want/need it to allow updates of the running system without complete > repartitioning. E.g. at wic creation time, the squashfs has a certain > size. Sometime in the future, there are needed e.g. 5 MiB more because > a new package was added or so. > Yep, I understood what you want. I think it's better achieved by setting partition size with --size option than artificially apply extra-space and overhead factor to the filesystem that doesn't support this. > I need to reserve space in the partition so that I can 'dd' the new > image without a complete repartitioning. > > --extra-space/--overhead has the meaning which I want (e.g. to say > "reserve +20% for changes in the future"). > So far extra-space and overhead factor were used to allocate additional space on the file system. This is different from the way you want to use them. You're not allocating space on squashfs, you're allocating unformatted space on the partition. It's better to use --size for this. > >> I would keep it as is. It is a non trivial function which is reusable. > >> Perhaps, somebody adds CRAMFS or another file system which does not > >> support extending its size. > >> > > > > I personally find this much more understandable and logical: > > me not ;) As I said, functionality is perfect for putting into an extra > method and it makes code much more readable. When somebody implements > CRAMFS he can use this method instead of copy&pasting the code. > When somebody implements something that would require this functionality he/she can create this method instead of copy&pasting the code. Let's not try to solve problems before they arise as they may never arise :) > > Then introducing a method that will be called for all supported > > filesystems except squashfs without any effect. > > You mean, function should be called at the end of prepare_rootfs() > (after "method(...)")? Yes, it would reduce code duplication but other > methods needs to be touch too (to give information whether rootfs size > has been adjusted already). > I mean that method is called for all possible filesystems, but makes sense only for squashfs. It makes code less understandable for no reason. When code is used in the only place it's much more easy to understand from my point of view. -- Regards, Ed -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 3/3] wic: apply --extra-space + --overhead to squashfs
On Mon, Sep 11, 2017 at 04:04:40PM +0200, Enrico Scholz wrote: > Ed Bartosh writes: > > >> The --extra-space and --overhead option did not had an effect to squashfs > >> partitions. Although squashfs is read-only, it can be useful to allocate > >> more space for the on-disk partition to avoid repartitioning of the whole > >> disk when a new (and larger) squashfs image is written on later updates. > > > > Is it possible to just use --size or --fixed-size in .wks to allocate > > partition of certain size? > > --fixed-size works with this patch (tested); --size should work. Sorry for not being clear. Why do we need this if we can use --size to explicity specify partition size? --extra-space and --overhead have the same meaning as IMAGE_ROOTFS_EXTRA_SPACE and IMAGE_OVERHEAD_SIZE variables. From my point of view their purpose is to allocate additional space on the file system. This doesn't make sense for squashfs. That's the reason those options are not used for squashfs. Using them to implicitly make bigger partition is a questionable and possible confusing approach. > > >> +def __extend_rootfs_image(self, rootfs): > > > > Do we really need to mangle name of this method? > > ok; I will make it _extend_rootfs_image > > > > As this function is not going to be used anywhere I'd just use this > > code in prepare_rootfs_squashfs. It would be less generic, but much > > more readable and understandable. > > I would keep it as is. It is a non trivial function which is reusable. > Perhaps, somebody adds CRAMFS or another file system which does not > support extending its size. > I personally find this much more understandable and logical: --- a/scripts/lib/wic/partition.py +++ b/scripts/lib/wic/partition.py @@ -338,6 +338,14 @@ class Partition(): (rootfs_dir, rootfs, extraopts) exec_native_cmd(squashfs_cmd, native_sysroot, pseudo=pseudo) +# Enlarges the image to make bigger partition +sz = (os.stat(rootfs).st_size + 1023) // 1024 +pad_sz = self.get_rootfs_size(sz) + +if pad_sz > sz: +with open(rootfs, 'a') as fimage: + fimage.truncate(pad_sz * 1024) + def prepare_empty_partition_ext(self, rootfs, oe_builddir, native_sysroot): Then introducing a method that will be called for all supported filesystems except squashfs without any effect. > partition.py is full of redundant code (look for all the "du" calls or > generally at the beginning of the prepare_rootfs_*() methods) where > probably somebody thought that it is not going to be used anywhere else > ;) Feel free to send a patch then. I'd not consider it a good excuse for the above new method though :) -- Regards, Ed -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 3/3] wic: apply --extra-space + --overhead to squashfs
On Fri, Sep 08, 2017 at 07:33:03PM +0200, Enrico Scholz wrote: > From: Enrico Scholz > > The --extra-space and --overhead option did not had an effect to squashfs > partitions. Although squashfs is read-only, it can be useful to allocate > more space for the on-disk partition to avoid repartitioning of the whole > disk when a new (and larger) squashfs image is written on later updates. > Is it possible to just use --size or --fixed-size in .wks to allocate partition of certain size? > Patch calls get_rootfs_size() *after* creating the image and truncates > it then. > > Signed-off-by: Enrico Scholz > --- > scripts/lib/wic/partition.py | 13 + > 1 file changed, 13 insertions(+) > > diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py > index b623bb9..da651f8 100644 > --- a/scripts/lib/wic/partition.py > +++ b/scripts/lib/wic/partition.py > @@ -195,6 +195,17 @@ class Partition(): > "larger (%d kB) than its allowed size %d kB" % > (self.mountpoint, self.size, self.fixed_size)) > > +def __extend_rootfs_image(self, rootfs): Do we really need to mangle name of this method? Please, consider reading this for further details: http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html#naming As this function is not going to be used anywhere I'd just use this code in prepare_rootfs_squashfs. It would be less generic, but much more readable and understandable. > +"""Enlarges the rootfs so that it fulfills size/overhead-factor > +constraints""" > + > +sz = (os.stat(rootfs).st_size + 1023) // 1024 > +pad_sz = self.get_rootfs_size(sz) > + > +if pad_sz > sz: > +with open(rootfs, 'a') as f: > +os.ftruncate(f.fileno(), pad_sz * 1024) > + > def prepare_rootfs(self, cr_workdir, oe_builddir, rootfs_dir, > native_sysroot): > """ > @@ -338,6 +349,8 @@ class Partition(): > (rootfs_dir, rootfs, extraopts) > exec_native_cmd(squashfs_cmd, native_sysroot, pseudo=pseudo) > > +self.__extend_rootfs_image(rootfs) > + > def prepare_empty_partition_ext(self, rootfs, oe_builddir, > native_sysroot): > """ -- Regards, Ed -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH v2 2/3] bmap-tools: switch to Python 3
bmap-tools is the only recipe in oe-core that still uses
Python 2. Switching it to Python 3 should help to get rid of
building native Python 2 and its dependencies.
[YOCTO #11891]
Signed-off-by: Ed Bartosh
---
meta/recipes-support/bmap-tools/bmap-tools_3.4.bb | 7 ++-
1 file changed, 2 insertions(+), 5 deletions(-)
diff --git a/meta/recipes-support/bmap-tools/bmap-tools_3.4.bb
b/meta/recipes-support/bmap-tools/bmap-tools_3.4.bb
index 3f75fad44b4..7454f9db751 100644
--- a/meta/recipes-support/bmap-tools/bmap-tools_3.4.bb
+++ b/meta/recipes-support/bmap-tools/bmap-tools_3.4.bb
@@ -16,10 +16,7 @@ S = "${WORKDIR}/git"
RDEPENDS_${PN} = "python-core python-compression python-mmap"
-inherit setuptools
+inherit python3native
+inherit setuptools3
BBCLASSEXTEND = "native"
-
-do_install_append_class-native() {
-sed -i -e 's|^#!.*/usr/bin/python-native/python|#! /usr/bin/env
nativepython|' ${D}${bindir}/bmaptool
-}
--
2.13.5
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH v2 3/3] wic: run bmaptool with native Python3
Modified wic code to run bmaptool using native Python3
from wic-tools native sysroot.
[YOCTO #11891]
Signed-off-by: Ed Bartosh
---
scripts/lib/wic/plugins/imager/direct.py | 6 --
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/scripts/lib/wic/plugins/imager/direct.py
b/scripts/lib/wic/plugins/imager/direct.py
index 5765bbb527b..a6abc3d09ef 100644
--- a/scripts/lib/wic/plugins/imager/direct.py
+++ b/scripts/lib/wic/plugins/imager/direct.py
@@ -213,8 +213,10 @@ class DirectPlugin(ImagerPlugin):
# Generate .bmap
if self.bmap:
logger.debug("Generating bmap file for %s", disk_name)
-exec_native_cmd("bmaptool create %s -o %s.bmap" % (full_path,
full_path),
-self.native_sysroot)
+python = os.path.join(self.native_sysroot,
'usr/bin/python3-native/python3')
+bmaptool = os.path.join(self.native_sysroot, 'usr/bin/bmaptool')
+exec_native_cmd("%s %s create %s -o %s.bmap" % \
+(python, bmaptool, full_path, full_path),
self.native_sysroot)
# Compress the image
if self.compressor:
logger.debug("Compressing disk %s with %s", disk_name,
self.compressor)
--
2.13.5
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH v2 1/3] bmap-tools: upgrade to v3.4
Upgraded to the latest upstream release.
Signed-off-by: Ed Bartosh
---
.../bmap-tools/{bmap-tools_3.2.bb => bmap-tools_3.4.bb} | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
rename meta/recipes-support/bmap-tools/{bmap-tools_3.2.bb =>
bmap-tools_3.4.bb} (84%)
diff --git a/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb
b/meta/recipes-support/bmap-tools/bmap-tools_3.4.bb
similarity index 84%
rename from meta/recipes-support/bmap-tools/bmap-tools_3.2.bb
rename to meta/recipes-support/bmap-tools/bmap-tools_3.4.bb
index b8ebe630815..3f75fad44b4 100644
--- a/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb
+++ b/meta/recipes-support/bmap-tools/bmap-tools_3.4.bb
@@ -10,7 +10,7 @@ LICENSE = "GPLv2"
LIC_FILES_CHKSUM = "file://COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263"
SRC_URI = "git://github.com/01org/bmap-tools.git"
-SRCREV = "96702a869220ab20830db916ec4ac595e1d97f92"
+SRCREV = "9dad724104df265442226972a1e310813f9ffcba"
S = "${WORKDIR}/git"
@@ -21,5 +21,5 @@ inherit setuptools
BBCLASSEXTEND = "native"
do_install_append_class-native() {
-sed -i -e 's|^#!.*/usr/bin/env python|#! /usr/bin/env nativepython|'
${D}${bindir}/bmaptool
+sed -i -e 's|^#!.*/usr/bin/python-native/python|#! /usr/bin/env
nativepython|' ${D}${bindir}/bmaptool
}
--
2.13.5
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH v2 0/3] #11891 - Port bmaptools to Python 3
Hi,
This patchset switches bmap-tools to usage of native Python 3.
As bmap-tools is the only recipe in oe-core that uses Python 2
this change should help to stop building native Python 2.
Changes in v2: Used commit id instead of release tag (thanks to Ross)
Fixed wrong bug spec in commit messages (thanks to Patchwork)
The following changes since commit 091b759dcec3068eb0b6bffdc5763fd6cc84d0a9:
wic: selftest: add test_wic_rm_ext test case (2017-09-06 13:02:02 +0300)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib ed/wip
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/wip
Ed Bartosh (3):
bmap-tools: upgrade to v3.4
bmap-tools: switch to Python 3
wic: run bmaptool with native Python3
.../bmap-tools/{bmap-tools_3.2.bb => bmap-tools_3.4.bb} | 9 +++--
scripts/lib/wic/plugins/imager/direct.py | 6 --
2 files changed, 7 insertions(+), 8 deletions(-)
rename meta/recipes-support/bmap-tools/{bmap-tools_3.2.bb =>
bmap-tools_3.4.bb} (79%)
--
Regards,
Ed
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH 3/3] wic: run bmaptool with native Python3
Modified wic code to run bmaptool using native Python3
from wic-tools native sysroot.
[YOCTO: #11891]
Signed-off-by: Ed Bartosh
---
scripts/lib/wic/plugins/imager/direct.py | 6 --
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/scripts/lib/wic/plugins/imager/direct.py
b/scripts/lib/wic/plugins/imager/direct.py
index 5765bbb527b..a6abc3d09ef 100644
--- a/scripts/lib/wic/plugins/imager/direct.py
+++ b/scripts/lib/wic/plugins/imager/direct.py
@@ -213,8 +213,10 @@ class DirectPlugin(ImagerPlugin):
# Generate .bmap
if self.bmap:
logger.debug("Generating bmap file for %s", disk_name)
-exec_native_cmd("bmaptool create %s -o %s.bmap" % (full_path,
full_path),
-self.native_sysroot)
+python = os.path.join(self.native_sysroot,
'usr/bin/python3-native/python3')
+bmaptool = os.path.join(self.native_sysroot, 'usr/bin/bmaptool')
+exec_native_cmd("%s %s create %s -o %s.bmap" % \
+(python, bmaptool, full_path, full_path),
self.native_sysroot)
# Compress the image
if self.compressor:
logger.debug("Compressing disk %s with %s", disk_name,
self.compressor)
--
2.13.5
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH 2/3] bmap-tools: switch to Python 3
bmap-tools is the only recipe in oe-core that still uses
Python 2. Switching it to Python 3 should help to get rid of
building native Python 2 and its dependencies.
[YOCTO: #11891]
Signed-off-by: Ed Bartosh
---
meta/recipes-support/bmap-tools/bmap-tools_3.4.bb | 7 ++-
1 file changed, 2 insertions(+), 5 deletions(-)
diff --git a/meta/recipes-support/bmap-tools/bmap-tools_3.4.bb
b/meta/recipes-support/bmap-tools/bmap-tools_3.4.bb
index 9735707f85b..f52fa53b4f3 100644
--- a/meta/recipes-support/bmap-tools/bmap-tools_3.4.bb
+++ b/meta/recipes-support/bmap-tools/bmap-tools_3.4.bb
@@ -16,10 +16,7 @@ S = "${WORKDIR}/git"
RDEPENDS_${PN} = "python-core python-compression python-mmap"
-inherit setuptools
+inherit python3native
+inherit setuptools3
BBCLASSEXTEND = "native"
-
-do_install_append_class-native() {
-sed -i -e 's|^#!.*/usr/bin/python-native/python|#! /usr/bin/env
nativepython|' ${D}${bindir}/bmaptool
-}
--
2.13.5
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH 1/3] bmap-tools: upgrade to v3.4
Upgraded to the latest upstream release.
Signed-off-by: Ed Bartosh
---
.../bmap-tools/{bmap-tools_3.2.bb => bmap-tools_3.4.bb} | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
rename meta/recipes-support/bmap-tools/{bmap-tools_3.2.bb =>
bmap-tools_3.4.bb} (85%)
diff --git a/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb
b/meta/recipes-support/bmap-tools/bmap-tools_3.4.bb
similarity index 85%
rename from meta/recipes-support/bmap-tools/bmap-tools_3.2.bb
rename to meta/recipes-support/bmap-tools/bmap-tools_3.4.bb
index b8ebe630815..9735707f85b 100644
--- a/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb
+++ b/meta/recipes-support/bmap-tools/bmap-tools_3.4.bb
@@ -10,7 +10,7 @@ LICENSE = "GPLv2"
LIC_FILES_CHKSUM = "file://COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263"
SRC_URI = "git://github.com/01org/bmap-tools.git"
-SRCREV = "96702a869220ab20830db916ec4ac595e1d97f92"
+SRCREV = "v3.4"
S = "${WORKDIR}/git"
@@ -21,5 +21,5 @@ inherit setuptools
BBCLASSEXTEND = "native"
do_install_append_class-native() {
-sed -i -e 's|^#!.*/usr/bin/env python|#! /usr/bin/env nativepython|'
${D}${bindir}/bmaptool
+sed -i -e 's|^#!.*/usr/bin/python-native/python|#! /usr/bin/env
nativepython|' ${D}${bindir}/bmaptool
}
--
2.13.5
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH 0/3] #11891 - Port bmaptools to Python 3
Hi,
This patchset switches bmap-tools to usage of native Python 3.
As bmap-tools is the only recipe in oe-core that uses Python 2
this change should help to stop building native Python 2.
The following changes since commit 7a5d997dc264ca8913c5ea57a2c5120a2dc41528:
wic: selftest: add test_wic_rm_ext test case (2017-09-05 14:28:41 +0300)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib ed/wip
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/wip
Ed Bartosh (3):
bmap-tools: upgrade to v3.4
bmap-tools: switch to Python 3
wic: run bmaptool with native Python3
.../bmap-tools/{bmap-tools_3.2.bb => bmap-tools_3.4.bb} | 9 +++--
scripts/lib/wic/plugins/imager/direct.py | 6 --
2 files changed, 7 insertions(+), 8 deletions(-)
rename meta/recipes-support/bmap-tools/{bmap-tools_3.2.bb =>
bmap-tools_3.4.bb} (79%)
--
Regards,
Ed
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH 4/7] wic: update help content
Added ext* partitions to the description of 'wic ls', 'wic cp' and 'wic rm' commands. Signed-off-by: Ed Bartosh --- scripts/lib/wic/help.py | 64 - 1 file changed, 32 insertions(+), 32 deletions(-) diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py index ccd33823244..83bd86e7c51 100644 --- a/scripts/lib/wic/help.py +++ b/scripts/lib/wic/help.py @@ -288,10 +288,10 @@ wic_ls_usage = """ List content of a partitioned image - usage: wic ls [:[]] [--native-sysroot ] + usage: wic ls [:[]] [--native-sysroot ] This command outputs either list of image partitions or directory contents - of vfat partitions. + of vfat and ext* partitions. See 'wic help ls' for more detailed instructions. @@ -300,17 +300,17 @@ wic_ls_usage = """ wic_ls_help = """ NAME -wic ls - List contents of partitioned image or vfat partitions +wic ls - List contents of partitioned image or partition SYNOPSIS wic ls -wic ls : -wic ls : -wic ls : --native-sysroot +wic ls : +wic ls : +wic ls : --native-sysroot DESCRIPTION This command lists either partitions of the image or directory contents -of vfat partitions. +of vfat or ext* partitions. The first form it lists partitions of the image. For example: @@ -319,7 +319,7 @@ DESCRIPTION 11048576 24438783 23390208 fat16 2 25165824 50315263 25149440 ext4 -Second and third form list directory content of vfat partition: +Second and third form list directory content of the partition: $ wic ls tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1 Volume in drive : is boot Volume Serial Number is 2DF2-5F02 @@ -351,12 +351,12 @@ DESCRIPTION wic_cp_usage = """ - Copy files and directories to the vfat partitions + Copy files and directories to the vfat or ext* partition - usage: wic cp :[] [--native-sysroot ] + usage: wic cp :[] [--native-sysroot ] - This command copies local files or directories to the vfat partitions of partitioned - image. + This command copies local files or directories to the vfat or ext* partitions +of partitioned image. See 'wic help cp' for more detailed instructions. @@ -365,19 +365,19 @@ wic_cp_usage = """ wic_cp_help = """ NAME -wic cp - copy files and directories to the vfat partitions +wic cp - copy files and directories to the vfat or ext* partitions SYNOPSIS -wic cp : -wic cp : -wic cp : --native-sysroot +wic cp : +wic cp : +wic cp : --native-sysroot DESCRIPTION -This command copies files and directories to the vfat partition of the -wic image. +This command copies files and directories to the vfat or ext* partition of +the partitioned image. The first form of it copies file or directory to the root directory of -the vfat partition: +the partition: $ wic cp test.wks tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1 $ wic ls tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1 Volume in drive : is boot @@ -393,7 +393,7 @@ DESCRIPTION 15 677 440 bytes free The second form of the command copies file or directory to the specified directory -on the vfat partition: +on the partition: $ wic cp test tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1/efi/ $ wic ls tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1/efi/ Volume in drive : is boot @@ -413,12 +413,12 @@ DESCRIPTION wic_rm_usage = """ - Remove files or directories from the vfat partitions + Remove files or directories from the vfat or ext* partitions - usage: wic rm : [--native-sysroot ] + usage: wic rm : [--native-sysroot ] - This command removes files or directories from the vfat partitions of partitioned - image. + This command removes files or directories from the vfat or ext* partitions of + the partitioned image. See 'wic help rm' for more detailed instructions. @@ -427,15 +427,15 @@ wic_rm_usage = """ wic_rm_help = """ NAME -wic rm - remove files or directories from the vfat partitions +wic rm - remove files or directories from the vfat or ext* partitions SYNOPSIS -wic rm : -wic rm : --native-sysroot +wic rm : +wic rm : --native-sysroot DESCRIPTION -This command removes files or directories from the vfat partition of the -wic image: +This command removes files or directories from the vfat or ext* partition of the +partitioned image: $ wic ls ./tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1 Volume in dr
[OE-core] [PATCH 2/7] wic: implement ext fs support for 'wic cp'
Implemented copying files to the ext partition
using debugfs tool.
Signed-off-by: Ed Bartosh
---
scripts/lib/wic/engine.py | 13 +
1 file changed, 9 insertions(+), 4 deletions(-)
diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index c6a63f2080a..9ebd93ae278 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -325,10 +325,15 @@ class Disk:
def copy(self, src, pnum, path):
"""Copy partition image into wic image."""
-cmd = "{} -i {} -snop {} ::{}".format(self.mcopy,
- self._get_part_image(pnum),
- src, path)
-exec_cmd(cmd)
+if self.partitions[pnum].fstype.startswith('ext'):
+cmd = "echo -e 'cd {}\nwrite {} {}' | {} -w {}".\
+ format(path, src, os.path.basename(src),
+ self.debugfs, self._get_part_image(pnum))
+else: # fat
+cmd = "{} -i {} -snop {} ::{}".format(self.mcopy,
+ self._get_part_image(pnum),
+ src, path)
+exec_cmd(cmd, as_shell=True)
self._put_part_image(pnum)
def remove(self, pnum, path):
--
2.13.5
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH 6/7] wic: selftest: add test_wic_cp_ext test case
Tested if 'wic cp' correctly copies files to the ext4 partition
of the wic image.
Signed-off-by: Ed Bartosh
---
meta/lib/oeqa/selftest/cases/wic.py | 30 ++
1 file changed, 30 insertions(+)
diff --git a/meta/lib/oeqa/selftest/cases/wic.py
b/meta/lib/oeqa/selftest/cases/wic.py
index 95170b946c6..1f0383c091f 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -1007,3 +1007,33 @@ part /etc --source rootfs --ondisk mmcblk0 --fstype=ext4
--exclude-path bin/ --r
self.assertEqual(0, result.status)
self.assertTrue(set(['bin', 'home', 'proc', 'usr', 'var', 'dev',
'lib', 'sbin']).issubset(
set(line.split()[-1] for line in
result.output.split('\n') if line)))
+
+def test_wic_cp_ext(self):
+"""Test copy files and directories to the ext partition."""
+self.assertEqual(0, runCmd("wic create wictestdisk "
+ "--image-name=core-image-minimal "
+ "-D -o %s" % self.resultdir).status)
+images = glob(self.resultdir + "wictestdisk-*.direct")
+self.assertEqual(1, len(images))
+
+sysroot = get_bb_var('RECIPE_SYSROOT_NATIVE', 'wic-tools')
+
+# list directory content of the ext4 partition
+result = runCmd("wic ls %s:2/ -n %s" % (images[0], sysroot))
+self.assertEqual(0, result.status)
+dirs = set(line.split()[-1] for line in result.output.split('\n') if
line)
+self.assertTrue(set(['bin', 'home', 'proc', 'usr', 'var', 'dev',
'lib', 'sbin']).issubset(dirs))
+
+with NamedTemporaryFile("w", suffix=".wic-cp") as testfile:
+testfile.write("test")
+
+# copy file to the partition
+result = runCmd("wic cp %s %s:2/ -n %s" % (testfile.name,
images[0], sysroot))
+self.assertEqual(0, result.status)
+
+# check if file is there
+result = runCmd("wic ls %s:2/ -n %s" % (images[0], sysroot))
+self.assertEqual(0, result.status)
+newdirs = set(line.split()[-1] for line in
result.output.split('\n') if line)
+self.assertEqual(newdirs.difference(dirs),
set([os.path.basename(testfile.name)]))
+
--
2.13.5
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH 7/7] wic: selftest: add test_wic_rm_ext test case
Tested if 'wic rm' correctly removes files from the ext4 partition
of the wic image.
Signed-off-by: Ed Bartosh
---
meta/lib/oeqa/selftest/cases/wic.py | 23 +++
1 file changed, 23 insertions(+)
diff --git a/meta/lib/oeqa/selftest/cases/wic.py
b/meta/lib/oeqa/selftest/cases/wic.py
index 1f0383c091f..e416a3724ab 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -1037,3 +1037,26 @@ part /etc --source rootfs --ondisk mmcblk0 --fstype=ext4
--exclude-path bin/ --r
newdirs = set(line.split()[-1] for line in
result.output.split('\n') if line)
self.assertEqual(newdirs.difference(dirs),
set([os.path.basename(testfile.name)]))
+def test_wic_rm_ext(self):
+"""Test removing files from the ext partition."""
+self.assertEqual(0, runCmd("wic create mkefidisk "
+ "--image-name=core-image-minimal "
+ "-D -o %s" % self.resultdir).status)
+images = glob(self.resultdir + "mkefidisk-*.direct")
+self.assertEqual(1, len(images))
+
+sysroot = get_bb_var('RECIPE_SYSROOT_NATIVE', 'wic-tools')
+
+# list directory content of the /etc directory on ext4 partition
+result = runCmd("wic ls %s:2/etc/ -n %s" % (images[0], sysroot))
+self.assertEqual(0, result.status)
+self.assertTrue('fstab' in [line.split()[-1] for line in
result.output.split('\n') if line])
+
+# remove file
+result = runCmd("wic rm %s:2/etc/fstab -n %s" % (images[0], sysroot))
+self.assertEqual(0, result.status)
+
+# check if it's removed
+result = runCmd("wic ls %s:2/etc/ -n %s" % (images[0], sysroot))
+self.assertEqual(0, result.status)
+self.assertTrue('fstab' not in [line.split()[-1] for line in
result.output.split('\n') if line])
--
2.13.5
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH 1/7] wic: implement ext fs support for 'wic ls'
Implemented listing directory contents for ext file
system using debugfs tool.
Signed-off-by: Ed Bartosh
---
scripts/lib/wic/engine.py | 15 ++-
1 file changed, 10 insertions(+), 5 deletions(-)
diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index eafc6c783ec..c6a63f2080a 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -234,7 +234,7 @@ def wic_list(args, scripts_path):
class Disk:
-def __init__(self, imagepath, native_sysroot, fstypes=('fat',)):
+def __init__(self, imagepath, native_sysroot, fstypes=('fat', 'ext')):
self.imagepath = imagepath
self.native_sysroot = native_sysroot
self.fstypes = fstypes
@@ -280,7 +280,7 @@ class Disk:
def __getattr__(self, name):
"""Get path to the executable in a lazy way."""
if name in ("mdir", "mcopy", "mdel", "mdeltree", "sfdisk", "e2fsck",
-"resize2fs", "mkswap", "mkdosfs"):
+"resize2fs", "mkswap", "mkdosfs", "debugfs"):
aname = "_%s" % name
if aname not in self.__dict__:
setattr(self, aname, find_executable(name, self.paths))
@@ -314,9 +314,14 @@ class Disk:
seek=self.partitions[pnum].start)
def dir(self, pnum, path):
-return exec_cmd("{} -i {} ::{}".format(self.mdir,
- self._get_part_image(pnum),
- path))
+if self.partitions[pnum].fstype.startswith('ext'):
+return exec_cmd("{} {} -R 'ls -l {}'".format(self.debugfs,
+
self._get_part_image(pnum),
+ path), as_shell=True)
+else: # fat
+return exec_cmd("{} -i {} ::{}".format(self.mdir,
+ self._get_part_image(pnum),
+ path))
def copy(self, src, pnum, path):
"""Copy partition image into wic image."""
--
2.13.5
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH 3/7] wic: implement ext fs support for 'wic rm'
Implemented removing files or directories from the ext
partition using debugfs tool.
Signed-off-by: Ed Bartosh
---
scripts/lib/wic/engine.py | 27 ---
1 file changed, 16 insertions(+), 11 deletions(-)
diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index 9ebd93ae278..edcfab39ef5 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -339,18 +339,23 @@ class Disk:
def remove(self, pnum, path):
"""Remove files/dirs from the partition."""
partimg = self._get_part_image(pnum)
-cmd = "{} -i {} ::{}".format(self.mdel, partimg, path)
-try:
-exec_cmd(cmd)
-except WicError as err:
-if "not found" in str(err) or "non empty" in str(err):
-# mdel outputs 'File ... not found' or 'directory .. non empty"
-# try to use mdeltree as path could be a directory
-cmd = "{} -i {} ::{}".format(self.mdeltree,
- partimg, path)
+if self.partitions[pnum].fstype.startswith('ext'):
+exec_cmd("{} {} -wR 'rm {}'".format(self.debugfs,
+self._get_part_image(pnum),
+path), as_shell=True)
+else: # fat
+cmd = "{} -i {} ::{}".format(self.mdel, partimg, path)
+try:
exec_cmd(cmd)
-else:
-raise err
+except WicError as err:
+if "not found" in str(err) or "non empty" in str(err):
+# mdel outputs 'File ... not found' or 'directory .. non
empty"
+# try to use mdeltree as path could be a directory
+cmd = "{} -i {} ::{}".format(self.mdeltree,
+ partimg, path)
+exec_cmd(cmd)
+else:
+raise err
self._put_part_image(pnum)
def write(self, target, expand):
--
2.13.5
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH 5/7] wic: selftest: add test_wic_ls_ext test case
Tested if 'wic ls' correctly lists directory contents
of the ext* partition.
Signed-off-by: Ed Bartosh
---
meta/lib/oeqa/selftest/cases/wic.py | 16
1 file changed, 16 insertions(+)
diff --git a/meta/lib/oeqa/selftest/cases/wic.py
b/meta/lib/oeqa/selftest/cases/wic.py
index aa73ba4f7ea..95170b946c6 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -991,3 +991,19 @@ part /etc --source rootfs --ondisk mmcblk0 --fstype=ext4
--exclude-path bin/ --r
os.unlink(new_image_path)
if os.path.exists(image_path + '.bak'):
os.rename(image_path + '.bak', image_path)
+
+def test_wic_ls_ext(self):
+"""Test listing content of the ext partition using 'wic ls'"""
+self.assertEqual(0, runCmd("wic create wictestdisk "
+ "--image-name=core-image-minimal "
+ "-D -o %s" % self.resultdir).status)
+images = glob(self.resultdir + "wictestdisk-*.direct")
+self.assertEqual(1, len(images))
+
+sysroot = get_bb_var('RECIPE_SYSROOT_NATIVE', 'wic-tools')
+
+# list directory content of the second ext4 partition
+result = runCmd("wic ls %s:2/ -n %s" % (images[0], sysroot))
+self.assertEqual(0, result.status)
+self.assertTrue(set(['bin', 'home', 'proc', 'usr', 'var', 'dev',
'lib', 'sbin']).issubset(
+set(line.split()[-1] for line in
result.output.split('\n') if line)))
--
2.13.5
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH 0/7] implement wic ls/cp/rm for ext partitions
Hi, This patchset implements listing directory contents, copying and removing files on ext[2,3,4] partitions of partitioned images. The following changes since commit ee5bce2d11e783c0921df47b629025a6b67c44bf: kernel-dev: Removed bad reference for creating patches. (2017-09-02 00:52:49 +0100) are available in the git repository at: git://git.yoctoproject.org/poky-contrib ed/wip http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/wip Ed Bartosh (7): wic: implement ext fs support for 'wic ls' wic: implement ext fs support for 'wic cp' wic: implement ext fs support for 'wic rm' wic: update help content wic: selftest: add test_wic_ls_ext test case wic: selftest: add test_wic_cp_ext test case wic: selftest: add test_wic_rm_ext test case meta/lib/oeqa/selftest/cases/wic.py | 69 + scripts/lib/wic/engine.py | 55 ++--- scripts/lib/wic/help.py | 64 +- 3 files changed, 136 insertions(+), 52 deletions(-) -- Regards, Ed -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] Running wic on target?
On Thu, Aug 31, 2017 at 09:17:03AM +0200, Mike Looijmans wrote: > I noticed that the "wic" recipes appear to focus on the host only. > > Would it be possible to run "wic" on target? > > For example, it would be really convenient to be able to program the eMMC > using wic and auto-expanding the root or data partition to cover the whole > device. > Wic depends on too many things that are not present on target image. pseudo and bitbake are two hardest cases. Making wic less dependent on them would end up in much more complex code than wic has currently. However, 'wic write [--expand]' functionality can be copied to simple script relatively easy as it only depends on bunch of tools like parted, sfdisk, resize2fs etc. I'd suggest to keep this functionality in wic as it's handy to use wic to flash target media. This would create a bit of overlap in functionality but it worth it from my point of view. -- Regards, Ed -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH v2 7/8] wic: always read image partitions
Got rid of lazy evaluation of self.partitions property.
It's not needed because partitions of the source image should
be always read.
Signed-off-by: Ed Bartosh
---
scripts/lib/wic/engine.py | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index 303f323..eafc6c7 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -254,12 +254,13 @@ class Disk:
if not self.parted:
raise WicError("Can't find executable parted")
+self.partitions = self.get_partitions()
+
def __del__(self):
for path in self._partimages.values():
os.unlink(path)
-@property
-def partitions(self):
+def get_partitions(self):
if self._partitions is None:
self._partitions = OrderedDict()
out = exec_cmd("%s -sm %s unit B print" % (self.parted,
self.imagepath))
--
2.1.4
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH v2 8/8] wic: setlftest: test expanding MBR image
Added test_expand_mbr_image test case to the wic oe-selftest suite.
The test expands directdisk wic image to 1Gb target, checks if it's
expanded correctly and boots it in qemu to make sure the image
is bootable, i.e. bootloader, kernel, boot and root partitions
are still functional.
Signed-off-by: Ed Bartosh
---
meta/lib/oeqa/selftest/cases/wic.py | 53 +
1 file changed, 53 insertions(+)
diff --git a/meta/lib/oeqa/selftest/cases/wic.py
b/meta/lib/oeqa/selftest/cases/wic.py
index aaefd4f..aa73ba4 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -938,3 +938,56 @@ part /etc --source rootfs --ondisk mmcblk0 --fstype=ext4
--exclude-path bin/ --r
wksname = os.path.splitext(os.path.basename(wks.name))[0]
out = glob(self.resultdir + "%s-*direct" % wksname)
self.assertEqual(1, len(out))
+
+def test_expand_mbr_image(self):
+"""Test wic write --expand command for mbr image"""
+# build an image
+config = 'IMAGE_FSTYPES = "wic"\nWKS_FILE = "directdisk.wks"\n'
+self.append_config(config)
+self.assertEqual(0, bitbake('core-image-minimal').status)
+
+# get path to the image
+bb_vars = get_bb_vars(['DEPLOY_DIR_IMAGE', 'MACHINE'])
+deploy_dir = bb_vars['DEPLOY_DIR_IMAGE']
+machine = bb_vars['MACHINE']
+image_path = os.path.join(deploy_dir, 'core-image-minimal-%s.wic' %
machine)
+
+self.remove_config(config)
+
+try:
+# expand image to 1G
+new_image_path = None
+with NamedTemporaryFile(mode='wb', suffix='.wic.exp',
+dir=deploy_dir, delete=False) as sparse:
+sparse.truncate(1024 ** 3)
+new_image_path = sparse.name
+
+sysroot = get_bb_var('RECIPE_SYSROOT_NATIVE', 'wic-tools')
+cmd = "wic write -n %s --expand 1:0 %s %s" % (sysroot, image_path,
new_image_path)
+self.assertEqual(0, runCmd(cmd).status)
+
+# check if partitions are expanded
+orig = runCmd("wic ls %s -n %s" % (image_path, sysroot))
+exp = runCmd("wic ls %s -n %s" % (new_image_path, sysroot))
+orig_sizes = [int(line.split()[3]) for line in
orig.output.split('\n')[1:]]
+exp_sizes = [int(line.split()[3]) for line in
exp.output.split('\n')[1:]]
+self.assertEqual(orig_sizes[0], exp_sizes[0]) # first partition is
not resized
+self.assertTrue(orig_sizes[1] < exp_sizes[1])
+
+# Check if all free space is partitioned
+result = runCmd("%s/usr/sbin/sfdisk -F %s" % (sysroot,
new_image_path))
+self.assertTrue("0 B, 0 bytes, 0 sectors" in result.output)
+
+os.rename(image_path, image_path + '.bak')
+os.rename(new_image_path, image_path)
+
+# Check if it boots in qemu
+with runqemu('core-image-minimal', ssh=False) as qemu:
+cmd = "ls /etc/"
+status, output = qemu.run_serial('true')
+self.assertEqual(1, status, 'Failed to run command "%s": %s' %
(cmd, output))
+finally:
+if os.path.exists(new_image_path):
+os.unlink(new_image_path)
+if os.path.exists(image_path + '.bak'):
+os.rename(image_path + '.bak', image_path)
--
2.1.4
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH v2 6/8] wic: implement 'wic write' command
This command writes image to the media or another file with
the possibility to expand partitions to fill free target space.
[YOCTO #11278]
Signed-off-by: Ed Bartosh
---
scripts/lib/wic/engine.py | 146 ++
scripts/lib/wic/help.py | 40 +
scripts/wic | 55 +
3 files changed, 241 insertions(+)
diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index 4ffb08d..303f323 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -31,6 +31,8 @@
import logging
import os
import tempfile
+import json
+import subprocess
from collections import namedtuple, OrderedDict
from distutils.spawn import find_executable
@@ -340,6 +342,143 @@ class Disk:
raise err
self._put_part_image(pnum)
+def write(self, target, expand):
+"""Write disk image to the media or file."""
+def write_sfdisk_script(outf, parts):
+for key, val in parts['partitiontable'].items():
+if key in ("partitions", "device", "firstlba", "lastlba"):
+continue
+if key == "id":
+key = "label-id"
+outf.write("{}: {}\n".format(key, val))
+outf.write("\n")
+for part in parts['partitiontable']['partitions']:
+line = ''
+for name in ('attrs', 'name', 'size', 'type', 'uuid'):
+if name == 'size' and part['type'] == 'f':
+# don't write size for extended partition
+continue
+val = part.get(name)
+if val:
+line += '{}={}, '.format(name, val)
+if line:
+line = line[:-2] # strip ', '
+if part.get('bootable'):
+line += ' ,bootable'
+outf.write("{}\n".format(line))
+outf.flush()
+
+def read_ptable(path):
+out = exec_cmd("{} -dJ {}".format(self.sfdisk, path))
+return json.loads(out)
+
+def write_ptable(parts, target):
+with tempfile.NamedTemporaryFile(prefix="wic-sfdisk-", mode='w')
as outf:
+write_sfdisk_script(outf, parts)
+cmd = "{} --no-reread {} < {} 2>/dev/null".format(self.sfdisk,
target, outf.name)
+try:
+subprocess.check_output(cmd, shell=True)
+except subprocess.CalledProcessError as err:
+raise WicError("Can't run '{}' command: {}".format(cmd,
err))
+
+if expand is None:
+sparse_copy(self.imagepath, target)
+else:
+# copy first sectors that may contain bootloader
+sparse_copy(self.imagepath, target, length=2048 *
self._lsector_size)
+
+# copy source partition table to the target
+parts = read_ptable(self.imagepath)
+write_ptable(parts, target)
+
+# get size of unpartitioned space
+free = None
+for line in exec_cmd("{} -F {}".format(self.sfdisk,
target)).splitlines():
+if line.startswith("Unpartitioned space ") and
line.endswith("sectors"):
+free = int(line.split()[-2])
+if free is None:
+raise WicError("Can't get size of unpartitioned space")
+
+# calculate expanded partitions sizes
+sizes = {}
+for num, part in enumerate(parts['partitiontable']['partitions'],
1):
+if num in expand:
+if expand[num] != 0: # don't resize partition if size is
set to 0
+sectors = expand[num] // self._lsector_size
+free -= sectors - part['size']
+part['size'] = sectors
+sizes[num] = sectors
+elif part['type'] != 'f':
+sizes[num] = -1
+
+for num, part in enumerate(parts['partitiontable']['partitions'],
1):
+if sizes.get(num) == -1:
+part['size'] += free // len(sizes)
+
+# write resized partition table to the target
+write_ptable(parts, target)
+
+# read resized partition table
+parts = read_ptable(target)
+
+# copy partitions content
+for num, part in enumerate(parts['partitio
[OE-core] [PATCH v2 5/8] wic: extend list of used tools
Added sfdisk, e2fsck, mkswap, resize2fs, mkdosfs to the
list of used tools in Disk class. They're going to be used
in 'wic write' implementation.
Added dependency to util-linux to wic-tools to ensure that
sfdisk and mkswap are available from wic-tools native sysroot.
Signed-off-by: Ed Bartosh
---
meta/recipes-core/meta/wic-tools.bb | 3 ++-
scripts/lib/wic/engine.py | 3 ++-
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/meta/recipes-core/meta/wic-tools.bb
b/meta/recipes-core/meta/wic-tools.bb
index 428befe..57dd37a 100644
--- a/meta/recipes-core/meta/wic-tools.bb
+++ b/meta/recipes-core/meta/wic-tools.bb
@@ -5,7 +5,8 @@ LICENSE = "MIT"
DEPENDS = "\
parted-native syslinux-native gptfdisk-native dosfstools-native \
mtools-native bmap-tools-native grub-efi-native cdrtools-native \
- btrfs-tools-native squashfs-tools-native pseudo-native
e2fsprogs-native \
+ btrfs-tools-native squashfs-tools-native pseudo-native \
+ e2fsprogs-native util-linux-native \
"
DEPENDS_append_x86 = " syslinux grub-efi systemd-boot"
DEPENDS_append_x86-64 = " syslinux grub-efi systemd-boot"
diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index 9f2e87f..4ffb08d 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -276,7 +276,8 @@ class Disk:
def __getattr__(self, name):
"""Get path to the executable in a lazy way."""
-if name in ("mdir", "mcopy", "mdel", "mdeltree"):
+if name in ("mdir", "mcopy", "mdel", "mdeltree", "sfdisk", "e2fsck",
+"resize2fs", "mkswap", "mkdosfs"):
aname = "_%s" % name
if aname not in self.__dict__:
setattr(self, aname, find_executable(name, self.paths))
--
2.1.4
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH v2 4/8] wic: added 'fstypes' parameter to Disk.__init__
This parameter specifies list of supported filesystems.
So far only 'fat' is supported, but 'wic write' is going
to support at least 'fat', 'ext' and 'swap'.
Signed-off-by: Ed Bartosh
---
scripts/lib/wic/engine.py | 9 +++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index e169147..9f2e87f 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -232,9 +232,10 @@ def wic_list(args, scripts_path):
class Disk:
-def __init__(self, imagepath, native_sysroot):
+def __init__(self, imagepath, native_sysroot, fstypes=('fat',)):
self.imagepath = imagepath
self.native_sysroot = native_sysroot
+self.fstypes = fstypes
self._partitions = None
self._partimages = {}
self._lsector_size = None
@@ -288,7 +289,11 @@ class Disk:
if pnum not in self.partitions:
raise WicError("Partition %s is not in the image")
part = self.partitions[pnum]
-if not part.fstype.startswith("fat"):
+# check if fstype is supported
+for fstype in self.fstypes:
+if part.fstype.startswith(fstype):
+break
+else:
raise WicError("Not supported fstype: {}".format(part.fstype))
if pnum not in self._partimages:
tmpf = tempfile.NamedTemporaryFile(prefix="wic-part")
--
2.1.4
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH v2 3/8] wic: convert partition number to int
Converted partition number to int in order to use
it as an index in the list of partitions.
Signed-off-by: Ed Bartosh
---
scripts/lib/wic/engine.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index a965b8b..e169147 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -267,7 +267,7 @@ class Disk:
self._psector_size = int(psector_size)
for line in splitted[2:]:
pnum, start, end, size, fstype = line.split(':')[:5]
-partition = parttype(pnum, int(start[:-1]), int(end[:-1]),
+partition = parttype(int(pnum), int(start[:-1]), int(end[:-1]),
int(size[:-1]), fstype)
self._partitions[pnum] = partition
@@ -341,7 +341,7 @@ def wic_ls(args, native_sysroot):
if disk.partitions:
print('Num StartEnd Size Fstype')
for part in disk.partitions.values():
-print("{:2s} {:12d} {:12d} {:12d} {}".format(\
+print("{:2d} {:12d} {:12d} {:12d} {}".format(\
part.pnum, part.start, part.end,
part.size, part.fstype))
else:
--
2.1.4
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH v2 2/8] wic: get more info from the 'parted print' output
Got partition type and sector sizes from the output
of 'parted print'. This info may be used in the implementation
of 'wic write' command.
Signed-off-by: Ed Bartosh
---
scripts/lib/wic/engine.py | 9 -
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index b23dd65..a965b8b 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -237,6 +237,9 @@ class Disk:
self.native_sysroot = native_sysroot
self._partitions = None
self._partimages = {}
+self._lsector_size = None
+self._psector_size = None
+self._ptable_format = None
# find parted
self.paths = "/bin:/usr/bin:/usr/sbin:/sbin/"
@@ -258,7 +261,11 @@ class Disk:
self._partitions = OrderedDict()
out = exec_cmd("%s -sm %s unit B print" % (self.parted,
self.imagepath))
parttype = namedtuple("Part", "pnum start end size fstype")
-for line in out.splitlines()[2:]:
+splitted = out.splitlines()
+lsector_size, psector_size, self._ptable_format =
splitted[1].split(":")[3:6]
+self._lsector_size = int(lsector_size)
+self._psector_size = int(psector_size)
+for line in splitted[2:]:
pnum, start, end, size, fstype = line.split(':')[:5]
partition = parttype(pnum, int(start[:-1]), int(end[:-1]),
int(size[:-1]), fstype)
--
2.1.4
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH v2 0/8] #11278 - wic: create an option to use entire disk
Hi, This patchset implements wic write [--expand auto|:] subcommand to write partitioned image to the target media or file with the optional possibility to expand partitions to either fill the target media(--expand auto) or using specified partition sizes (--expand :) 'wic write' uses bmaptool API, so it's much faster than dd. It doesn't require root privileges, just write permissions for the target file or device. 'wic write' can also be used to flash unpartitioned images. Changes in v2: squashed 'wic write' implementation in one commit fixed typo in commit message The following changes since commit a087e0bc765ade6386720f22d842e2fc0bd5f128: maintainers.inc: assign newly added recipes (2017-08-24 16:49:57 +0100) are available in the git repository at: git://git.yoctoproject.org/poky-contrib ed/wip http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/wip Ed Bartosh (8): wic: reimplement getting paths of used tools wic: get more info from the 'parted print' output wic: convert partition number to int wic: added 'fstypes' parameter to Disk.__init__ wic: extend list of used tools wic: implement 'wic write' command wic: always read image partitions wic: setlftest: test expanding MBR image meta/lib/oeqa/selftest/cases/wic.py | 53 + meta/recipes-core/meta/wic-tools.bb | 3 +- scripts/lib/wic/engine.py | 210 ++-- scripts/lib/wic/help.py | 40 +++ scripts/wic | 55 ++ 5 files changed, 326 insertions(+), 35 deletions(-) -- 2.1.4 -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH v2 1/8] wic: reimplement getting paths of used tools
So far every used tool have to have separate property and
private attribute in the Disk class. This is too verbose,
considering that there will be much more tools used.
Reimplemented getting tools paths using custom __getattr__
method. This is much more compact and readable.
Signed-off-by: Ed Bartosh
---
scripts/lib/wic/engine.py | 36 +---
1 file changed, 9 insertions(+), 27 deletions(-)
diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index 2dc2fd5..b23dd65 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -236,10 +236,6 @@ class Disk:
self.imagepath = imagepath
self.native_sysroot = native_sysroot
self._partitions = None
-self._mdir = None
-self._mcopy = None
-self._mdel = None
-self._mdeltree = None
self._partimages = {}
# find parted
@@ -270,30 +266,16 @@ class Disk:
return self._partitions
-def _prop(self, name):
+def __getattr__(self, name):
"""Get path to the executable in a lazy way."""
-aname = "_%s" % name
-if getattr(self, aname) is None:
-setattr(self, aname, find_executable(name, self.paths))
-if not getattr(self, aname):
-raise WicError("Can't find executable {}".format(name))
-return getattr(self, aname)
-
-@property
-def mdir(self):
-return self._prop('mdir')
-
-@property
-def mcopy(self):
-return self._prop("mcopy")
-
-@property
-def mdel(self):
-return self._prop("mdel")
-
-@property
-def mdeltree(self):
-return self._prop("mdeltree")
+if name in ("mdir", "mcopy", "mdel", "mdeltree"):
+aname = "_%s" % name
+if aname not in self.__dict__:
+setattr(self, aname, find_executable(name, self.paths))
+if aname not in self.__dict__:
+raise WicError("Can't find executable {}".format(name))
+return self.__dict__[aname]
+return self.__dict__[name]
def _get_part_image(self, pnum):
if pnum not in self.partitions:
--
2.1.4
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH 10/10] wic: setlftest: test expanding MBR image
Added test_expand_mbr_image test case to the wic oe-selftest suite.
The test expands directrisk wic image to 1Gb target, checks if it's
expanded correctly and boots it in qemu to make sure the image
is bootable, i.e. bootloader, kernel, boot and root partitions
are still functional.
Signed-off-by: Ed Bartosh
---
meta/lib/oeqa/selftest/cases/wic.py | 53 +
1 file changed, 53 insertions(+)
diff --git a/meta/lib/oeqa/selftest/cases/wic.py
b/meta/lib/oeqa/selftest/cases/wic.py
index aaefd4f..aa73ba4 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -938,3 +938,56 @@ part /etc --source rootfs --ondisk mmcblk0 --fstype=ext4
--exclude-path bin/ --r
wksname = os.path.splitext(os.path.basename(wks.name))[0]
out = glob(self.resultdir + "%s-*direct" % wksname)
self.assertEqual(1, len(out))
+
+def test_expand_mbr_image(self):
+"""Test wic write --expand command for mbr image"""
+# build an image
+config = 'IMAGE_FSTYPES = "wic"\nWKS_FILE = "directdisk.wks"\n'
+self.append_config(config)
+self.assertEqual(0, bitbake('core-image-minimal').status)
+
+# get path to the image
+bb_vars = get_bb_vars(['DEPLOY_DIR_IMAGE', 'MACHINE'])
+deploy_dir = bb_vars['DEPLOY_DIR_IMAGE']
+machine = bb_vars['MACHINE']
+image_path = os.path.join(deploy_dir, 'core-image-minimal-%s.wic' %
machine)
+
+self.remove_config(config)
+
+try:
+# expand image to 1G
+new_image_path = None
+with NamedTemporaryFile(mode='wb', suffix='.wic.exp',
+dir=deploy_dir, delete=False) as sparse:
+sparse.truncate(1024 ** 3)
+new_image_path = sparse.name
+
+sysroot = get_bb_var('RECIPE_SYSROOT_NATIVE', 'wic-tools')
+cmd = "wic write -n %s --expand 1:0 %s %s" % (sysroot, image_path,
new_image_path)
+self.assertEqual(0, runCmd(cmd).status)
+
+# check if partitions are expanded
+orig = runCmd("wic ls %s -n %s" % (image_path, sysroot))
+exp = runCmd("wic ls %s -n %s" % (new_image_path, sysroot))
+orig_sizes = [int(line.split()[3]) for line in
orig.output.split('\n')[1:]]
+exp_sizes = [int(line.split()[3]) for line in
exp.output.split('\n')[1:]]
+self.assertEqual(orig_sizes[0], exp_sizes[0]) # first partition is
not resized
+self.assertTrue(orig_sizes[1] < exp_sizes[1])
+
+# Check if all free space is partitioned
+result = runCmd("%s/usr/sbin/sfdisk -F %s" % (sysroot,
new_image_path))
+self.assertTrue("0 B, 0 bytes, 0 sectors" in result.output)
+
+os.rename(image_path, image_path + '.bak')
+os.rename(new_image_path, image_path)
+
+# Check if it boots in qemu
+with runqemu('core-image-minimal', ssh=False) as qemu:
+cmd = "ls /etc/"
+status, output = qemu.run_serial('true')
+self.assertEqual(1, status, 'Failed to run command "%s": %s' %
(cmd, output))
+finally:
+if os.path.exists(new_image_path):
+os.unlink(new_image_path)
+if os.path.exists(image_path + '.bak'):
+os.rename(image_path + '.bak', image_path)
--
2.1.4
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH 08/10] wic: implement 'wic write' command
This command writes image to the media or another file with
the possibility to expand partitions to fill free target space.
Signed-off-by: Ed Bartosh
---
scripts/lib/wic/engine.py | 142 +-
1 file changed, 141 insertions(+), 1 deletion(-)
diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index 3c1f0d3..303f323 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -31,6 +31,8 @@
import logging
import os
import tempfile
+import json
+import subprocess
from collections import namedtuple, OrderedDict
from distutils.spawn import find_executable
@@ -340,6 +342,143 @@ class Disk:
raise err
self._put_part_image(pnum)
+def write(self, target, expand):
+"""Write disk image to the media or file."""
+def write_sfdisk_script(outf, parts):
+for key, val in parts['partitiontable'].items():
+if key in ("partitions", "device", "firstlba", "lastlba"):
+continue
+if key == "id":
+key = "label-id"
+outf.write("{}: {}\n".format(key, val))
+outf.write("\n")
+for part in parts['partitiontable']['partitions']:
+line = ''
+for name in ('attrs', 'name', 'size', 'type', 'uuid'):
+if name == 'size' and part['type'] == 'f':
+# don't write size for extended partition
+continue
+val = part.get(name)
+if val:
+line += '{}={}, '.format(name, val)
+if line:
+line = line[:-2] # strip ', '
+if part.get('bootable'):
+line += ' ,bootable'
+outf.write("{}\n".format(line))
+outf.flush()
+
+def read_ptable(path):
+out = exec_cmd("{} -dJ {}".format(self.sfdisk, path))
+return json.loads(out)
+
+def write_ptable(parts, target):
+with tempfile.NamedTemporaryFile(prefix="wic-sfdisk-", mode='w')
as outf:
+write_sfdisk_script(outf, parts)
+cmd = "{} --no-reread {} < {} 2>/dev/null".format(self.sfdisk,
target, outf.name)
+try:
+subprocess.check_output(cmd, shell=True)
+except subprocess.CalledProcessError as err:
+raise WicError("Can't run '{}' command: {}".format(cmd,
err))
+
+if expand is None:
+sparse_copy(self.imagepath, target)
+else:
+# copy first sectors that may contain bootloader
+sparse_copy(self.imagepath, target, length=2048 *
self._lsector_size)
+
+# copy source partition table to the target
+parts = read_ptable(self.imagepath)
+write_ptable(parts, target)
+
+# get size of unpartitioned space
+free = None
+for line in exec_cmd("{} -F {}".format(self.sfdisk,
target)).splitlines():
+if line.startswith("Unpartitioned space ") and
line.endswith("sectors"):
+free = int(line.split()[-2])
+if free is None:
+raise WicError("Can't get size of unpartitioned space")
+
+# calculate expanded partitions sizes
+sizes = {}
+for num, part in enumerate(parts['partitiontable']['partitions'],
1):
+if num in expand:
+if expand[num] != 0: # don't resize partition if size is
set to 0
+sectors = expand[num] // self._lsector_size
+free -= sectors - part['size']
+part['size'] = sectors
+sizes[num] = sectors
+elif part['type'] != 'f':
+sizes[num] = -1
+
+for num, part in enumerate(parts['partitiontable']['partitions'],
1):
+if sizes.get(num) == -1:
+part['size'] += free // len(sizes)
+
+# write resized partition table to the target
+write_ptable(parts, target)
+
+# read resized partition table
+parts = read_ptable(target)
+
+# copy partitions content
+for num, part in enumerate(parts['partitiontable']['partitions'],
1):
+pnum = str(num)
+fst
[OE-core] [PATCH 07/10] wic: extend list of used tools
Added sfdisk, e2fsck, mkswap, resize2fs, mkdosfs to the
list of used tools in Disk class. They're going to be used
in 'wic write' implementation.
Added dependency to util-linux to wic-tools to ensure that
sfdisk and mkswap are available from wic-tools native sysroot.
Signed-off-by: Ed Bartosh
---
meta/recipes-core/meta/wic-tools.bb | 3 ++-
scripts/lib/wic/engine.py | 3 ++-
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/meta/recipes-core/meta/wic-tools.bb
b/meta/recipes-core/meta/wic-tools.bb
index 428befe..57dd37a 100644
--- a/meta/recipes-core/meta/wic-tools.bb
+++ b/meta/recipes-core/meta/wic-tools.bb
@@ -5,7 +5,8 @@ LICENSE = "MIT"
DEPENDS = "\
parted-native syslinux-native gptfdisk-native dosfstools-native \
mtools-native bmap-tools-native grub-efi-native cdrtools-native \
- btrfs-tools-native squashfs-tools-native pseudo-native
e2fsprogs-native \
+ btrfs-tools-native squashfs-tools-native pseudo-native \
+ e2fsprogs-native util-linux-native \
"
DEPENDS_append_x86 = " syslinux grub-efi systemd-boot"
DEPENDS_append_x86-64 = " syslinux grub-efi systemd-boot"
diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index a6ea362..3c1f0d3 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -276,7 +276,8 @@ class Disk:
def __getattr__(self, name):
"""Get path to the executable in a lazy way."""
-if name in ("mdir", "mcopy", "mdel", "mdeltree"):
+if name in ("mdir", "mcopy", "mdel", "mdeltree", "sfdisk", "e2fsck",
+"resize2fs", "mkswap", "mkdosfs"):
aname = "_%s" % name
if aname not in self.__dict__:
setattr(self, aname, find_executable(name, self.paths))
--
2.1.4
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH 09/10] wic: always read image partitions
Got rid of lazy evaluation of self.partitions property.
It's not needed because partitions of the source image should
be always read.
Signed-off-by: Ed Bartosh
---
scripts/lib/wic/engine.py | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index 303f323..eafc6c7 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -254,12 +254,13 @@ class Disk:
if not self.parted:
raise WicError("Can't find executable parted")
+self.partitions = self.get_partitions()
+
def __del__(self):
for path in self._partimages.values():
os.unlink(path)
-@property
-def partitions(self):
+def get_partitions(self):
if self._partitions is None:
self._partitions = OrderedDict()
out = exec_cmd("%s -sm %s unit B print" % (self.parted,
self.imagepath))
--
2.1.4
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH 05/10] wic: convert partition number to int
Converted partition number to int in order to use
it as an index in the list of partitions.
Signed-off-by: Ed Bartosh
---
scripts/lib/wic/engine.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index b8fd3ae..83e633d 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -267,7 +267,7 @@ class Disk:
self._psector_size = int(psector_size)
for line in splitted[2:]:
pnum, start, end, size, fstype = line.split(':')[:5]
-partition = parttype(pnum, int(start[:-1]), int(end[:-1]),
+partition = parttype(int(pnum), int(start[:-1]), int(end[:-1]),
int(size[:-1]), fstype)
self._partitions[pnum] = partition
@@ -341,7 +341,7 @@ def wic_ls(args, native_sysroot):
if disk.partitions:
print('Num StartEnd Size Fstype')
for part in disk.partitions.values():
-print("{:2s} {:12d} {:12d} {:12d} {}".format(\
+print("{:2d} {:12d} {:12d} {:12d} {}".format(\
part.pnum, part.start, part.end,
part.size, part.fstype))
else:
--
2.1.4
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH 03/10] wic: reimplement getting paths of used tools
So far every used tool have to have separate property and
private attribute in the Disk class. This is too verbose,
considering that there will be much more tools used.
Reimplemented getting tools paths using custom __getattr__
method. This is much more compact and readable.
Signed-off-by: Ed Bartosh
---
scripts/lib/wic/engine.py | 36 +---
1 file changed, 9 insertions(+), 27 deletions(-)
diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index d33c9b3..a030ca2 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -236,10 +236,6 @@ class Disk:
self.imagepath = imagepath
self.native_sysroot = native_sysroot
self._partitions = None
-self._mdir = None
-self._mcopy = None
-self._mdel = None
-self._mdeltree = None
self._partimages = {}
# find parted
@@ -270,30 +266,16 @@ class Disk:
return self._partitions
-def _prop(self, name):
+def __getattr__(self, name):
"""Get path to the executable in a lazy way."""
-aname = "_%s" % name
-if getattr(self, aname) is None:
-setattr(self, aname, find_executable(name, self.paths))
-if not getattr(self, aname):
-raise WicError("Can't find executable {}".format(name))
-return getattr(self, aname)
-
-@property
-def mdir(self):
-return self._prop('mdir')
-
-@property
-def mcopy(self):
-return self._prop("mcopy")
-
-@property
-def mdel(self):
-return self._prop("mdel")
-
-@property
-def mdeltree(self):
-return self._prop("mdeltree")
+if name in ("mdir", "mcopy", "mdel", "mdeltree"):
+aname = "_%s" % name
+if aname not in self.__dict__:
+setattr(self, aname, find_executable(name, self.paths))
+if aname not in self.__dict__:
+raise WicError("Can't find executable {}".format(name))
+return self.__dict__[aname]
+return self.__dict__[name]
def _get_part_image(self, pnum):
if pnum not in self.partitions:
--
2.1.4
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH 01/10] wic: add help and usage content for 'wic write'
Added wic_write_help and wic_write_usage variables to the help.py module. These variables contain help content that will be used in 'wic write help' and 'wic write --help' output. [YOCTO #11278] Signed-off-by: Ed Bartosh --- scripts/lib/wic/help.py | 40 1 file changed, 40 insertions(+) diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py index 99912cd..ccd3382 100644 --- a/scripts/lib/wic/help.py +++ b/scripts/lib/wic/help.py @@ -468,6 +468,46 @@ DESCRIPTION containing the tools(parted and mtools) to use. """ +wic_write_usage = """ + + Write image to a device + + usage: wic write [--expand [rules]] [--native-sysroot ] + + This command writes wic image to a target device (USB stick, SD card etc). + + See 'wic help write' for more detailed instructions. + +""" + +wic_write_help = """ + +NAME +wic write - write wic image to a device + +SYNOPSIS +wic write +wic write --expand auto +wic write --expand 1:100M-2:300M +wic write --native-sysroot + +DESCRIPTION +This command writes wic image to a target device (USB stick, SD card etc) + +$ wic write ./tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic /dev/sdb + +The --expand option is used to resize image partitions. +--expand auto expands partitions to occupy all free space available on the target device. +It's also possible to specify expansion rules in a format +:[-:...] for one or more partitions. +Specifying size 0 will keep partition unmodified. +Note: Resizing boot partition can result in non-bootable image for non-EFI images. It is +recommended to use size 0 for boot partition to keep image bootable. + +The --native-sysroot option is used to specify the path to the native sysroot +containing the tools(parted, resize2fs) to use. +""" + wic_plugins_help = """ NAME -- 2.1.4 -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH 02/10] wic: add 'wic write' command
Added empty 'wic write' command that does nothing.
The functionality will be added by the next commits.
[YOCTO #11278]
Signed-off-by: Ed Bartosh
---
scripts/lib/wic/engine.py | 6 ++
scripts/wic | 55 +++
2 files changed, 61 insertions(+)
diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index 2dc2fd5..d33c9b3 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -375,6 +375,12 @@ def wic_rm(args, native_sysroot):
disk = Disk(args.path.image, native_sysroot)
disk.remove(args.path.part, args.path.path)
+def wic_write(args, native_sysroot):
+"""
+Write image to a target device.
+"""
+pass
+
def find_canned(scripts_path, file_name):
"""
Find a file either by its path or by name in the canned files dir.
diff --git a/scripts/wic b/scripts/wic
index 02bc82c..592a0e4 100755
--- a/scripts/wic
+++ b/scripts/wic
@@ -257,6 +257,13 @@ def wic_rm_subcommand(args, usage_str):
"""
engine.wic_rm(args, args.native_sysroot)
+def wic_write_subcommand(args, usage_str):
+"""
+Command-line handling for writing images.
+The real work is done by engine.wic_write()
+"""
+engine.wic_write(args, args.native_sysroot)
+
def wic_help_subcommand(args, usage_str):
"""
Command-line handling for help subcommand to keep the current
@@ -298,6 +305,9 @@ helptopics = {
"rm":[wic_help_topic_subcommand,
wic_help_topic_usage,
hlp.wic_rm_help],
+"write": [wic_help_topic_subcommand,
+ wic_help_topic_usage,
+ hlp.wic_write_help],
"list": [wic_help_topic_subcommand,
wic_help_topic_usage,
hlp.wic_list_help]
@@ -397,6 +407,47 @@ def wic_init_parser_rm(subparser):
subparser.add_argument("-n", "--native-sysroot",
help="path to the native sysroot containing the tools")
+def expandtype(rules):
+"""
+Custom type for ArgumentParser
+Converts expand rules to the dictionary {: size}
+"""
+if rules == 'auto':
+return {}
+result = {}
+for rule in rules.split('-'):
+try:
+part, size = rule.split(':')
+except ValueError:
+raise argparse.ArgumentTypeError("Incorrect rule format: %s" %
rule)
+
+if not part.isdigit():
+raise argparse.ArgumentTypeError("Rule '%s': partition number must
be integer" % rule)
+
+# validate size
+multiplier = 1
+for suffix, mult in [('K', 1024), ('M', 1024 * 1024), ('G', 1024 *
1024 * 1024)]:
+if size.upper().endswith(suffix):
+multiplier = mult
+size = size[:-1]
+break
+if not size.isdigit():
+raise argparse.ArgumentTypeError("Rule '%s': size must be integer"
% rule)
+
+result[int(part)] = int(size) * multiplier
+
+return result
+
+def wic_init_parser_write(subparser):
+subparser.add_argument("image",
+help="path to the wic image")
+subparser.add_argument("target",
+help="target file or device")
+subparser.add_argument("-e", "--expand", type=expandtype,
+help="expand rules: auto or
:[,:]")
+subparser.add_argument("-n", "--native-sysroot",
+help="path to the native sysroot containing the tools")
+
def wic_init_parser_help(subparser):
helpparsers = subparser.add_subparsers(dest='help_topic',
help=hlp.wic_usage)
for helptopic in helptopics:
@@ -425,6 +476,10 @@ subcommands = {
hlp.wic_rm_usage,
hlp.wic_rm_help,
wic_init_parser_rm],
+"write": [wic_write_subcommand,
+ hlp.wic_write_usage,
+ hlp.wic_write_help,
+ wic_init_parser_write],
"help": [wic_help_subcommand,
wic_help_topic_usage,
hlp.wic_help_help,
--
2.1.4
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH 06/10] wic: added 'fstypes' parameter to Disk.__init__
This parameter specifies list of supported filesystems.
So far only 'fat' is supported, but 'wic write' is going
to support at least 'fat', 'ext' and 'swap'.
Signed-off-by: Ed Bartosh
---
scripts/lib/wic/engine.py | 9 +++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index 83e633d..a6ea362 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -232,9 +232,10 @@ def wic_list(args, scripts_path):
class Disk:
-def __init__(self, imagepath, native_sysroot):
+def __init__(self, imagepath, native_sysroot, fstypes=('fat',)):
self.imagepath = imagepath
self.native_sysroot = native_sysroot
+self.fstypes = fstypes
self._partitions = None
self._partimages = {}
self._lsector_size = None
@@ -288,7 +289,11 @@ class Disk:
if pnum not in self.partitions:
raise WicError("Partition %s is not in the image")
part = self.partitions[pnum]
-if not part.fstype.startswith("fat"):
+# check if fstype is supported
+for fstype in self.fstypes:
+if part.fstype.startswith(fstype):
+break
+else:
raise WicError("Not supported fstype: {}".format(part.fstype))
if pnum not in self._partimages:
tmpf = tempfile.NamedTemporaryFile(prefix="wic-part")
--
2.1.4
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH 04/10] wic: get more info from the 'parted print' output
Got partition type and sector sizes from the output
of 'parted print'. This info may be used in the implementation
of 'wic write' command.
Signed-off-by: Ed Bartosh
---
scripts/lib/wic/engine.py | 9 -
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index a030ca2..b8fd3ae 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -237,6 +237,9 @@ class Disk:
self.native_sysroot = native_sysroot
self._partitions = None
self._partimages = {}
+self._lsector_size = None
+self._psector_size = None
+self._ptable_format = None
# find parted
self.paths = "/bin:/usr/bin:/usr/sbin:/sbin/"
@@ -258,7 +261,11 @@ class Disk:
self._partitions = OrderedDict()
out = exec_cmd("%s -sm %s unit B print" % (self.parted,
self.imagepath))
parttype = namedtuple("Part", "pnum start end size fstype")
-for line in out.splitlines()[2:]:
+splitted = out.splitlines()
+lsector_size, psector_size, self._ptable_format =
splitted[1].split(":")[3:6]
+self._lsector_size = int(lsector_size)
+self._psector_size = int(psector_size)
+for line in splitted[2:]:
pnum, start, end, size, fstype = line.split(':')[:5]
partition = parttype(pnum, int(start[:-1]), int(end[:-1]),
int(size[:-1]), fstype)
--
2.1.4
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH 00/10] #11278 - wic: create an option to use entire disk
Hi, This patchset implements wic write [--expand auto|:] subcommand to write partitioned image to the target media or file with the optional possibility to expand partitions to either fill the target media(--expand auto) or using specified partition sizes (--expand :) 'wic write' uses bmaptool API, so it's much faster than dd. It doesn't require root privileges, just write permissions for the target file or device. 'wic write' can also be used to flash unpartitioned images. Please, review. The following changes since commit a087e0bc765ade6386720f22d842e2fc0bd5f128: maintainers.inc: assign newly added recipes (2017-08-24 16:49:57 +0100) are available in the git repository at: git://git.yoctoproject.org/poky-contrib ed/wip http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/wip Ed Bartosh (10): wic: add help and usage content for 'wic write' wic: add 'wic write' command wic: reimplement getting paths of used tools wic: get more info from the 'parted print' output wic: convert partition number to int wic: added 'fstypes' parameter to Disk.__init__ wic: extend list of used tools wic: implement 'wic write' command wic: always read image partitions wic: setlftest: test expanding MBR image meta/lib/oeqa/selftest/cases/wic.py | 53 + meta/recipes-core/meta/wic-tools.bb | 3 +- scripts/lib/wic/engine.py | 210 ++-- scripts/lib/wic/help.py | 40 +++ scripts/wic | 55 ++ 5 files changed, 326 insertions(+), 35 deletions(-) -- Regards, Ed -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/4] wic-image-minimal: drop WKS_FILE_DEPENDS definition
On Tue, Aug 01, 2017 at 07:13:47PM +0200, liu.min...@gmail.com wrote:
> From: Ming Liu
>
> It's being set in image_types_wic.bbclass, not need to double set it
> here, furthermore, the dependencies are incomplete by missing
> e2fsprogs-native.
>
> Signed-off-by: Ming Liu
> ---
> meta-selftest/recipes-test/images/wic-image-minimal.bb | 2 --
> 1 file changed, 2 deletions(-)
>
> diff --git a/meta-selftest/recipes-test/images/wic-image-minimal.bb
> b/meta-selftest/recipes-test/images/wic-image-minimal.bb
> index e1da203..af40d96 100644
> --- a/meta-selftest/recipes-test/images/wic-image-minimal.bb
> +++ b/meta-selftest/recipes-test/images/wic-image-minimal.bb
> @@ -6,8 +6,6 @@ IMAGE_INSTALL = "packagegroup-core-boot"
>
> IMAGE_FSTYPES = "wic"
>
> -WKS_FILE_DEPENDS = "syslinux syslinux-native dosfstools-native mtools-native
> gptfdisk-native"
> -
> LIC_FILES_CHKSUM =
> "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
>
> IMAGE_ROOTFS_EXTRA_SPACE = "2000"
Please, consider adding e2fsprogs-native to the list instead of building
extra packages that are not going to be used.
--
Regards,
Ed
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 3/4] qemux86.conf: drop do_image_wic task dependencies
On Tue, Aug 01, 2017 at 07:13:49PM +0200, liu.min...@gmail.com wrote: > From: Ming Liu > > They are being controlled by WKS_FILE_DEPENDS_DEFAULT, the code here > is redundant and incomplete as well. > > Signed-off-by: Ming Liu > --- > meta/conf/machine/qemux86.conf | 1 - > 1 file changed, 1 deletion(-) > > diff --git a/meta/conf/machine/qemux86.conf b/meta/conf/machine/qemux86.conf > index c26dda2..f0bebbf 100644 > --- a/meta/conf/machine/qemux86.conf > +++ b/meta/conf/machine/qemux86.conf > @@ -29,4 +29,3 @@ MACHINE_FEATURES += "x86" > MACHINE_ESSENTIAL_EXTRA_RDEPENDS += "v86d" > > WKS_FILE ?= "directdisk.wks" > -do_image_wic[depends] += "syslinux:do_populate_sysroot > syslinux-native:do_populate_sysroot mtools-native:do_populate_sysroot > dosfstools-native:do_populate_sysroot" -1 WKS_FILE_DEPENDS_DEFAULT is a termporary setting aimed to easy transition from using wic-tools to using WKS_FILE_DEPENDS. It contains the whole set of packages that is not usually needed. Please, consider setting WKS_FILE_DEPENDS with the minimum set of packages required by directdisk.wks. This will faster the build as unneeded packages will not be built. You may want to look at "image_types_wic: set default WKS_FILE_DEPENDS" commit for further details. -- Regards, Ed -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 2/4] image_types_wic.bbclass: deal with dependencies only with WKS_FILE_DEPENDS_DEFAULT
On Tue, Aug 01, 2017 at 07:13:48PM +0200, liu.min...@gmail.com wrote:
> From: Ming Liu
>
> Drop do_image_wic depends varflags, move the dependencies to
> WKS_FILE_DEPENDS_DEFAULT. I can not see the need to maintain them
> in two different ways, besides that it's out of USING_WIC control.
>
> Signed-off-by: Ming Liu
> ---
> meta/classes/image_types_wic.bbclass | 3 +--
> 1 file changed, 1 insertion(+), 2 deletions(-)
>
> diff --git a/meta/classes/image_types_wic.bbclass
> b/meta/classes/image_types_wic.bbclass
> index b825b47..7b0a1c7 100644
> --- a/meta/classes/image_types_wic.bbclass
> +++ b/meta/classes/image_types_wic.bbclass
> @@ -39,9 +39,8 @@ IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES"
> USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + '
> '.join('wic.%s' % c for c in '${CONVERSIONTYPES}'.split()), '1', '', d)}"
> WKS_FILE_CHECKSUM = "${@'${WKS_FULL_PATH}:%s' %
> os.path.exists('${WKS_FULL_PATH}') if '${USING_WIC}' else ''}"
> do_image_wic[file-checksums] += "${WKS_FILE_CHECKSUM}"
> -do_image_wic[depends] += "${@' '.join('%s-native:do_populate_sysroot' % r
> for r in ('parted', 'gptfdisk', 'dosfstools', 'mtools'))}"
>
> -WKS_FILE_DEPENDS_DEFAULT = "syslinux-native bmap-tools-native
> cdrtools-native btrfs-tools-native squashfs-tools-native e2fsprogs-native"
> +WKS_FILE_DEPENDS_DEFAULT = "parted-native gptfdisk-native dosfstools-native
> mtools-native syslinux-native bmap-tools-native cdrtools-native
> btrfs-tools-native squashfs-tools-native e2fsprogs-native"
> WKS_FILE_DEPENDS_BOOTLOADERS = ""
> WKS_FILE_DEPENDS_BOOTLOADERS_x86 = "syslinux grub-efi systemd-boot"
> WKS_FILE_DEPENDS_BOOTLOADERS_x86-64 = "syslinux grub-efi systemd-boot"
This breakes wic build on qemux86-64 target on my machine with the
error: "ERROR: A native program parted required to build the image was not
found (see details above)."
--
Regards,
Ed
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH v3] wic: ensure generated disk system identifier is non-zero
On Mon, Jul 31, 2017 at 06:04:47PM +1000, Jonathan Liu wrote: > > > > How about random.SystemRandom().randrange(1, 0x) ? > > > > random.SystemRandom().randint(1, 0x) actually > This looks ok to me. Thanks. -- Regards, Ed -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH v3] wic: ensure generated disk system identifier is non-zero
On Sun, Jul 30, 2017 at 08:37:26PM +1000, Jonathan Liu wrote: > Hi Ed, > > On 30 July 2017 at 20:02, Ed Bartosh wrote: > > On Sat, Jul 29, 2017 at 12:45:27AM +1000, Jonathan Liu wrote: > >> Zero may be interpreted as no MBR signature present and another > >> partitioning program might install a new MBR signature. > >> > >> Signed-off-by: Jonathan Liu > >> --- > >> scripts/lib/wic/plugins/imager/direct.py | 2 +- > >> 1 file changed, 1 insertion(+), 1 deletion(-) > >> > >> diff --git a/scripts/lib/wic/plugins/imager/direct.py > >> b/scripts/lib/wic/plugins/imager/direct.py > >> index f20d8433f1..fe9b688ab0 100644 > >> --- a/scripts/lib/wic/plugins/imager/direct.py > >> +++ b/scripts/lib/wic/plugins/imager/direct.py > >> @@ -297,7 +297,7 @@ class PartitionedImage(): > >># all partitions (in bytes) > >> self.ptable_format = ptable_format # Partition table format > >> # Disk system identifier > >> -self.identifier = int.from_bytes(os.urandom(4), 'little') > >> +self.identifier = int.from_bytes(os.urandom(4), 'little') or > >> 0x > >> > > > > Can we generate random identifier by calling os.urandom again if > > self.identifier is zero? Something like this, may be? > > > > while True: > > self.identifier = int.from_bytes(os.urandom(4), 'little') > > if self.identifier: > > break > > > > Assigning constant 0x can potentially create nonunique identifiers. > > > > There is no guarantee that urandom will create unique identifiers. We can't have a guarantee with urandom, true. My point was that if you need random value it's better to call urandom again than using a constant value. > Infinite loop needs a timeout and error in the case it is unable to > generate a suitable identifier. I'm not sure it's needed, but it's not a big deal to add this. I really doubt os.urandom can generate long enough sequence of zeros. > It is only 0x if it is 0. That > means 0x is twice as likely to occur (2 in 4294967296 instead > of 1 in 4294967296) so there is tiny bias. Note that this is how it is > implemented for the DISK_SIGNATURE variable in OE and in GNU Parted as > well. Does this mean we shouldn't do better? > >> self.partitions = partitions > >> self.partimages = [] > >> -- > >> 2.13.2 > >> > >> -- > >> ___ > >> Openembedded-core mailing list > >> Openembedded-core@lists.openembedded.org > >> http://lists.openembedded.org/mailman/listinfo/openembedded-core > > Regards, > Jonathan -- Regards, Ed -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH] x86-base.inc: enable live image type
live image type was replaced by hddimg recently. This made NOHDD and NOISO options ineffective as they only influence live builds. It also causes image building failure for image sizes >4Gb Returned back live image type and disabled building iso image. This doesn't change result (hddimg is built), but it makes NOHDD and NOISO working as expected. [YOCTO #11842] Signed-off-by: Ed Bartosh --- meta/conf/machine/include/x86-base.inc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/meta/conf/machine/include/x86-base.inc b/meta/conf/machine/include/x86-base.inc index a29641a..bc0c27b 100644 --- a/meta/conf/machine/include/x86-base.inc +++ b/meta/conf/machine/include/x86-base.inc @@ -10,7 +10,8 @@ MACHINE_FEATURES += "screen keyboard pci usbhost ext2 ext3 x86 \ MACHINE_EXTRA_RRECOMMENDS += "kernel-modules" -IMAGE_FSTYPES ?= "hddimg" +IMAGE_FSTYPES ?= "live" +NOISO ?= "1" KERNEL_IMAGETYPE ?= "bzImage" -- 2.1.4 -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH v3] wic: ensure generated disk system identifier is non-zero
On Sat, Jul 29, 2017 at 12:45:27AM +1000, Jonathan Liu wrote: > Zero may be interpreted as no MBR signature present and another > partitioning program might install a new MBR signature. > > Signed-off-by: Jonathan Liu > --- > scripts/lib/wic/plugins/imager/direct.py | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/scripts/lib/wic/plugins/imager/direct.py > b/scripts/lib/wic/plugins/imager/direct.py > index f20d8433f1..fe9b688ab0 100644 > --- a/scripts/lib/wic/plugins/imager/direct.py > +++ b/scripts/lib/wic/plugins/imager/direct.py > @@ -297,7 +297,7 @@ class PartitionedImage(): ># all partitions (in bytes) > self.ptable_format = ptable_format # Partition table format > # Disk system identifier > -self.identifier = int.from_bytes(os.urandom(4), 'little') > +self.identifier = int.from_bytes(os.urandom(4), 'little') or > 0x > Can we generate random identifier by calling os.urandom again if self.identifier is zero? Something like this, may be? while True: self.identifier = int.from_bytes(os.urandom(4), 'little') if self.identifier: break Assigning constant 0x can potentially create nonunique identifiers. > self.partitions = partitions > self.partimages = [] > -- > 2.13.2 > > -- > ___ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-core -- -- Regards, Ed -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] image: Convert vmdk/vdi/qcow2 to strict CONVERSION_CMD types
sha512sum bmap u-boot ${COMPRESSIONTYPES}"
> >> > +CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum
> >> > sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2
> >> > ${COMPRESSIONTYPES}"
> >> > CONVERSION_CMD_lzma = "lzma -k -f -7
> >> > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
> >> > CONVERSION_CMD_gz = "gzip -f -9 -c
> >> > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >
> >> > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
> >> > CONVERSION_CMD_bz2 = "pbzip2 -f -k
> >> > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
> >> > @@ -287,6 +283,9 @@ CONVERSION_CMD_sha384sum = "sha384sum
> >> > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}
> >> > CONVERSION_CMD_sha512sum = "sha512sum
> >> > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >
> >> > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha512sum"
> >> > CONVERSION_CMD_bmap = "bmaptool create
> >> > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} -o
> >> > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.bmap"
> >> > CONVERSION_CMD_u-boot = "mkimage -A ${UBOOT_ARCH} -O linux -T ramdisk
> >> > -C none -n ${IMAGE_NAME} -d ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}
> >> > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.u-boot"
> >> > +CONVERSION_CMD_vmdk = "qemu-img convert -O vmdk
> >> > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}
> >> > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.vmdk"
> >> > +CONVERSION_CMD_vdi = "qemu-img convert -O vdi
> >> > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}
> >> > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.vdi"
> >> > +CONVERSION_CMD_qcow2 = "qemu-img convert -O qcow2
> >> > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}
> >> > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.qcow2"
> >> > CONVERSION_DEPENDS_lzma = "xz-native"
> >> > CONVERSION_DEPENDS_gz = "pigz-native"
> >> > CONVERSION_DEPENDS_bz2 = "pbzip2-native"
> >> > @@ -297,6 +296,9 @@ CONVERSION_DEPENDS_zip = "zip-native"
> >> > CONVERSION_DEPENDS_sum = "mtd-utils-native"
> >> > CONVERSION_DEPENDS_bmap = "bmap-tools-native"
> >> > CONVERSION_DEPENDS_u-boot = "u-boot-mkimage-native"
> >> > +CONVERSION_DEPENDS_vmdk = "qemu-native"
> >> > +CONVERSION_DEPENDS_vdi = "qemu-native"
> >> > +CONVERSION_DEPENDS_qcow2 = "qemu-native"
> >> >
> >> > RUNNABLE_IMAGE_TYPES ?= "ext2 ext3 ext4"
> >> > RUNNABLE_MACHINE_PATTERNS ?= "qemu"
> >> > diff --git a/meta/recipes-core/images/build-appliance-image_15.0.0.bb
> >> > b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
> >> > index f145b5e..927a931 100644
> >> > --- a/meta/recipes-core/images/build-appliance-image_15.0.0.bb
> >> > +++ b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
> >> > @@ -19,7 +19,7 @@ IMAGE_ROOTFS_EXTRA_SPACE = "41943040"
> >> > APPEND += "rootfstype=ext4 quiet"
> >> >
> >> > DEPENDS = "zip-native python3-pip-native"
> >> > -IMAGE_FSTYPES = "vmdk"
> >> > +IMAGE_FSTYPES = "wic.vmdk"
> >> >
> >> > inherit core-image module-base setuptools3
> >> >
> >> > @@ -120,7 +120,7 @@ create_bundle_files () {
> >> > cd ${WORKDIR}
> >> > mkdir -p Yocto_Build_Appliance
> >> > cp *.vmx* Yocto_Build_Appliance
> >> > - ln -sf ${IMGDEPLOYDIR}/${IMAGE_NAME}.vmdk
> >> > Yocto_Build_Appliance/Yocto_Build_Appliance.vmdk
> >> > + ln -sf
> >> > ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.wic.vmdk
> >> > Yocto_Build_Appliance/Yocto_Build_Appliance.vmdk
> >> > zip -r ${IMGDEPLOYDIR}/Yocto_Build_Appliance-${DATETIME}.zip
> >> > Yocto_Build_Appliance
> >> > ln -sf Yocto_Build_Appliance-${DATETIME}.zip
> >> > ${IMGDEPLOYDIR}/Yocto_Build_Appliance.zip
> >> > }
> >> > @@ -130,4 +130,4 @@ python do_bundle_files() {
> >> > bb.build.exec_func('create_bundle_files', d)
> >> > }
> >> >
> >> > -addtask bundle_files after do_vmimg before do_image_complete
> >> > +addtask bundle_files after do_image_wic before do_image_complete
> >>
> >> I know some people rely on DISK_SIGNATURE. Does DISK_SIGNATURE still
> >> work after this change?
> >
> > It does not, because the code is gone. Can you point me to users of it?
> > We could move it to the "common" file if there's users still. FWIW, it
> > may be better to leverage the UUID support in wic.
> >
> > --
> > Tom
>
> Here is an example of a user having an issue with DISK_SIGNATURE:
> http://lists.openembedded.org/pipermail/openembedded-core/2017-May/136940.html
>
> I fixed the issue in:
> http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=d2390c0646eb6c66ea2bbce69cea9d2c412bdd93
>
> Does wic support 32-bit MBR disk signature (byte offset 440 in MBR
> partition table)?
>
Yes, it does.
--
Regards,
Ed
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH v5 4/7] wic: rootfs: fix rootfs path reporting
wic gets rootfs paths from partition object property
'rootfs_dir' and shows them in final report.
rootfs plugin sets this property to the temporary path,
which causes temporary paths appearing in the report.
Changed the code to prevent storing temporary rootfs path
in part.rootfs_dir. This should fix the report.
Signed-off-by: Ed Bartosh
---
scripts/lib/wic/plugins/source/rootfs.py | 10 --
1 file changed, 4 insertions(+), 6 deletions(-)
diff --git a/scripts/lib/wic/plugins/source/rootfs.py
b/scripts/lib/wic/plugins/source/rootfs.py
index c08f760..e438158 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -81,8 +81,9 @@ class RootfsPlugin(SourcePlugin):
raise WicError("Couldn't find --rootfs-dir=%s connection or "
"it is not a valid path, exiting" %
part.rootfs_dir)
-real_rootfs_dir = cls.__get_rootfs_dir(rootfs_dir)
+part.rootfs_dir = cls.__get_rootfs_dir(rootfs_dir)
+new_rootfs = None
# Handle excluded paths.
if part.exclude_path is not None:
# We need a new rootfs directory we can delete files from. Copy to
@@ -92,9 +93,7 @@ class RootfsPlugin(SourcePlugin):
if os.path.lexists(new_rootfs):
shutil.rmtree(os.path.join(new_rootfs))
-copyhardlinktree(real_rootfs_dir, new_rootfs)
-
-real_rootfs_dir = new_rootfs
+copyhardlinktree(part.rootfs_dir, new_rootfs)
for orig_path in part.exclude_path:
path = orig_path
@@ -123,6 +122,5 @@ class RootfsPlugin(SourcePlugin):
# Delete whole directory.
shutil.rmtree(full_path)
-part.rootfs_dir = real_rootfs_dir
part.prepare_rootfs(cr_workdir, oe_builddir,
-real_rootfs_dir, native_sysroot)
+new_rootfs or part.rootfs_dir, native_sysroot)
--
2.1.4
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH v5 3/7] wic: use absolute paths in rootfs plugin
Using relative paths can cause copyhardlinktree API to fail as
it changes current directory when working. Converted all paths
to absolute paths using os.path.realpath.
Signed-off-by: Ed Bartosh
---
scripts/lib/wic/plugins/source/rootfs.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/scripts/lib/wic/plugins/source/rootfs.py
b/scripts/lib/wic/plugins/source/rootfs.py
index 4dc4cb8..c08f760 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -48,7 +48,7 @@ class RootfsPlugin(SourcePlugin):
@staticmethod
def __get_rootfs_dir(rootfs_dir):
if os.path.isdir(rootfs_dir):
-return rootfs_dir
+return os.path.realpath(rootfs_dir)
image_rootfs_dir = get_bitbake_var("IMAGE_ROOTFS", rootfs_dir)
if not os.path.isdir(image_rootfs_dir):
@@ -56,7 +56,7 @@ class RootfsPlugin(SourcePlugin):
"named %s has been found at %s, exiting." %
(rootfs_dir, image_rootfs_dir))
-return image_rootfs_dir
+return os.path.realpath(image_rootfs_dir)
@classmethod
def do_prepare_partition(cls, part, source_params, cr, cr_workdir,
--
2.1.4
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH v5 7/7] oe-selftest: wic: fix test_quemu
This test case boots the image in qemu and checks for mounted partitions. As /boot is mounted automatically the test case fails. Fixed this by adding /boot to the list of mounted partitions. Signed-off-by: Ed Bartosh --- meta/lib/oeqa/selftest/cases/wic.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/lib/oeqa/selftest/cases/wic.py b/meta/lib/oeqa/selftest/cases/wic.py index 7c2c877..aaefd4f 100644 --- a/meta/lib/oeqa/selftest/cases/wic.py +++ b/meta/lib/oeqa/selftest/cases/wic.py @@ -637,7 +637,7 @@ part /etc --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path bin/ --r cmd = "mount |grep '^/dev/' | cut -f1,3 -d ' '" status, output = qemu.run_serial(cmd) self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output)) -self.assertEqual(output, '/dev/root /\r\n/dev/sda3 /mnt') +self.assertEqual(output, '/dev/root /\r\n/dev/sda1 /boot\r\n/dev/sda3 /mnt') @only_for_arch(['i586', 'i686', 'x86_64']) @OETestID(1852) -- 2.1.4 -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH v5 5/7] wic: rootfs: make copied rootfs unique
Used unique suffix (line number from .wks file) for the copied rootfs directory to avoid possible conflicts. Signed-off-by: Ed Bartosh --- scripts/lib/wic/plugins/source/rootfs.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py index e438158..aec720f 100644 --- a/scripts/lib/wic/plugins/source/rootfs.py +++ b/scripts/lib/wic/plugins/source/rootfs.py @@ -88,7 +88,7 @@ class RootfsPlugin(SourcePlugin): if part.exclude_path is not None: # We need a new rootfs directory we can delete files from. Copy to # workdir. -new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs")) +new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs%d" % part.lineno)) if os.path.lexists(new_rootfs): shutil.rmtree(os.path.join(new_rootfs)) -- 2.1.4 -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH v5 6/7] wic: add /boot mount point to fstab by default
wic avoided adding /boot to fstab for no reason.
This exception was hardcoded in the wic code.
There is no need for this as mountpoint in .wks file is an optional
field. It can be used only if user wants to have partitions
automatically mounted on system boot.
[YOCTO #11662]
Signed-off-by: Ed Bartosh
---
scripts/lib/wic/plugins/imager/direct.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/lib/wic/plugins/imager/direct.py
b/scripts/lib/wic/plugins/imager/direct.py
index 9b31c9e..9cb2c46 100644
--- a/scripts/lib/wic/plugins/imager/direct.py
+++ b/scripts/lib/wic/plugins/imager/direct.py
@@ -135,7 +135,7 @@ class DirectPlugin(ImagerPlugin):
updated = False
for part in parts:
if not part.realnum or not part.mountpoint \
- or part.mountpoint in ("/", "/boot"):
+ or part.mountpoint == "/":
continue
# mmc device partitions are named mmcblk0p1, mmcblk0p2..
--
2.1.4
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH v5 2/7] wic: copy rootfs directory before changing fstab
wic updates /etc/fstab on root partition if there are
valid mount points in .wks
When wic runs from bitbake this can cause incorrect results
or even breakage of other tasks working with the same rootfs
directory in parallel with do_image_wic.
Implemented hardlinking rootfs directory to a temporary location
before updating fstab to avoid conflicts with other tasks working
with the same rootfs directory.
Signed-off-by: Ed Bartosh
---
scripts/lib/wic/plugins/imager/direct.py | 24 +++-
1 file changed, 15 insertions(+), 9 deletions(-)
diff --git a/scripts/lib/wic/plugins/imager/direct.py
b/scripts/lib/wic/plugins/imager/direct.py
index f20d843..9b31c9e 100644
--- a/scripts/lib/wic/plugins/imager/direct.py
+++ b/scripts/lib/wic/plugins/imager/direct.py
@@ -115,12 +115,20 @@ class DirectPlugin(ImagerPlugin):
fstab_lines = fstab.readlines()
if self._update_fstab(fstab_lines, self.parts):
-shutil.copyfile(fstab_path, fstab_path + ".orig")
+# hardlink rootfs content to the temporary directory to update
fstab
+# as rootfs can be used by other tasks and can't be modified
+new_rootfs = os.path.realpath(os.path.join(self.workdir,
"rootfs_copy"))
+exec_cmd("cp -al %s %s" % (image_rootfs, new_rootfs),
+ self.native_sysroot)
+
+fstab_path = os.path.join(new_rootfs, 'etc/fstab')
+
+os.unlink(fstab_path)
with open(fstab_path, "w") as fstab:
fstab.writelines(fstab_lines)
-return fstab_path
+return new_rootfs
def _update_fstab(self, fstab_lines, parts):
"""Assume partition order same as in wks"""
@@ -156,7 +164,10 @@ class DirectPlugin(ImagerPlugin):
filesystems from the artifacts directly and combine them into
a partitioned image.
"""
-fstab_path = self._write_fstab(self.rootfs_dir.get("ROOTFS_DIR"))
+new_rootfs = self._write_fstab(self.rootfs_dir.get("ROOTFS_DIR"))
+if new_rootfs:
+# rootfs was copied to update fstab
+self.rootfs_dir['ROOTFS_DIR'] = new_rootfs
for part in self.parts:
# get rootfs size from bitbake variable if it's not set in .ks file
@@ -172,12 +183,7 @@ class DirectPlugin(ImagerPlugin):
if rsize_bb:
part.size = int(round(float(rsize_bb)))
-try:
-self._image.prepare(self)
-finally:
-if fstab_path:
-shutil.move(fstab_path + ".orig", fstab_path)
-
+self._image.prepare(self)
self._image.layout_partitions()
self._image.create()
--
2.1.4
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH v5 1/7] image_types.bbclass: ignore tar exit code 1
tar exists with 1 and produces warning "file changed as we read it"
if content is changed while tar archives it. Even hardlinking content
causes tar to fail this way as it changes file ctime.
Other tasks running in parallel with do_image_tar may need to hardlink
rootfs content in order to change it, e.g. do_image_wic does this to
update etc/fstab.
Ignored tar exit code 1 to be able to hardlink rootfs content while
do_rootfs_tar is tarring it.
Signed-off-by: Ed Bartosh
---
meta/classes/image_types.bbclass | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index cf946a6..2365ce8 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -119,7 +119,8 @@ IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS}
${IMGDEPLOYDIR}/${IMAGE_NAM
# In practice, it turned out to be not needed when creating archives and
# required when extracting, but it seems prudent to use it in both cases.
IMAGE_CMD_TAR ?= "tar"
-IMAGE_CMD_tar = "${IMAGE_CMD_TAR} -cvf
${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.tar -C ${IMAGE_ROOTFS} ."
+# ignore return code 1 "file changed as we read it" as other tasks(e.g.
do_image_wic) may be hardlinking rootfs
+IMAGE_CMD_tar = "${IMAGE_CMD_TAR} -cf
${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.tar -C ${IMAGE_ROOTFS} . ||
[[ $? -eq 1 ]]"
do_image_cpio[cleandirs] += "${WORKDIR}/cpio_append"
IMAGE_CMD_cpio () {
--
2.1.4
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH v5 0/7] #11662 - wic should mount /boot
Hi, This patchset adds /boot to the /etc/fstab of root partition, making it mounted on boot. It also fixes reporting and testing issues caused by this change. The patchset also fixes long standing bug: wic updated fstab inplace in rootfs directory. This causes other tasks working with rootfs directory to produce incorrect results or crash. This is fixed by hadlinking rootfs content to the temporary directory before updating fstab. This approach caused do_image_tar to fail with the error "file changed as we read it" as hardlinking changes files ctime. In order to solve this we had to modify do_image_tar to ignore file changes. Changes in v2: squashed patches by reviewer's request Changes in v3: unlink /etc/fstab in rootfs copy before updating it Changes in v4: used 'cp -a' instead of copyhardlinktree to avoid do_image_tar failure due to changed ctime Changes in v5: back to hardlinking. ignored tar exit code 1. The following changes since commit b73f5e088a543775a2a94b60302f750edfffbd10: wic-tools: add dependency to e2fsprogs-native (2017-07-27 16:07:26 +0300) are available in the git repository at: git://git.yoctoproject.org/poky-contrib ed/wip http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/wip Ed Bartosh (7): image_types.bbclass: ignore tar exit code 1 wic: copy rootfs directory before changing fstab wic: use absolute paths in rootfs plugin wic: rootfs: fix rootfs path reporting wic: rootfs: make copied rootfs unique wic: add /boot mount point to fstab by default oe-selftest: wic: fix test_quemu meta/classes/image_types.bbclass | 3 ++- meta/lib/oeqa/selftest/cases/wic.py | 2 +- scripts/lib/wic/plugins/imager/direct.py | 26 -- scripts/lib/wic/plugins/source/rootfs.py | 16 +++- 4 files changed, 26 insertions(+), 21 deletions(-) -- 2.1.4 -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCHv2 1/5] image: Convert vmdk/vdi/qcow2 to strict CONVERSION_CMD types
On Wed, Jul 26, 2017 at 10:06:09PM -0400, Tom Rini wrote:
> The vmdk/vdi/qcow2 IMAGE_FSTYPEs predate wic. As such, they provide
> some similar underlying functionality in order to produce a "disk" image
> that in turn can be converted into different formats that various
> hypervisor types work with. They do not however provide the ability for
> other disk image types to be converted into these same output types.
> Furthermore, they are less flexible than what wic does provide. This
> drops the old style vmdk/vdi/qcow2 types and re-introduces them under
> the CONVERSION_CMD framework. The equivalent of vmdk is now wic.vmdk
> and so forth for the other types.
>
> Signed-off-by: Tom Rini
Acked-by: Ed Bartosh
> ---
> Changes in v2:
> - Update runqemu help (code matches .wic.vmdk as vmdk, does right thing,
> tested).
> - Supply an update to yocto-docs separately, to be merged once these
> changes are live.
> ---
> meta/classes/image-vm.bbclass | 171
> -
> meta/classes/image.bbclass | 3 -
> meta/classes/image_types.bbclass | 12 +-
> .../images/build-appliance-image_15.0.0.bb | 6 +-
> scripts/runqemu| 4 +-
> 5 files changed, 12 insertions(+), 184 deletions(-)
> delete mode 100644 meta/classes/image-vm.bbclass
>
> diff --git a/meta/classes/image-vm.bbclass b/meta/classes/image-vm.bbclass
> deleted file mode 100644
> index b52df9fbf5a1..
> --- a/meta/classes/image-vm.bbclass
> +++ /dev/null
> @@ -1,171 +0,0 @@
> -# image-vm.bbclass
> -# (loosly based off image-live.bbclass Copyright (C) 2004, Advanced Micro
> Devices, Inc.)
> -#
> -# Create an image which can be placed directly onto a harddisk using dd and
> then
> -# booted.
> -#
> -# This uses syslinux. extlinux would have been nice but required the ext2/3
> -# partition to be mounted. grub requires to run itself as part of the install
> -# process.
> -#
> -# The end result is a 512 boot sector populated with an MBR and partition
> table
> -# followed by an msdos fat16 partition containing syslinux and a linux kernel
> -# completed by the ext2/3 rootfs.
> -#
> -# We have to push the msdos parition table size > 16MB so fat 16 is used as
> parted
> -# won't touch fat12 partitions.
> -
> -inherit live-vm-common
> -
> -do_bootdirectdisk[depends] += "dosfstools-native:do_populate_sysroot \
> - virtual/kernel:do_deploy \
> - syslinux:do_populate_sysroot \
> - syslinux-native:do_populate_sysroot \
> - parted-native:do_populate_sysroot \
> - mtools-native:do_populate_sysroot \
> - ${PN}:do_image_${VM_ROOTFS_TYPE} \
> - "
> -
> -IMAGE_TYPEDEP_vmdk = "${VM_ROOTFS_TYPE}"
> -IMAGE_TYPEDEP_vdi = "${VM_ROOTFS_TYPE}"
> -IMAGE_TYPEDEP_qcow2 = "${VM_ROOTFS_TYPE}"
> -IMAGE_TYPEDEP_hdddirect = "${VM_ROOTFS_TYPE}"
> -IMAGE_TYPES_MASKED += "vmdk vdi qcow2 hdddirect"
> -
> -VM_ROOTFS_TYPE ?= "ext4"
> -ROOTFS ?= "${IMGDEPLOYDIR}/${IMAGE_LINK_NAME}.${VM_ROOTFS_TYPE}"
> -
> -# Used by bootloader
> -LABELS_VM ?= "boot"
> -ROOT_VM ?= "root=/dev/sda2"
> -# Using an initramfs is optional. Enable it by setting INITRD_IMAGE_VM.
> -INITRD_IMAGE_VM ?= ""
> -INITRD_VM ?= "${@'${IMGDEPLOYDIR}/${INITRD_IMAGE_VM}-${MACHINE}.cpio.gz' if
> '${INITRD_IMAGE_VM}' else ''}"
> -do_bootdirectdisk[depends] += "${@'${INITRD_IMAGE_VM}:do_image_complete' if
> '${INITRD_IMAGE_VM}' else ''}"
> -
> -BOOTDD_VOLUME_ID ?= "boot"
> -BOOTDD_EXTRA_SPACE ?= "16384"
> -
> -DISK_SIGNATURE ?= "${DISK_SIGNATURE_GENERATED}"
> -DISK_SIGNATURE[vardepsexclude] = "DISK_SIGNATURE_GENERATED"
> -
> -build_boot_dd() {
> - HDDDIR="${S}/hdd/boot"
> - HDDIMG="${S}/hdd.image"
> - IMAGE=${IMGDEPLOYDIR}/${IMAGE_NAME}.hdddirect
> -
> - populate_kernel $HDDDIR
> -
> - if [ "${PCBIOS}" = "1" ]; then
> - syslinux_hddimg_populate $HDDDIR
> - fi
> - if [ "${EFI}" = "1" ]; then
> - efi_hddimg_populate $HDDDIR
> - fi
> -
> - BLOCKS=`du -bks $HDDDIR | cut -f 1`
> - BLOCKS=`expr $BLOCKS + ${BOOTDD_EXTRA_SPACE}`
> -
> - # Remove it since mkdosfs would fail when it exists
> -
[OE-core] [PATCH v4 2/6] wic: use absolute paths in rootfs plugin
Using relative paths can cause copyhardlinktree API to fail as
it changes current directory when working. Converted all paths
to absolute paths using os.path.realpath.
Signed-off-by: Ed Bartosh
---
scripts/lib/wic/plugins/source/rootfs.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/scripts/lib/wic/plugins/source/rootfs.py
b/scripts/lib/wic/plugins/source/rootfs.py
index 4dc4cb8..c08f760 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -48,7 +48,7 @@ class RootfsPlugin(SourcePlugin):
@staticmethod
def __get_rootfs_dir(rootfs_dir):
if os.path.isdir(rootfs_dir):
-return rootfs_dir
+return os.path.realpath(rootfs_dir)
image_rootfs_dir = get_bitbake_var("IMAGE_ROOTFS", rootfs_dir)
if not os.path.isdir(image_rootfs_dir):
@@ -56,7 +56,7 @@ class RootfsPlugin(SourcePlugin):
"named %s has been found at %s, exiting." %
(rootfs_dir, image_rootfs_dir))
-return image_rootfs_dir
+return os.path.realpath(image_rootfs_dir)
@classmethod
def do_prepare_partition(cls, part, source_params, cr, cr_workdir,
--
2.1.4
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH v4 6/6] oe-selftest: wic: fix test_quemu
This test case boots the image in qemu and checks for mounted partitions. As /boot is mounted automatically the test case fails. Fixed this by adding /boot to the list of mounted partitions. Signed-off-by: Ed Bartosh --- meta/lib/oeqa/selftest/cases/wic.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/lib/oeqa/selftest/cases/wic.py b/meta/lib/oeqa/selftest/cases/wic.py index 7c2c877..aaefd4f 100644 --- a/meta/lib/oeqa/selftest/cases/wic.py +++ b/meta/lib/oeqa/selftest/cases/wic.py @@ -637,7 +637,7 @@ part /etc --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path bin/ --r cmd = "mount |grep '^/dev/' | cut -f1,3 -d ' '" status, output = qemu.run_serial(cmd) self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output)) -self.assertEqual(output, '/dev/root /\r\n/dev/sda3 /mnt') +self.assertEqual(output, '/dev/root /\r\n/dev/sda1 /boot\r\n/dev/sda3 /mnt') @only_for_arch(['i586', 'i686', 'x86_64']) @OETestID(1852) -- 2.1.4 -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH v4 5/6] wic: add /boot mount point to fstab by default
wic avoided adding /boot to fstab for no reason.
This exception was hardcoded in the wic code.
There is no need for this as mountpoint in .wks file is an optional
field. It can be used only if user wants to have partitions
automatically mounted on system boot.
[YOCTO #11662]
Signed-off-by: Ed Bartosh
---
scripts/lib/wic/plugins/imager/direct.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/lib/wic/plugins/imager/direct.py
b/scripts/lib/wic/plugins/imager/direct.py
index 07e3f3e..61500ab 100644
--- a/scripts/lib/wic/plugins/imager/direct.py
+++ b/scripts/lib/wic/plugins/imager/direct.py
@@ -133,7 +133,7 @@ class DirectPlugin(ImagerPlugin):
updated = False
for part in parts:
if not part.realnum or not part.mountpoint \
- or part.mountpoint in ("/", "/boot"):
+ or part.mountpoint == "/":
continue
# mmc device partitions are named mmcblk0p1, mmcblk0p2..
--
2.1.4
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH v4 3/6] wic: rootfs: fix rootfs path reporting
wic gets rootfs paths from partition object property
'rootfs_dir' and shows them in final report.
rootfs plugin sets this property to the temporary path,
which causes temporary paths appearing in the report.
Changed the code to prevent storing temporary rootfs path
in part.rootfs_dir. This should fix the report.
Signed-off-by: Ed Bartosh
---
scripts/lib/wic/plugins/source/rootfs.py | 10 --
1 file changed, 4 insertions(+), 6 deletions(-)
diff --git a/scripts/lib/wic/plugins/source/rootfs.py
b/scripts/lib/wic/plugins/source/rootfs.py
index c08f760..e438158 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -81,8 +81,9 @@ class RootfsPlugin(SourcePlugin):
raise WicError("Couldn't find --rootfs-dir=%s connection or "
"it is not a valid path, exiting" %
part.rootfs_dir)
-real_rootfs_dir = cls.__get_rootfs_dir(rootfs_dir)
+part.rootfs_dir = cls.__get_rootfs_dir(rootfs_dir)
+new_rootfs = None
# Handle excluded paths.
if part.exclude_path is not None:
# We need a new rootfs directory we can delete files from. Copy to
@@ -92,9 +93,7 @@ class RootfsPlugin(SourcePlugin):
if os.path.lexists(new_rootfs):
shutil.rmtree(os.path.join(new_rootfs))
-copyhardlinktree(real_rootfs_dir, new_rootfs)
-
-real_rootfs_dir = new_rootfs
+copyhardlinktree(part.rootfs_dir, new_rootfs)
for orig_path in part.exclude_path:
path = orig_path
@@ -123,6 +122,5 @@ class RootfsPlugin(SourcePlugin):
# Delete whole directory.
shutil.rmtree(full_path)
-part.rootfs_dir = real_rootfs_dir
part.prepare_rootfs(cr_workdir, oe_builddir,
-real_rootfs_dir, native_sysroot)
+new_rootfs or part.rootfs_dir, native_sysroot)
--
2.1.4
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH v4 4/6] wic: rootfs: make copied rootfs unique
Used unique suffix (line number from .wks file) for the copied rootfs directory to avoid possible conflicts. Signed-off-by: Ed Bartosh --- scripts/lib/wic/plugins/source/rootfs.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py index e438158..aec720f 100644 --- a/scripts/lib/wic/plugins/source/rootfs.py +++ b/scripts/lib/wic/plugins/source/rootfs.py @@ -88,7 +88,7 @@ class RootfsPlugin(SourcePlugin): if part.exclude_path is not None: # We need a new rootfs directory we can delete files from. Copy to # workdir. -new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs")) +new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs%d" % part.lineno)) if os.path.lexists(new_rootfs): shutil.rmtree(os.path.join(new_rootfs)) -- 2.1.4 -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH v4 1/6] wic: copy rootfs directory before changing fstab
wic updates /etc/fstab on root partition if there are
valid mount points in .wks
When wic runs from bitbake this can cause incorrect results
or even breakage of other tasks working with the same rootfs
directory in parallel with do_image_wic.
Implemented copying rootfs directory to a temporary location
before updating fstab to avoid conflicts with other tasks working
with the same rootfs directory.
Signed-off-by: Ed Bartosh
---
scripts/lib/wic/plugins/imager/direct.py | 22 +-
1 file changed, 13 insertions(+), 9 deletions(-)
diff --git a/scripts/lib/wic/plugins/imager/direct.py
b/scripts/lib/wic/plugins/imager/direct.py
index f20d843..07e3f3e 100644
--- a/scripts/lib/wic/plugins/imager/direct.py
+++ b/scripts/lib/wic/plugins/imager/direct.py
@@ -115,12 +115,18 @@ class DirectPlugin(ImagerPlugin):
fstab_lines = fstab.readlines()
if self._update_fstab(fstab_lines, self.parts):
-shutil.copyfile(fstab_path, fstab_path + ".orig")
+# copy rootfs dir to workdir to update fstab
+# as rootfs can be used by other tasks and can't be modified
+new_rootfs = os.path.realpath(os.path.join(self.workdir,
"rootfs_copy"))
+exec_cmd("cp -a %s %s" % (image_rootfs, new_rootfs),
+ self.native_sysroot)
+
+fstab_path = os.path.join(new_rootfs, 'etc/fstab')
with open(fstab_path, "w") as fstab:
fstab.writelines(fstab_lines)
-return fstab_path
+return new_rootfs
def _update_fstab(self, fstab_lines, parts):
"""Assume partition order same as in wks"""
@@ -156,7 +162,10 @@ class DirectPlugin(ImagerPlugin):
filesystems from the artifacts directly and combine them into
a partitioned image.
"""
-fstab_path = self._write_fstab(self.rootfs_dir.get("ROOTFS_DIR"))
+new_rootfs = self._write_fstab(self.rootfs_dir.get("ROOTFS_DIR"))
+if new_rootfs:
+# rootfs was copied to update fstab
+self.rootfs_dir['ROOTFS_DIR'] = new_rootfs
for part in self.parts:
# get rootfs size from bitbake variable if it's not set in .ks file
@@ -172,12 +181,7 @@ class DirectPlugin(ImagerPlugin):
if rsize_bb:
part.size = int(round(float(rsize_bb)))
-try:
-self._image.prepare(self)
-finally:
-if fstab_path:
-shutil.move(fstab_path + ".orig", fstab_path)
-
+self._image.prepare(self)
self._image.layout_partitions()
self._image.create()
--
2.1.4
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH v4 0/6] #11662 - wic should mount /boot
Hi, This patchset adds /boot to the /etc/fstab of root partition, making it mounted on boot. It also fixes reporting and testing issues caused by this change. The patchset also fixes long standing bug: wic updated fstab inplace in rootfs directory. This causes other tasks working with rootfs directory to produce incorrect results or crash. This is fixed by copying rootfs content to the temporary directory before updating fstab. In previous versions of this patchset temporary rootfs directory was created using copyhardlinktree API. This broke do_image_tar as creating hardlinks changes file ctime and causes tar to exit with the error "file changed as we read it". It's fixes in v4 by using copying instead of hardlinking. This is slower, but it shouldn't influence overall build time too much as even for a rootfs size 500Mb it takes only 1.5 sec Changes in v2: squashed patches by reviewer's request Changes in v3: unlink /etc/fstab in rootfs copy before updating it Changes in v4: used 'cp -a' instead of copyhardlinktree to avoid do_image_tar failure due to changed ctime The following changes since commit b73f5e088a543775a2a94b60302f750edfffbd10: wic-tools: add dependency to e2fsprogs-native (2017-07-27 16:07:26 +0300) are available in the git repository at: git://git.yoctoproject.org/poky-contrib ed/wip http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/wip Ed Bartosh (6): wic: copy rootfs directory before changing fstab wic: use absolute paths in rootfs plugin wic: rootfs: fix rootfs path reporting wic: rootfs: make copied rootfs unique wic: add /boot mount point to fstab by default oe-selftest: wic: fix test_quemu meta/lib/oeqa/selftest/cases/wic.py | 2 +- scripts/lib/wic/plugins/imager/direct.py | 24 ++-- scripts/lib/wic/plugins/source/rootfs.py | 16 +++- 3 files changed, 22 insertions(+), 20 deletions(-) -- Regards, Ed -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH] wic-tools: add dependency to e2fsprogs-native
Added e2fsprogs-native to the list of dependencies for wic-tools as all fs-related utilities have to be in this list. Signed-off-by: Ed Bartosh --- meta/recipes-core/meta/wic-tools.bb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/recipes-core/meta/wic-tools.bb b/meta/recipes-core/meta/wic-tools.bb index d908e48..428befe 100644 --- a/meta/recipes-core/meta/wic-tools.bb +++ b/meta/recipes-core/meta/wic-tools.bb @@ -5,7 +5,7 @@ LICENSE = "MIT" DEPENDS = "\ parted-native syslinux-native gptfdisk-native dosfstools-native \ mtools-native bmap-tools-native grub-efi-native cdrtools-native \ - btrfs-tools-native squashfs-tools-native pseudo-native \ + btrfs-tools-native squashfs-tools-native pseudo-native e2fsprogs-native \ " DEPENDS_append_x86 = " syslinux grub-efi systemd-boot" DEPENDS_append_x86-64 = " syslinux grub-efi systemd-boot" -- 2.1.4 -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH v2] oe-selftest: wic: change mkfs.btrfs options
test_mkfs_extraopts test case fails on ab with error caused
by using -K --mixed options:
output: extent-tree.c:2696: btrfs_reserve_extent: BUG_ON `ret` triggered,
value -28
For this test case it's not important to use particular options,
so changing options to anything less influential is OK.
Changed extra options for mkfs.btrfs to '--quiet' to fix the failure.
Signed-off-by: Ed Bartosh
---
meta/lib/oeqa/selftest/cases/wic.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/lib/oeqa/selftest/cases/wic.py
b/meta/lib/oeqa/selftest/cases/wic.py
index ceb92ae..aaefd4f 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -926,7 +926,7 @@ part /etc --source rootfs --ondisk mmcblk0 --fstype=ext4
--exclude-path bin/ --r
with NamedTemporaryFile("w", suffix=".wks") as wks:
wks.writelines(
['part ext2 --fstype ext2 --source rootfs
--mkfs-extraopts "-D -F -i 8192"\n',
- 'part btrfs --fstype btrfs--source rootfs --size 40M
--mkfs-extraopts "--mixed -K"\n',
+ "part btrfs --fstype btrfs--source rootfs --size 40M
--mkfs-extraopts='--quiet'\n",
'part squash --fstype squashfs --source rootfs
--mkfs-extraopts "-no-sparse -b 4096"\n',
'part emptyvfat --fstype vfat --size 1M --mkfs-extraopts
"-S 1024 -s 64"\n',
'part emptymsdos --fstype msdos --size 1M --mkfs-extraopts
"-S 1024 -s 64"\n',
--
2.1.4
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH] image_types_wic: add dependency to e2fsprogs-native
Added e2fsprogs-native to the list of default dependencies for
wic (WKS_FILE_DEPENDS_DEFAULT) as all fs-related utilities
have to be in this list.
Thanks to Patrick Ohly for noticing this.
[YOCTO #11817]
Signed-off-by: Ed Bartosh
---
meta/classes/image_types_wic.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/classes/image_types_wic.bbclass
b/meta/classes/image_types_wic.bbclass
index 57ba646..b825b47 100644
--- a/meta/classes/image_types_wic.bbclass
+++ b/meta/classes/image_types_wic.bbclass
@@ -41,7 +41,7 @@ WKS_FILE_CHECKSUM = "${@'${WKS_FULL_PATH}:%s' %
os.path.exists('${WKS_FULL_PATH}
do_image_wic[file-checksums] += "${WKS_FILE_CHECKSUM}"
do_image_wic[depends] += "${@' '.join('%s-native:do_populate_sysroot' % r for
r in ('parted', 'gptfdisk', 'dosfstools', 'mtools'))}"
-WKS_FILE_DEPENDS_DEFAULT = "syslinux-native bmap-tools-native cdrtools-native
btrfs-tools-native squashfs-tools-native"
+WKS_FILE_DEPENDS_DEFAULT = "syslinux-native bmap-tools-native cdrtools-native
btrfs-tools-native squashfs-tools-native e2fsprogs-native"
WKS_FILE_DEPENDS_BOOTLOADERS = ""
WKS_FILE_DEPENDS_BOOTLOADERS_x86 = "syslinux grub-efi systemd-boot"
WKS_FILE_DEPENDS_BOOTLOADERS_x86-64 = "syslinux grub-efi systemd-boot"
--
2.1.4
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] wic: Switch to using --use-uuid by default
On Wed, Jul 26, 2017 at 10:29:09AM +0300, Ed Bartosh wrote: > On Tue, Jul 25, 2017 at 03:58:22PM -0400, Tom Rini wrote: > > The most portable way to specifiy a root device in a disk image that we > > create is to use PARTUUID rather than /dev/sda2. As background, both > > GPT and MBR tables provide valid UUID values for each partition and the > > Linux Kernel contains the logic to parse this value. With this change > > we can now boot the default disk images when used as any valid block > > device that the included kernel uses. This for example means that > > VirtualBox can be used to run vmdk without changes as it uses IDE for > > the virtual disk controller. > > > I suspect oe-selftest needs to be adjusted for this change. > > Please run "oe-selftest -r wic" to ensure all tests pass. > I've just run all wic tests in the branch with your changes - everything works, no breakages. Acked-by: Ed Bartosh > > Cc: Ed Bartosh > > Cc: Matt Porter > > Signed-off-by: Tom Rini > > --- > > scripts/lib/wic/canned-wks/common.wks.inc | 2 +- > > scripts/lib/wic/canned-wks/directdisk-bootloader-config.cfg | 8 > > scripts/lib/wic/canned-wks/qemux86-directdisk.wks | 2 +- > > 3 files changed, 6 insertions(+), 6 deletions(-) > > > > diff --git a/scripts/lib/wic/canned-wks/common.wks.inc > > b/scripts/lib/wic/canned-wks/common.wks.inc > > index 5cf2fd1..89880b4 100644 > > --- a/scripts/lib/wic/canned-wks/common.wks.inc > > +++ b/scripts/lib/wic/canned-wks/common.wks.inc > > @@ -1,3 +1,3 @@ > > # This file is included into 3 canned wks files from this directory > > part /boot --source bootimg-pcbios --ondisk sda --label boot --active > > --align 1024 > > -part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align > > 1024 > > +part / --source rootfs --use-uuid --fstype=ext4 --label platform --align > > 1024 > > diff --git a/scripts/lib/wic/canned-wks/directdisk-bootloader-config.cfg > > b/scripts/lib/wic/canned-wks/directdisk-bootloader-config.cfg > > index d5a07d2..c58e74a 100644 > > --- a/scripts/lib/wic/canned-wks/directdisk-bootloader-config.cfg > > +++ b/scripts/lib/wic/canned-wks/directdisk-bootloader-config.cfg > > @@ -12,16 +12,16 @@ DEFAULT Graphics console boot > > > > LABEL Graphics console boot > > KERNEL /vmlinuz > > -APPEND label=boot root=/dev/sda2 rootwait > > +APPEND label=boot rootwait > > > > LABEL Serial console boot > > KERNEL /vmlinuz > > -APPEND label=boot root=/dev/sda2 rootwait console=ttyS0,115200 > > +APPEND label=boot rootwait console=ttyS0,115200 > > > > LABEL Graphics console install > > KERNEL /vmlinuz > > -APPEND label=install root=/dev/sda2 rootwait > > +APPEND label=install rootwait > > > > LABEL Serial console install > > KERNEL /vmlinuz > > -APPEND label=install root=/dev/sda2 rootwait console=ttyS0,115200 > > +APPEND label=install rootwait console=ttyS0,115200 > > diff --git a/scripts/lib/wic/canned-wks/qemux86-directdisk.wks > > b/scripts/lib/wic/canned-wks/qemux86-directdisk.wks > > index db30bbc..1f8466a 100644 > > --- a/scripts/lib/wic/canned-wks/qemux86-directdisk.wks > > +++ b/scripts/lib/wic/canned-wks/qemux86-directdisk.wks > > @@ -4,5 +4,5 @@ > > > > include common.wks.inc > > > > -bootloader --timeout=0 --append="vga=0 uvesafb.mode_option=640x480-32 > > root=/dev/sda2 rw mem=256M ip=192.168.7.2::192.168.7.1:255.255.255.0 > > oprofile.timer=1 rootfstype=ext4 " > > +bootloader --timeout=0 --append="vga=0 uvesafb.mode_option=640x480-32 rw > > mem=256M ip=192.168.7.2::192.168.7.1:255.255.255.0 oprofile.timer=1 > > rootfstype=ext4 " > > > > -- > > 2.1.4 > > > > -- > -- > Regards, > Ed > -- > ___ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-core -- -- Regards, Ed -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH] oe-selftest: wic: change mkfs.btrfs options
test_mkfs_extraopts test case fails on ab with error caused
by using -K --mixed options:
output: extent-tree.c:2696: btrfs_reserve_extent: BUG_ON `ret` triggered,
value -28
For this test case it's not important to use particular options,
so changing options to anything less influential is OK.
Changed extra options for mkfs.btrfs to '--quiet' to fix the failure.
Signed-off-by: Ed Bartosh
---
meta/lib/oeqa/selftest/cases/wic.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/lib/oeqa/selftest/cases/wic.py
b/meta/lib/oeqa/selftest/cases/wic.py
index ceb92ae..daa4067 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -926,7 +926,7 @@ part /etc --source rootfs --ondisk mmcblk0 --fstype=ext4
--exclude-path bin/ --r
with NamedTemporaryFile("w", suffix=".wks") as wks:
wks.writelines(
['part ext2 --fstype ext2 --source rootfs
--mkfs-extraopts "-D -F -i 8192"\n',
- 'part btrfs --fstype btrfs--source rootfs --size 40M
--mkfs-extraopts "--mixed -K"\n',
+ 'part btrfs --fstype btrfs--source rootfs --size 40M
--mkfs-extraopts "--quiet"\n',
'part squash --fstype squashfs --source rootfs
--mkfs-extraopts "-no-sparse -b 4096"\n',
'part emptyvfat --fstype vfat --size 1M --mkfs-extraopts
"-S 1024 -s 64"\n',
'part emptymsdos --fstype msdos --size 1M --mkfs-extraopts
"-S 1024 -s 64"\n',
--
2.1.4
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] wic: Switch to using --use-uuid by default
On Tue, Jul 25, 2017 at 03:58:22PM -0400, Tom Rini wrote: > The most portable way to specifiy a root device in a disk image that we > create is to use PARTUUID rather than /dev/sda2. As background, both > GPT and MBR tables provide valid UUID values for each partition and the > Linux Kernel contains the logic to parse this value. With this change > we can now boot the default disk images when used as any valid block > device that the included kernel uses. This for example means that > VirtualBox can be used to run vmdk without changes as it uses IDE for > the virtual disk controller. > I suspect oe-selftest needs to be adjusted for this change. Please run "oe-selftest -r wic" to ensure all tests pass. > Cc: Ed Bartosh > Cc: Matt Porter > Signed-off-by: Tom Rini > --- > scripts/lib/wic/canned-wks/common.wks.inc | 2 +- > scripts/lib/wic/canned-wks/directdisk-bootloader-config.cfg | 8 > scripts/lib/wic/canned-wks/qemux86-directdisk.wks | 2 +- > 3 files changed, 6 insertions(+), 6 deletions(-) > > diff --git a/scripts/lib/wic/canned-wks/common.wks.inc > b/scripts/lib/wic/canned-wks/common.wks.inc > index 5cf2fd1..89880b4 100644 > --- a/scripts/lib/wic/canned-wks/common.wks.inc > +++ b/scripts/lib/wic/canned-wks/common.wks.inc > @@ -1,3 +1,3 @@ > # This file is included into 3 canned wks files from this directory > part /boot --source bootimg-pcbios --ondisk sda --label boot --active > --align 1024 > -part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align > 1024 > +part / --source rootfs --use-uuid --fstype=ext4 --label platform --align 1024 > diff --git a/scripts/lib/wic/canned-wks/directdisk-bootloader-config.cfg > b/scripts/lib/wic/canned-wks/directdisk-bootloader-config.cfg > index d5a07d2..c58e74a 100644 > --- a/scripts/lib/wic/canned-wks/directdisk-bootloader-config.cfg > +++ b/scripts/lib/wic/canned-wks/directdisk-bootloader-config.cfg > @@ -12,16 +12,16 @@ DEFAULT Graphics console boot > > LABEL Graphics console boot > KERNEL /vmlinuz > -APPEND label=boot root=/dev/sda2 rootwait > +APPEND label=boot rootwait > > LABEL Serial console boot > KERNEL /vmlinuz > -APPEND label=boot root=/dev/sda2 rootwait console=ttyS0,115200 > +APPEND label=boot rootwait console=ttyS0,115200 > > LABEL Graphics console install > KERNEL /vmlinuz > -APPEND label=install root=/dev/sda2 rootwait > +APPEND label=install rootwait > > LABEL Serial console install > KERNEL /vmlinuz > -APPEND label=install root=/dev/sda2 rootwait console=ttyS0,115200 > +APPEND label=install rootwait console=ttyS0,115200 > diff --git a/scripts/lib/wic/canned-wks/qemux86-directdisk.wks > b/scripts/lib/wic/canned-wks/qemux86-directdisk.wks > index db30bbc..1f8466a 100644 > --- a/scripts/lib/wic/canned-wks/qemux86-directdisk.wks > +++ b/scripts/lib/wic/canned-wks/qemux86-directdisk.wks > @@ -4,5 +4,5 @@ > > include common.wks.inc > > -bootloader --timeout=0 --append="vga=0 uvesafb.mode_option=640x480-32 > root=/dev/sda2 rw mem=256M ip=192.168.7.2::192.168.7.1:255.255.255.0 > oprofile.timer=1 rootfstype=ext4 " > +bootloader --timeout=0 --append="vga=0 uvesafb.mode_option=640x480-32 rw > mem=256M ip=192.168.7.2::192.168.7.1:255.255.255.0 oprofile.timer=1 > rootfstype=ext4 " > > -- > 2.1.4 > -- -- Regards, Ed -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] image: Convert vmdk/vdi/qcow2 to strict CONVERSION_CMD types
On Tue, Jul 25, 2017 at 03:58:16PM -0400, Tom Rini wrote:
> The vmdk/vdi/qcow2 IMAGE_FSTYPEs predate wic. As such, they provide
> some similar underlying functionality in order to produce a "disk" image
> that in turn can be converted into different formats that various
> hypervisor types work with. They do not however provide the ability for
> other disk image types to be converted into these same output types.
> Furthermore, they are less flexible than what wic does provide. This
> drops the old style vmdk/vdi/qcow2 types and re-introduces them under
> the CONVERSION_CMD framework. The equivalent of vmdk is now wic.vmdk
> and so forth for the other types.
>
> Signed-off-by: Tom Rini
> ---
> This depends on my previous series to correct chaining compression
> support. I had attempted to keep the vmdk/vdi/qcow2 IMAGE_FSTYPES for
> compatibility sake using IMAGE_TYPEDEP_vmdk = "wic" and introducing an
> oe_qemuimg function to run to do the conversion. This was working, but
> I could not get it to have the symlinks created automatically. At this
> timeI think it makes most sense to not hide that the underlying disk
> content has changed at least slightly from the old vmdk type to the new
> type and that we can simply handle this change in documentation. As
> such, if there's agreement about dropping the types I'll include some
> documentation changes in the next version.
>
Great work! Thank you!
Acked-by: Ed Bartosh
How about adding couple of test cases to oe-selftest to test these
conversion types? You can see some examples in wic testing suite
in meta/lib/oeqa/selftest/cases/wic.py
> meta/classes/image-vm.bbclass | 171
> -
> meta/classes/image.bbclass | 3 -
> meta/classes/image_types.bbclass | 12 +-
> .../images/build-appliance-image_15.0.0.bb | 6 +-
> 4 files changed, 10 insertions(+), 182 deletions(-)
> delete mode 100644 meta/classes/image-vm.bbclass
>
> diff --git a/meta/classes/image-vm.bbclass b/meta/classes/image-vm.bbclass
> deleted file mode 100644
> index b52df9f..000
> --- a/meta/classes/image-vm.bbclass
> +++ /dev/null
> @@ -1,171 +0,0 @@
> -# image-vm.bbclass
> -# (loosly based off image-live.bbclass Copyright (C) 2004, Advanced Micro
> Devices, Inc.)
> -#
> -# Create an image which can be placed directly onto a harddisk using dd and
> then
> -# booted.
> -#
> -# This uses syslinux. extlinux would have been nice but required the ext2/3
> -# partition to be mounted. grub requires to run itself as part of the install
> -# process.
> -#
> -# The end result is a 512 boot sector populated with an MBR and partition
> table
> -# followed by an msdos fat16 partition containing syslinux and a linux kernel
> -# completed by the ext2/3 rootfs.
> -#
> -# We have to push the msdos parition table size > 16MB so fat 16 is used as
> parted
> -# won't touch fat12 partitions.
> -
> -inherit live-vm-common
> -
> -do_bootdirectdisk[depends] += "dosfstools-native:do_populate_sysroot \
> - virtual/kernel:do_deploy \
> - syslinux:do_populate_sysroot \
> - syslinux-native:do_populate_sysroot \
> - parted-native:do_populate_sysroot \
> - mtools-native:do_populate_sysroot \
> - ${PN}:do_image_${VM_ROOTFS_TYPE} \
> - "
> -
> -IMAGE_TYPEDEP_vmdk = "${VM_ROOTFS_TYPE}"
> -IMAGE_TYPEDEP_vdi = "${VM_ROOTFS_TYPE}"
> -IMAGE_TYPEDEP_qcow2 = "${VM_ROOTFS_TYPE}"
> -IMAGE_TYPEDEP_hdddirect = "${VM_ROOTFS_TYPE}"
> -IMAGE_TYPES_MASKED += "vmdk vdi qcow2 hdddirect"
> -
> -VM_ROOTFS_TYPE ?= "ext4"
> -ROOTFS ?= "${IMGDEPLOYDIR}/${IMAGE_LINK_NAME}.${VM_ROOTFS_TYPE}"
> -
> -# Used by bootloader
> -LABELS_VM ?= "boot"
> -ROOT_VM ?= "root=/dev/sda2"
> -# Using an initramfs is optional. Enable it by setting INITRD_IMAGE_VM.
> -INITRD_IMAGE_VM ?= ""
> -INITRD_VM ?= "${@'${IMGDEPLOYDIR}/${INITRD_IMAGE_VM}-${MACHINE}.cpio.gz' if
> '${INITRD_IMAGE_VM}' else ''}"
> -do_bootdirectdisk[depends] += "${@'${INITRD_IMAGE_VM}:do_image_complete' if
> '${INITRD_IMAGE_VM}' else ''}"
> -
> -BOOTDD_VOLUME_ID ?= "boot"
> -BOOTDD_EXTRA_SPACE ?= "16384"
> -
> -DISK_SIGNATURE ?= "${DISK_SIGNATURE_GENERATED}"
> -DISK_SIGNATURE[vardepsexclude] = "DISK_SIGNATURE_GENERATED"
> -
>
Re: [OE-core] [PATCH 1/2] image.bbclass: Correct chaining compression support
On Fri, Jul 21, 2017 at 06:06:33PM -0400, Tom Rini wrote:
> When chaining of compression/conversion types was added, we had a new
> way to handle doing things like "ext4.bz2.sha256sum" or
> "ext2.gz.u-boot". However, because the U-Boot image class isn't
> included normally, it wasn't properly converted at the time. After the
> support was added the "clean" argument that the .u-boot code uses no
> longer functions. The fix for this inadvertently broke chaining
> compression/conversion. First, correct the u-boot conversion code.
>
> Fixes: 46bc438374de ("image.bbclass: do exact match for rootfs type")
> Cc: Zhenhua Luo
> Cc: Richard Purdie
> Cc: Patrick Ohly
> Signed-off-by: Tom Rini
Acked-by: Ed Bartosh
Any chance to have this functionality covered by oe-selftest?
That would help to ensure it will not be broken again.
> ---
> This change is fairly important and, imho, innocuous and should be
> populated to pyro/etc, once merged to master. The next part of the
> series is clean-up and while with my U-Boot hat on, I would say should
> be pushed more widely, I am biased.
> ---
> meta/classes/image.bbclass | 2 +-
> meta/classes/image_types_uboot.bbclass | 13 +
> 2 files changed, 6 insertions(+), 9 deletions(-)
>
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index de535ce6fcff..bd6a5b7b810a 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -453,7 +453,7 @@ python () {
> rm_tmp_images = set()
> def gen_conversion_cmds(bt):
> for ctype in ctypes:
> -if bt[bt.find('.') + 1:] == ctype:
> +if bt.endswith("." + ctype):
> type = bt[0:-len(ctype) - 1]
> if type.startswith("debugfs_"):
> type = type[8:]
> diff --git a/meta/classes/image_types_uboot.bbclass
> b/meta/classes/image_types_uboot.bbclass
> index 5dfa39287dab..8db436efb14b 100644
> --- a/meta/classes/image_types_uboot.bbclass
> +++ b/meta/classes/image_types_uboot.bbclass
> @@ -3,9 +3,6 @@ inherit image_types kernel-arch
> oe_mkimage () {
> mkimage -A ${UBOOT_ARCH} -O linux -T ramdisk -C $2 -n ${IMAGE_NAME} \
> -d ${IMGDEPLOYDIR}/$1 ${IMGDEPLOYDIR}/$1.u-boot
> -if [ x$3 = x"clean" ]; then
> -rm $1
> -fi
> }
>
> CONVERSIONTYPES += "gz.u-boot bz2.u-boot lz4.u-boot lzma.u-boot lzo.u-boot
> u-boot"
> @@ -14,19 +11,19 @@ CONVERSION_DEPENDS_u-boot = "u-boot-mkimage-native"
> CONVERSION_CMD_u-boot = "oe_mkimage ${IMAGE_NAME}.rootfs.${type} none"
>
> CONVERSION_DEPENDS_gz.u-boot = "u-boot-mkimage-native"
> -CONVERSION_CMD_gz.u-boot = "${CONVERSION_CMD_gz}; oe_mkimage
> ${IMAGE_NAME}.rootfs.${type}.gz gzip clean"
> +CONVERSION_CMD_gz.u-boot = "${CONVERSION_CMD_gz}; oe_mkimage
> ${IMAGE_NAME}.rootfs.${type}.gz gzip"
>
> CONVERSION_DEPENDS_bz2.u-boot = "u-boot-mkimage-native"
> -CONVERSION_CMD_bz2.u-boot = "${CONVERSION_CMD_bz2}; oe_mkimage
> ${IMAGE_NAME}.rootfs.${type}.bz2 bzip2 clean"
> +CONVERSION_CMD_bz2.u-boot = "${CONVERSION_CMD_bz2}; oe_mkimage
> ${IMAGE_NAME}.rootfs.${type}.bz2 bzip2"
>
> CONVERSION_DEPENDS_lz4.u-boot = "u-boot-mkimage-native"
> -CONVERSION_CMD_lz4.u-boot = "${CONVERSION_CMD_lz4_legacy}; oe_mkimage
> ${IMAGE_NAME}.rootfs.${type}.lz4 lz4 clean"
> +CONVERSION_CMD_lz4.u-boot = "${CONVERSION_CMD_lz4_legacy}; oe_mkimage
> ${IMAGE_NAME}.rootfs.${type}.lz4 lz4"
>
> CONVERSION_DEPENDS_lzma.u-boot = "u-boot-mkimage-native"
> -CONVERSION_CMD_lzma.u-boot = "${CONVERSION_CMD_lzma}; oe_mkimage
> ${IMAGE_NAME}.rootfs.${type}.lzma lzma clean"
> +CONVERSION_CMD_lzma.u-boot = "${CONVERSION_CMD_lzma}; oe_mkimage
> ${IMAGE_NAME}.rootfs.${type}.lzma lzma"
>
> CONVERSION_DEPENDS_lzo.u-boot = "u-boot-mkimage-native"
> -CONVERSION_CMD_lzo.u-boot = "${CONVERSION_CMD_lzo}; oe_mkimage
> ${IMAGE_NAME}.rootfs.${type}.lzo lzo clean"
> +CONVERSION_CMD_lzo.u-boot = "${CONVERSION_CMD_lzo}; oe_mkimage
> ${IMAGE_NAME}.rootfs.${type}.lzo lzo"
>
> IMAGE_TYPES += "ext2.u-boot ext2.gz.u-boot ext2.bz2.u-boot ext2.lzma.u-boot
> ext3.gz.u-boot ext4.gz.u-boot cpio.gz.u-boot"
>
> --
> 1.9.1
>
> --
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
--
Regards,
Ed
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH v3 5/6] wic: add /boot mount point to fstab by default
wic avoided adding /boot to fstab for no reason.
This exception was hardcoded in the wic code.
There is no need for this as mountpoint in .wks file is an optional
field. It can be used only if user wants to have partitions
automatically mounted on system boot.
[YOCTO #11662]
Signed-off-by: Ed Bartosh
---
scripts/lib/wic/plugins/imager/direct.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/lib/wic/plugins/imager/direct.py
b/scripts/lib/wic/plugins/imager/direct.py
index 4f441c1..0a65a9c 100644
--- a/scripts/lib/wic/plugins/imager/direct.py
+++ b/scripts/lib/wic/plugins/imager/direct.py
@@ -135,7 +135,7 @@ class DirectPlugin(ImagerPlugin):
updated = False
for part in parts:
if not part.realnum or not part.mountpoint \
- or part.mountpoint in ("/", "/boot"):
+ or part.mountpoint == "/":
continue
# mmc device partitions are named mmcblk0p1, mmcblk0p2..
--
2.1.4
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH v3 6/6] oe-selftest: wic: fix test_quemu
This test case boots the image in qemu and checks for mounted partitions. As /boot is mounted automatically the test case fails. Fixed this by adding /boot to the list of mounted partitions. Signed-off-by: Ed Bartosh --- meta/lib/oeqa/selftest/cases/wic.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/lib/oeqa/selftest/cases/wic.py b/meta/lib/oeqa/selftest/cases/wic.py index df205e8..ceb92ae 100644 --- a/meta/lib/oeqa/selftest/cases/wic.py +++ b/meta/lib/oeqa/selftest/cases/wic.py @@ -637,7 +637,7 @@ part /etc --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path bin/ --r cmd = "mount |grep '^/dev/' | cut -f1,3 -d ' '" status, output = qemu.run_serial(cmd) self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output)) -self.assertEqual(output, '/dev/root /\r\n/dev/sda3 /mnt') +self.assertEqual(output, '/dev/root /\r\n/dev/sda1 /boot\r\n/dev/sda3 /mnt') @only_for_arch(['i586', 'i686', 'x86_64']) @OETestID(1852) -- 2.1.4 -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH v3 4/6] wic: rootfs: make copied rootfs unique
Used unique suffix (line number from .wks file) for the copied rootfs directory to avoid possible conflicts. Signed-off-by: Ed Bartosh --- scripts/lib/wic/plugins/source/rootfs.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py index e438158..aec720f 100644 --- a/scripts/lib/wic/plugins/source/rootfs.py +++ b/scripts/lib/wic/plugins/source/rootfs.py @@ -88,7 +88,7 @@ class RootfsPlugin(SourcePlugin): if part.exclude_path is not None: # We need a new rootfs directory we can delete files from. Copy to # workdir. -new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs")) +new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs%d" % part.lineno)) if os.path.lexists(new_rootfs): shutil.rmtree(os.path.join(new_rootfs)) -- 2.1.4 -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
