[OE-core] [PATCH v2 1/3] systemd: add PACKAGECONFIG for bpf-framework
The bpf-framework is used to pre-compile eBPFs that required for the systemd.resource-control features RestrictFileSystems=[1] and RestrictNetworkInterfaces=[2] to work. Apart from 'clang-native' to compile the eBPFs, the required kernel switches are described in [3]. Link: https://www.freedesktop.org/software/systemd/man/latest/systemd.resource-control.html#RestrictFileSystems= Link: https://www.freedesktop.org/software/systemd/man/latest/systemd.resource-control.html#RestrictNetworkInterfaces= Link: https://kinvolk.io/blog/2021/04/extending-systemd-security-features-with-ebpf/ Signed-off-by: Johannes Schneider --- meta/recipes-core/systemd/systemd_255.6.bb | 1 + 1 file changed, 1 insertion(+) diff --git a/meta/recipes-core/systemd/systemd_255.6.bb b/meta/recipes-core/systemd/systemd_255.6.bb index 15b631fc07..defdb15620 100644 --- a/meta/recipes-core/systemd/systemd_255.6.bb +++ b/meta/recipes-core/systemd/systemd_255.6.bb @@ -132,6 +132,7 @@ PACKAGECONFIG[acl] = "-Dacl=true,-Dacl=false,acl" PACKAGECONFIG[audit] = "-Daudit=true,-Daudit=false,audit" PACKAGECONFIG[backlight] = "-Dbacklight=true,-Dbacklight=false" PACKAGECONFIG[binfmt] = "-Dbinfmt=true,-Dbinfmt=false" +PACKAGECONFIG[bpf-framework] = "-Dbpf-framework=true,-Dbpf-framework=false,clang-native bpftool-native libbpf,libbpf" PACKAGECONFIG[bzip2] = "-Dbzip2=true,-Dbzip2=false,bzip2" PACKAGECONFIG[cgroupv2] = "-Ddefault-hierarchy=unified,-Ddefault-hierarchy=hybrid" PACKAGECONFIG[coredump] = "-Dcoredump=true,-Dcoredump=false" -- 2.34.1 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#200947): https://lists.openembedded.org/g/openembedded-core/message/200947 Mute This Topic: https://lists.openembedded.org/mt/106778247/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2 0/3] systemd and RestrictNetworkInterfaces
Systemd has eBPF based resource-control features to limit file-system and network-interface access [1][2] For these to be usable the corresponding eBPFs that come with systemd need to be compiled an deployed to the system - this could now be done by setting the PACKAGECONFIG+="bpf-framework" in the target build-setup. Then clang-native (from meta-clang [3]) could be used to provide the bpf-compiler, that then systemds build-setup uses to pre-compile these eBPFs; the other build requirements are bpftool-native and libbpf (both form meta-openembedded/meta-oe). On the system the only run-time dependency is then libbpf to load these pre-compiled filters. And "some kernel switches" [4]: CONFIG_BPF CONFIG_BPF_SYSCALL CONFIG_CGROUP_BPF To use/test these run for example: $> systemd-run -t -p RestrictNetworkInterfaces=enp0s3 ping 8.8.8.8 which would result in 100% packet-loss, if the default route goes over another interface. Link: https://www.freedesktop.org/software/systemd/man/latest/systemd.resource-control.html#RestrictFileSystems= Link: https://www.freedesktop.org/software/systemd/man/latest/systemd.resource-control.html#RestrictNetworkInterfaces= Link: https://github.com/kraj/meta-clang Link: https://kinvolk.io/blog/2021/04/extending-systemd-security-features-with-ebpf/ = changes with v2: rework hardcoded recipe-sysroot = oe-specific patch, to an upstreamed patch in systemd's meson.build see: https://github.com/systemd/systemd/pull/33427 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#200948): https://lists.openembedded.org/g/openembedded-core/message/200948 Mute This Topic: https://lists.openembedded.org/mt/106778248/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2 3/3] systemd: bpf-framework: pass 'recipe-sysroot' to BPF compiler
Pass the "recipe-sysroot" path via the CFLAGS=--sysroot= to the
compiler used by systemd to build the BPF, so that it can find the
needed system includes.
Signed-off-by: Johannes Schneider
---
meta/recipes-core/systemd/systemd_255.6.bb | 4
1 file changed, 4 insertions(+)
diff --git a/meta/recipes-core/systemd/systemd_255.6.bb
b/meta/recipes-core/systemd/systemd_255.6.bb
index d2c5e09697..0376412f61 100644
--- a/meta/recipes-core/systemd/systemd_255.6.bb
+++ b/meta/recipes-core/systemd/systemd_255.6.bb
@@ -234,6 +234,10 @@ PACKAGECONFIG[zstd] = "-Dzstd=true,-Dzstd=false,zstd"
RESOLV_CONF ??= ""
+# bpf-framework: pass the recipe-sysroot to the compiler used to build
+# the eBPFs, so that it can find needed system includes in there.
+CFLAGS:append = " --sysroot=${STAGING_DIR_TARGET}"
+
# Helper variables to clarify locations. This mirrors the logic in systemd's
# build system.
rootprefix ?= "${root_prefix}"
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#200950):
https://lists.openembedded.org/g/openembedded-core/message/200950
Mute This Topic: https://lists.openembedded.org/mt/106778250/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v2 2/3] systemd: bpf-framework: 'propagate' the '--sysroot=' for crosscompilation
The eBPFs are pre-compiled during the systemd-build with a different
compiler than the cross-compiler used to build systemd itself.
This is either a 'clang-native' or a gcc (bpf-unknown-none) which do
not see the BUILD_CFLAGS, that point to the correct include search
patch. To address this have systemd's meson.build "propagate" the
--system from the C_FLAGS into the BPF compiler call.
Signed-off-by: Johannes Schneider
---
...pagate-sysroot-for-cross-compilation.patch | 31 +++
meta/recipes-core/systemd/systemd_255.6.bb| 1 +
2 files changed, 32 insertions(+)
create mode 100644
meta/recipes-core/systemd/systemd/0023-meson-bpf-propagate-sysroot-for-cross-compilation.patch
diff --git
a/meta/recipes-core/systemd/systemd/0023-meson-bpf-propagate-sysroot-for-cross-compilation.patch
b/meta/recipes-core/systemd/systemd/0023-meson-bpf-propagate-sysroot-for-cross-compilation.patch
new file mode 100644
index 00..e8116b0540
--- /dev/null
+++
b/meta/recipes-core/systemd/systemd/0023-meson-bpf-propagate-sysroot-for-cross-compilation.patch
@@ -0,0 +1,31 @@
+From 7463b382bcaf26aacc60b73f98f0262aa41db3ee Mon Sep 17 00:00:00 2001
+From: Johannes Schneider
+Date: Thu, 20 Jun 2024 12:32:18 +0200
+Subject: [PATCH] meson: bpf: propagate 'sysroot' for cross compilation
+
+During cross-compilation of systemd, the compiler used to build the bpf's needs
+to be pointed at the correct include searchpath. Which can be done by passing
+the corresponding directory in through the cflags; for example in yocto/bitbake
+this would work: CFLAGS += "--sysroot=${STAGING_DIR_TARGET}"
+
+Upstream-Status: Submitted [https://github.com/systemd/systemd/pull/33427]
+Signed-off-by: Johannes Schneider
+---
+ meson.build | 1 +
+ 1 file changed, 1 insertion(+)
+
+diff --git a/meson.build b/meson.build
+index 187e7b216d..15df058ab6 100644
+--- a/meson.build
b/meson.build
+@@ -1691,6 +1691,7 @@ if conf.get('BPF_FRAMEWORK') == 1
+ '-ffile-prefix-map=',
+ '-fdebug-prefix-map=',
+ '-fmacro-prefix-map=',
++'--sysroot=',
+ ]
+
+ foreach opt : c_args
+--
+2.34.1
+
diff --git a/meta/recipes-core/systemd/systemd_255.6.bb
b/meta/recipes-core/systemd/systemd_255.6.bb
index defdb15620..d2c5e09697 100644
--- a/meta/recipes-core/systemd/systemd_255.6.bb
+++ b/meta/recipes-core/systemd/systemd_255.6.bb
@@ -28,6 +28,7 @@ SRC_URI += " \
file://systemd-pager.sh \
file://0002-binfmt-Don-t-install-dependency-links-at-install-tim.patch \
file://0008-implment-systemd-sysv-install-for-OE.patch \
+ file://0023-meson-bpf-propagate-sysroot-for-cross-compilation.patch
\
"
# patches needed by musl
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#200949):
https://lists.openembedded.org/g/openembedded-core/message/200949
Mute This Topic: https://lists.openembedded.org/mt/106778249/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v1 2/2] systemd: bpf-framework: pass STAGING_INCDIR to eBPF compiler
Hoi,
> >
> > I went to have a quick look at this but can’t built bpftool:
> >
> > ERROR: bpftool-native-1.0-r0 do_compile: oe_runmake failed
> > ERROR: bpftool-native-1.0-r0 do_compile:
> > ExecutionError('/work/ross/build/tmp/work/aarch64-linux/bpftool-native/1.0/temp/run.do_compi)
> > ERROR: Logfile of failure stored in:
> > /work/ross/build/tmp/work/aarch64-linux/bpftool-native/1.0/temp/log.do_compile.1052635
> > Log data follows:
> > | DEBUG: Executing shell function do_compile
> > | NOTE: make -j 64 V=1 -C
> > /work/ross/build/tmp/work-shared/qemuarm64/kernel-source/tools/bpf/bpftool
> > O=/work/ross/build/tmp/work/aarcn
> > | make: Entering directory
> > '/work/ross/build/tmp/work-shared/qemuarm64/kernel-source/tools/bpf/bpftool'
> > | gcc: error: unrecognized command-line option ‘-fcanon-prefix-map’; did
> > you mean ‘-fmacro-prefix-map=’?
> > |
> > | Auto-detecting system features:
> > | ... clang-bpf-co-re: [ OFF ]
> > | ...llvm: [ OFF ]
> > | ... libcap: [ OFF ]
> > | ... libbfd: [ OFF ]
> > |
> > | make -C
> > /work/ross/build/tmp/work-shared/qemuarm64/kernel-source/tools/lib/bpf
> > OUTPUT=/work/ross/build/tmp/work/aarch64-linux/bpfto\
> > |
> > DESTDIR=/work/ross/build/tmp/work/aarch64-linux/bpftool-native/1.0/bpftool-1.0/libbpf
> > prefix= /work/ross/build/tmp/work/aarchs
> > | make[1]: Entering directory
> > '/work/ross/build/tmp/work-shared/qemuarm64/kernel-source/tools/lib/bpf'
> > | gcc: error: unrecognized command-line option ‘-fcanon-prefix-map’; did
> > you mean ‘-fmacro-prefix-map=’?
> > | gcc: error: unrecognized command-line option ‘-fcanon-prefix-map’; did
> > you mean ‘-fmacro-prefix-map=’?
> >
> > Looks like a problem with the host compiler being passed target flags.
> > Have you seen this?
>
> Yes, bpftool is passing DEBUG_PREFIX_MAP to the host compiler when building
> natively, which won’t work.
>
> I worked around that and had a look. A less ugly and probably upstreamable
> solution might be to add —sysroot to propagate_cflags in meson.build, and add
> the sysroot to CFLAGS directly in the recipe. For historical reasons we pass
> the sysroot via CC, but specifying it twice won’t hurt.
>
> Ross
>
just tried out your hint with the '--sysroot=`, works wonderfully
-> i'll rework the patchset into v2 to use that instead
thanks!
gruß
Johannes
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#200944):
https://lists.openembedded.org/g/openembedded-core/message/200944
Mute This Topic: https://lists.openembedded.org/mt/106572377/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v11 0/3] pkg-database and systemd-sysext image
Hoi, and thanks for taking the time to look into this patch-set! > > Hello, > > On 13/06/2024 07:44:59+0100, Richard Purdie wrote: > > On Tue, 2024-06-04 at 08:50 +0200, Johannes Schneider wrote: > > > systemd-sysext allows to overlay another image (or multiple) ontop of > > > a "base-image" = the current rootfs, via the use of overlayfs; to add > > > tools and features meant for development purposes. > > > > > > To quote the documentation on systemd-sysext: > > > " ...addition in order to make debugging/development easier). System > > > extension images should not be misunderstood as a generic software > > > packaging framework, ..." > > > > > > To build a lean image, that only holds packages that are not already > > > part of the base-image, a snapshot of the package-database is taken > > > after the installation of the base-rootfs is done, and picked up > > > again > > > when collecting the rootfs of such a extension image. > > > > > > with all this in place an example usage could look like this: > > > some-core-image.bb > > > inherit core-image > > > IMAGE_GEN_PKGDBFS = "1" > > > > > > extending-image.bb > > > inherit image-sysext > > > IMAGE_FSTYPES = "squashfs" > > > IMAGE_BASE_PKGDB = "some-core-image" > > > # the above pointing at a package-db similar to: > > > # build/deploy/images/$MACHINE/some-core-image-$MACHINE- > > > 20240210172305-pkgdb.rootfs.tar.gz > > > > > > then on the device, running some-core-image, with the extension image > > > placed at FN: > > > $> ln -s "$FN" /run/extensions/$(basename $FN).raw > > > $> systemd-sysext list > > > $> SYSTEMD_LOG_LEVEL=debug systemd-sysext merge > > > > > > As long as the VERSION_ID of the extension image matches the os- > > > release > > > in the base image, the above commands return sucessfully; > > > for details on the compativility check see the docs for systemd- > > > sysext. > > > > I'm unsure what to so with this series/change. I'm a bit worried that > > it is copy and pasting the debugfs image code to another form and once > > we have this form, I suspect others will then also want things to be > > added for other image update use cases or similar. That code is already > > hard to read and this is not going to improve that. I can understand > > the use case for the code though and I can certainly see why you'd want > > this code upstream as it would be hard to maintain standalone. Having > > the tests do help but the also illustrate this all feels a bit fragile. > > IIRC you mentioned in a past mail that you're not "too fond" of the debug-fs or it's (current] implementation - is it an inherited/legacy functionality thtat's hard to move elsewhere or what is the background/history there? Granted, the code looks similar because it hooks in at the same point in the image generation, also operates on the package(s) level, and was in part based on how the debug data is handled. The "feels a bit fragile" is most likely because i have/had to iterate a lot over these patches because i've got not a full/good grasph of the complex test&build infrastructure (and can't easily replicate/reproduce it locally :-S) At the last iteration almost all build variants where 'Ok', with one exception around an "already present" ipkg in the build environment image. > > > > I've just seen further failures in testing: mangled links > > will probably fail too but is currently still building. > > > > What really worries me about these failures is that there isn't a good > > error message, so if this happens some time in the future we're going > > to be scratching our heads wondering what is wrong. I'm worrying this > > is going to be particularly hard to maintain and keep working in the > > future. > > Im very open for suggestions on how to increase the verbosity = pointers/hints are appreciated :-) I've tried to be verbose/to the point with error messages in the actual code; or are you looking for more verbosity or different messages in the tests (what and how? - since i'm unsure what would be helpful in the build/test infrastructure; it seems to me that not all messages make it through to the logs visible in the web-ui). > > > > In many ways I'm wishing there was an API you could hook into so that > > the core project didn't need to take on the responsibility for this > > complexity. > > Like "image generation step hooks"? im not sure that all necessary information can easily be exposed, passed along, or tested! - that code could end up being more complicated than the current form that does it "just in place directly". If there where an API, where would the code then live - if not in the core? (and who would test it how?) > > Regardless, unfortunately we're still not to the bottom of the failures > > as evidenced above :( > > I'm actually surprised it failed for you as v11 was successful on the AB > for me. Anyway, In the meantime, I dropped the series. Aww, we were getting close; so the last iteration fixed the one remaining/troubling exce
[OE-core] [PATCH v1 2/2] systemd: bpf-framework: pass STAGING_INCDIR to eBPF compiler
The eBPFs are pre-compiled during the systemd-build with a different
compiler than the cross-compiler used to build systemd itself.
This is either a 'clang-native' or a gcc (bpf-unknown-none) which do
not see the BUILD_CFLAGS set -isystem ${STAGING_INCDIR}. For this the
meson.build file constructing the compiler call had to be patched to
handle this bitbake specific setup.
Signed-off-by: Johannes Schneider
---
...bitbake-pass-STAGING_INCDIR-to-clang.patch | 42 +++
meta/recipes-core/systemd/systemd_255.6.bb| 1 +
2 files changed, 43 insertions(+)
create mode 100644
meta/recipes-core/systemd/systemd/0023-bitbake-pass-STAGING_INCDIR-to-clang.patch
diff --git
a/meta/recipes-core/systemd/systemd/0023-bitbake-pass-STAGING_INCDIR-to-clang.patch
b/meta/recipes-core/systemd/systemd/0023-bitbake-pass-STAGING_INCDIR-to-clang.patch
new file mode 100644
index 00..a6d4d7a81d
--- /dev/null
+++
b/meta/recipes-core/systemd/systemd/0023-bitbake-pass-STAGING_INCDIR-to-clang.patch
@@ -0,0 +1,42 @@
+From 4f4b3d662e5475b375808cea3a6a90d61b6c9522 Mon Sep 17 00:00:00 2001
+From: Johannes Schneider
+Date: Sat, 8 Jun 2024 18:19:39 +0200
+Subject: [PATCH] bitbake: pass STAGING_INCDIR to clang
+
+Pass bitbakes STAGING_INCDIR as header searchpath to clang/gcc, where
+it can&should look for the libbpf headers when it compiles the BPFs.
+
+This is "the least ugly hack", compared to other alternatives like for example
+what '-Defi-includedir=...' did (creating an extra option to smuggle in bitbake
+variables).
+
+Ideally the compiler would use the BUILD_CFLAGS set by bitbake, which
+set among other things also the corretc -isystem...; but haven't found
+a good way to do so in meson (yet) :-P
+
+Upstream-Status: Inappropriate [oe specific]
+Signed-off-by: Johannes Schneider
+---
+ meson.build | 6 +-
+ 1 file changed, 5 insertions(+), 1 deletion(-)
+
+diff --git a/meson.build b/meson.build
+index 187e7b216d..dc1a464504 100644
+--- a/meson.build
b/meson.build
+@@ -1746,7 +1746,11 @@ if conf.get('BPF_FRAMEWORK') == 1
+
+ bpf_o_unstripped_cmd += ['-I.']
+
+-if not meson.is_cross_build()
++if meson.is_cross_build()
++## point 'clang'/'gcc' to the bitbake recipe sysroot, to find
the libbpf headers there.
++staging_incdir = join_paths(meson.source_root(),
'../recipe-sysroot/usr/include')
++bpf_o_unstripped_cmd += [ '-isystem', staging_incdir ]
++else
+ target_triplet_cmd = run_command('gcc', '-dumpmachine',
check: false)
+ if target_triplet_cmd.returncode() == 0
+ target_triplet = target_triplet_cmd.stdout().strip()
+--
+2.34.1
+
diff --git a/meta/recipes-core/systemd/systemd_255.6.bb
b/meta/recipes-core/systemd/systemd_255.6.bb
index e70c9c738d..3fd4d5bc5b 100644
--- a/meta/recipes-core/systemd/systemd_255.6.bb
+++ b/meta/recipes-core/systemd/systemd_255.6.bb
@@ -28,6 +28,7 @@ SRC_URI += " \
file://systemd-pager.sh \
file://0002-binfmt-Don-t-install-dependency-links-at-install-tim.patch \
file://0008-implment-systemd-sysv-install-for-OE.patch \
+ file://0023-bitbake-pass-STAGING_INCDIR-to-clang.patch \
"
# patches needed by musl
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#200456):
https://lists.openembedded.org/g/openembedded-core/message/200456
Mute This Topic: https://lists.openembedded.org/mt/106572377/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v1 1/2] systemd: add PACKAGECONFIG for bpf-framework
The bpf-framework is used to pre-compile eBPFs that required for the systemd.resource-control features RestrictFileSystems=[1] and RestrictNetworkInterfaces=[2] to work. Apart from 'clang-native' to compile the eBPFs, the required kernel switches are described in [3]. Link: https://www.freedesktop.org/software/systemd/man/latest/systemd.resource-control.html#RestrictFileSystems= Link: https://www.freedesktop.org/software/systemd/man/latest/systemd.resource-control.html#RestrictNetworkInterfaces= Link: https://kinvolk.io/blog/2021/04/extending-systemd-security-features-with-ebpf/ Signed-off-by: Johannes Schneider --- meta/recipes-core/systemd/systemd_255.6.bb | 1 + 1 file changed, 1 insertion(+) diff --git a/meta/recipes-core/systemd/systemd_255.6.bb b/meta/recipes-core/systemd/systemd_255.6.bb index 62842d43c8..e70c9c738d 100644 --- a/meta/recipes-core/systemd/systemd_255.6.bb +++ b/meta/recipes-core/systemd/systemd_255.6.bb @@ -132,6 +132,7 @@ PACKAGECONFIG[acl] = "-Dacl=true,-Dacl=false,acl" PACKAGECONFIG[audit] = "-Daudit=true,-Daudit=false,audit" PACKAGECONFIG[backlight] = "-Dbacklight=true,-Dbacklight=false" PACKAGECONFIG[binfmt] = "-Dbinfmt=true,-Dbinfmt=false" +PACKAGECONFIG[bpf-framework] = "-Dbpf-framework=true,-Dbpf-framework=false,clang-native bpftool-native libbpf,libbpf" PACKAGECONFIG[bzip2] = "-Dbzip2=true,-Dbzip2=false,bzip2" PACKAGECONFIG[cgroupv2] = "-Ddefault-hierarchy=unified,-Ddefault-hierarchy=hybrid" PACKAGECONFIG[coredump] = "-Dcoredump=true,-Dcoredump=false" -- 2.34.1 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#200453): https://lists.openembedded.org/g/openembedded-core/message/200453 Mute This Topic: https://lists.openembedded.org/mt/106572376/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v1 2/2] systemd: bpf-framework: pass STAGING_INCDIR to eBPF compiler
The eBPFs are pre-compiled during the systemd-build with a different
compiler than the cross-compiler used to build systemd itself.
This is either a 'clang-native' or a gcc (bpf-unknown-none) which do
not see the BUILD_CFLAGS set -isystem ${STAGING_INCDIR}. For this the
meson.build file constructing the compiler call had to be patched to
handle this bitbake specific setup.
Signed-off-by: Johannes Schneider
---
...bitbake-pass-STAGING_INCDIR-to-clang.patch | 41 +++
meta/recipes-core/systemd/systemd_255.6.bb| 1 +
2 files changed, 42 insertions(+)
create mode 100644
meta/recipes-core/systemd/systemd/0023-bitbake-pass-STAGING_INCDIR-to-clang.patch
diff --git
a/meta/recipes-core/systemd/systemd/0023-bitbake-pass-STAGING_INCDIR-to-clang.patch
b/meta/recipes-core/systemd/systemd/0023-bitbake-pass-STAGING_INCDIR-to-clang.patch
new file mode 100644
index 00..b907fd79b7
--- /dev/null
+++
b/meta/recipes-core/systemd/systemd/0023-bitbake-pass-STAGING_INCDIR-to-clang.patch
@@ -0,0 +1,41 @@
+From 4f4b3d662e5475b375808cea3a6a90d61b6c9522 Mon Sep 17 00:00:00 2001
+From: Johannes Schneider
+Date: Sat, 8 Jun 2024 18:19:39 +0200
+Subject: [PATCH] bitbake: pass STAGING_INCDIR to clang
+
+Pass bitbakes STAGING_INCDIR as header searchpath to clang/gcc, where
+it can&should look for the libbpf headers when it compiles the BPFs.
+
+This is "the least ugly hack", compared to other alternatives like for example
+what '-Defi-includedir=...' did (creating an extra option to smuggle in bitbake
+variables).
+
+Ideally the compiler would use the BUILD_CFLAGS set by bitbake, which
+set among other things also the corretc -isystem...; but haven't found
+a good way to do so in meson (yet) :-P
+
+Signed-off-by: Johannes Schneider
+---
+ meson.build | 6 +-
+ 1 file changed, 5 insertions(+), 1 deletion(-)
+
+diff --git a/meson.build b/meson.build
+index 187e7b216d..dc1a464504 100644
+--- a/meson.build
b/meson.build
+@@ -1746,7 +1746,11 @@ if conf.get('BPF_FRAMEWORK') == 1
+
+ bpf_o_unstripped_cmd += ['-I.']
+
+-if not meson.is_cross_build()
++if meson.is_cross_build()
++## point 'clang'/'gcc' to the bitbake recipe sysroot, to find
the libbpf headers there.
++staging_incdir = join_paths(meson.source_root(),
'../recipe-sysroot/usr/include')
++bpf_o_unstripped_cmd += [ '-isystem', staging_incdir ]
++else
+ target_triplet_cmd = run_command('gcc', '-dumpmachine',
check: false)
+ if target_triplet_cmd.returncode() == 0
+ target_triplet = target_triplet_cmd.stdout().strip()
+--
+2.34.1
+
diff --git a/meta/recipes-core/systemd/systemd_255.6.bb
b/meta/recipes-core/systemd/systemd_255.6.bb
index e70c9c738d..3fd4d5bc5b 100644
--- a/meta/recipes-core/systemd/systemd_255.6.bb
+++ b/meta/recipes-core/systemd/systemd_255.6.bb
@@ -28,6 +28,7 @@ SRC_URI += " \
file://systemd-pager.sh \
file://0002-binfmt-Don-t-install-dependency-links-at-install-tim.patch \
file://0008-implment-systemd-sysv-install-for-OE.patch \
+ file://0023-bitbake-pass-STAGING_INCDIR-to-clang.patch \
"
# patches needed by musl
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#200454):
https://lists.openembedded.org/g/openembedded-core/message/200454
Mute This Topic: https://lists.openembedded.org/mt/106572377/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v1 0/2] systemd and RestrictNetworkInterfaces
Systemd has eBPF based resource-control features to limit file-system and network-interface access [1][2] For these to be usable the corresponding eBPFs that come with systemd need to be compiled an deployed to the system - this could now be done by setting the PACKAGECONFIG+="bpf-framework" in the target build-setup. Then clang-native (from meta-clang [3]) could be used to provide the bpf-compiler, that then systemds build-setup uses to pre-compile these eBPFs; the other build requirements are bpftool-native and libbpf (both form meta-openembedded/meta-oe). On the system the only run-time dependency is then libbpf to load these pre-compiled filters. And "some kernel switches" [4]: CONFIG_BPF CONFIG_BPF_SYSCALL CONFIG_CGROUP_BPF To use/test these run for example: $> systemd-run -t -p RestrictNetworkInterfaces=enp0s3 ping 8.8.8.8 which would result in 100% packet-loss, if the default route goes over another interface. Link: https://www.freedesktop.org/software/systemd/man/latest/systemd.resource-control.html#RestrictFileSystems= Link: https://www.freedesktop.org/software/systemd/man/latest/systemd.resource-control.html#RestrictNetworkInterfaces= Link: https://github.com/kraj/meta-clang Link: https://kinvolk.io/blog/2021/04/extending-systemd-security-features-with-ebpf/ = -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#200452): https://lists.openembedded.org/g/openembedded-core/message/200452 Mute This Topic: https://lists.openembedded.org/mt/106572375/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v11 3/3] classes: add a systemd-sysext image class
systemd-sysext can load a raw-image containing usr/ and opt/ folders
to mount them as RO overlay over the rootfs, to "extend" the systems.
This class provides the necessary changes/additions to the enclosed
file-system so that systemd-sysext accepts the extension for "merge"
into the rootfs.
With such an created image, placed into the correct folder (see [1]),
`systemd-sysext list` should be able to list the "extension" and
`systemd-sysext merge` should enable the overlay. On both commands a
preceding "SYSTEMD_LOG_LEVEL=debug" can aide in figuring out what is
amiss.
The strict name checking systemd-sysext does against the name of
extension-release.NAME file, is disabled, as there is only one such in
the resulting image. This is done to allow a user to freely rename the
resulting image file.
Note that for e.g. squashfs, the kernel needs CONFIG_SQUASHFS_XATTR=y
Link:
https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html
Link:
https://0pointer.net/blog/testing-my-system-code-in-usr-without-modifying-usr.html
Signed-off-by: Johannes Schneider
---
meta/classes-recipe/image-sysext.bbclass | 43
1 file changed, 43 insertions(+)
create mode 100644 meta/classes-recipe/image-sysext.bbclass
diff --git a/meta/classes-recipe/image-sysext.bbclass
b/meta/classes-recipe/image-sysext.bbclass
new file mode 100644
index 00..bc3e4d52b5
--- /dev/null
+++ b/meta/classes-recipe/image-sysext.bbclass
@@ -0,0 +1,43 @@
+# SPDX-License-Identifier: MIT
+#
+# Copyright Leica Geosystems AG
+#
+
+# systemd-sysext [1] has a simple mechanism for version compatibility:
+# the extension to be loaded has to contain a
+# /usr/lib/extension-release.d/extension-release.NAME
+# with "NAME" *exactly* matching the filename of the extensions
+# raw-device filename/
+#
+# from the extension-release file the "ID" and "VERSION_ID" fields are
+# matched against the etc/os-release and the extension is only "merged"
+# if no mismatches between NAME, ID, and VERSION_ID.
+#
+# Link:
https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html
+
+inherit image
+
+IMAGE_NAME_SUFFIX = ".sysext"
+EXTENSION_NAME = "${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${IMAGE_FSTYPES}"
+IMAGE_LINK_NAME:append = ".sysext"
+
+DEPENDS += " os-release"
+
+sysext_image_mangle_rootfs() {
+R=${IMAGE_ROOTFS}
+
+# pull a copy of the rootfs version information, which systemd-sysext
matches against
+cp -av ${RECIPE_SYSROOT}/${nonarch_libdir}/os-release
${WORKDIR}/extension-release.base
+
+echo 'EXTENSION_RELOAD_MANAGER=1' >> ${WORKDIR}/extension-release.base
+
+install -d $R${nonarch_libdir}/extension-release.d
+install -m 0644 ${WORKDIR}/extension-release.base \
+
$R${nonarch_libdir}/extension-release.d/extension-release.${EXTENSION_NAME}
+
+# disable systemd-sysext's strict name checking, so that the image file
can be renamed, while still being 'merge'-able
+setfattr -n user.extension-release.strict -v false \
+
$R${nonarch_libdir}/extension-release.d/extension-release.${EXTENSION_NAME}
+}
+
+ROOTFS_POSTPROCESS_COMMAND += " sysext_image_mangle_rootfs; "
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#200293):
https://lists.openembedded.org/g/openembedded-core/message/200293
Mute This Topic: https://lists.openembedded.org/mt/106478183/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v11 2/3] image.bbclass/rootfs: set and unpack package-database
set the package-database of a "lower image" to unpack and build upon
when installing packages for the current image. This way a lean image
will be created, which only holds the packages that are not already
present in the lower image.
An image build such could then be used with overlayfs or systemd-
sysext to extend the "lower image" on demand; for development purposes
on a device running the "lower image" in RO mode for example.
A configuration could look as follows:
some-core-image.bb
inherit image
IMAGE_GEN_PKGDBFS = "1"
extending-image.bb
inherit image
IMAGE_BASE_PKGDB = "some-core-image"
Signed-off-by: Johannes Schneider
---
meta/classes-recipe/image.bbclass | 23 ++--
meta/conf/documentation.conf | 3 +-
meta/lib/oe/package_manager/deb/rootfs.py | 2 +
meta/lib/oe/package_manager/ipk/rootfs.py | 6 ++-
meta/lib/oe/package_manager/rpm/rootfs.py | 7 ++-
meta/lib/oe/rootfs.py | 18 +++
meta/lib/oeqa/selftest/cases/imagefeatures.py | 52 ++-
7 files changed, 102 insertions(+), 9 deletions(-)
diff --git a/meta/classes-recipe/image.bbclass
b/meta/classes-recipe/image.bbclass
index 3ccaaa17b8..c573c37cd8 100644
--- a/meta/classes-recipe/image.bbclass
+++ b/meta/classes-recipe/image.bbclass
@@ -42,8 +42,16 @@ IMAGE_FEATURES ?= ""
IMAGE_FEATURES[type] = "list"
IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs
read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password
allow-empty-password allow-root-login serial-autologin-root
post-install-logging overlayfs-etc"
-# Generate snapshot of the package database?
+# Image layering:
+# a "base image" would create a snapshot of the package-database after the
+# installation of all packages into the rootfs is done. The next/other image
+# "layered on-top" of the former would then import that database and install
+# further packages; without reinstalling packages/dependencies that are already
+# installed in the layer below.
+# Set to '1' in a "base image" recipe, to preserve a snapshot of the package
database.
IMAGE_GEN_PKGDBFS ?= "0"
+# "PN" of a "base image", upon which the current image is to be built upon.
+IMAGE_BASE_PKGDB ?= ""
# Generate companion debugfs?
IMAGE_GEN_DEBUGFS ?= "0"
@@ -118,6 +126,15 @@ do_rootfs[depends] += " \
"
do_rootfs[recrdeptask] += "do_packagedata"
+python () {
+# make sure that the 'base image' has been queued in before this
+# image wants to unpack and build upon the formers pgkdb
+base_image = d.getVar('IMAGE_BASE_PKGDB')
+pn = d.getVar('PN')
+if base_image and base_image != pn:
+d.appendVarFlag("do_rootfs", 'depends', ' '+ base_image +
':do_image_complete')
+}
+
def rootfs_command_variables(d):
return
['ROOTFS_POSTPROCESS_COMMAND','ROOTFS_PREPROCESS_COMMAND','ROOTFS_POSTINSTALL_COMMAND','ROOTFS_POSTUNINSTALL_COMMAND','OPKG_PREPROCESS_COMMANDS','OPKG_POSTPROCESS_COMMANDS','IMAGE_POSTPROCESS_COMMAND',
'IMAGE_PREPROCESS_COMMAND','RPM_PREPROCESS_COMMANDS','RPM_POSTPROCESS_COMMANDS','DEB_PREPROCESS_COMMANDS','DEB_POSTPROCESS_COMMANDS']
@@ -134,8 +151,8 @@ def rootfs_variables(d):
'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
'IMAGE_LINGUAS_COMPLEMENTARY', 'IMAGE_LOCALES_ARCHIVE',
'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
- 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS', 'IMAGE_GEN_DEBUGFS',
'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY',
'REPRODUCIBLE_TIMESTAMP_ROOTFS',
- 'IMAGE_INSTALL_DEBUGFS']
+ 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS', 'IMAGE_BASE_PKGDB',
'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR',
'PACKAGE_EXCLUDE_COMPLEMENTARY',
+ 'REPRODUCIBLE_TIMESTAMP_ROOTFS', 'IMAGE_INSTALL_DEBUGFS']
variables.extend(rootfs_command_variables(d))
variables.extend(variable_depends(d))
return " ".join(variables)
diff --git a/meta/conf/documentation.conf b/meta/conf/documentation.conf
index 36aebb59ab..9f493cfe96 100644
--- a/meta/conf/documentation.conf
+++ b/meta/conf/documentation.conf
@@ -208,6 +208,7 @@ ICECC_PATH[doc] = "The location of the icecc binary."
ICECC_CLASS_DISABLE[doc] = "Identifies user classes that you do not want the
Icecream distributed compile support to consider."
ICECC_RECIPE_DISABLE[doc] = "Identifies user recipes that you do not want the
Icecream distributed compile support to consider."
ICECC_RECIPE_ENABLE[doc] = "Identifies user recipes that use an
[OE-core] [PATCH v11 1/3] image.bbclass/rootfs: archive and deploy package database
archive the package database after the rootfs has been put together as
*rootfs-pkdbfs.tar.gz, and put it into the deploy folder.
This creates a snapshot of the package mangers state at the point in
time when all dependencies have been resolved and installed; which
could be used by "extension images" to built upon.
Signed-off-by: Johannes Schneider
---
meta/classes-recipe/image.bbclass | 44 +-
meta/classes-recipe/image_types.bbclass | 1 +
meta/conf/documentation.conf | 1 +
meta/lib/oe/package_manager/deb/rootfs.py | 1 +
meta/lib/oe/package_manager/ipk/rootfs.py | 1 +
meta/lib/oe/package_manager/rpm/rootfs.py | 1 +
meta/lib/oe/rootfs.py | 20 +++
meta/lib/oeqa/selftest/cases/imagefeatures.py | 146 ++
8 files changed, 214 insertions(+), 1 deletion(-)
diff --git a/meta/classes-recipe/image.bbclass
b/meta/classes-recipe/image.bbclass
index 28be6c6362..3ccaaa17b8 100644
--- a/meta/classes-recipe/image.bbclass
+++ b/meta/classes-recipe/image.bbclass
@@ -42,6 +42,9 @@ IMAGE_FEATURES ?= ""
IMAGE_FEATURES[type] = "list"
IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs
read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password
allow-empty-password allow-root-login serial-autologin-root
post-install-logging overlayfs-etc"
+# Generate snapshot of the package database?
+IMAGE_GEN_PKGDBFS ?= "0"
+
# Generate companion debugfs?
IMAGE_GEN_DEBUGFS ?= "0"
@@ -131,7 +134,8 @@ def rootfs_variables(d):
'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
'IMAGE_LINGUAS_COMPLEMENTARY', 'IMAGE_LOCALES_ARCHIVE',
'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
- 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED',
'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY',
'REPRODUCIBLE_TIMESTAMP_ROOTFS', 'IMAGE_INSTALL_DEBUGFS']
+ 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS', 'IMAGE_GEN_DEBUGFS',
'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY',
'REPRODUCIBLE_TIMESTAMP_ROOTFS',
+ 'IMAGE_INSTALL_DEBUGFS']
variables.extend(rootfs_command_variables(d))
variables.extend(variable_depends(d))
return " ".join(variables)
@@ -337,6 +341,17 @@ python do_image_qa_setscene () {
}
addtask do_image_qa_setscene
+def setup_pkgdbfs_variables(d):
+d.appendVar('IMAGE_ROOTFS', '-pkgdb')
+if d.getVar('IMAGE_LINK_NAME'):
+d.appendVar('IMAGE_LINK_NAME', '-pkgdb')
+d.appendVar('IMAGE_NAME','-pkgdb')
+d.setVar('IMAGE_FSTYPES', 'tar.gz')
+
+python setup_pkgdbfs () {
+setup_pkgdbfs_variables(d)
+}
+
def setup_debugfs_variables(d):
d.appendVar('IMAGE_ROOTFS', '-dbg')
if d.getVar('IMAGE_LINK_NAME'):
@@ -381,6 +396,11 @@ python () {
alltypes = d.getVar('IMAGE_FSTYPES').split()
typedeps = {}
+if d.getVar('IMAGE_GEN_PKGDBFS') == "1":
+pkgdbfs_fstypes = ['tar.gz']
+for t in pkgdbfs_fstypes:
+alltypes.append("pkgdbfs_" + t)
+
if d.getVar('IMAGE_GEN_DEBUGFS') == "1":
debugfs_fstypes = d.getVar('IMAGE_FSTYPES_DEBUGFS').split()
for t in debugfs_fstypes:
@@ -393,6 +413,10 @@ python () {
basetypes[baset]= []
if t not in basetypes[baset]:
basetypes[baset].append(t)
+pkgdb = ""
+if t.startswith("pkgdbfs_"):
+t = t[8:]
+pkgdb = "pkgdbfs_"
debug = ""
if t.startswith("debugfs_"):
t = t[8:]
@@ -401,6 +425,13 @@ python () {
vardeps.add('IMAGE_TYPEDEP:' + t)
if baset not in typedeps:
typedeps[baset] = set()
+deps = [pkgdb + dep for dep in deps]
+for dep in deps:
+if dep not in alltypes:
+alltypes.append(dep)
+_add_type(dep)
+basedep = _image_base_type(dep)
+typedeps[baset].add(basedep)
deps = [debug + dep for dep in deps]
for dep in deps:
if dep not in alltypes:
@@ -419,6 +450,7 @@ python () {
maskedtypes = (d.getVar('IMAGE_TYPES_MASKED') or "").split()
maskedtypes = [dbg + t for t in maskedtypes for dbg in ("", "debugfs_")]
+maskedtypes = [pkgdb + t for t in maskedtypes for pkgdb in ("",
"pkgdbfs_")]
for t in basetypes:
vardeps = set()
@@ -430,6 +462,11 @@ python () {
continue
localdata = bb.data.createCopy(d)
+pkgdb = ""
+if t.startswith("pkgdbfs_"):
+setup_pkgdbfs_variabl
[OE-core] [PATCH v11 0/3] pkg-database and systemd-sysext image
systemd-sysext allows to overlay another image (or multiple) ontop of a "base-image" = the current rootfs, via the use of overlayfs; to add tools and features meant for development purposes. To quote the documentation on systemd-sysext: " ...addition in order to make debugging/development easier). System extension images should not be misunderstood as a generic software packaging framework, ..." To build a lean image, that only holds packages that are not already part of the base-image, a snapshot of the package-database is taken after the installation of the base-rootfs is done, and picked up again when collecting the rootfs of such a extension image. with all this in place an example usage could look like this: some-core-image.bb inherit core-image IMAGE_GEN_PKGDBFS = "1" extending-image.bb inherit image-sysext IMAGE_FSTYPES = "squashfs" IMAGE_BASE_PKGDB = "some-core-image" # the above pointing at a package-db similar to: # build/deploy/images/$MACHINE/some-core-image-$MACHINE-20240210172305-pkgdb.rootfs.tar.gz then on the device, running some-core-image, with the extension image placed at FN: $> ln -s "$FN" /run/extensions/$(basename $FN).raw $> systemd-sysext list $> SYSTEMD_LOG_LEVEL=debug systemd-sysext merge As long as the VERSION_ID of the extension image matches the os-release in the base image, the above commands return sucessfully; for details on the compativility check see the docs for systemd-sysext. = changes with v2: rebase from 'kirkstone' onto 'master' changes with v3: incorporate review suggestions for simplification add task dependency handling add oe-selftest for the pkgdb handling add variable documentation and some more comments, and examples in the commit-msg changes with v4: rebase onto 'master' => no functional changes fixed patchtest findings changes with v5: rebase onto 'master' add '.sysext' to the deployed symlink name sidenote on the tests and autobuilder failure: run locally they succeed, e.g.: #> oe-selftest --verbose -r imagefeatures.ImageFeatures.test_image_gen_pkgdbfs changes with v6: tests: restructure to call 'bitbake' only once in the testcase itself (in hopes of solving the autobuilder problem; local test runs succeed) changes with v7: tests: undo the restructuring of done in v6, in favour of explicitly forcing steps: '-c rootfs -f'. since concurrency of the autobuilder, and reusing artifacts/sstate-caching is probably the issue? changes with v8: tests: fixed missing feature=PACKAGE_CLASSES during 'test_image_gen_pkgdbfs' which is now split into three, one for each of the three available packagemanager: ipk, deb, rpm changes with v9: tests: fix singular failure in 'oe-selftest-armhost' which was due to the built image not containing a 'kernel-image...', which the gen-pkgdb sanity check expected, but 'packagegroup-core-boot' instead changes with v10: tests: fix runCmd usage in relation to native-dirs due to which only the use-pkgdbfs-ipk test would fail in a qemu run armhost environment changes with v11: tests: fix runCmd usage: replace image->STAGING_DIR_NATIVE with package-manager->RECIPE_SYSROOT_NATIVE as native_sysroot the tests for test_image_gen_pkgdbfs_deb and *_rpm would succeed, but the one for *_ipk would fail only when doing the armhost=quemuarm64 build, because in the rootfs-image used as build enviroinmane happens to be an opkg already presaent in the PATH - which would be called instead -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#200292): https://lists.openembedded.org/g/openembedded-core/message/200292 Mute This Topic: https://lists.openembedded.org/mt/106478182/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v10 0/3] pkg-database and systemd-sysext image
Hoi, yes - v10 is the current one; and thanks for getting back with... yet another build failure :`-) i'm still struggling to get a setup to reproduce the issue with that particular build host... too many layers of MACHINEs and virtualization but this time i might have a clue/an idea: the error " Command 'opkg list-installed' returned non-zero exit status 255: " 255 -> usually is a 'foreign architecture' error; e.g. the tests happens to call the opkg from within the core-image-minimal-quemuarm64 instead of the correct one = opkg-native this would also explain why the dpkg and rpm tests succeed - there the correct sysroot is used, and neither dpkg nor rpm are part of the image running the build; but the ipk test fails because there happens to be already a "wrong"/other opkg in PATH i'll compare other users of runCmd, and dig some more -> expect a v11 :-D gruß Johannes From: Alexandre Belloni Sent: Wednesday, May 29, 2024 12:34 To: SCHNEIDER Johannes Cc: openembedded-core@lists.openembedded.org; richard.pur...@linuxfoundation.org; alex.kana...@gmail.com Subject: Re: [OE-core] [PATCH v10 0/3] pkg-database and systemd-sysext image This email is not from Hexagon’s Office 365 instance. Please be careful while clicking links, opening attachments, or replying to this email. Hello, I believe I tested the proper series but still had the failure on armhost: https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/3398/steps/15/logs/stdio On 22/05/2024 16:11:49+0200, Johannes Schneider via lists.openembedded.org wrote: > systemd-sysext allows to overlay another image (or multiple) ontop of > a "base-image" = the current rootfs, via the use of overlayfs; to add > tools and features meant for development purposes. > > To quote the documentation on systemd-sysext: > " ...addition in order to make debugging/development easier). System > extension images should not be misunderstood as a generic software > packaging framework, ..." > > To build a lean image, that only holds packages that are not already > part of the base-image, a snapshot of the package-database is taken > after the installation of the base-rootfs is done, and picked up again > when collecting the rootfs of such a extension image. > > with all this in place an example usage could look like this: > some-core-image.bb > inherit core-image > IMAGE_GEN_PKGDBFS = "1" > > extending-image.bb > inherit image-sysext > IMAGE_FSTYPES = "squashfs" > IMAGE_BASE_PKGDB = "some-core-image" > # the above pointing at a package-db similar to: > # > build/deploy/images/$MACHINE/some-core-image-$MACHINE-20240210172305-pkgdb.rootfs.tar.gz > > then on the device, running some-core-image, with the extension image placed > at FN: > $> ln -s "$FN" /run/extensions/$(basename $FN).raw > $> systemd-sysext list > $> SYSTEMD_LOG_LEVEL=debug systemd-sysext merge > > As long as the VERSION_ID of the extension image matches the os-release > in the base image, the above commands return sucessfully; > for details on the compativility check see the docs for systemd-sysext. > > = > > changes with v2: > rebase from 'kirkstone' onto 'master' > > changes with v3: > incorporate review suggestions for simplification > add task dependency handling > add oe-selftest for the pkgdb handling > add variable documentation and > some more comments, and examples in the commit-msg > > changes with v4: > rebase onto 'master' => no functional changes > fixed patchtest findings > > changes with v5: > rebase onto 'master' > add '.sysext' to the deployed symlink name > sidenote on the tests and autobuilder failure: run locally they > succeed, e.g.: > #> oe-selftest --verbose -r > imagefeatures.ImageFeatures.test_image_gen_pkgdbfs > > changes with v6: > tests: restructure to call 'bitbake' only once in the testcase itself > (in hopes of solving the autobuilder problem; local test runs succeed) > > changes with v7: > tests: undo the restructuring of done in v6, in favour of explicitly > forcing steps: > '-c rootfs -f'. since concurrency of the autobuilder, and reusing > artifacts/sstate-caching is probably the issue? > > changes with v8: > tests: fixed missing feature=PACKAGE_CLASSES during > 'test_image_gen_pkgdbfs' > which is now split into three, one for each of the three available > packagemanager: ipk, deb, rpm > > changes with v9: >
[OE-core] [PATCH v10 1/3] image.bbclass/rootfs: archive and deploy package database
archive the package database after the rootfs has been put together as
*rootfs-pkdbfs.tar.gz, and put it into the deploy folder.
This creates a snapshot of the package mangers state at the point in
time when all dependencies have been resolved and installed; which
could be used by "extension images" to built upon.
Signed-off-by: Johannes Schneider
---
meta/classes-recipe/image.bbclass | 44 +-
meta/classes-recipe/image_types.bbclass | 1 +
meta/conf/documentation.conf | 1 +
meta/lib/oe/package_manager/deb/rootfs.py | 1 +
meta/lib/oe/package_manager/ipk/rootfs.py | 1 +
meta/lib/oe/package_manager/rpm/rootfs.py | 1 +
meta/lib/oe/rootfs.py | 20 +++
meta/lib/oeqa/selftest/cases/imagefeatures.py | 149 ++
8 files changed, 217 insertions(+), 1 deletion(-)
diff --git a/meta/classes-recipe/image.bbclass
b/meta/classes-recipe/image.bbclass
index 28be6c6362..3ccaaa17b8 100644
--- a/meta/classes-recipe/image.bbclass
+++ b/meta/classes-recipe/image.bbclass
@@ -42,6 +42,9 @@ IMAGE_FEATURES ?= ""
IMAGE_FEATURES[type] = "list"
IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs
read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password
allow-empty-password allow-root-login serial-autologin-root
post-install-logging overlayfs-etc"
+# Generate snapshot of the package database?
+IMAGE_GEN_PKGDBFS ?= "0"
+
# Generate companion debugfs?
IMAGE_GEN_DEBUGFS ?= "0"
@@ -131,7 +134,8 @@ def rootfs_variables(d):
'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
'IMAGE_LINGUAS_COMPLEMENTARY', 'IMAGE_LOCALES_ARCHIVE',
'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
- 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED',
'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY',
'REPRODUCIBLE_TIMESTAMP_ROOTFS', 'IMAGE_INSTALL_DEBUGFS']
+ 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS', 'IMAGE_GEN_DEBUGFS',
'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY',
'REPRODUCIBLE_TIMESTAMP_ROOTFS',
+ 'IMAGE_INSTALL_DEBUGFS']
variables.extend(rootfs_command_variables(d))
variables.extend(variable_depends(d))
return " ".join(variables)
@@ -337,6 +341,17 @@ python do_image_qa_setscene () {
}
addtask do_image_qa_setscene
+def setup_pkgdbfs_variables(d):
+d.appendVar('IMAGE_ROOTFS', '-pkgdb')
+if d.getVar('IMAGE_LINK_NAME'):
+d.appendVar('IMAGE_LINK_NAME', '-pkgdb')
+d.appendVar('IMAGE_NAME','-pkgdb')
+d.setVar('IMAGE_FSTYPES', 'tar.gz')
+
+python setup_pkgdbfs () {
+setup_pkgdbfs_variables(d)
+}
+
def setup_debugfs_variables(d):
d.appendVar('IMAGE_ROOTFS', '-dbg')
if d.getVar('IMAGE_LINK_NAME'):
@@ -381,6 +396,11 @@ python () {
alltypes = d.getVar('IMAGE_FSTYPES').split()
typedeps = {}
+if d.getVar('IMAGE_GEN_PKGDBFS') == "1":
+pkgdbfs_fstypes = ['tar.gz']
+for t in pkgdbfs_fstypes:
+alltypes.append("pkgdbfs_" + t)
+
if d.getVar('IMAGE_GEN_DEBUGFS') == "1":
debugfs_fstypes = d.getVar('IMAGE_FSTYPES_DEBUGFS').split()
for t in debugfs_fstypes:
@@ -393,6 +413,10 @@ python () {
basetypes[baset]= []
if t not in basetypes[baset]:
basetypes[baset].append(t)
+pkgdb = ""
+if t.startswith("pkgdbfs_"):
+t = t[8:]
+pkgdb = "pkgdbfs_"
debug = ""
if t.startswith("debugfs_"):
t = t[8:]
@@ -401,6 +425,13 @@ python () {
vardeps.add('IMAGE_TYPEDEP:' + t)
if baset not in typedeps:
typedeps[baset] = set()
+deps = [pkgdb + dep for dep in deps]
+for dep in deps:
+if dep not in alltypes:
+alltypes.append(dep)
+_add_type(dep)
+basedep = _image_base_type(dep)
+typedeps[baset].add(basedep)
deps = [debug + dep for dep in deps]
for dep in deps:
if dep not in alltypes:
@@ -419,6 +450,7 @@ python () {
maskedtypes = (d.getVar('IMAGE_TYPES_MASKED') or "").split()
maskedtypes = [dbg + t for t in maskedtypes for dbg in ("", "debugfs_")]
+maskedtypes = [pkgdb + t for t in maskedtypes for pkgdb in ("",
"pkgdbfs_")]
for t in basetypes:
vardeps = set()
@@ -430,6 +462,11 @@ python () {
continue
localdata = bb.data.createCopy(d)
+pkgdb = ""
+if t.startswith("pkgdbfs_"):
+setup_pkgdbfs_variabl
[OE-core] [PATCH v10 2/3] image.bbclass/rootfs: set and unpack package-database
set the package-database of a "lower image" to unpack and build upon
when installing packages for the current image. This way a lean image
will be created, which only holds the packages that are not already
present in the lower image.
An image build such could then be used with overlayfs or systemd-
sysext to extend the "lower image" on demand; for development purposes
on a device running the "lower image" in RO mode for example.
A configuration could look as follows:
some-core-image.bb
inherit image
IMAGE_GEN_PKGDBFS = "1"
extending-image.bb
inherit image
IMAGE_BASE_PKGDB = "some-core-image"
Signed-off-by: Johannes Schneider
---
meta/classes-recipe/image.bbclass | 23 ++--
meta/conf/documentation.conf | 3 +-
meta/lib/oe/package_manager/deb/rootfs.py | 2 +
meta/lib/oe/package_manager/ipk/rootfs.py | 6 ++-
meta/lib/oe/package_manager/rpm/rootfs.py | 7 ++-
meta/lib/oe/rootfs.py | 18 +++
meta/lib/oeqa/selftest/cases/imagefeatures.py | 52 ++-
7 files changed, 102 insertions(+), 9 deletions(-)
diff --git a/meta/classes-recipe/image.bbclass
b/meta/classes-recipe/image.bbclass
index 3ccaaa17b8..c573c37cd8 100644
--- a/meta/classes-recipe/image.bbclass
+++ b/meta/classes-recipe/image.bbclass
@@ -42,8 +42,16 @@ IMAGE_FEATURES ?= ""
IMAGE_FEATURES[type] = "list"
IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs
read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password
allow-empty-password allow-root-login serial-autologin-root
post-install-logging overlayfs-etc"
-# Generate snapshot of the package database?
+# Image layering:
+# a "base image" would create a snapshot of the package-database after the
+# installation of all packages into the rootfs is done. The next/other image
+# "layered on-top" of the former would then import that database and install
+# further packages; without reinstalling packages/dependencies that are already
+# installed in the layer below.
+# Set to '1' in a "base image" recipe, to preserve a snapshot of the package
database.
IMAGE_GEN_PKGDBFS ?= "0"
+# "PN" of a "base image", upon which the current image is to be built upon.
+IMAGE_BASE_PKGDB ?= ""
# Generate companion debugfs?
IMAGE_GEN_DEBUGFS ?= "0"
@@ -118,6 +126,15 @@ do_rootfs[depends] += " \
"
do_rootfs[recrdeptask] += "do_packagedata"
+python () {
+# make sure that the 'base image' has been queued in before this
+# image wants to unpack and build upon the formers pgkdb
+base_image = d.getVar('IMAGE_BASE_PKGDB')
+pn = d.getVar('PN')
+if base_image and base_image != pn:
+d.appendVarFlag("do_rootfs", 'depends', ' '+ base_image +
':do_image_complete')
+}
+
def rootfs_command_variables(d):
return
['ROOTFS_POSTPROCESS_COMMAND','ROOTFS_PREPROCESS_COMMAND','ROOTFS_POSTINSTALL_COMMAND','ROOTFS_POSTUNINSTALL_COMMAND','OPKG_PREPROCESS_COMMANDS','OPKG_POSTPROCESS_COMMANDS','IMAGE_POSTPROCESS_COMMAND',
'IMAGE_PREPROCESS_COMMAND','RPM_PREPROCESS_COMMANDS','RPM_POSTPROCESS_COMMANDS','DEB_PREPROCESS_COMMANDS','DEB_POSTPROCESS_COMMANDS']
@@ -134,8 +151,8 @@ def rootfs_variables(d):
'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
'IMAGE_LINGUAS_COMPLEMENTARY', 'IMAGE_LOCALES_ARCHIVE',
'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
- 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS', 'IMAGE_GEN_DEBUGFS',
'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY',
'REPRODUCIBLE_TIMESTAMP_ROOTFS',
- 'IMAGE_INSTALL_DEBUGFS']
+ 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS', 'IMAGE_BASE_PKGDB',
'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR',
'PACKAGE_EXCLUDE_COMPLEMENTARY',
+ 'REPRODUCIBLE_TIMESTAMP_ROOTFS', 'IMAGE_INSTALL_DEBUGFS']
variables.extend(rootfs_command_variables(d))
variables.extend(variable_depends(d))
return " ".join(variables)
diff --git a/meta/conf/documentation.conf b/meta/conf/documentation.conf
index 36aebb59ab..9f493cfe96 100644
--- a/meta/conf/documentation.conf
+++ b/meta/conf/documentation.conf
@@ -208,6 +208,7 @@ ICECC_PATH[doc] = "The location of the icecc binary."
ICECC_CLASS_DISABLE[doc] = "Identifies user classes that you do not want the
Icecream distributed compile support to consider."
ICECC_RECIPE_DISABLE[doc] = "Identifies user recipes that you do not want the
Icecream distributed compile support to consider."
ICECC_RECIPE_ENABLE[doc] = "Identifies user recipes that use an
[OE-core] [PATCH v10 3/3] classes: add a systemd-sysext image class
systemd-sysext can load a raw-image containing usr/ and opt/ folders
to mount them as RO overlay over the rootfs, to "extend" the systems.
This class provides the necessary changes/additions to the enclosed
file-system so that systemd-sysext accepts the extension for "merge"
into the rootfs.
With such an created image, placed into the correct folder (see [1]),
`systemd-sysext list` should be able to list the "extension" and
`systemd-sysext merge` should enable the overlay. On both commands a
preceding "SYSTEMD_LOG_LEVEL=debug" can aide in figuring out what is
amiss.
The strict name checking systemd-sysext does against the name of
extension-release.NAME file, is disabled, as there is only one such in
the resulting image. This is done to allow a user to freely rename the
resulting image file.
Note that for e.g. squashfs, the kernel needs CONFIG_SQUASHFS_XATTR=y
Link:
https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html
Link:
https://0pointer.net/blog/testing-my-system-code-in-usr-without-modifying-usr.html
Signed-off-by: Johannes Schneider
---
meta/classes-recipe/image-sysext.bbclass | 43
1 file changed, 43 insertions(+)
create mode 100644 meta/classes-recipe/image-sysext.bbclass
diff --git a/meta/classes-recipe/image-sysext.bbclass
b/meta/classes-recipe/image-sysext.bbclass
new file mode 100644
index 00..bc3e4d52b5
--- /dev/null
+++ b/meta/classes-recipe/image-sysext.bbclass
@@ -0,0 +1,43 @@
+# SPDX-License-Identifier: MIT
+#
+# Copyright Leica Geosystems AG
+#
+
+# systemd-sysext [1] has a simple mechanism for version compatibility:
+# the extension to be loaded has to contain a
+# /usr/lib/extension-release.d/extension-release.NAME
+# with "NAME" *exactly* matching the filename of the extensions
+# raw-device filename/
+#
+# from the extension-release file the "ID" and "VERSION_ID" fields are
+# matched against the etc/os-release and the extension is only "merged"
+# if no mismatches between NAME, ID, and VERSION_ID.
+#
+# Link:
https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html
+
+inherit image
+
+IMAGE_NAME_SUFFIX = ".sysext"
+EXTENSION_NAME = "${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${IMAGE_FSTYPES}"
+IMAGE_LINK_NAME:append = ".sysext"
+
+DEPENDS += " os-release"
+
+sysext_image_mangle_rootfs() {
+R=${IMAGE_ROOTFS}
+
+# pull a copy of the rootfs version information, which systemd-sysext
matches against
+cp -av ${RECIPE_SYSROOT}/${nonarch_libdir}/os-release
${WORKDIR}/extension-release.base
+
+echo 'EXTENSION_RELOAD_MANAGER=1' >> ${WORKDIR}/extension-release.base
+
+install -d $R${nonarch_libdir}/extension-release.d
+install -m 0644 ${WORKDIR}/extension-release.base \
+
$R${nonarch_libdir}/extension-release.d/extension-release.${EXTENSION_NAME}
+
+# disable systemd-sysext's strict name checking, so that the image file
can be renamed, while still being 'merge'-able
+setfattr -n user.extension-release.strict -v false \
+
$R${nonarch_libdir}/extension-release.d/extension-release.${EXTENSION_NAME}
+}
+
+ROOTFS_POSTPROCESS_COMMAND += " sysext_image_mangle_rootfs; "
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#199759):
https://lists.openembedded.org/g/openembedded-core/message/199759
Mute This Topic: https://lists.openembedded.org/mt/106243263/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v10 0/3] pkg-database and systemd-sysext image
systemd-sysext allows to overlay another image (or multiple) ontop of a "base-image" = the current rootfs, via the use of overlayfs; to add tools and features meant for development purposes. To quote the documentation on systemd-sysext: " ...addition in order to make debugging/development easier). System extension images should not be misunderstood as a generic software packaging framework, ..." To build a lean image, that only holds packages that are not already part of the base-image, a snapshot of the package-database is taken after the installation of the base-rootfs is done, and picked up again when collecting the rootfs of such a extension image. with all this in place an example usage could look like this: some-core-image.bb inherit core-image IMAGE_GEN_PKGDBFS = "1" extending-image.bb inherit image-sysext IMAGE_FSTYPES = "squashfs" IMAGE_BASE_PKGDB = "some-core-image" # the above pointing at a package-db similar to: # build/deploy/images/$MACHINE/some-core-image-$MACHINE-20240210172305-pkgdb.rootfs.tar.gz then on the device, running some-core-image, with the extension image placed at FN: $> ln -s "$FN" /run/extensions/$(basename $FN).raw $> systemd-sysext list $> SYSTEMD_LOG_LEVEL=debug systemd-sysext merge As long as the VERSION_ID of the extension image matches the os-release in the base image, the above commands return sucessfully; for details on the compativility check see the docs for systemd-sysext. = changes with v2: rebase from 'kirkstone' onto 'master' changes with v3: incorporate review suggestions for simplification add task dependency handling add oe-selftest for the pkgdb handling add variable documentation and some more comments, and examples in the commit-msg changes with v4: rebase onto 'master' => no functional changes fixed patchtest findings changes with v5: rebase onto 'master' add '.sysext' to the deployed symlink name sidenote on the tests and autobuilder failure: run locally they succeed, e.g.: #> oe-selftest --verbose -r imagefeatures.ImageFeatures.test_image_gen_pkgdbfs changes with v6: tests: restructure to call 'bitbake' only once in the testcase itself (in hopes of solving the autobuilder problem; local test runs succeed) changes with v7: tests: undo the restructuring of done in v6, in favour of explicitly forcing steps: '-c rootfs -f'. since concurrency of the autobuilder, and reusing artifacts/sstate-caching is probably the issue? changes with v8: tests: fixed missing feature=PACKAGE_CLASSES during 'test_image_gen_pkgdbfs' which is now split into three, one for each of the three available packagemanager: ipk, deb, rpm changes with v9: tests: fix singular failure in 'oe-selftest-armhost' which was due to the built image not containing a 'kernel-image...', which the gen-pkgdb sanity check expected, but 'packagegroup-core-boot' instead changes with v10: tests: fix runCmd usage in relation to native-dirs due to which only the use-pkgdbfs-ipk test would fail in a qemu run armhost environment -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#199756): https://lists.openembedded.org/g/openembedded-core/message/199756 Mute This Topic: https://lists.openembedded.org/mt/106243260/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v9 0/3] pkg-database and systemd-sysext image
Hoi,
> If you scroll further up, there's:
ah... stupid me relied on the search function in the webui (which caches only
part of the output...) instead of looking at/searching through the raw log;
sorry!
> Also note that this happened only on the arm host, and not on x86
hosts. Which does make it tricky to reproduce, but at least gives a
lead:
curiously if i use an "actual arm host" = a debian arm64 schroot (see below),
building and running the tests in there also succeeds
-> so my current lead is that there is sth "wrong" with the qemu disk-image :-S
i'll have to pick apart how the yocto-autobuilder builds itself qemu-arm, a
disk image and then runs the tests inside of that... convoluted :-D
since this most likely ends up in a special case/exception inside the test;
would excluding that machine+test be an option?
gruß
Johannes
debian-arm schroot:
#> apt install qemu-user-static
#> mkdir -p debian-arm64/usr/bin/; cp /usr/bin/qemu-arm-static
debian-arm64/usr/bin/
#> debootstrap --arch=arm64 bookworm debian-arm64 http://deb.debian.org/debian/
#chroot> apt install -y gawk wget git diffstat unzip texinfo gcc
build-essential chrpath socat cpio python3 python3-pip python3-pexpect xz-utils
debianutils iputils-ping python3-git python3-jinja2 python3-subunit zstd
liblz4-tool file locales libacl1
#chroot /tmp/openembedded-core/build> SDKMACHINE=aarch64 MACHINE=qemuarm64
PACKAGE_CLASSES=package_rpm oe-selftest --verbose -r
imagefeatures.ImageFeatures.test_image_gen_pkgdbfs_ipk
2024-05-22 05:15:52,357 - oe-selftest - DEBUG - running cmd:
/tmp/openembedded-core/build-st/tmp-glibc/sysroots/aarch64/usr/bin/opkg
--volatile-cache
--offline-root=/tmp/openembedded-core/build-st/tmp-glibc/deploy/images/qemuarm64
--add-arch qemuarm64:11 --add-arch qemuarm64:12 list-installed
2024-05-22 05:16:06,765 - oe-selftest - INFO - oe-selftest - OK - All required
tests passed (successes=1, skipped=0, failures=0, errors=0)
From: Alexander Kanavin
Sent: Tuesday, May 21, 2024 12:08
To: SCHNEIDER Johannes
Cc: Alexandre Belloni; openembedded-core@lists.openembedded.org;
richard.pur...@linuxfoundation.org
Subject: Re: [OE-core] [PATCH v9 0/3] pkg-database and systemd-sysext image
This email is not from Hexagon’s Office 365 instance. Please be careful while
clicking links, opening attachments, or replying to this email.
If you scroll further up, there's:
2024-05-16 22:05:57,101 - oe-selftest - INFO - FAIL:
imagefeatures.ImageFeatures.test_image_gen_pkgdbfs_ipk
(subunit.RemotedTestCase)
2024-05-16 22:05:57,102 - oe-selftest - INFO -
--
2024-05-16 22:05:57,102 - oe-selftest - INFO -
testtools.testresult.real._StringException: Traceback (most recent
call last):
File
"/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/lib/oeqa/selftest/cases/imagefeatures.py",
line 344, in test_image_gen_pkgdbfs_ipk
result = runCmd(cmd)
File
"/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/lib/oeqa/utils/commands.py",
line 212, in runCmd
raise AssertionError("Command '%s' returned non-zero exit status
%d:\n%s" % (command, result.status, exc_output))
AssertionError: Command
'/home/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-4137794/tmp/sysroots/aarch64/usr/bin/opkg
--volatile-cache
--offline-root=/home/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-4137794/tmp/deploy/images/qemuarm64
--add-arch qemuarm64:11 --add-arch qemuarm64:12 list-installed'
returned non-zero exit status 255:
Also note that this happened only on the arm host, and not on x86
hosts. Which does make it tricky to reproduce, but at least gives a
lead:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6921
Alex
On Sat, 18 May 2024 at 08:30, SCHNEIDER Johannes
wrote:
>
> Hoi Alexandre,
>
>
> is there a way to get more output out of the autobuilder?
> the logs only tell "that" the test failed, not "why" = which assertion was
> missed
>
> if i run the test locally with the differing variables set like this:
> #> SDKMACHINE=aarch64 MACHINE=qemuarm64 oe-selftest --verbose -r
> imagefeatures.ImageFeatures.test_image_gen_pkgdbfs_ipk
>
> it succeeds :-S
> ... ideas/thoughts?
>
>
> gruß
> Johannes
>
>
> From: Alexandre Belloni
> Sent: Friday, May 17, 2024 20:33
> To: SCHNEIDER Johannes
> Cc: openembedded-core@lists.openembedded.org;
> richard.pur...@linuxfoundation.org; alex.kana...@gmail.com
> Subject: Re: [OE-core] [PATCH v9 0/3] pkg-database and systemd-sysext image
>
> This email is not from Hexagon’s Office 365 instance. Please be careful while
> clicking links, opening attachments, or replying to this email
Re: [OE-core] [PATCH v9 0/3] pkg-database and systemd-sysext image
Hoi Alexandre, is there a way to get more output out of the autobuilder? the logs only tell "that" the test failed, not "why" = which assertion was missed if i run the test locally with the differing variables set like this: #> SDKMACHINE=aarch64 MACHINE=qemuarm64 oe-selftest --verbose -r imagefeatures.ImageFeatures.test_image_gen_pkgdbfs_ipk it succeeds :-S ... ideas/thoughts? gruß Johannes From: Alexandre Belloni Sent: Friday, May 17, 2024 20:33 To: SCHNEIDER Johannes Cc: openembedded-core@lists.openembedded.org; richard.pur...@linuxfoundation.org; alex.kana...@gmail.com Subject: Re: [OE-core] [PATCH v9 0/3] pkg-database and systemd-sysext image This email is not from Hexagon’s Office 365 instance. Please be careful while clicking links, opening attachments, or replying to this email. Hello, deb and rpm seem successful but ipk still fails: https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/3335/steps/14/logs/stdio On 16/05/2024 00:34:57+0200, Johannes Schneider via lists.openembedded.org wrote: > systemd-sysext allows to overlay another image (or multiple) ontop of > a "base-image" = the current rootfs, via the use of overlayfs; to add > tools and features meant for development purposes. > > To quote the documentation on systemd-sysext: > " ...addition in order to make debugging/development easier). System > extension images should not be misunderstood as a generic software > packaging framework, ..." > > To build a lean image, that only holds packages that are not already > part of the base-image, a snapshot of the package-database is taken > after the installation of the base-rootfs is done, and picked up again > when collecting the rootfs of such a extension image. > > with all this in place an example usage could look like this: > some-core-image.bb > inherit core-image > IMAGE_GEN_PKGDBFS = "1" > > extending-image.bb > inherit image-sysext > IMAGE_FSTYPES = "squashfs" > IMAGE_BASE_PKGDB = "some-core-image" > # the above pointing at a package-db similar to: > # > build/deploy/images/$MACHINE/some-core-image-$MACHINE-20240210172305-pkgdb.rootfs.tar.gz > > then on the device, running some-core-image, with the extension image placed > at FN: > $> ln -s "$FN" /run/extensions/$(basename $FN).raw > $> systemd-sysext list > $> SYSTEMD_LOG_LEVEL=debug systemd-sysext merge > > As long as the VERSION_ID of the extension image matches the os-release > in the base image, the above commands return sucessfully; > for details on the compativility check see the docs for systemd-sysext. > > = > > changes with v2: > rebase from 'kirkstone' onto 'master' > > changes with v3: > incorporate review suggestions for simplification > add task dependency handling > add oe-selftest for the pkgdb handling > add variable documentation and > some more comments, and examples in the commit-msg > > changes with v4: > rebase onto 'master' => no functional changes > fixed patchtest findings > > changes with v5: > rebase onto 'master' > add '.sysext' to the deployed symlink name > sidenote on the tests and autobuilder failure: run locally they > succeed, e.g.: > #> oe-selftest --verbose -r > imagefeatures.ImageFeatures.test_image_gen_pkgdbfs > > changes with v6: > tests: restructure to call 'bitbake' only once in the testcase itself > (in hopes of solving the autobuilder problem; local test runs succeed) > > changes with v7: > tests: undo the restructuring of done in v6, in favour of explicitly > forcing steps: > '-c rootfs -f'. since concurrency of the autobuilder, and reusing > artifacts/sstate-caching is probably the issue? > > changes with v8: > tests: fixed missing feature=PACKAGE_CLASSES during > 'test_image_gen_pkgdbfs' > which is now split into three, one for each of the three available > packagemanager: ipk, deb, rpm > > changes with v9: > tests: fix singular failure in 'oe-selftest-armhost' > which was due to the built image not containing a 'kernel-image...', > which the gen-pkgdb sanity > check expected, but 'packagegroup-core-boot' instead > > > -- Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com/ -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#199529): https://lists.openembedded.org/g/openembedded-core/message/199529 Mute This Topic: https://lists.openembedded.org/mt/106124365/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v8 0/3] pkg-database and systemd-sysext image
Hoi Alexandre,
and thanks again for patiently pointing out the autobuilder problems :-)
"only" the test_image_gen_pkgdbfs_{ipk,deb,rpm} fail, the
test_image_use_pkgdbfs succeeds
picking apart the yocto-autobuilder-helper/config.json showed the difference:
basically all oe-selftest-* use
"MACHINE" : "qemux86-64"
**but** oe-selftest-armhost sets:
"MACHINE": "qemuarm64"
the test_image_gen_pkgdbfs_ checks to sanity check the packagamanager state
looked for a "kernel-image" package in the listing... but for quemuarm64
"packagegroup-core-boot" replaces that
v9 of the patchstack adapts that particular assert(s) to look for either the
kernel-image or core-boot depending on the set MACHINE
gruß
Johannes
From: Alexandre Belloni
Sent: Wednesday, May 15, 2024 21:09
To: SCHNEIDER Johannes
Cc: openembedded-core@lists.openembedded.org;
richard.pur...@linuxfoundation.org; alex.kana...@gmail.com
Subject: Re: [OE-core] [PATCH v8 0/3] pkg-database and systemd-sysext image
This email is not from Hexagon’s Office 365 instance. Please be careful while
clicking links, opening attachments, or replying to this email.
Hello,
It seems that this reliably fails oe-selftest-armhost. I didn't really
pay attention until now beause we had other issues but in this build,
those are the only failing tests:
https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/3322/steps/15/logs/stdio
On 12/05/2024 08:31:56+0200, Johannes Schneider via lists.openembedded.org
wrote:
> systemd-sysext allows to overlay another image (or multiple) ontop of
> a "base-image" = the current rootfs, via the use of overlayfs; to add
> tools and features meant for development purposes.
>
> To quote the documentation on systemd-sysext:
> " ...addition in order to make debugging/development easier). System
> extension images should not be misunderstood as a generic software
> packaging framework, ..."
>
> To build a lean image, that only holds packages that are not already
> part of the base-image, a snapshot of the package-database is taken
> after the installation of the base-rootfs is done, and picked up again
> when collecting the rootfs of such a extension image.
>
> with all this in place an example usage could look like this:
> some-core-image.bb
> inherit core-image
> IMAGE_GEN_PKGDBFS = "1"
>
> extending-image.bb
> inherit image-sysext
> IMAGE_FSTYPES = "squashfs"
> IMAGE_BASE_PKGDB = "some-core-image"
> # the above pointing at a package-db similar to:
> #
> build/deploy/images/$MACHINE/some-core-image-$MACHINE-20240210172305-pkgdb.rootfs.tar.gz
>
> then on the device, running some-core-image, with the extension image placed
> at FN:
> $> ln -s "$FN" /run/extensions/$(basename $FN).raw
> $> systemd-sysext list
> $> SYSTEMD_LOG_LEVEL=debug systemd-sysext merge
>
> As long as the VERSION_ID of the extension image matches the os-release
> in the base image, the above commands return sucessfully;
> for details on the compativility check see the docs for systemd-sysext.
>
> =
>
> changes with v2:
> rebase from 'kirkstone' onto 'master'
>
> changes with v3:
> incorporate review suggestions for simplification
> add task dependency handling
> add oe-selftest for the pkgdb handling
> add variable documentation and
> some more comments, and examples in the commit-msg
>
> changes with v4:
> rebase onto 'master' => no functional changes
> fixed patchtest findings
>
> changes with v5:
> rebase onto 'master'
> add '.sysext' to the deployed symlink name
> sidenote on the tests and autobuilder failure: run locally they
> succeed, e.g.:
> #> oe-selftest --verbose -r
> imagefeatures.ImageFeatures.test_image_gen_pkgdbfs
>
> changes with v6:
> tests: restructure to call 'bitbake' only once in the testcase itself
> (in hopes of solving the autobuilder problem; local test runs succeed)
>
> changes with v7:
> tests: undo the restructuring of done in v6, in favour of explicitly
> forcing steps:
> '-c rootfs -f'. since concurrency of the autobuilder, and reusing
> artifacts/sstate-caching is probably the issue?
>
> changes with v8:
> tests: fixed missing feature=PACKAGE_CLASSES during
> 'test_image_gen_pkgdbfs'
> which is now split into three, one for each of the three available
> packagemanager: ipk, deb, rpm
>
>
>
--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com/
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#199434):
https://lists.openembedded.org/g/openembedded-core/message/199434
Mute This Topic: https://lists.openembedded.org/mt/106051545/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v9 3/3] classes: add a systemd-sysext image class
systemd-sysext can load a raw-image containing usr/ and opt/ folders
to mount them as RO overlay over the rootfs, to "extend" the systems.
This class provides the necessary changes/additions to the enclosed
file-system so that systemd-sysext accepts the extension for "merge"
into the rootfs.
With such an created image, placed into the correct folder (see [1]),
`systemd-sysext list` should be able to list the "extension" and
`systemd-sysext merge` should enable the overlay. On both commands a
preceding "SYSTEMD_LOG_LEVEL=debug" can aide in figuring out what is
amiss.
The strict name checking systemd-sysext does against the name of
extension-release.NAME file, is disabled, as there is only one such in
the resulting image. This is done to allow a user to freely rename the
resulting image file.
Note that for e.g. squashfs, the kernel needs CONFIG_SQUASHFS_XATTR=y
Link:
https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html
Link:
https://0pointer.net/blog/testing-my-system-code-in-usr-without-modifying-usr.html
Signed-off-by: Johannes Schneider
---
meta/classes-recipe/image-sysext.bbclass | 43
1 file changed, 43 insertions(+)
create mode 100644 meta/classes-recipe/image-sysext.bbclass
diff --git a/meta/classes-recipe/image-sysext.bbclass
b/meta/classes-recipe/image-sysext.bbclass
new file mode 100644
index 00..bc3e4d52b5
--- /dev/null
+++ b/meta/classes-recipe/image-sysext.bbclass
@@ -0,0 +1,43 @@
+# SPDX-License-Identifier: MIT
+#
+# Copyright Leica Geosystems AG
+#
+
+# systemd-sysext [1] has a simple mechanism for version compatibility:
+# the extension to be loaded has to contain a
+# /usr/lib/extension-release.d/extension-release.NAME
+# with "NAME" *exactly* matching the filename of the extensions
+# raw-device filename/
+#
+# from the extension-release file the "ID" and "VERSION_ID" fields are
+# matched against the etc/os-release and the extension is only "merged"
+# if no mismatches between NAME, ID, and VERSION_ID.
+#
+# Link:
https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html
+
+inherit image
+
+IMAGE_NAME_SUFFIX = ".sysext"
+EXTENSION_NAME = "${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${IMAGE_FSTYPES}"
+IMAGE_LINK_NAME:append = ".sysext"
+
+DEPENDS += " os-release"
+
+sysext_image_mangle_rootfs() {
+R=${IMAGE_ROOTFS}
+
+# pull a copy of the rootfs version information, which systemd-sysext
matches against
+cp -av ${RECIPE_SYSROOT}/${nonarch_libdir}/os-release
${WORKDIR}/extension-release.base
+
+echo 'EXTENSION_RELOAD_MANAGER=1' >> ${WORKDIR}/extension-release.base
+
+install -d $R${nonarch_libdir}/extension-release.d
+install -m 0644 ${WORKDIR}/extension-release.base \
+
$R${nonarch_libdir}/extension-release.d/extension-release.${EXTENSION_NAME}
+
+# disable systemd-sysext's strict name checking, so that the image file
can be renamed, while still being 'merge'-able
+setfattr -n user.extension-release.strict -v false \
+
$R${nonarch_libdir}/extension-release.d/extension-release.${EXTENSION_NAME}
+}
+
+ROOTFS_POSTPROCESS_COMMAND += " sysext_image_mangle_rootfs; "
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#199433):
https://lists.openembedded.org/g/openembedded-core/message/199433
Mute This Topic: https://lists.openembedded.org/mt/106124370/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v9 2/3] image.bbclass/rootfs: set and unpack package-database
set the package-database of a "lower image" to unpack and build upon
when installing packages for the current image. This way a lean image
will be created, which only holds the packages that are not already
present in the lower image.
An image build such could then be used with overlayfs or systemd-
sysext to extend the "lower image" on demand; for development purposes
on a device running the "lower image" in RO mode for example.
A configuration could look as follows:
some-core-image.bb
inherit image
IMAGE_GEN_PKGDBFS = "1"
extending-image.bb
inherit image
IMAGE_BASE_PKGDB = "some-core-image"
Signed-off-by: Johannes Schneider
---
meta/classes-recipe/image.bbclass | 23 ++--
meta/conf/documentation.conf | 3 +-
meta/lib/oe/package_manager/deb/rootfs.py | 2 +
meta/lib/oe/package_manager/ipk/rootfs.py | 6 ++-
meta/lib/oe/package_manager/rpm/rootfs.py | 7 ++-
meta/lib/oe/rootfs.py | 18 +++
meta/lib/oeqa/selftest/cases/imagefeatures.py | 52 ++-
7 files changed, 102 insertions(+), 9 deletions(-)
diff --git a/meta/classes-recipe/image.bbclass
b/meta/classes-recipe/image.bbclass
index 3ccaaa17b8..c573c37cd8 100644
--- a/meta/classes-recipe/image.bbclass
+++ b/meta/classes-recipe/image.bbclass
@@ -42,8 +42,16 @@ IMAGE_FEATURES ?= ""
IMAGE_FEATURES[type] = "list"
IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs
read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password
allow-empty-password allow-root-login serial-autologin-root
post-install-logging overlayfs-etc"
-# Generate snapshot of the package database?
+# Image layering:
+# a "base image" would create a snapshot of the package-database after the
+# installation of all packages into the rootfs is done. The next/other image
+# "layered on-top" of the former would then import that database and install
+# further packages; without reinstalling packages/dependencies that are already
+# installed in the layer below.
+# Set to '1' in a "base image" recipe, to preserve a snapshot of the package
database.
IMAGE_GEN_PKGDBFS ?= "0"
+# "PN" of a "base image", upon which the current image is to be built upon.
+IMAGE_BASE_PKGDB ?= ""
# Generate companion debugfs?
IMAGE_GEN_DEBUGFS ?= "0"
@@ -118,6 +126,15 @@ do_rootfs[depends] += " \
"
do_rootfs[recrdeptask] += "do_packagedata"
+python () {
+# make sure that the 'base image' has been queued in before this
+# image wants to unpack and build upon the formers pgkdb
+base_image = d.getVar('IMAGE_BASE_PKGDB')
+pn = d.getVar('PN')
+if base_image and base_image != pn:
+d.appendVarFlag("do_rootfs", 'depends', ' '+ base_image +
':do_image_complete')
+}
+
def rootfs_command_variables(d):
return
['ROOTFS_POSTPROCESS_COMMAND','ROOTFS_PREPROCESS_COMMAND','ROOTFS_POSTINSTALL_COMMAND','ROOTFS_POSTUNINSTALL_COMMAND','OPKG_PREPROCESS_COMMANDS','OPKG_POSTPROCESS_COMMANDS','IMAGE_POSTPROCESS_COMMAND',
'IMAGE_PREPROCESS_COMMAND','RPM_PREPROCESS_COMMANDS','RPM_POSTPROCESS_COMMANDS','DEB_PREPROCESS_COMMANDS','DEB_POSTPROCESS_COMMANDS']
@@ -134,8 +151,8 @@ def rootfs_variables(d):
'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
'IMAGE_LINGUAS_COMPLEMENTARY', 'IMAGE_LOCALES_ARCHIVE',
'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
- 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS', 'IMAGE_GEN_DEBUGFS',
'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY',
'REPRODUCIBLE_TIMESTAMP_ROOTFS',
- 'IMAGE_INSTALL_DEBUGFS']
+ 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS', 'IMAGE_BASE_PKGDB',
'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR',
'PACKAGE_EXCLUDE_COMPLEMENTARY',
+ 'REPRODUCIBLE_TIMESTAMP_ROOTFS', 'IMAGE_INSTALL_DEBUGFS']
variables.extend(rootfs_command_variables(d))
variables.extend(variable_depends(d))
return " ".join(variables)
diff --git a/meta/conf/documentation.conf b/meta/conf/documentation.conf
index 36aebb59ab..9f493cfe96 100644
--- a/meta/conf/documentation.conf
+++ b/meta/conf/documentation.conf
@@ -208,6 +208,7 @@ ICECC_PATH[doc] = "The location of the icecc binary."
ICECC_CLASS_DISABLE[doc] = "Identifies user classes that you do not want the
Icecream distributed compile support to consider."
ICECC_RECIPE_DISABLE[doc] = "Identifies user recipes that you do not want the
Icecream distributed compile support to consider."
ICECC_RECIPE_ENABLE[doc] = "Identifies user recipes that use an
[OE-core] [PATCH v9 1/3] image.bbclass/rootfs: archive and deploy package database
archive the package database after the rootfs has been put together as
*rootfs-pkdbfs.tar.gz, and put it into the deploy folder.
This creates a snapshot of the package mangers state at the point in
time when all dependencies have been resolved and installed; which
could be used by "extension images" to built upon.
Signed-off-by: Johannes Schneider
---
meta/classes-recipe/image.bbclass | 44 -
meta/classes-recipe/image_types.bbclass | 1 +
meta/conf/documentation.conf | 1 +
meta/lib/oe/package_manager/deb/rootfs.py | 1 +
meta/lib/oe/package_manager/ipk/rootfs.py | 1 +
meta/lib/oe/package_manager/rpm/rootfs.py | 1 +
meta/lib/oe/rootfs.py | 20 +++
meta/lib/oeqa/selftest/cases/imagefeatures.py | 159 ++
8 files changed, 227 insertions(+), 1 deletion(-)
diff --git a/meta/classes-recipe/image.bbclass
b/meta/classes-recipe/image.bbclass
index 28be6c6362..3ccaaa17b8 100644
--- a/meta/classes-recipe/image.bbclass
+++ b/meta/classes-recipe/image.bbclass
@@ -42,6 +42,9 @@ IMAGE_FEATURES ?= ""
IMAGE_FEATURES[type] = "list"
IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs
read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password
allow-empty-password allow-root-login serial-autologin-root
post-install-logging overlayfs-etc"
+# Generate snapshot of the package database?
+IMAGE_GEN_PKGDBFS ?= "0"
+
# Generate companion debugfs?
IMAGE_GEN_DEBUGFS ?= "0"
@@ -131,7 +134,8 @@ def rootfs_variables(d):
'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
'IMAGE_LINGUAS_COMPLEMENTARY', 'IMAGE_LOCALES_ARCHIVE',
'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
- 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED',
'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY',
'REPRODUCIBLE_TIMESTAMP_ROOTFS', 'IMAGE_INSTALL_DEBUGFS']
+ 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS', 'IMAGE_GEN_DEBUGFS',
'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY',
'REPRODUCIBLE_TIMESTAMP_ROOTFS',
+ 'IMAGE_INSTALL_DEBUGFS']
variables.extend(rootfs_command_variables(d))
variables.extend(variable_depends(d))
return " ".join(variables)
@@ -337,6 +341,17 @@ python do_image_qa_setscene () {
}
addtask do_image_qa_setscene
+def setup_pkgdbfs_variables(d):
+d.appendVar('IMAGE_ROOTFS', '-pkgdb')
+if d.getVar('IMAGE_LINK_NAME'):
+d.appendVar('IMAGE_LINK_NAME', '-pkgdb')
+d.appendVar('IMAGE_NAME','-pkgdb')
+d.setVar('IMAGE_FSTYPES', 'tar.gz')
+
+python setup_pkgdbfs () {
+setup_pkgdbfs_variables(d)
+}
+
def setup_debugfs_variables(d):
d.appendVar('IMAGE_ROOTFS', '-dbg')
if d.getVar('IMAGE_LINK_NAME'):
@@ -381,6 +396,11 @@ python () {
alltypes = d.getVar('IMAGE_FSTYPES').split()
typedeps = {}
+if d.getVar('IMAGE_GEN_PKGDBFS') == "1":
+pkgdbfs_fstypes = ['tar.gz']
+for t in pkgdbfs_fstypes:
+alltypes.append("pkgdbfs_" + t)
+
if d.getVar('IMAGE_GEN_DEBUGFS') == "1":
debugfs_fstypes = d.getVar('IMAGE_FSTYPES_DEBUGFS').split()
for t in debugfs_fstypes:
@@ -393,6 +413,10 @@ python () {
basetypes[baset]= []
if t not in basetypes[baset]:
basetypes[baset].append(t)
+pkgdb = ""
+if t.startswith("pkgdbfs_"):
+t = t[8:]
+pkgdb = "pkgdbfs_"
debug = ""
if t.startswith("debugfs_"):
t = t[8:]
@@ -401,6 +425,13 @@ python () {
vardeps.add('IMAGE_TYPEDEP:' + t)
if baset not in typedeps:
typedeps[baset] = set()
+deps = [pkgdb + dep for dep in deps]
+for dep in deps:
+if dep not in alltypes:
+alltypes.append(dep)
+_add_type(dep)
+basedep = _image_base_type(dep)
+typedeps[baset].add(basedep)
deps = [debug + dep for dep in deps]
for dep in deps:
if dep not in alltypes:
@@ -419,6 +450,7 @@ python () {
maskedtypes = (d.getVar('IMAGE_TYPES_MASKED') or "").split()
maskedtypes = [dbg + t for t in maskedtypes for dbg in ("", "debugfs_")]
+maskedtypes = [pkgdb + t for t in maskedtypes for pkgdb in ("",
"pkgdbfs_")]
for t in basetypes:
vardeps = set()
@@ -430,6 +462,11 @@ python () {
continue
localdata = bb.data.createCopy(d)
+pkgdb = ""
+if t.startswith("pkgdbfs_"):
+setup_pkgdbfs_variable
[OE-core] [PATCH v9 0/3] pkg-database and systemd-sysext image
systemd-sysext allows to overlay another image (or multiple) ontop of a "base-image" = the current rootfs, via the use of overlayfs; to add tools and features meant for development purposes. To quote the documentation on systemd-sysext: " ...addition in order to make debugging/development easier). System extension images should not be misunderstood as a generic software packaging framework, ..." To build a lean image, that only holds packages that are not already part of the base-image, a snapshot of the package-database is taken after the installation of the base-rootfs is done, and picked up again when collecting the rootfs of such a extension image. with all this in place an example usage could look like this: some-core-image.bb inherit core-image IMAGE_GEN_PKGDBFS = "1" extending-image.bb inherit image-sysext IMAGE_FSTYPES = "squashfs" IMAGE_BASE_PKGDB = "some-core-image" # the above pointing at a package-db similar to: # build/deploy/images/$MACHINE/some-core-image-$MACHINE-20240210172305-pkgdb.rootfs.tar.gz then on the device, running some-core-image, with the extension image placed at FN: $> ln -s "$FN" /run/extensions/$(basename $FN).raw $> systemd-sysext list $> SYSTEMD_LOG_LEVEL=debug systemd-sysext merge As long as the VERSION_ID of the extension image matches the os-release in the base image, the above commands return sucessfully; for details on the compativility check see the docs for systemd-sysext. = changes with v2: rebase from 'kirkstone' onto 'master' changes with v3: incorporate review suggestions for simplification add task dependency handling add oe-selftest for the pkgdb handling add variable documentation and some more comments, and examples in the commit-msg changes with v4: rebase onto 'master' => no functional changes fixed patchtest findings changes with v5: rebase onto 'master' add '.sysext' to the deployed symlink name sidenote on the tests and autobuilder failure: run locally they succeed, e.g.: #> oe-selftest --verbose -r imagefeatures.ImageFeatures.test_image_gen_pkgdbfs changes with v6: tests: restructure to call 'bitbake' only once in the testcase itself (in hopes of solving the autobuilder problem; local test runs succeed) changes with v7: tests: undo the restructuring of done in v6, in favour of explicitly forcing steps: '-c rootfs -f'. since concurrency of the autobuilder, and reusing artifacts/sstate-caching is probably the issue? changes with v8: tests: fixed missing feature=PACKAGE_CLASSES during 'test_image_gen_pkgdbfs' which is now split into three, one for each of the three available packagemanager: ipk, deb, rpm changes with v9: tests: fix singular failure in 'oe-selftest-armhost' which was due to the built image not containing a 'kernel-image...', which the gen-pkgdb sanity check expected, but 'packagegroup-core-boot' instead -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#199430): https://lists.openembedded.org/g/openembedded-core/message/199430 Mute This Topic: https://lists.openembedded.org/mt/106124365/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v8 3/3] classes: add a systemd-sysext image class
systemd-sysext can load a raw-image containing usr/ and opt/ folders
to mount them as RO overlay over the rootfs, to "extend" the systems.
This class provides the necessary changes/additions to the enclosed
file-system so that systemd-sysext accepts the extension for "merge"
into the rootfs.
With such an created image, placed into the correct folder (see [1]),
`systemd-sysext list` should be able to list the "extension" and
`systemd-sysext merge` should enable the overlay. On both commands a
preceding "SYSTEMD_LOG_LEVEL=debug" can aide in figuring out what is
amiss.
The strict name checking systemd-sysext does against the name of
extension-release.NAME file, is disabled, as there is only one such in
the resulting image. This is done to allow a user to freely rename the
resulting image file.
Note that for e.g. squashfs, the kernel needs CONFIG_SQUASHFS_XATTR=y
Link:
https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html
Link:
https://0pointer.net/blog/testing-my-system-code-in-usr-without-modifying-usr.html
Signed-off-by: Johannes Schneider
---
meta/classes-recipe/image-sysext.bbclass | 43
1 file changed, 43 insertions(+)
create mode 100644 meta/classes-recipe/image-sysext.bbclass
diff --git a/meta/classes-recipe/image-sysext.bbclass
b/meta/classes-recipe/image-sysext.bbclass
new file mode 100644
index 00..bc3e4d52b5
--- /dev/null
+++ b/meta/classes-recipe/image-sysext.bbclass
@@ -0,0 +1,43 @@
+# SPDX-License-Identifier: MIT
+#
+# Copyright Leica Geosystems AG
+#
+
+# systemd-sysext [1] has a simple mechanism for version compatibility:
+# the extension to be loaded has to contain a
+# /usr/lib/extension-release.d/extension-release.NAME
+# with "NAME" *exactly* matching the filename of the extensions
+# raw-device filename/
+#
+# from the extension-release file the "ID" and "VERSION_ID" fields are
+# matched against the etc/os-release and the extension is only "merged"
+# if no mismatches between NAME, ID, and VERSION_ID.
+#
+# Link:
https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html
+
+inherit image
+
+IMAGE_NAME_SUFFIX = ".sysext"
+EXTENSION_NAME = "${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${IMAGE_FSTYPES}"
+IMAGE_LINK_NAME:append = ".sysext"
+
+DEPENDS += " os-release"
+
+sysext_image_mangle_rootfs() {
+R=${IMAGE_ROOTFS}
+
+# pull a copy of the rootfs version information, which systemd-sysext
matches against
+cp -av ${RECIPE_SYSROOT}/${nonarch_libdir}/os-release
${WORKDIR}/extension-release.base
+
+echo 'EXTENSION_RELOAD_MANAGER=1' >> ${WORKDIR}/extension-release.base
+
+install -d $R${nonarch_libdir}/extension-release.d
+install -m 0644 ${WORKDIR}/extension-release.base \
+
$R${nonarch_libdir}/extension-release.d/extension-release.${EXTENSION_NAME}
+
+# disable systemd-sysext's strict name checking, so that the image file
can be renamed, while still being 'merge'-able
+setfattr -n user.extension-release.strict -v false \
+
$R${nonarch_libdir}/extension-release.d/extension-release.${EXTENSION_NAME}
+}
+
+ROOTFS_POSTPROCESS_COMMAND += " sysext_image_mangle_rootfs; "
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#199211):
https://lists.openembedded.org/g/openembedded-core/message/199211
Mute This Topic: https://lists.openembedded.org/mt/106051547/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v8 0/3] pkg-database and systemd-sysext image
systemd-sysext allows to overlay another image (or multiple) ontop of a "base-image" = the current rootfs, via the use of overlayfs; to add tools and features meant for development purposes. To quote the documentation on systemd-sysext: " ...addition in order to make debugging/development easier). System extension images should not be misunderstood as a generic software packaging framework, ..." To build a lean image, that only holds packages that are not already part of the base-image, a snapshot of the package-database is taken after the installation of the base-rootfs is done, and picked up again when collecting the rootfs of such a extension image. with all this in place an example usage could look like this: some-core-image.bb inherit core-image IMAGE_GEN_PKGDBFS = "1" extending-image.bb inherit image-sysext IMAGE_FSTYPES = "squashfs" IMAGE_BASE_PKGDB = "some-core-image" # the above pointing at a package-db similar to: # build/deploy/images/$MACHINE/some-core-image-$MACHINE-20240210172305-pkgdb.rootfs.tar.gz then on the device, running some-core-image, with the extension image placed at FN: $> ln -s "$FN" /run/extensions/$(basename $FN).raw $> systemd-sysext list $> SYSTEMD_LOG_LEVEL=debug systemd-sysext merge As long as the VERSION_ID of the extension image matches the os-release in the base image, the above commands return sucessfully; for details on the compativility check see the docs for systemd-sysext. = changes with v2: rebase from 'kirkstone' onto 'master' changes with v3: incorporate review suggestions for simplification add task dependency handling add oe-selftest for the pkgdb handling add variable documentation and some more comments, and examples in the commit-msg changes with v4: rebase onto 'master' => no functional changes fixed patchtest findings changes with v5: rebase onto 'master' add '.sysext' to the deployed symlink name sidenote on the tests and autobuilder failure: run locally they succeed, e.g.: #> oe-selftest --verbose -r imagefeatures.ImageFeatures.test_image_gen_pkgdbfs changes with v6: tests: restructure to call 'bitbake' only once in the testcase itself (in hopes of solving the autobuilder problem; local test runs succeed) changes with v7: tests: undo the restructuring of done in v6, in favour of explicitly forcing steps: '-c rootfs -f'. since concurrency of the autobuilder, and reusing artifacts/sstate-caching is probably the issue? changes with v8: tests: fixed missing feature=PACKAGE_CLASSES during 'test_image_gen_pkgdbfs' which is now split into three, one for each of the three available packagemanager: ipk, deb, rpm -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#199209): https://lists.openembedded.org/g/openembedded-core/message/199209 Mute This Topic: https://lists.openembedded.org/mt/106051545/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v8 1/3] image.bbclass/rootfs: archive and deploy package database
archive the package database after the rootfs has been put together as
*rootfs-pkdbfs.tar.gz, and put it into the deploy folder.
This creates a snapshot of the package mangers state at the point in
time when all dependencies have been resolved and installed; which
could be used by "extension images" to built upon.
Signed-off-by: Johannes Schneider
---
meta/classes-recipe/image.bbclass | 44 -
meta/classes-recipe/image_types.bbclass | 1 +
meta/conf/documentation.conf | 1 +
meta/lib/oe/package_manager/deb/rootfs.py | 1 +
meta/lib/oe/package_manager/ipk/rootfs.py | 1 +
meta/lib/oe/package_manager/rpm/rootfs.py | 1 +
meta/lib/oe/rootfs.py | 20 +++
meta/lib/oeqa/selftest/cases/imagefeatures.py | 150 ++
8 files changed, 218 insertions(+), 1 deletion(-)
diff --git a/meta/classes-recipe/image.bbclass
b/meta/classes-recipe/image.bbclass
index 28be6c6362..3ccaaa17b8 100644
--- a/meta/classes-recipe/image.bbclass
+++ b/meta/classes-recipe/image.bbclass
@@ -42,6 +42,9 @@ IMAGE_FEATURES ?= ""
IMAGE_FEATURES[type] = "list"
IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs
read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password
allow-empty-password allow-root-login serial-autologin-root
post-install-logging overlayfs-etc"
+# Generate snapshot of the package database?
+IMAGE_GEN_PKGDBFS ?= "0"
+
# Generate companion debugfs?
IMAGE_GEN_DEBUGFS ?= "0"
@@ -131,7 +134,8 @@ def rootfs_variables(d):
'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
'IMAGE_LINGUAS_COMPLEMENTARY', 'IMAGE_LOCALES_ARCHIVE',
'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
- 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED',
'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY',
'REPRODUCIBLE_TIMESTAMP_ROOTFS', 'IMAGE_INSTALL_DEBUGFS']
+ 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS', 'IMAGE_GEN_DEBUGFS',
'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY',
'REPRODUCIBLE_TIMESTAMP_ROOTFS',
+ 'IMAGE_INSTALL_DEBUGFS']
variables.extend(rootfs_command_variables(d))
variables.extend(variable_depends(d))
return " ".join(variables)
@@ -337,6 +341,17 @@ python do_image_qa_setscene () {
}
addtask do_image_qa_setscene
+def setup_pkgdbfs_variables(d):
+d.appendVar('IMAGE_ROOTFS', '-pkgdb')
+if d.getVar('IMAGE_LINK_NAME'):
+d.appendVar('IMAGE_LINK_NAME', '-pkgdb')
+d.appendVar('IMAGE_NAME','-pkgdb')
+d.setVar('IMAGE_FSTYPES', 'tar.gz')
+
+python setup_pkgdbfs () {
+setup_pkgdbfs_variables(d)
+}
+
def setup_debugfs_variables(d):
d.appendVar('IMAGE_ROOTFS', '-dbg')
if d.getVar('IMAGE_LINK_NAME'):
@@ -381,6 +396,11 @@ python () {
alltypes = d.getVar('IMAGE_FSTYPES').split()
typedeps = {}
+if d.getVar('IMAGE_GEN_PKGDBFS') == "1":
+pkgdbfs_fstypes = ['tar.gz']
+for t in pkgdbfs_fstypes:
+alltypes.append("pkgdbfs_" + t)
+
if d.getVar('IMAGE_GEN_DEBUGFS') == "1":
debugfs_fstypes = d.getVar('IMAGE_FSTYPES_DEBUGFS').split()
for t in debugfs_fstypes:
@@ -393,6 +413,10 @@ python () {
basetypes[baset]= []
if t not in basetypes[baset]:
basetypes[baset].append(t)
+pkgdb = ""
+if t.startswith("pkgdbfs_"):
+t = t[8:]
+pkgdb = "pkgdbfs_"
debug = ""
if t.startswith("debugfs_"):
t = t[8:]
@@ -401,6 +425,13 @@ python () {
vardeps.add('IMAGE_TYPEDEP:' + t)
if baset not in typedeps:
typedeps[baset] = set()
+deps = [pkgdb + dep for dep in deps]
+for dep in deps:
+if dep not in alltypes:
+alltypes.append(dep)
+_add_type(dep)
+basedep = _image_base_type(dep)
+typedeps[baset].add(basedep)
deps = [debug + dep for dep in deps]
for dep in deps:
if dep not in alltypes:
@@ -419,6 +450,7 @@ python () {
maskedtypes = (d.getVar('IMAGE_TYPES_MASKED') or "").split()
maskedtypes = [dbg + t for t in maskedtypes for dbg in ("", "debugfs_")]
+maskedtypes = [pkgdb + t for t in maskedtypes for pkgdb in ("",
"pkgdbfs_")]
for t in basetypes:
vardeps = set()
@@ -430,6 +462,11 @@ python () {
continue
localdata = bb.data.createCopy(d)
+pkgdb = ""
+if t.startswith("pkgdbfs_"):
+setup_pkgdbfs_variable
[OE-core] [PATCH v8 2/3] image.bbclass/rootfs: set and unpack package-database
set the package-database of a "lower image" to unpack and build upon
when installing packages for the current image. This way a lean image
will be created, which only holds the packages that are not already
present in the lower image.
An image build such could then be used with overlayfs or systemd-
sysext to extend the "lower image" on demand; for development purposes
on a device running the "lower image" in RO mode for example.
A configuration could look as follows:
some-core-image.bb
inherit image
IMAGE_GEN_PKGDBFS = "1"
extending-image.bb
inherit image
IMAGE_BASE_PKGDB = "some-core-image"
Signed-off-by: Johannes Schneider
---
meta/classes-recipe/image.bbclass | 23 ++--
meta/conf/documentation.conf | 3 +-
meta/lib/oe/package_manager/deb/rootfs.py | 2 +
meta/lib/oe/package_manager/ipk/rootfs.py | 6 ++-
meta/lib/oe/package_manager/rpm/rootfs.py | 7 ++-
meta/lib/oe/rootfs.py | 18 +++
meta/lib/oeqa/selftest/cases/imagefeatures.py | 52 ++-
7 files changed, 102 insertions(+), 9 deletions(-)
diff --git a/meta/classes-recipe/image.bbclass
b/meta/classes-recipe/image.bbclass
index 3ccaaa17b8..c573c37cd8 100644
--- a/meta/classes-recipe/image.bbclass
+++ b/meta/classes-recipe/image.bbclass
@@ -42,8 +42,16 @@ IMAGE_FEATURES ?= ""
IMAGE_FEATURES[type] = "list"
IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs
read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password
allow-empty-password allow-root-login serial-autologin-root
post-install-logging overlayfs-etc"
-# Generate snapshot of the package database?
+# Image layering:
+# a "base image" would create a snapshot of the package-database after the
+# installation of all packages into the rootfs is done. The next/other image
+# "layered on-top" of the former would then import that database and install
+# further packages; without reinstalling packages/dependencies that are already
+# installed in the layer below.
+# Set to '1' in a "base image" recipe, to preserve a snapshot of the package
database.
IMAGE_GEN_PKGDBFS ?= "0"
+# "PN" of a "base image", upon which the current image is to be built upon.
+IMAGE_BASE_PKGDB ?= ""
# Generate companion debugfs?
IMAGE_GEN_DEBUGFS ?= "0"
@@ -118,6 +126,15 @@ do_rootfs[depends] += " \
"
do_rootfs[recrdeptask] += "do_packagedata"
+python () {
+# make sure that the 'base image' has been queued in before this
+# image wants to unpack and build upon the formers pgkdb
+base_image = d.getVar('IMAGE_BASE_PKGDB')
+pn = d.getVar('PN')
+if base_image and base_image != pn:
+d.appendVarFlag("do_rootfs", 'depends', ' '+ base_image +
':do_image_complete')
+}
+
def rootfs_command_variables(d):
return
['ROOTFS_POSTPROCESS_COMMAND','ROOTFS_PREPROCESS_COMMAND','ROOTFS_POSTINSTALL_COMMAND','ROOTFS_POSTUNINSTALL_COMMAND','OPKG_PREPROCESS_COMMANDS','OPKG_POSTPROCESS_COMMANDS','IMAGE_POSTPROCESS_COMMAND',
'IMAGE_PREPROCESS_COMMAND','RPM_PREPROCESS_COMMANDS','RPM_POSTPROCESS_COMMANDS','DEB_PREPROCESS_COMMANDS','DEB_POSTPROCESS_COMMANDS']
@@ -134,8 +151,8 @@ def rootfs_variables(d):
'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
'IMAGE_LINGUAS_COMPLEMENTARY', 'IMAGE_LOCALES_ARCHIVE',
'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
- 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS', 'IMAGE_GEN_DEBUGFS',
'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY',
'REPRODUCIBLE_TIMESTAMP_ROOTFS',
- 'IMAGE_INSTALL_DEBUGFS']
+ 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS', 'IMAGE_BASE_PKGDB',
'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR',
'PACKAGE_EXCLUDE_COMPLEMENTARY',
+ 'REPRODUCIBLE_TIMESTAMP_ROOTFS', 'IMAGE_INSTALL_DEBUGFS']
variables.extend(rootfs_command_variables(d))
variables.extend(variable_depends(d))
return " ".join(variables)
diff --git a/meta/conf/documentation.conf b/meta/conf/documentation.conf
index 36aebb59ab..9f493cfe96 100644
--- a/meta/conf/documentation.conf
+++ b/meta/conf/documentation.conf
@@ -208,6 +208,7 @@ ICECC_PATH[doc] = "The location of the icecc binary."
ICECC_CLASS_DISABLE[doc] = "Identifies user classes that you do not want the
Icecream distributed compile support to consider."
ICECC_RECIPE_DISABLE[doc] = "Identifies user recipes that you do not want the
Icecream distributed compile support to consider."
ICECC_RECIPE_ENABLE[doc] = "Identifies user recipes that use an
Re: [OE-core] [PATCH v6 0/3] pkg-database and systemd-sysext image
Hoi > > This still fails on the autobuilders, I'm pretty sure the config for the > build folder has: > PACKAGE_CLASSES = "package_rpm" > aha! that is the missing puzzle piece - thanks! thought i read somewhere that ipk is the default and geared the tests towards that... "always check yous assumptions" > > This explains this: > > build-st-4166614$ tree tmp/deploy/images/qemux86-64/var/lib/ > tmp/deploy/images/qemux86-64/var/lib/ > ├── dnf > │ ├── history.sqlite > │ ├── history.sqlite-shm > │ └── history.sqlite-wal > └── rpm > ├── rpmdb.sqlite > ├── rpmdb.sqlite-shm > └── rpmdb.sqlite-wal > > 3 directories, 6 files > ...which make me realize that there needs to be a test for all three packagemanagers; at least on the "gen_pkgdb_fs" side -> expect a v8 > > Are you testing with plain poky? > yes, i've checked out just the 'openembeddec-core' plus 'bitbake' layers; and run the tests "vanilla" on that setup - which default to using 'package_ipk' = no wonder my local test runs always succeeded :-D feature request for the autobuilder logs: include a config dump at the beginning? (or a link to the config git repo + sha; or the docs?) gruß Johannes -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#199208): https://lists.openembedded.org/g/openembedded-core/message/199208 Mute This Topic: https://lists.openembedded.org/mt/106014867/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v6 0/3] pkg-database and systemd-sysext image
systemd-sysext allows to overlay another image (or multiple) ontop of a "base-image" = the current rootfs, via the use of overlayfs; to add tools and features meant for development purposes. To quote the documentation on systemd-sysext: " ...addition in order to make debugging/development easier). System extension images should not be misunderstood as a generic software packaging framework, ..." To build a lean image, that only holds packages that are not already part of the base-image, a snapshot of the package-database is taken after the installation of the base-rootfs is done, and picked up again when collecting the rootfs of such a extension image. with all this in place an example usage could look like this: some-core-image.bb inherit core-image IMAGE_GEN_PKGDBFS = "1" extending-image.bb inherit image-sysext IMAGE_FSTYPES = "squashfs" IMAGE_BASE_PKGDB = "some-core-image" # the above pointing at a package-db similar to: # build/deploy/images/$MACHINE/some-core-image-$MACHINE-20240210172305-pkgdb.rootfs.tar.gz then on the device, running some-core-image, with the extension image placed at FN: $> ln -s "$FN" /run/extensions/$(basename $FN).raw $> systemd-sysext list $> SYSTEMD_LOG_LEVEL=debug systemd-sysext merge As long as the VERSION_ID of the extension image matches the os-release in the base image, the above commands return sucessfully; for details on the compativility check see the docs for systemd-sysext. = changes with v2: rebase from 'kirkstone' onto 'master' changes with v3: incorporate review suggestions for simplification add task dependency handling add oe-selftest for the pkgdb handling add variable documentation and some more comments, and examples in the commit-msg changes with v4: rebase onto 'master' => no functional changes fixed patchtest findings changes with v5: rebase onto 'master' add '.sysext' to the deployed symlink name sidenote on the tests and autobuilder failure: run locally they succeed, e.g.: #> oe-selftest --verbose -r imagefeatures.ImageFeatures.test_image_gen_pkgdbfs changes with v6: tests: restructure to call 'bitbake' only once in the testcase itself (in hopes of solving the autobuilder problem; local test runs succeed) changes with v7: tests: undo the restructuring of done in v6, in favour of explicitly forcing steps: '-c rootfs -f'. since concurrency of the autobuilder, and reusing artifacts/sstate-caching is probably the issue? -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#199187): https://lists.openembedded.org/g/openembedded-core/message/199187 Mute This Topic: https://lists.openembedded.org/mt/106014867/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v7 3/3] classes: add a systemd-sysext image class
systemd-sysext can load a raw-image containing usr/ and opt/ folders
to mount them as RO overlay over the rootfs, to "extend" the systems.
This class provides the necessary changes/additions to the enclosed
file-system so that systemd-sysext accepts the extension for "merge"
into the rootfs.
With such an created image, placed into the correct folder (see [1]),
`systemd-sysext list` should be able to list the "extension" and
`systemd-sysext merge` should enable the overlay. On both commands a
preceding "SYSTEMD_LOG_LEVEL=debug" can aide in figuring out what is
amiss.
The strict name checking systemd-sysext does against the name of
extension-release.NAME file, is disabled, as there is only one such in
the resulting image. This is done to allow a user to freely rename the
resulting image file.
Note that for e.g. squashfs, the kernel needs CONFIG_SQUASHFS_XATTR=y
Link:
https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html
Link:
https://0pointer.net/blog/testing-my-system-code-in-usr-without-modifying-usr.html
Signed-off-by: Johannes Schneider
---
meta/classes-recipe/image-sysext.bbclass | 43
1 file changed, 43 insertions(+)
create mode 100644 meta/classes-recipe/image-sysext.bbclass
diff --git a/meta/classes-recipe/image-sysext.bbclass
b/meta/classes-recipe/image-sysext.bbclass
new file mode 100644
index 00..bc3e4d52b5
--- /dev/null
+++ b/meta/classes-recipe/image-sysext.bbclass
@@ -0,0 +1,43 @@
+# SPDX-License-Identifier: MIT
+#
+# Copyright Leica Geosystems AG
+#
+
+# systemd-sysext [1] has a simple mechanism for version compatibility:
+# the extension to be loaded has to contain a
+# /usr/lib/extension-release.d/extension-release.NAME
+# with "NAME" *exactly* matching the filename of the extensions
+# raw-device filename/
+#
+# from the extension-release file the "ID" and "VERSION_ID" fields are
+# matched against the etc/os-release and the extension is only "merged"
+# if no mismatches between NAME, ID, and VERSION_ID.
+#
+# Link:
https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html
+
+inherit image
+
+IMAGE_NAME_SUFFIX = ".sysext"
+EXTENSION_NAME = "${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${IMAGE_FSTYPES}"
+IMAGE_LINK_NAME:append = ".sysext"
+
+DEPENDS += " os-release"
+
+sysext_image_mangle_rootfs() {
+R=${IMAGE_ROOTFS}
+
+# pull a copy of the rootfs version information, which systemd-sysext
matches against
+cp -av ${RECIPE_SYSROOT}/${nonarch_libdir}/os-release
${WORKDIR}/extension-release.base
+
+echo 'EXTENSION_RELOAD_MANAGER=1' >> ${WORKDIR}/extension-release.base
+
+install -d $R${nonarch_libdir}/extension-release.d
+install -m 0644 ${WORKDIR}/extension-release.base \
+
$R${nonarch_libdir}/extension-release.d/extension-release.${EXTENSION_NAME}
+
+# disable systemd-sysext's strict name checking, so that the image file
can be renamed, while still being 'merge'-able
+setfattr -n user.extension-release.strict -v false \
+
$R${nonarch_libdir}/extension-release.d/extension-release.${EXTENSION_NAME}
+}
+
+ROOTFS_POSTPROCESS_COMMAND += " sysext_image_mangle_rootfs; "
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#199190):
https://lists.openembedded.org/g/openembedded-core/message/199190
Mute This Topic: https://lists.openembedded.org/mt/106014870/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v7 2/3] image.bbclass/rootfs: set and unpack package-database
set the package-database of a "lower image" to unpack and build upon
when installing packages for the current image. This way a lean image
will be created, which only holds the packages that are not already
present in the lower image.
An image build such could then be used with overlayfs or systemd-
sysext to extend the "lower image" on demand; for development purposes
on a device running the "lower image" in RO mode for example.
A configuration could look as follows:
some-core-image.bb
inherit image
IMAGE_GEN_PKGDBFS = "1"
extending-image.bb
inherit image
IMAGE_BASE_PKGDB = "some-core-image"
Signed-off-by: Johannes Schneider
---
meta/classes-recipe/image.bbclass | 23 +++--
meta/conf/documentation.conf | 3 +-
meta/lib/oe/package_manager/deb/rootfs.py | 2 +
meta/lib/oe/package_manager/ipk/rootfs.py | 6 ++-
meta/lib/oe/package_manager/rpm/rootfs.py | 7 ++-
meta/lib/oe/rootfs.py | 18 +++
meta/lib/oeqa/selftest/cases/imagefeatures.py | 49 +++
7 files changed, 100 insertions(+), 8 deletions(-)
diff --git a/meta/classes-recipe/image.bbclass
b/meta/classes-recipe/image.bbclass
index 3ccaaa17b8..c573c37cd8 100644
--- a/meta/classes-recipe/image.bbclass
+++ b/meta/classes-recipe/image.bbclass
@@ -42,8 +42,16 @@ IMAGE_FEATURES ?= ""
IMAGE_FEATURES[type] = "list"
IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs
read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password
allow-empty-password allow-root-login serial-autologin-root
post-install-logging overlayfs-etc"
-# Generate snapshot of the package database?
+# Image layering:
+# a "base image" would create a snapshot of the package-database after the
+# installation of all packages into the rootfs is done. The next/other image
+# "layered on-top" of the former would then import that database and install
+# further packages; without reinstalling packages/dependencies that are already
+# installed in the layer below.
+# Set to '1' in a "base image" recipe, to preserve a snapshot of the package
database.
IMAGE_GEN_PKGDBFS ?= "0"
+# "PN" of a "base image", upon which the current image is to be built upon.
+IMAGE_BASE_PKGDB ?= ""
# Generate companion debugfs?
IMAGE_GEN_DEBUGFS ?= "0"
@@ -118,6 +126,15 @@ do_rootfs[depends] += " \
"
do_rootfs[recrdeptask] += "do_packagedata"
+python () {
+# make sure that the 'base image' has been queued in before this
+# image wants to unpack and build upon the formers pgkdb
+base_image = d.getVar('IMAGE_BASE_PKGDB')
+pn = d.getVar('PN')
+if base_image and base_image != pn:
+d.appendVarFlag("do_rootfs", 'depends', ' '+ base_image +
':do_image_complete')
+}
+
def rootfs_command_variables(d):
return
['ROOTFS_POSTPROCESS_COMMAND','ROOTFS_PREPROCESS_COMMAND','ROOTFS_POSTINSTALL_COMMAND','ROOTFS_POSTUNINSTALL_COMMAND','OPKG_PREPROCESS_COMMANDS','OPKG_POSTPROCESS_COMMANDS','IMAGE_POSTPROCESS_COMMAND',
'IMAGE_PREPROCESS_COMMAND','RPM_PREPROCESS_COMMANDS','RPM_POSTPROCESS_COMMANDS','DEB_PREPROCESS_COMMANDS','DEB_POSTPROCESS_COMMANDS']
@@ -134,8 +151,8 @@ def rootfs_variables(d):
'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
'IMAGE_LINGUAS_COMPLEMENTARY', 'IMAGE_LOCALES_ARCHIVE',
'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
- 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS', 'IMAGE_GEN_DEBUGFS',
'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY',
'REPRODUCIBLE_TIMESTAMP_ROOTFS',
- 'IMAGE_INSTALL_DEBUGFS']
+ 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS', 'IMAGE_BASE_PKGDB',
'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR',
'PACKAGE_EXCLUDE_COMPLEMENTARY',
+ 'REPRODUCIBLE_TIMESTAMP_ROOTFS', 'IMAGE_INSTALL_DEBUGFS']
variables.extend(rootfs_command_variables(d))
variables.extend(variable_depends(d))
return " ".join(variables)
diff --git a/meta/conf/documentation.conf b/meta/conf/documentation.conf
index 36aebb59ab..9f493cfe96 100644
--- a/meta/conf/documentation.conf
+++ b/meta/conf/documentation.conf
@@ -208,6 +208,7 @@ ICECC_PATH[doc] = "The location of the icecc binary."
ICECC_CLASS_DISABLE[doc] = "Identifies user classes that you do not want the
Icecream distributed compile support to consider."
ICECC_RECIPE_DISABLE[doc] = "Identifies user recipes that you do not want the
Icecream distributed compile support to consider."
ICECC_RECIPE_ENABLE[doc] = "Identifies user recipes that use a
[OE-core] [PATCH v7 1/3] image.bbclass/rootfs: archive and deploy package database
archive the package database after the rootfs has been put together as
*rootfs-pkdbfs.tar.gz, and put it into the deploy folder.
This creates a snapshot of the package mangers state at the point in
time when all dependencies have been resolved and installed; which
could be used by "extension images" to built upon.
Signed-off-by: Johannes Schneider
---
meta/classes-recipe/image.bbclass | 44 ++-
meta/classes-recipe/image_types.bbclass | 1 +
meta/conf/documentation.conf | 1 +
meta/lib/oe/package_manager/deb/rootfs.py | 1 +
meta/lib/oe/package_manager/ipk/rootfs.py | 1 +
meta/lib/oe/package_manager/rpm/rootfs.py | 1 +
meta/lib/oe/rootfs.py | 20 +++
meta/lib/oeqa/selftest/cases/imagefeatures.py | 54 +++
8 files changed, 122 insertions(+), 1 deletion(-)
diff --git a/meta/classes-recipe/image.bbclass
b/meta/classes-recipe/image.bbclass
index 28be6c6362..3ccaaa17b8 100644
--- a/meta/classes-recipe/image.bbclass
+++ b/meta/classes-recipe/image.bbclass
@@ -42,6 +42,9 @@ IMAGE_FEATURES ?= ""
IMAGE_FEATURES[type] = "list"
IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs
read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password
allow-empty-password allow-root-login serial-autologin-root
post-install-logging overlayfs-etc"
+# Generate snapshot of the package database?
+IMAGE_GEN_PKGDBFS ?= "0"
+
# Generate companion debugfs?
IMAGE_GEN_DEBUGFS ?= "0"
@@ -131,7 +134,8 @@ def rootfs_variables(d):
'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
'IMAGE_LINGUAS_COMPLEMENTARY', 'IMAGE_LOCALES_ARCHIVE',
'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
- 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED',
'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY',
'REPRODUCIBLE_TIMESTAMP_ROOTFS', 'IMAGE_INSTALL_DEBUGFS']
+ 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS', 'IMAGE_GEN_DEBUGFS',
'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY',
'REPRODUCIBLE_TIMESTAMP_ROOTFS',
+ 'IMAGE_INSTALL_DEBUGFS']
variables.extend(rootfs_command_variables(d))
variables.extend(variable_depends(d))
return " ".join(variables)
@@ -337,6 +341,17 @@ python do_image_qa_setscene () {
}
addtask do_image_qa_setscene
+def setup_pkgdbfs_variables(d):
+d.appendVar('IMAGE_ROOTFS', '-pkgdb')
+if d.getVar('IMAGE_LINK_NAME'):
+d.appendVar('IMAGE_LINK_NAME', '-pkgdb')
+d.appendVar('IMAGE_NAME','-pkgdb')
+d.setVar('IMAGE_FSTYPES', 'tar.gz')
+
+python setup_pkgdbfs () {
+setup_pkgdbfs_variables(d)
+}
+
def setup_debugfs_variables(d):
d.appendVar('IMAGE_ROOTFS', '-dbg')
if d.getVar('IMAGE_LINK_NAME'):
@@ -381,6 +396,11 @@ python () {
alltypes = d.getVar('IMAGE_FSTYPES').split()
typedeps = {}
+if d.getVar('IMAGE_GEN_PKGDBFS') == "1":
+pkgdbfs_fstypes = ['tar.gz']
+for t in pkgdbfs_fstypes:
+alltypes.append("pkgdbfs_" + t)
+
if d.getVar('IMAGE_GEN_DEBUGFS') == "1":
debugfs_fstypes = d.getVar('IMAGE_FSTYPES_DEBUGFS').split()
for t in debugfs_fstypes:
@@ -393,6 +413,10 @@ python () {
basetypes[baset]= []
if t not in basetypes[baset]:
basetypes[baset].append(t)
+pkgdb = ""
+if t.startswith("pkgdbfs_"):
+t = t[8:]
+pkgdb = "pkgdbfs_"
debug = ""
if t.startswith("debugfs_"):
t = t[8:]
@@ -401,6 +425,13 @@ python () {
vardeps.add('IMAGE_TYPEDEP:' + t)
if baset not in typedeps:
typedeps[baset] = set()
+deps = [pkgdb + dep for dep in deps]
+for dep in deps:
+if dep not in alltypes:
+alltypes.append(dep)
+_add_type(dep)
+basedep = _image_base_type(dep)
+typedeps[baset].add(basedep)
deps = [debug + dep for dep in deps]
for dep in deps:
if dep not in alltypes:
@@ -419,6 +450,7 @@ python () {
maskedtypes = (d.getVar('IMAGE_TYPES_MASKED') or "").split()
maskedtypes = [dbg + t for t in maskedtypes for dbg in ("", "debugfs_")]
+maskedtypes = [pkgdb + t for t in maskedtypes for pkgdb in ("",
"pkgdbfs_")]
for t in basetypes:
vardeps = set()
@@ -430,6 +462,11 @@ python () {
continue
localdata = bb.data.createCopy(d)
+pkgdb = ""
+if t.startswith("pkgdbfs_"):
+setup_pkgdbfs_v
Re: [OE-core] [PATCH v6 0/3] pkg-database and systemd-sysext image
derIdentification ] This email is not from Hexagon’s Office 365 instance. Please be careful while clicking links, opening attachments, or replying to this email. On 15/04/2024 08:02:00+0200, Johannes Schneider via lists.openembedded.org wrote: > systemd-sysext allows to overlay another image (or multiple) ontop of > a "base-image" = the current rootfs, via the use of overlayfs; to add > tools and features meant for development purposes. > > To quote the documentation on systemd-sysext: > " ...addition in order to make debugging/development easier). System > extension images should not be misunderstood as a generic software > packaging framework, ..." > > To build a lean image, that only holds packages that are not already > part of the base-image, a snapshot of the package-database is taken > after the installation of the base-rootfs is done, and picked up again > when collecting the rootfs of such a extension image. > > with all this in place an example usage could look like this: > some-core-image.bb > inherit core-image > IMAGE_GEN_PKGDBFS = "1" > > extending-image.bb > inherit image-sysext > IMAGE_FSTYPES = "squashfs" > IMAGE_BASE_PKGDB = "some-core-image" > # the above pointing at a package-db similar to: > # > build/deploy/images/$MACHINE/some-core-image-$MACHINE-20240210172305-pkgdb.rootfs.tar.gz > > then on the device, running some-core-image, with the extension image placed > at FN: > $> ln -s "$FN" /run/extensions/$(basename $FN).raw > $> systemd-sysext list > $> SYSTEMD_LOG_LEVEL=debug systemd-sysext merge > > As long as the VERSION_ID of the extension image matches the os-release > in the base image, the above commands return sucessfully; > for details on the compativility check see the docs for systemd-sysext. > > = > > changes with v2: > rebase from 'kirkstone' onto 'master' > > changes with v3: > incorporate review suggestions for simplification > add task dependency handling > add oe-selftest for the pkgdb handling > add variable documentation and > some more comments, and examples in the commit-msg > > changes with v4: > rebase onto 'master' => no functional changes > fixed patchtest findings > > changes with v5: > rebase onto 'master' > add '.sysext' to the deployed symlink name > sidenote on the tests and autobuilder failure: run locally they > succeed, e.g.: > #> oe-selftest --verbose -r > imagefeatures.ImageFeatures.test_image_gen_pkgdbfs > > changes with v6: > restructure test to call 'bitbake' only once in the testcase itself > (in hopes of solving the autobuilder problem; local test runs succeed) This still fails on the autobuilders: https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/6607/steps/14/logs/stdio Reached login banner in 3.26 seconds (04/18/24 18:54:41) Couldn't login into serial console as root using blank password The output: root Password: Login timed out after 60 seconds. Poky (Yocto Project Reference Distro) 5.0+snapshot-0ee872ff1baf43e5436ccbd4c9ae4d733289e05e qemux86-64 /dev/ttyS1 qemux86-64 login: <<< run_serial(): command timed out after 120 seconds without output >>> 2024-04-18 19:17:59,061 - oe-selftest - INFO - 6: 20/46 379/588 (131.80s) (0 failed) (imagefeatures.ImageFeatures.test_image_gen_pkgdbfs) 2024-04-18 19:17:59,062 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/imagefeatures.py", line 342, in test_image_gen_pkgdbfs self.assertTrue(os.path.exists(os.path.join(img_vars['DEPLOY_DIR_IMAGE'], 'var/lib/opkg/status')), 'opkg\'s status file was not present in: %s' % tar_file) File "/usr/lib/python3.9/unittest/case.py", line 682, in assertTrue raise self.failureException(msg) AssertionError: False is not true : opkg's status file was not present in: /home/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3638382/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.rootfs-pkgdb.tar.gz > > > -- Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com/ -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#198580): https://lists.openembedded.org/g/openembedded-core/message/198580 Mute This Topic: https://lists.openembedded.org/mt/105530718/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v6 0/3] pkg-database and systemd-sysext image
systemd-sysext allows to overlay another image (or multiple) ontop of a "base-image" = the current rootfs, via the use of overlayfs; to add tools and features meant for development purposes. To quote the documentation on systemd-sysext: " ...addition in order to make debugging/development easier). System extension images should not be misunderstood as a generic software packaging framework, ..." To build a lean image, that only holds packages that are not already part of the base-image, a snapshot of the package-database is taken after the installation of the base-rootfs is done, and picked up again when collecting the rootfs of such a extension image. with all this in place an example usage could look like this: some-core-image.bb inherit core-image IMAGE_GEN_PKGDBFS = "1" extending-image.bb inherit image-sysext IMAGE_FSTYPES = "squashfs" IMAGE_BASE_PKGDB = "some-core-image" # the above pointing at a package-db similar to: # build/deploy/images/$MACHINE/some-core-image-$MACHINE-20240210172305-pkgdb.rootfs.tar.gz then on the device, running some-core-image, with the extension image placed at FN: $> ln -s "$FN" /run/extensions/$(basename $FN).raw $> systemd-sysext list $> SYSTEMD_LOG_LEVEL=debug systemd-sysext merge As long as the VERSION_ID of the extension image matches the os-release in the base image, the above commands return sucessfully; for details on the compativility check see the docs for systemd-sysext. = changes with v2: rebase from 'kirkstone' onto 'master' changes with v3: incorporate review suggestions for simplification add task dependency handling add oe-selftest for the pkgdb handling add variable documentation and some more comments, and examples in the commit-msg changes with v4: rebase onto 'master' => no functional changes fixed patchtest findings changes with v5: rebase onto 'master' add '.sysext' to the deployed symlink name sidenote on the tests and autobuilder failure: run locally they succeed, e.g.: #> oe-selftest --verbose -r imagefeatures.ImageFeatures.test_image_gen_pkgdbfs changes with v6: restructure test to call 'bitbake' only once in the testcase itself (in hopes of solving the autobuilder problem; local test runs succeed) -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#198219): https://lists.openembedded.org/g/openembedded-core/message/198219 Mute This Topic: https://lists.openembedded.org/mt/105530718/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v6 2/3] image.bbclass/rootfs: set and unpack package-database
set the package-database of a "lower image" to unpack and build upon
when installing packages for the current image. This way a lean image
will be created, which only holds the packages that are not already
present in the lower image.
An image build such could then be used with overlayfs or systemd-
sysext to extend the "lower image" on demand; for development purposes
on a device running the "lower image" in RO mode for example.
A configuration could look as follows:
some-core-image.bb
inherit image
IMAGE_GEN_PKGDBFS = "1"
extending-image.bb
inherit image
IMAGE_BASE_PKGDB = "some-core-image"
Signed-off-by: Johannes Schneider
---
meta/classes-recipe/image.bbclass | 23 --
meta/conf/documentation.conf | 3 +-
meta/lib/oe/package_manager/deb/rootfs.py | 2 +
meta/lib/oe/package_manager/ipk/rootfs.py | 6 ++-
meta/lib/oe/package_manager/rpm/rootfs.py | 7 ++-
meta/lib/oe/rootfs.py | 18
meta/lib/oeqa/selftest/cases/imagefeatures.py | 46 +++
7 files changed, 97 insertions(+), 8 deletions(-)
diff --git a/meta/classes-recipe/image.bbclass
b/meta/classes-recipe/image.bbclass
index 3ccaaa17b8..c573c37cd8 100644
--- a/meta/classes-recipe/image.bbclass
+++ b/meta/classes-recipe/image.bbclass
@@ -42,8 +42,16 @@ IMAGE_FEATURES ?= ""
IMAGE_FEATURES[type] = "list"
IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs
read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password
allow-empty-password allow-root-login serial-autologin-root
post-install-logging overlayfs-etc"
-# Generate snapshot of the package database?
+# Image layering:
+# a "base image" would create a snapshot of the package-database after the
+# installation of all packages into the rootfs is done. The next/other image
+# "layered on-top" of the former would then import that database and install
+# further packages; without reinstalling packages/dependencies that are already
+# installed in the layer below.
+# Set to '1' in a "base image" recipe, to preserve a snapshot of the package
database.
IMAGE_GEN_PKGDBFS ?= "0"
+# "PN" of a "base image", upon which the current image is to be built upon.
+IMAGE_BASE_PKGDB ?= ""
# Generate companion debugfs?
IMAGE_GEN_DEBUGFS ?= "0"
@@ -118,6 +126,15 @@ do_rootfs[depends] += " \
"
do_rootfs[recrdeptask] += "do_packagedata"
+python () {
+# make sure that the 'base image' has been queued in before this
+# image wants to unpack and build upon the formers pgkdb
+base_image = d.getVar('IMAGE_BASE_PKGDB')
+pn = d.getVar('PN')
+if base_image and base_image != pn:
+d.appendVarFlag("do_rootfs", 'depends', ' '+ base_image +
':do_image_complete')
+}
+
def rootfs_command_variables(d):
return
['ROOTFS_POSTPROCESS_COMMAND','ROOTFS_PREPROCESS_COMMAND','ROOTFS_POSTINSTALL_COMMAND','ROOTFS_POSTUNINSTALL_COMMAND','OPKG_PREPROCESS_COMMANDS','OPKG_POSTPROCESS_COMMANDS','IMAGE_POSTPROCESS_COMMAND',
'IMAGE_PREPROCESS_COMMAND','RPM_PREPROCESS_COMMANDS','RPM_POSTPROCESS_COMMANDS','DEB_PREPROCESS_COMMANDS','DEB_POSTPROCESS_COMMANDS']
@@ -134,8 +151,8 @@ def rootfs_variables(d):
'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
'IMAGE_LINGUAS_COMPLEMENTARY', 'IMAGE_LOCALES_ARCHIVE',
'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
- 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS', 'IMAGE_GEN_DEBUGFS',
'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY',
'REPRODUCIBLE_TIMESTAMP_ROOTFS',
- 'IMAGE_INSTALL_DEBUGFS']
+ 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS', 'IMAGE_BASE_PKGDB',
'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR',
'PACKAGE_EXCLUDE_COMPLEMENTARY',
+ 'REPRODUCIBLE_TIMESTAMP_ROOTFS', 'IMAGE_INSTALL_DEBUGFS']
variables.extend(rootfs_command_variables(d))
variables.extend(variable_depends(d))
return " ".join(variables)
diff --git a/meta/conf/documentation.conf b/meta/conf/documentation.conf
index 36aebb59ab..9f493cfe96 100644
--- a/meta/conf/documentation.conf
+++ b/meta/conf/documentation.conf
@@ -208,6 +208,7 @@ ICECC_PATH[doc] = "The location of the icecc binary."
ICECC_CLASS_DISABLE[doc] = "Identifies user classes that you do not want the
Icecream distributed compile support to consider."
ICECC_RECIPE_DISABLE[doc] = "Identifies user recipes that you do not want the
Icecream distributed compile support to consider."
ICECC_RECIPE_ENABLE[doc] = "Identifies user recipes that use
[OE-core] [PATCH v6 1/3] image.bbclass/rootfs: archive and deploy package database
archive the package database after the rootfs has been put together as
*rootfs-pkdbfs.tar.gz, and put it into the deploy folder.
This creates a snapshot of the package mangers state at the point in
time when all dependencies have been resolved and installed; which
could be used by "extension images" to built upon.
Signed-off-by: Johannes Schneider
---
meta/classes-recipe/image.bbclass | 44 +-
meta/classes-recipe/image_types.bbclass | 1 +
meta/conf/documentation.conf | 1 +
meta/lib/oe/package_manager/deb/rootfs.py | 1 +
meta/lib/oe/package_manager/ipk/rootfs.py | 1 +
meta/lib/oe/package_manager/rpm/rootfs.py | 1 +
meta/lib/oe/rootfs.py | 20 +++
meta/lib/oeqa/selftest/cases/imagefeatures.py | 60 +++
8 files changed, 128 insertions(+), 1 deletion(-)
diff --git a/meta/classes-recipe/image.bbclass
b/meta/classes-recipe/image.bbclass
index 28be6c6362..3ccaaa17b8 100644
--- a/meta/classes-recipe/image.bbclass
+++ b/meta/classes-recipe/image.bbclass
@@ -42,6 +42,9 @@ IMAGE_FEATURES ?= ""
IMAGE_FEATURES[type] = "list"
IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs
read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password
allow-empty-password allow-root-login serial-autologin-root
post-install-logging overlayfs-etc"
+# Generate snapshot of the package database?
+IMAGE_GEN_PKGDBFS ?= "0"
+
# Generate companion debugfs?
IMAGE_GEN_DEBUGFS ?= "0"
@@ -131,7 +134,8 @@ def rootfs_variables(d):
'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
'IMAGE_LINGUAS_COMPLEMENTARY', 'IMAGE_LOCALES_ARCHIVE',
'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
- 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED',
'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY',
'REPRODUCIBLE_TIMESTAMP_ROOTFS', 'IMAGE_INSTALL_DEBUGFS']
+ 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS', 'IMAGE_GEN_DEBUGFS',
'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY',
'REPRODUCIBLE_TIMESTAMP_ROOTFS',
+ 'IMAGE_INSTALL_DEBUGFS']
variables.extend(rootfs_command_variables(d))
variables.extend(variable_depends(d))
return " ".join(variables)
@@ -337,6 +341,17 @@ python do_image_qa_setscene () {
}
addtask do_image_qa_setscene
+def setup_pkgdbfs_variables(d):
+d.appendVar('IMAGE_ROOTFS', '-pkgdb')
+if d.getVar('IMAGE_LINK_NAME'):
+d.appendVar('IMAGE_LINK_NAME', '-pkgdb')
+d.appendVar('IMAGE_NAME','-pkgdb')
+d.setVar('IMAGE_FSTYPES', 'tar.gz')
+
+python setup_pkgdbfs () {
+setup_pkgdbfs_variables(d)
+}
+
def setup_debugfs_variables(d):
d.appendVar('IMAGE_ROOTFS', '-dbg')
if d.getVar('IMAGE_LINK_NAME'):
@@ -381,6 +396,11 @@ python () {
alltypes = d.getVar('IMAGE_FSTYPES').split()
typedeps = {}
+if d.getVar('IMAGE_GEN_PKGDBFS') == "1":
+pkgdbfs_fstypes = ['tar.gz']
+for t in pkgdbfs_fstypes:
+alltypes.append("pkgdbfs_" + t)
+
if d.getVar('IMAGE_GEN_DEBUGFS') == "1":
debugfs_fstypes = d.getVar('IMAGE_FSTYPES_DEBUGFS').split()
for t in debugfs_fstypes:
@@ -393,6 +413,10 @@ python () {
basetypes[baset]= []
if t not in basetypes[baset]:
basetypes[baset].append(t)
+pkgdb = ""
+if t.startswith("pkgdbfs_"):
+t = t[8:]
+pkgdb = "pkgdbfs_"
debug = ""
if t.startswith("debugfs_"):
t = t[8:]
@@ -401,6 +425,13 @@ python () {
vardeps.add('IMAGE_TYPEDEP:' + t)
if baset not in typedeps:
typedeps[baset] = set()
+deps = [pkgdb + dep for dep in deps]
+for dep in deps:
+if dep not in alltypes:
+alltypes.append(dep)
+_add_type(dep)
+basedep = _image_base_type(dep)
+typedeps[baset].add(basedep)
deps = [debug + dep for dep in deps]
for dep in deps:
if dep not in alltypes:
@@ -419,6 +450,7 @@ python () {
maskedtypes = (d.getVar('IMAGE_TYPES_MASKED') or "").split()
maskedtypes = [dbg + t for t in maskedtypes for dbg in ("", "debugfs_")]
+maskedtypes = [pkgdb + t for t in maskedtypes for pkgdb in ("",
"pkgdbfs_")]
for t in basetypes:
vardeps = set()
@@ -430,6 +462,11 @@ python () {
continue
localdata = bb.data.createCopy(d)
+pkgdb = ""
+if t.startswith("pkgdbfs_"):
+setup_pkgdbfs_va
[OE-core] [PATCH v6 3/3] classes: add a systemd-sysext image class
systemd-sysext can load a raw-image containing usr/ and opt/ folders
to mount them as RO overlay over the rootfs, to "extend" the systems.
This class provides the necessary changes/additions to the enclosed
file-system so that systemd-sysext accepts the extension for "merge"
into the rootfs.
With such an created image, placed into the correct folder (see [1]),
`systemd-sysext list` should be able to list the "extension" and
`systemd-sysext merge` should enable the overlay. On both commands a
preceding "SYSTEMD_LOG_LEVEL=debug" can aide in figuring out what is
amiss.
The strict name checking systemd-sysext does against the name of
extension-release.NAME file, is disabled, as there is only one such in
the resulting image. This is done to allow a user to freely rename the
resulting image file.
Note that for e.g. squashfs, the kernel needs CONFIG_SQUASHFS_XATTR=y
Link:
https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html
Link:
https://0pointer.net/blog/testing-my-system-code-in-usr-without-modifying-usr.html
Signed-off-by: Johannes Schneider
---
meta/classes-recipe/image-sysext.bbclass | 43
1 file changed, 43 insertions(+)
create mode 100644 meta/classes-recipe/image-sysext.bbclass
diff --git a/meta/classes-recipe/image-sysext.bbclass
b/meta/classes-recipe/image-sysext.bbclass
new file mode 100644
index 00..bc3e4d52b5
--- /dev/null
+++ b/meta/classes-recipe/image-sysext.bbclass
@@ -0,0 +1,43 @@
+# SPDX-License-Identifier: MIT
+#
+# Copyright Leica Geosystems AG
+#
+
+# systemd-sysext [1] has a simple mechanism for version compatibility:
+# the extension to be loaded has to contain a
+# /usr/lib/extension-release.d/extension-release.NAME
+# with "NAME" *exactly* matching the filename of the extensions
+# raw-device filename/
+#
+# from the extension-release file the "ID" and "VERSION_ID" fields are
+# matched against the etc/os-release and the extension is only "merged"
+# if no mismatches between NAME, ID, and VERSION_ID.
+#
+# Link:
https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html
+
+inherit image
+
+IMAGE_NAME_SUFFIX = ".sysext"
+EXTENSION_NAME = "${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${IMAGE_FSTYPES}"
+IMAGE_LINK_NAME:append = ".sysext"
+
+DEPENDS += " os-release"
+
+sysext_image_mangle_rootfs() {
+R=${IMAGE_ROOTFS}
+
+# pull a copy of the rootfs version information, which systemd-sysext
matches against
+cp -av ${RECIPE_SYSROOT}/${nonarch_libdir}/os-release
${WORKDIR}/extension-release.base
+
+echo 'EXTENSION_RELOAD_MANAGER=1' >> ${WORKDIR}/extension-release.base
+
+install -d $R${nonarch_libdir}/extension-release.d
+install -m 0644 ${WORKDIR}/extension-release.base \
+
$R${nonarch_libdir}/extension-release.d/extension-release.${EXTENSION_NAME}
+
+# disable systemd-sysext's strict name checking, so that the image file
can be renamed, while still being 'merge'-able
+setfattr -n user.extension-release.strict -v false \
+
$R${nonarch_libdir}/extension-release.d/extension-release.${EXTENSION_NAME}
+}
+
+ROOTFS_POSTPROCESS_COMMAND += " sysext_image_mangle_rootfs; "
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#198217):
https://lists.openembedded.org/g/openembedded-core/message/198217
Mute This Topic: https://lists.openembedded.org/mt/105530716/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v5 1/3] image.bbclass/rootfs: archive and deploy package database
archive the package database after the rootfs has been put together as
*rootfs-pkdbfs.tar.gz, and put it into the deploy folder.
This creates a snapshot of the package mangers state at the point in
time when all dependencies have been resolved and installed; which
could be used by "extension images" to built upon.
Signed-off-by: Johannes Schneider
---
meta/classes-recipe/image.bbclass | 44 +++-
meta/classes-recipe/image_types.bbclass | 1 +
meta/conf/documentation.conf | 1 +
meta/lib/oe/package_manager/deb/rootfs.py | 1 +
meta/lib/oe/package_manager/ipk/rootfs.py | 1 +
meta/lib/oe/package_manager/rpm/rootfs.py | 1 +
meta/lib/oe/rootfs.py | 20
meta/lib/oeqa/selftest/cases/imagefeatures.py | 51 +++
8 files changed, 119 insertions(+), 1 deletion(-)
diff --git a/meta/classes-recipe/image.bbclass
b/meta/classes-recipe/image.bbclass
index 28be6c6362..3ccaaa17b8 100644
--- a/meta/classes-recipe/image.bbclass
+++ b/meta/classes-recipe/image.bbclass
@@ -42,6 +42,9 @@ IMAGE_FEATURES ?= ""
IMAGE_FEATURES[type] = "list"
IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs
read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password
allow-empty-password allow-root-login serial-autologin-root
post-install-logging overlayfs-etc"
+# Generate snapshot of the package database?
+IMAGE_GEN_PKGDBFS ?= "0"
+
# Generate companion debugfs?
IMAGE_GEN_DEBUGFS ?= "0"
@@ -131,7 +134,8 @@ def rootfs_variables(d):
'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
'IMAGE_LINGUAS_COMPLEMENTARY', 'IMAGE_LOCALES_ARCHIVE',
'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
- 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED',
'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY',
'REPRODUCIBLE_TIMESTAMP_ROOTFS', 'IMAGE_INSTALL_DEBUGFS']
+ 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS', 'IMAGE_GEN_DEBUGFS',
'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY',
'REPRODUCIBLE_TIMESTAMP_ROOTFS',
+ 'IMAGE_INSTALL_DEBUGFS']
variables.extend(rootfs_command_variables(d))
variables.extend(variable_depends(d))
return " ".join(variables)
@@ -337,6 +341,17 @@ python do_image_qa_setscene () {
}
addtask do_image_qa_setscene
+def setup_pkgdbfs_variables(d):
+d.appendVar('IMAGE_ROOTFS', '-pkgdb')
+if d.getVar('IMAGE_LINK_NAME'):
+d.appendVar('IMAGE_LINK_NAME', '-pkgdb')
+d.appendVar('IMAGE_NAME','-pkgdb')
+d.setVar('IMAGE_FSTYPES', 'tar.gz')
+
+python setup_pkgdbfs () {
+setup_pkgdbfs_variables(d)
+}
+
def setup_debugfs_variables(d):
d.appendVar('IMAGE_ROOTFS', '-dbg')
if d.getVar('IMAGE_LINK_NAME'):
@@ -381,6 +396,11 @@ python () {
alltypes = d.getVar('IMAGE_FSTYPES').split()
typedeps = {}
+if d.getVar('IMAGE_GEN_PKGDBFS') == "1":
+pkgdbfs_fstypes = ['tar.gz']
+for t in pkgdbfs_fstypes:
+alltypes.append("pkgdbfs_" + t)
+
if d.getVar('IMAGE_GEN_DEBUGFS') == "1":
debugfs_fstypes = d.getVar('IMAGE_FSTYPES_DEBUGFS').split()
for t in debugfs_fstypes:
@@ -393,6 +413,10 @@ python () {
basetypes[baset]= []
if t not in basetypes[baset]:
basetypes[baset].append(t)
+pkgdb = ""
+if t.startswith("pkgdbfs_"):
+t = t[8:]
+pkgdb = "pkgdbfs_"
debug = ""
if t.startswith("debugfs_"):
t = t[8:]
@@ -401,6 +425,13 @@ python () {
vardeps.add('IMAGE_TYPEDEP:' + t)
if baset not in typedeps:
typedeps[baset] = set()
+deps = [pkgdb + dep for dep in deps]
+for dep in deps:
+if dep not in alltypes:
+alltypes.append(dep)
+_add_type(dep)
+basedep = _image_base_type(dep)
+typedeps[baset].add(basedep)
deps = [debug + dep for dep in deps]
for dep in deps:
if dep not in alltypes:
@@ -419,6 +450,7 @@ python () {
maskedtypes = (d.getVar('IMAGE_TYPES_MASKED') or "").split()
maskedtypes = [dbg + t for t in maskedtypes for dbg in ("", "debugfs_")]
+maskedtypes = [pkgdb + t for t in maskedtypes for pkgdb in ("",
"pkgdbfs_")]
for t in basetypes:
vardeps = set()
@@ -430,6 +462,11 @@ python () {
continue
localdata = bb.data.createCopy(d)
+pkgdb = ""
+if t.startswith("pkgdbfs_"):
+setup_pkgdbfs
[OE-core] [PATCH v5 3/3] classes: add a systemd-sysext image class
systemd-sysext can load a raw-image containing usr/ and opt/ folders
to mount them as RO overlay over the rootfs, to "extend" the systems.
This class provides the necessary changes/additions to the enclosed
file-system so that systemd-sysext accepts the extension for "merge"
into the rootfs.
With such an created image, placed into the correct folder (see [1]),
`systemd-sysext list` should be able to list the "extension" and
`systemd-sysext merge` should enable the overlay. On both commands a
preceding "SYSTEMD_LOG_LEVEL=debug" can aide in figuring out what is
amiss.
The strict name checking systemd-sysext does against the name of
extension-release.NAME file, is disabled, as there is only one such in
the resulting image. This is done to allow a user to freely rename the
resulting image file.
Note that for e.g. squashfs, the kernel needs CONFIG_SQUASHFS_XATTR=y
Link:
https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html
Link:
https://0pointer.net/blog/testing-my-system-code-in-usr-without-modifying-usr.html
Signed-off-by: Johannes Schneider
---
meta/classes-recipe/image-sysext.bbclass | 43
1 file changed, 43 insertions(+)
create mode 100644 meta/classes-recipe/image-sysext.bbclass
diff --git a/meta/classes-recipe/image-sysext.bbclass
b/meta/classes-recipe/image-sysext.bbclass
new file mode 100644
index 00..bc3e4d52b5
--- /dev/null
+++ b/meta/classes-recipe/image-sysext.bbclass
@@ -0,0 +1,43 @@
+# SPDX-License-Identifier: MIT
+#
+# Copyright Leica Geosystems AG
+#
+
+# systemd-sysext [1] has a simple mechanism for version compatibility:
+# the extension to be loaded has to contain a
+# /usr/lib/extension-release.d/extension-release.NAME
+# with "NAME" *exactly* matching the filename of the extensions
+# raw-device filename/
+#
+# from the extension-release file the "ID" and "VERSION_ID" fields are
+# matched against the etc/os-release and the extension is only "merged"
+# if no mismatches between NAME, ID, and VERSION_ID.
+#
+# Link:
https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html
+
+inherit image
+
+IMAGE_NAME_SUFFIX = ".sysext"
+EXTENSION_NAME = "${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${IMAGE_FSTYPES}"
+IMAGE_LINK_NAME:append = ".sysext"
+
+DEPENDS += " os-release"
+
+sysext_image_mangle_rootfs() {
+R=${IMAGE_ROOTFS}
+
+# pull a copy of the rootfs version information, which systemd-sysext
matches against
+cp -av ${RECIPE_SYSROOT}/${nonarch_libdir}/os-release
${WORKDIR}/extension-release.base
+
+echo 'EXTENSION_RELOAD_MANAGER=1' >> ${WORKDIR}/extension-release.base
+
+install -d $R${nonarch_libdir}/extension-release.d
+install -m 0644 ${WORKDIR}/extension-release.base \
+
$R${nonarch_libdir}/extension-release.d/extension-release.${EXTENSION_NAME}
+
+# disable systemd-sysext's strict name checking, so that the image file
can be renamed, while still being 'merge'-able
+setfattr -n user.extension-release.strict -v false \
+
$R${nonarch_libdir}/extension-release.d/extension-release.${EXTENSION_NAME}
+}
+
+ROOTFS_POSTPROCESS_COMMAND += " sysext_image_mangle_rootfs; "
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#198083):
https://lists.openembedded.org/g/openembedded-core/message/198083
Mute This Topic: https://lists.openembedded.org/mt/105437787/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v5 2/3] image.bbclass/rootfs: set and unpack package-database
set the package-database of a "lower image" to unpack and build upon
when installing packages for the current image. This way a lean image
will be created, which only holds the packages that are not already
present in the lower image.
An image build such could then be used with overlayfs or systemd-
sysext to extend the "lower image" on demand; for development purposes
on a device running the "lower image" in RO mode for example.
A configuration could look as follows:
some-core-image.bb
inherit image
IMAGE_GEN_PKGDBFS = "1"
extending-image.bb
inherit image
IMAGE_BASE_PKGDB = "some-core-image"
Signed-off-by: Johannes Schneider
---
meta/classes-recipe/image.bbclass | 23 --
meta/conf/documentation.conf | 3 +-
meta/lib/oe/package_manager/deb/rootfs.py | 2 +
meta/lib/oe/package_manager/ipk/rootfs.py | 6 ++-
meta/lib/oe/package_manager/rpm/rootfs.py | 7 ++-
meta/lib/oe/rootfs.py | 18
meta/lib/oeqa/selftest/cases/imagefeatures.py | 46 +++
7 files changed, 97 insertions(+), 8 deletions(-)
diff --git a/meta/classes-recipe/image.bbclass
b/meta/classes-recipe/image.bbclass
index 3ccaaa17b8..c573c37cd8 100644
--- a/meta/classes-recipe/image.bbclass
+++ b/meta/classes-recipe/image.bbclass
@@ -42,8 +42,16 @@ IMAGE_FEATURES ?= ""
IMAGE_FEATURES[type] = "list"
IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs
read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password
allow-empty-password allow-root-login serial-autologin-root
post-install-logging overlayfs-etc"
-# Generate snapshot of the package database?
+# Image layering:
+# a "base image" would create a snapshot of the package-database after the
+# installation of all packages into the rootfs is done. The next/other image
+# "layered on-top" of the former would then import that database and install
+# further packages; without reinstalling packages/dependencies that are already
+# installed in the layer below.
+# Set to '1' in a "base image" recipe, to preserve a snapshot of the package
database.
IMAGE_GEN_PKGDBFS ?= "0"
+# "PN" of a "base image", upon which the current image is to be built upon.
+IMAGE_BASE_PKGDB ?= ""
# Generate companion debugfs?
IMAGE_GEN_DEBUGFS ?= "0"
@@ -118,6 +126,15 @@ do_rootfs[depends] += " \
"
do_rootfs[recrdeptask] += "do_packagedata"
+python () {
+# make sure that the 'base image' has been queued in before this
+# image wants to unpack and build upon the formers pgkdb
+base_image = d.getVar('IMAGE_BASE_PKGDB')
+pn = d.getVar('PN')
+if base_image and base_image != pn:
+d.appendVarFlag("do_rootfs", 'depends', ' '+ base_image +
':do_image_complete')
+}
+
def rootfs_command_variables(d):
return
['ROOTFS_POSTPROCESS_COMMAND','ROOTFS_PREPROCESS_COMMAND','ROOTFS_POSTINSTALL_COMMAND','ROOTFS_POSTUNINSTALL_COMMAND','OPKG_PREPROCESS_COMMANDS','OPKG_POSTPROCESS_COMMANDS','IMAGE_POSTPROCESS_COMMAND',
'IMAGE_PREPROCESS_COMMAND','RPM_PREPROCESS_COMMANDS','RPM_POSTPROCESS_COMMANDS','DEB_PREPROCESS_COMMANDS','DEB_POSTPROCESS_COMMANDS']
@@ -134,8 +151,8 @@ def rootfs_variables(d):
'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
'IMAGE_LINGUAS_COMPLEMENTARY', 'IMAGE_LOCALES_ARCHIVE',
'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
- 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS', 'IMAGE_GEN_DEBUGFS',
'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY',
'REPRODUCIBLE_TIMESTAMP_ROOTFS',
- 'IMAGE_INSTALL_DEBUGFS']
+ 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS', 'IMAGE_BASE_PKGDB',
'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR',
'PACKAGE_EXCLUDE_COMPLEMENTARY',
+ 'REPRODUCIBLE_TIMESTAMP_ROOTFS', 'IMAGE_INSTALL_DEBUGFS']
variables.extend(rootfs_command_variables(d))
variables.extend(variable_depends(d))
return " ".join(variables)
diff --git a/meta/conf/documentation.conf b/meta/conf/documentation.conf
index 36aebb59ab..9f493cfe96 100644
--- a/meta/conf/documentation.conf
+++ b/meta/conf/documentation.conf
@@ -208,6 +208,7 @@ ICECC_PATH[doc] = "The location of the icecc binary."
ICECC_CLASS_DISABLE[doc] = "Identifies user classes that you do not want the
Icecream distributed compile support to consider."
ICECC_RECIPE_DISABLE[doc] = "Identifies user recipes that you do not want the
Icecream distributed compile support to consider."
ICECC_RECIPE_ENABLE[doc] = "Identifies user recipes that use
[OE-core] [PATCH v5 0/3] pkg-database and systemd-sysext image
systemd-sysext allows to overlay another image (or multiple) ontop of a "base-image" = the current rootfs, via the use of overlayfs; to add tools and features meant for development purposes. To quote the documentation on systemd-sysext: " ...addition in order to make debugging/development easier). System extension images should not be misunderstood as a generic software packaging framework, ..." To build a lean image, that only holds packages that are not already part of the base-image, a snapshot of the package-database is taken after the installation of the base-rootfs is done, and picked up again when collecting the rootfs of such a extension image. with all this in place an example usage could look like this: some-core-image.bb inherit core-image IMAGE_GEN_PKGDBFS = "1" extending-image.bb inherit image-sysext IMAGE_FSTYPES = "squashfs" IMAGE_BASE_PKGDB = "some-core-image" # the above pointing at a package-db similar to: # build/deploy/images/$MACHINE/some-core-image-$MACHINE-20240210172305-pkgdb.rootfs.tar.gz then on the device, running some-core-image, with the extension image placed at FN: $> ln -s "$FN" /run/extensions/$(basename $FN).raw $> systemd-sysext list $> SYSTEMD_LOG_LEVEL=debug systemd-sysext merge As long as the VERSION_ID of the extension image matches the os-release in the base image, the above commands return sucessfully; for details on the compativility check see the docs for systemd-sysext. = changes with v2: rebase from 'kirkstone' onto 'master' changes with v3: incorporate review suggestions for simplification add task dependency handling add oe-selftest for the pkgdb handling add variable documentation and some more comments, and examples in the commit-msg changes with v4: rebase onto 'master' => no functional changes fixed patchtest findings changes with v5: rebase onto 'master' add '.sysext' to the deployed symlink name sidenote on the tests and autobuilder failure: run locally they succeed, e.g.: #> oe-selftest --verbose -r imagefeatures.ImageFeatures.test_image_gen_pkgdbfs -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#198080): https://lists.openembedded.org/g/openembedded-core/message/198080 Mute This Topic: https://lists.openembedded.org/mt/105437784/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v4 0/3] pkg-database and systemd-sysext image
Hoi, thanks for pointing the build failure out, Im trying to reproduce it locally but fail so far to see the problem :-S rebased on the latest master, when the two selftests added with the patches are run selecively, they pass: #> oe-selftest -r imagefeatures.ImageFeatures.test_image_gen_pkgdbfs imagefeatures.ImageFeatures.test_image_use_pkgdbfs but running the whole test-routine, same way the buildserver does in that log, currently fails me for other reasons (lack of resources? or i might be calling it the wrong way) any hints/ideas/pointers would be appreciated :-) gruß Johannes From: Alexandre Belloni Sent: Saturday, March 23, 2024 14:40 To: SCHNEIDER Johannes Cc: openembedded-core@lists.openembedded.org; richard.pur...@linuxfoundation.org Subject: Re: [OE-core] [PATCH v4 0/3] pkg-database and systemd-sysext image [You don't often get email from alexandre.bell...@bootlin.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] This email is not from Hexagon’s Office 365 instance. Please be careful while clicking links, opening attachments, or replying to this email. Hello, The series causes failures on the autobuilders: https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6548/steps/14/logs/stdio 2024-03-21 21:05:49,848 - oe-selftest - INFO - 6: 20/46 500/587 (137.73s) (0 failed) (imagefeatures.ImageFeatures.test_image_gen_pkgdbfs) 2024-03-21 21:05:49,848 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/imagefeatures.py", line 327, in test_image_gen_pkgdbfs self.assertTrue(os.path.exists(os.path.join(img_vars['DEPLOY_DIR_IMAGE'], 'var/lib/opkg/status')), 'opkg\'s status file was not present in: %s' % tar_file) File "/usr/lib64/python3.9/unittest/case.py", line 688, in assertTrue raise self.failureException(msg) AssertionError: False is not true : opkg's status file was not present in: /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-3332915/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.rootfs-pkgdb.tar.gz On 04/03/2024 07:15:50+0100, Johannes Schneider via lists.openembedded.org wrote: > systemd-sysext allows to overlay another image (or multiple) ontop of > a "base-image" = the current rootfs, via the use of overlayfs; to add > tools and features meant for development purposes. > > To quote the documentation on systemd-sysext: > " ...addition in order to make debugging/development easier). System > extension images should not be misunderstood as a generic software > packaging framework, ..." > > To build a lean image, that only holds packages that are not already > part of the base-image, a snapshot of the package-database is taken > after the installation of the base-rootfs is done, and picked up again > when collecting the rootfs of such a extension image. > > with all this in place an example usage could look like this: > some-core-image.bb > inherit core-image > IMAGE_GEN_PKGDBFS = "1" > > extending-image.bb > inherit image-sysext > IMAGE_FSTYPES = "squashfs" > IMAGE_BASE_PKGDB = "some-core-image" > # the above pointing at a package-db similar to: > # > build/deploy/images/$MACHINE/some-core-image-$MACHINE-20240210172305-pkgdb.rootfs.tar.gz > > then on the device, running some-core-image, with the extension image placed > at FN: > $> ln -s "$FN" /run/extensions/$(basename $FN).raw > $> systemd-sysext list > $> SYSTEMD_LOG_LEVEL=debug systemd-sysext merge > > As long as the VERSION_ID of the extension image matches the os-release > in the base image, the above commands return sucessfully; > for details on the compativility check see the docs for systemd-sysext. > > = > > changes with v2: > rebase from 'kirkstone' onto 'master' > > changes with v3; > incorporate review suggestions for simplification > add task dependency handling > add oe-selftest for the pkgdb handling > add variable documentation and > some more comments, and examples in the commit-msg > > changes with v4; > rebase onto 'master' => no functional changes > fixed patchtest findings > > > -- Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com/ -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#197485): https://lists.openembedded.org/g/openembedded-core/message/197485 Mute This Topic: https://lists.openembedded.org/mt/104717642/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v4 0/3] pkg-database and systemd-sysext image
Hoi Alex, thanks for the heads up - i'll patiently wait then :-) gruß Johannes From: Alexander Kanavin Sent: Monday, March 4, 2024 10:25 To: SCHNEIDER Johannes Cc: openembedded-core@lists.openembedded.org; richard.pur...@linuxfoundation.org Subject: Re: [OE-core] [PATCH v4 0/3] pkg-database and systemd-sysext image [You don't often get email from alex.kana...@gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] This email is not from Hexagon’s Office 365 instance. Please be careful while clicking links, opening attachments, or replying to this email. There is a feature freeze now, so this might have to wait for after that. Alex On Mon, 4 Mar 2024 at 07:16, Johannes Schneider via lists.openembedded.org wrote: > > systemd-sysext allows to overlay another image (or multiple) ontop of > a "base-image" = the current rootfs, via the use of overlayfs; to add > tools and features meant for development purposes. > > To quote the documentation on systemd-sysext: > " ...addition in order to make debugging/development easier). System > extension images should not be misunderstood as a generic software > packaging framework, ..." > > To build a lean image, that only holds packages that are not already > part of the base-image, a snapshot of the package-database is taken > after the installation of the base-rootfs is done, and picked up again > when collecting the rootfs of such a extension image. > > with all this in place an example usage could look like this: > some-core-image.bb > inherit core-image > IMAGE_GEN_PKGDBFS = "1" > > extending-image.bb > inherit image-sysext > IMAGE_FSTYPES = "squashfs" > IMAGE_BASE_PKGDB = "some-core-image" > # the above pointing at a package-db similar to: > # > build/deploy/images/$MACHINE/some-core-image-$MACHINE-20240210172305-pkgdb.rootfs.tar.gz > > then on the device, running some-core-image, with the extension image placed > at FN: > $> ln -s "$FN" /run/extensions/$(basename $FN).raw > $> systemd-sysext list > $> SYSTEMD_LOG_LEVEL=debug systemd-sysext merge > > As long as the VERSION_ID of the extension image matches the os-release > in the base image, the above commands return sucessfully; > for details on the compativility check see the docs for systemd-sysext. > > = > > changes with v2: > rebase from 'kirkstone' onto 'master' > > changes with v3; > incorporate review suggestions for simplification > add task dependency handling > add oe-selftest for the pkgdb handling > add variable documentation and > some more comments, and examples in the commit-msg > > changes with v4; > rebase onto 'master' => no functional changes > fixed patchtest findings > > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#196593): https://lists.openembedded.org/g/openembedded-core/message/196593 Mute This Topic: https://lists.openembedded.org/mt/104717642/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v4 0/3] pkg-database and systemd-sysext image
systemd-sysext allows to overlay another image (or multiple) ontop of a "base-image" = the current rootfs, via the use of overlayfs; to add tools and features meant for development purposes. To quote the documentation on systemd-sysext: " ...addition in order to make debugging/development easier). System extension images should not be misunderstood as a generic software packaging framework, ..." To build a lean image, that only holds packages that are not already part of the base-image, a snapshot of the package-database is taken after the installation of the base-rootfs is done, and picked up again when collecting the rootfs of such a extension image. with all this in place an example usage could look like this: some-core-image.bb inherit core-image IMAGE_GEN_PKGDBFS = "1" extending-image.bb inherit image-sysext IMAGE_FSTYPES = "squashfs" IMAGE_BASE_PKGDB = "some-core-image" # the above pointing at a package-db similar to: # build/deploy/images/$MACHINE/some-core-image-$MACHINE-20240210172305-pkgdb.rootfs.tar.gz then on the device, running some-core-image, with the extension image placed at FN: $> ln -s "$FN" /run/extensions/$(basename $FN).raw $> systemd-sysext list $> SYSTEMD_LOG_LEVEL=debug systemd-sysext merge As long as the VERSION_ID of the extension image matches the os-release in the base image, the above commands return sucessfully; for details on the compativility check see the docs for systemd-sysext. = changes with v2: rebase from 'kirkstone' onto 'master' changes with v3; incorporate review suggestions for simplification add task dependency handling add oe-selftest for the pkgdb handling add variable documentation and some more comments, and examples in the commit-msg changes with v4; rebase onto 'master' => no functional changes fixed patchtest findings -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#196588): https://lists.openembedded.org/g/openembedded-core/message/196588 Mute This Topic: https://lists.openembedded.org/mt/104717642/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v4 3/3] classes: add a systemd-sysext image class
systemd-sysext can load a raw-image containing usr/ and opt/ folders
to mount them as RO overlay over the rootfs, to "extend" the systems.
This class provides the necessary changes/additions to the enclosed
file-system so that systemd-sysext accepts the extension for "merge"
into the rootfs.
With such an created image, placed into the correct folder (see [1]),
`systemd-sysext list` should be able to list the "extension" and
`systemd-sysext merge` should enable the overlay. On both commands a
preceding "SYSTEMD_LOG_LEVEL=debug" can aide in figuring out what is
amiss.
The strict name checking systemd-sysext does against the name of
extension-release.NAME file, is disabled, as there is only one such in
the resulting image. This is done to allow a user to freely rename the
resulting image file.
Note that for e.g. squashfs, the kernel needs CONFIG_SQUASHFS_XATTR=y
Link:
https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html
Link:
https://0pointer.net/blog/testing-my-system-code-in-usr-without-modifying-usr.html
Signed-off-by: Johannes Schneider
---
meta/classes-recipe/image-sysext.bbclass | 42
1 file changed, 42 insertions(+)
create mode 100644 meta/classes-recipe/image-sysext.bbclass
diff --git a/meta/classes-recipe/image-sysext.bbclass
b/meta/classes-recipe/image-sysext.bbclass
new file mode 100644
index 00..012db83353
--- /dev/null
+++ b/meta/classes-recipe/image-sysext.bbclass
@@ -0,0 +1,42 @@
+# SPDX-License-Identifier: MIT
+#
+# Copyright Leica Geosystems AG
+#
+
+# systemd-sysext [1] has a simple mechanism for version compatibility:
+# the extension to be loaded has to contain a
+# /usr/lib/extension-release.d/extension-release.NAME
+# with "NAME" *exactly* matching the filename of the extensions
+# raw-device filename/
+#
+# from the extension-release file the "ID" and "VERSION_ID" fields are
+# matched against the etc/os-release and the extension is only "merged"
+# if no mismatches between NAME, ID, and VERSION_ID.
+#
+# Link:
https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html
+
+inherit image
+
+IMAGE_NAME_SUFFIX = ".sysext"
+EXTENSION_NAME = "${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${IMAGE_FSTYPES}"
+
+DEPENDS += " os-release"
+
+sysext_image_mangle_rootfs() {
+R=${IMAGE_ROOTFS}
+
+# pull a copy of the rootfs version information, which systemd-sysext
matches against
+cp -av ${RECIPE_SYSROOT}/${nonarch_libdir}/os-release
${WORKDIR}/extension-release.base
+
+echo 'EXTENSION_RELOAD_MANAGER=1' >> ${WORKDIR}/extension-release.base
+
+install -d $R${nonarch_libdir}/extension-release.d
+install -m 0644 ${WORKDIR}/extension-release.base \
+
$R${nonarch_libdir}/extension-release.d/extension-release.${EXTENSION_NAME}
+
+# disable systemd-sysext's strict name checking, so that the image file
can be renamed, while still being 'merge'-able
+setfattr -n user.extension-release.strict -v false \
+
$R${nonarch_libdir}/extension-release.d/extension-release.${EXTENSION_NAME}
+}
+
+ROOTFS_POSTPROCESS_COMMAND += " sysext_image_mangle_rootfs; "
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#196589):
https://lists.openembedded.org/g/openembedded-core/message/196589
Mute This Topic: https://lists.openembedded.org/mt/104717643/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v4 2/3] image.bbclass/rootfs: set and unpack package-database
set the package-database of a "lower image" to unpack and build upon
when installing packages for the current image. This way a lean image
will be created, which only holds the packages that are not already
present in the lower image.
An image build such could then be used with overlayfs or systemd-
sysext to extend the "lower image" on demand; for development purposes
on a device running the "lower image" in RO mode for example.
A configuration could look as follows:
some-core-image.bb
inherit image
IMAGE_GEN_PKGDBFS = "1"
extending-image.bb
inherit image
IMAGE_BASE_PKGDB = "some-core-image"
Signed-off-by: Johannes Schneider
---
meta/classes-recipe/image.bbclass | 23 --
meta/conf/documentation.conf | 3 +-
meta/lib/oe/package_manager/deb/rootfs.py | 2 +
meta/lib/oe/package_manager/ipk/rootfs.py | 6 ++-
meta/lib/oe/package_manager/rpm/rootfs.py | 7 ++-
meta/lib/oe/rootfs.py | 18
meta/lib/oeqa/selftest/cases/imagefeatures.py | 46 +++
7 files changed, 97 insertions(+), 8 deletions(-)
diff --git a/meta/classes-recipe/image.bbclass
b/meta/classes-recipe/image.bbclass
index 3ccaaa17b8..c573c37cd8 100644
--- a/meta/classes-recipe/image.bbclass
+++ b/meta/classes-recipe/image.bbclass
@@ -42,8 +42,16 @@ IMAGE_FEATURES ?= ""
IMAGE_FEATURES[type] = "list"
IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs
read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password
allow-empty-password allow-root-login serial-autologin-root
post-install-logging overlayfs-etc"
-# Generate snapshot of the package database?
+# Image layering:
+# a "base image" would create a snapshot of the package-database after the
+# installation of all packages into the rootfs is done. The next/other image
+# "layered on-top" of the former would then import that database and install
+# further packages; without reinstalling packages/dependencies that are already
+# installed in the layer below.
+# Set to '1' in a "base image" recipe, to preserve a snapshot of the package
database.
IMAGE_GEN_PKGDBFS ?= "0"
+# "PN" of a "base image", upon which the current image is to be built upon.
+IMAGE_BASE_PKGDB ?= ""
# Generate companion debugfs?
IMAGE_GEN_DEBUGFS ?= "0"
@@ -118,6 +126,15 @@ do_rootfs[depends] += " \
"
do_rootfs[recrdeptask] += "do_packagedata"
+python () {
+# make sure that the 'base image' has been queued in before this
+# image wants to unpack and build upon the formers pgkdb
+base_image = d.getVar('IMAGE_BASE_PKGDB')
+pn = d.getVar('PN')
+if base_image and base_image != pn:
+d.appendVarFlag("do_rootfs", 'depends', ' '+ base_image +
':do_image_complete')
+}
+
def rootfs_command_variables(d):
return
['ROOTFS_POSTPROCESS_COMMAND','ROOTFS_PREPROCESS_COMMAND','ROOTFS_POSTINSTALL_COMMAND','ROOTFS_POSTUNINSTALL_COMMAND','OPKG_PREPROCESS_COMMANDS','OPKG_POSTPROCESS_COMMANDS','IMAGE_POSTPROCESS_COMMAND',
'IMAGE_PREPROCESS_COMMAND','RPM_PREPROCESS_COMMANDS','RPM_POSTPROCESS_COMMANDS','DEB_PREPROCESS_COMMANDS','DEB_POSTPROCESS_COMMANDS']
@@ -134,8 +151,8 @@ def rootfs_variables(d):
'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
'IMAGE_LINGUAS_COMPLEMENTARY', 'IMAGE_LOCALES_ARCHIVE',
'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
- 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS', 'IMAGE_GEN_DEBUGFS',
'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY',
'REPRODUCIBLE_TIMESTAMP_ROOTFS',
- 'IMAGE_INSTALL_DEBUGFS']
+ 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS', 'IMAGE_BASE_PKGDB',
'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR',
'PACKAGE_EXCLUDE_COMPLEMENTARY',
+ 'REPRODUCIBLE_TIMESTAMP_ROOTFS', 'IMAGE_INSTALL_DEBUGFS']
variables.extend(rootfs_command_variables(d))
variables.extend(variable_depends(d))
return " ".join(variables)
diff --git a/meta/conf/documentation.conf b/meta/conf/documentation.conf
index 36aebb59ab..9f493cfe96 100644
--- a/meta/conf/documentation.conf
+++ b/meta/conf/documentation.conf
@@ -208,6 +208,7 @@ ICECC_PATH[doc] = "The location of the icecc binary."
ICECC_CLASS_DISABLE[doc] = "Identifies user classes that you do not want the
Icecream distributed compile support to consider."
ICECC_RECIPE_DISABLE[doc] = "Identifies user recipes that you do not want the
Icecream distributed compile support to consider."
ICECC_RECIPE_ENABLE[doc] = "Identifies user recipes that use
[OE-core] [PATCH v4 1/3] image.bbclass/rootfs: archive and deploy package database
archive the package database after the rootfs has been put together as
*rootfs-pkdbfs.tar.gz, and put it into the deploy folder.
This creates a snapshot of the package mangers state at the point in
time when all dependencies have been resolved and installed; which
could be used by "extension images" to built upon.
Signed-off-by: Johannes Schneider
---
meta/classes-recipe/image.bbclass | 44 +++-
meta/classes-recipe/image_types.bbclass | 1 +
meta/conf/documentation.conf | 1 +
meta/lib/oe/package_manager/deb/rootfs.py | 1 +
meta/lib/oe/package_manager/ipk/rootfs.py | 1 +
meta/lib/oe/package_manager/rpm/rootfs.py | 1 +
meta/lib/oe/rootfs.py | 20
meta/lib/oeqa/selftest/cases/imagefeatures.py | 51 +++
8 files changed, 119 insertions(+), 1 deletion(-)
diff --git a/meta/classes-recipe/image.bbclass
b/meta/classes-recipe/image.bbclass
index 28be6c6362..3ccaaa17b8 100644
--- a/meta/classes-recipe/image.bbclass
+++ b/meta/classes-recipe/image.bbclass
@@ -42,6 +42,9 @@ IMAGE_FEATURES ?= ""
IMAGE_FEATURES[type] = "list"
IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs
read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password
allow-empty-password allow-root-login serial-autologin-root
post-install-logging overlayfs-etc"
+# Generate snapshot of the package database?
+IMAGE_GEN_PKGDBFS ?= "0"
+
# Generate companion debugfs?
IMAGE_GEN_DEBUGFS ?= "0"
@@ -131,7 +134,8 @@ def rootfs_variables(d):
'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
'IMAGE_LINGUAS_COMPLEMENTARY', 'IMAGE_LOCALES_ARCHIVE',
'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
- 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED',
'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY',
'REPRODUCIBLE_TIMESTAMP_ROOTFS', 'IMAGE_INSTALL_DEBUGFS']
+ 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS', 'IMAGE_GEN_DEBUGFS',
'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY',
'REPRODUCIBLE_TIMESTAMP_ROOTFS',
+ 'IMAGE_INSTALL_DEBUGFS']
variables.extend(rootfs_command_variables(d))
variables.extend(variable_depends(d))
return " ".join(variables)
@@ -337,6 +341,17 @@ python do_image_qa_setscene () {
}
addtask do_image_qa_setscene
+def setup_pkgdbfs_variables(d):
+d.appendVar('IMAGE_ROOTFS', '-pkgdb')
+if d.getVar('IMAGE_LINK_NAME'):
+d.appendVar('IMAGE_LINK_NAME', '-pkgdb')
+d.appendVar('IMAGE_NAME','-pkgdb')
+d.setVar('IMAGE_FSTYPES', 'tar.gz')
+
+python setup_pkgdbfs () {
+setup_pkgdbfs_variables(d)
+}
+
def setup_debugfs_variables(d):
d.appendVar('IMAGE_ROOTFS', '-dbg')
if d.getVar('IMAGE_LINK_NAME'):
@@ -381,6 +396,11 @@ python () {
alltypes = d.getVar('IMAGE_FSTYPES').split()
typedeps = {}
+if d.getVar('IMAGE_GEN_PKGDBFS') == "1":
+pkgdbfs_fstypes = ['tar.gz']
+for t in pkgdbfs_fstypes:
+alltypes.append("pkgdbfs_" + t)
+
if d.getVar('IMAGE_GEN_DEBUGFS') == "1":
debugfs_fstypes = d.getVar('IMAGE_FSTYPES_DEBUGFS').split()
for t in debugfs_fstypes:
@@ -393,6 +413,10 @@ python () {
basetypes[baset]= []
if t not in basetypes[baset]:
basetypes[baset].append(t)
+pkgdb = ""
+if t.startswith("pkgdbfs_"):
+t = t[8:]
+pkgdb = "pkgdbfs_"
debug = ""
if t.startswith("debugfs_"):
t = t[8:]
@@ -401,6 +425,13 @@ python () {
vardeps.add('IMAGE_TYPEDEP:' + t)
if baset not in typedeps:
typedeps[baset] = set()
+deps = [pkgdb + dep for dep in deps]
+for dep in deps:
+if dep not in alltypes:
+alltypes.append(dep)
+_add_type(dep)
+basedep = _image_base_type(dep)
+typedeps[baset].add(basedep)
deps = [debug + dep for dep in deps]
for dep in deps:
if dep not in alltypes:
@@ -419,6 +450,7 @@ python () {
maskedtypes = (d.getVar('IMAGE_TYPES_MASKED') or "").split()
maskedtypes = [dbg + t for t in maskedtypes for dbg in ("", "debugfs_")]
+maskedtypes = [pkgdb + t for t in maskedtypes for pkgdb in ("",
"pkgdbfs_")]
for t in basetypes:
vardeps = set()
@@ -430,6 +462,11 @@ python () {
continue
localdata = bb.data.createCopy(d)
+pkgdb = ""
+if t.startswith("pkgdbfs_"):
+setup_pkgdbfs
[OE-core] [PATCH v3 0/3] pkg-database and systemd-sysext image
systemd-sysext allows to overlay another image (or multiple) ontop of a "base-image" = the current rootfs, via the use of overlayfs; to add tools and features meant for development purposes. To quote the documentation on systemd-sysext: " ...addition in order to make debugging/development easier). System extension images should not be misunderstood as a generic software packaging framework, ..." To build a lean image, that only holds packages that are not already part of the base-image, a snapshot of the package-database is taken after the installation of the base-rootfs is done, and picked up again when collecting the rootfs of such a extension image. with all this in place an example usage could look like this: some-core-image.bb inherit core-image IMAGE_GEN_PKGDBFS = "1" extending-image.bb inherit image-sysext IMAGE_FSTYPES = "squashfs" IMAGE_BASE_PKGDB = "some-core-image" # the above pointing at a package-db similar to: # build/deploy/images/$MACHINE/some-core-image-$MACHINE-20240210172305-pkgdb.rootfs.tar.gz then on the device, running some-core-image, with the extension image placed at FN: $> ln -s "$FN" /run/extensions/$(basename $FN).raw $> systemd-sysext list $> SYSTEMD_LOG_LEVEL=debug systemd-sysext merge As long as the VERSION_ID of the extension image matches the os-release in the base image, the above commands return sucessfully; for details on the compativility check see the docs for systemd-sysext. = changes with v2: rebase from 'kirkstone' onto 'master' changes with v3; incorporate review suggestions for simplification add task dependency handling add oe-selftest for the pkgdb handling add variable documentation and some more comments, and examples in the commit-msg changes with v4; rebase onto 'master' => no functional changes fixed patchtest findings -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#196585): https://lists.openembedded.org/g/openembedded-core/message/196585 Mute This Topic: https://lists.openembedded.org/mt/104717633/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v3 3/3] classes: add a systemd-sysext image class
systemd-sysext can load a raw-image containing usr/ and opt/ folders
to mount them as RO overlay over the rootfs, to "extend" the systems.
This class provides the necessary changes/additions to the enclosed
file-system so that systemd-sysext accepts the extension for "merge"
into the rootfs.
With such an created image, placed into the correct folder (see [1]),
`systemd-sysext list` should be able to list the "extension" and
`systemd-sysext merge` should enable the overlay. On both commands a
preceding "SYSTEMD_LOG_LEVEL=debug" can aide in figuring out what is
amiss.
The strict name checking systemd-sysext does against the name of
extension-release.NAME file, is disabled, as there is only one such in
the resulting image. This is done to allow a user to freely rename the
resulting image file.
Note that for e.g. squashfs, the kernel needs CONFIG_SQUASHFS_XATTR=y
Link:
https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html
Link:
https://0pointer.net/blog/testing-my-system-code-in-usr-without-modifying-usr.html
Signed-off-by: Johannes Schneider
---
meta/classes-recipe/image-sysext.bbclass | 42
1 file changed, 42 insertions(+)
create mode 100644 meta/classes-recipe/image-sysext.bbclass
diff --git a/meta/classes-recipe/image-sysext.bbclass
b/meta/classes-recipe/image-sysext.bbclass
new file mode 100644
index 00..012db83353
--- /dev/null
+++ b/meta/classes-recipe/image-sysext.bbclass
@@ -0,0 +1,42 @@
+# SPDX-License-Identifier: MIT
+#
+# Copyright Leica Geosystems AG
+#
+
+# systemd-sysext [1] has a simple mechanism for version compatibility:
+# the extension to be loaded has to contain a
+# /usr/lib/extension-release.d/extension-release.NAME
+# with "NAME" *exactly* matching the filename of the extensions
+# raw-device filename/
+#
+# from the extension-release file the "ID" and "VERSION_ID" fields are
+# matched against the etc/os-release and the extension is only "merged"
+# if no mismatches between NAME, ID, and VERSION_ID.
+#
+# Link:
https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html
+
+inherit image
+
+IMAGE_NAME_SUFFIX = ".sysext"
+EXTENSION_NAME = "${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${IMAGE_FSTYPES}"
+
+DEPENDS += " os-release"
+
+sysext_image_mangle_rootfs() {
+R=${IMAGE_ROOTFS}
+
+# pull a copy of the rootfs version information, which systemd-sysext
matches against
+cp -av ${RECIPE_SYSROOT}/${nonarch_libdir}/os-release
${WORKDIR}/extension-release.base
+
+echo 'EXTENSION_RELOAD_MANAGER=1' >> ${WORKDIR}/extension-release.base
+
+install -d $R${nonarch_libdir}/extension-release.d
+install -m 0644 ${WORKDIR}/extension-release.base \
+
$R${nonarch_libdir}/extension-release.d/extension-release.${EXTENSION_NAME}
+
+# disable systemd-sysext's strict name checking, so that the image file
can be renamed, while still being 'merge'-able
+setfattr -n user.extension-release.strict -v false \
+
$R${nonarch_libdir}/extension-release.d/extension-release.${EXTENSION_NAME}
+}
+
+ROOTFS_POSTPROCESS_COMMAND += " sysext_image_mangle_rootfs; "
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#196003):
https://lists.openembedded.org/g/openembedded-core/message/196003
Mute This Topic: https://lists.openembedded.org/mt/104500433/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] classes: add a systemd-sysext image class
systemd-sysext can load a raw-image containing usr/ and opt/ folders
to mount them as RO overlay over the rootfs, to "extend" the systems.
This class provides the necessary changes/additions to the enclosed
file-system so that systemd-sysext accepts the extension for "merge"
into the rootfs.
With such an created image, placed into the correct folder (see [1]),
`systemd-sysext list` should be able to list the "extension" and
`systemd-sysext merge` should enable the overlay. On both commands a
preceding "SYSTEMD_LOG_LEVEL=debug" can aide in figuring out what is
amiss.
The strict name checking systemd-sysext does against the name of
extension-release.NAME file, is disabled, as there is only one such in
the resulting image. This is done to allow a user to freely rename the
resulting image file.
Note that for e.g. squashfs, the kernel needs CONFIG_SQUASHFS_XATTR=y
Link:
https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html
Link:
https://0pointer.net/blog/testing-my-system-code-in-usr-without-modifying-usr.html
Signed-off-by: Johannes Schneider
---
meta/classes-recipe/image-sysext.bbclass | 42
1 file changed, 42 insertions(+)
create mode 100644 meta/classes-recipe/image-sysext.bbclass
diff --git a/meta/classes-recipe/image-sysext.bbclass
b/meta/classes-recipe/image-sysext.bbclass
new file mode 100644
index 00..012db83353
--- /dev/null
+++ b/meta/classes-recipe/image-sysext.bbclass
@@ -0,0 +1,42 @@
+# SPDX-License-Identifier: MIT
+#
+# Copyright Leica Geosystems AG
+#
+
+# systemd-sysext [1] has a simple mechanism for version compatibility:
+# the extension to be loaded has to contain a
+# /usr/lib/extension-release.d/extension-release.NAME
+# with "NAME" *exactly* matching the filename of the extensions
+# raw-device filename/
+#
+# from the extension-release file the "ID" and "VERSION_ID" fields are
+# matched against the etc/os-release and the extension is only "merged"
+# if no mismatches between NAME, ID, and VERSION_ID.
+#
+# Link:
https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html
+
+inherit image
+
+IMAGE_NAME_SUFFIX = ".sysext"
+EXTENSION_NAME = "${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${IMAGE_FSTYPES}"
+
+DEPENDS += " os-release"
+
+sysext_image_mangle_rootfs() {
+R=${IMAGE_ROOTFS}
+
+# pull a copy of the rootfs version information, which systemd-sysext
matches against
+cp -av ${RECIPE_SYSROOT}/${nonarch_libdir}/os-release
${WORKDIR}/extension-release.base
+
+echo 'EXTENSION_RELOAD_MANAGER=1' >> ${WORKDIR}/extension-release.base
+
+install -d $R${nonarch_libdir}/extension-release.d
+install -m 0644 ${WORKDIR}/extension-release.base \
+
$R${nonarch_libdir}/extension-release.d/extension-release.${EXTENSION_NAME}
+
+# disable systemd-sysext's strict name checking, so that the image file
can be renamed, while still being 'merge'-able
+setfattr -n user.extension-release.strict -v false \
+
$R${nonarch_libdir}/extension-release.d/extension-release.${EXTENSION_NAME}
+}
+
+ROOTFS_POSTPROCESS_COMMAND += " sysext_image_mangle_rootfs; "
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#196002):
https://lists.openembedded.org/g/openembedded-core/message/196002
Mute This Topic: https://lists.openembedded.org/mt/104500432/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v3 2/3] image.bbclass/rootfs: set and unpack package-database
set the package-database of a "lower image" to unpack and build upon
when installing packages for the current image. This way a lean image
will be created, which only holds the packages that are not already
present in the lower image.
An image build such could then be used with overlayfs or systemd-
sysext to extend the "lower image" on demand; for development purposes
on a device running the "lower image" in RO mode for example.
A configuration could look as follows:
some-core-image.bb
inherit image
IMAGE_GEN_PKGDBFS = "1"
extending-image.bb
inherit image
IMAGE_BASE_PKGDB = "some-core-image"
Signed-off-by: Johannes Schneider
---
meta/classes-recipe/image.bbclass | 23 --
meta/conf/documentation.conf | 3 +-
meta/lib/oe/package_manager/deb/rootfs.py | 2 +
meta/lib/oe/package_manager/ipk/rootfs.py | 6 ++-
meta/lib/oe/package_manager/rpm/rootfs.py | 7 ++-
meta/lib/oe/rootfs.py | 18
meta/lib/oeqa/selftest/cases/imagefeatures.py | 46 +++
7 files changed, 97 insertions(+), 8 deletions(-)
diff --git a/meta/classes-recipe/image.bbclass
b/meta/classes-recipe/image.bbclass
index 3ccaaa17b8..c573c37cd8 100644
--- a/meta/classes-recipe/image.bbclass
+++ b/meta/classes-recipe/image.bbclass
@@ -42,8 +42,16 @@ IMAGE_FEATURES ?= ""
IMAGE_FEATURES[type] = "list"
IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs
read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password
allow-empty-password allow-root-login serial-autologin-root
post-install-logging overlayfs-etc"
-# Generate snapshot of the package database?
+# Image layering:
+# a "base image" would create a snapshot of the package-database after the
+# installation of all packages into the rootfs is done. The next/other image
+# "layered on-top" of the former would then import that database and install
+# further packages; without reinstalling packages/dependencies that are already
+# installed in the layer below.
+# Set to '1' in a "base image" recipe, to preserve a snapshot of the package
database.
IMAGE_GEN_PKGDBFS ?= "0"
+# "PN" of a "base image", upon which the current image is to be built upon.
+IMAGE_BASE_PKGDB ?= ""
# Generate companion debugfs?
IMAGE_GEN_DEBUGFS ?= "0"
@@ -118,6 +126,15 @@ do_rootfs[depends] += " \
"
do_rootfs[recrdeptask] += "do_packagedata"
+python () {
+# make sure that the 'base image' has been queued in before this
+# image wants to unpack and build upon the formers pgkdb
+base_image = d.getVar('IMAGE_BASE_PKGDB')
+pn = d.getVar('PN')
+if base_image and base_image != pn:
+d.appendVarFlag("do_rootfs", 'depends', ' '+ base_image +
':do_image_complete')
+}
+
def rootfs_command_variables(d):
return
['ROOTFS_POSTPROCESS_COMMAND','ROOTFS_PREPROCESS_COMMAND','ROOTFS_POSTINSTALL_COMMAND','ROOTFS_POSTUNINSTALL_COMMAND','OPKG_PREPROCESS_COMMANDS','OPKG_POSTPROCESS_COMMANDS','IMAGE_POSTPROCESS_COMMAND',
'IMAGE_PREPROCESS_COMMAND','RPM_PREPROCESS_COMMANDS','RPM_POSTPROCESS_COMMANDS','DEB_PREPROCESS_COMMANDS','DEB_POSTPROCESS_COMMANDS']
@@ -134,8 +151,8 @@ def rootfs_variables(d):
'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
'IMAGE_LINGUAS_COMPLEMENTARY', 'IMAGE_LOCALES_ARCHIVE',
'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
- 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS', 'IMAGE_GEN_DEBUGFS',
'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY',
'REPRODUCIBLE_TIMESTAMP_ROOTFS',
- 'IMAGE_INSTALL_DEBUGFS']
+ 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS', 'IMAGE_BASE_PKGDB',
'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR',
'PACKAGE_EXCLUDE_COMPLEMENTARY',
+ 'REPRODUCIBLE_TIMESTAMP_ROOTFS', 'IMAGE_INSTALL_DEBUGFS']
variables.extend(rootfs_command_variables(d))
variables.extend(variable_depends(d))
return " ".join(variables)
diff --git a/meta/conf/documentation.conf b/meta/conf/documentation.conf
index 7ec6f07dfc..9013556c04 100644
--- a/meta/conf/documentation.conf
+++ b/meta/conf/documentation.conf
@@ -208,6 +208,7 @@ ICECC_PATH[doc] = "The location of the icecc binary."
ICECC_CLASS_DISABLE[doc] = "Identifies user classes that you do not want the
Icecream distributed compile support to consider."
ICECC_RECIPE_DISABLE[doc] = "Identifies user recipes that you do not want the
Icecream distributed compile support to consider."
ICECC_RECIPE_ENABLE[doc] = "Identifies user recipes that use
[OE-core] [PATCH v3 0/3] pkg-database and systemd-sysext image
systemd-sysext allows to overlay another image (or multiple) ontop of a "base-image" = the current rootfs, via the use of overlayfs; to add tools and features meant for development purposes. To quote the documentation on systemd-sysext: " ...addition in order to make debugging/development easier). System extension images should not be misunderstood as a generic software packaging framework, ..." To build a lean image, that only holds packages that are not already part of the base-image, a snapshot of the package-database is taken after the installation of the base-rootfs is done, and picked up again when collecting the rootfs of such a extension image. with all this in place an example usage could look like this: some-core-image.bb inherit core-image IMAGE_GEN_PKGDBFS = "1" extending-image.bb inherit image-sysext IMAGE_FSTYPES = "squashfs" IMAGE_BASE_PKGDB = "some-core-image" # the above pointing at a package-db similar to: # build/deploy/images/$MACHINE/some-core-image-$MACHINE-20240210172305-pkgdb.rootfs.tar.gz then on the device, running some-core-image, with the extension image placed at FN: $> ln -s "$FN" /run/extensions/$(basename $FN).raw $> systemd-sysext list $> SYSTEMD_LOG_LEVEL=debug systemd-sysext merge As long as the VERSION_ID of the extension image matches the os-release in the base image, the above commands return sucessfully; for details on the compativility check see the docs for systemd-sysext. = changes with v2: rebase from 'kirkstone' onto 'master' changes with v3; incorporate review suggestions for simplification add task dependency handling add oe-selftest for the pkgdb handling add variable documentation and some more comments, and examples in the commit-msg -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#195999): https://lists.openembedded.org/g/openembedded-core/message/195999 Mute This Topic: https://lists.openembedded.org/mt/104500428/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH v3 1/3] image.bbclass/rootfs: archive and deploy package database
archive the package database after the rootfs has been put together as
*rootfs-pkdbfs.tar.gz, and put it into the deploy folder.
This creates a snapshot of the package mangers state at the point in
time when all dependencies have been resolved and installed; which
could be used by "extension images" to built upon.
Signed-off-by: Johannes Schneider
---
meta/classes-recipe/image.bbclass | 44 -
meta/classes-recipe/image_types.bbclass | 1 +
meta/conf/documentation.conf | 1 +
meta/lib/oe/package_manager/deb/rootfs.py | 1 +
meta/lib/oe/package_manager/ipk/rootfs.py | 1 +
meta/lib/oe/package_manager/rpm/rootfs.py | 1 +
meta/lib/oe/rootfs.py | 20
meta/lib/oeqa/selftest/cases/imagefeatures.py | 47 +++
8 files changed, 115 insertions(+), 1 deletion(-)
diff --git a/meta/classes-recipe/image.bbclass
b/meta/classes-recipe/image.bbclass
index 28be6c6362..3ccaaa17b8 100644
--- a/meta/classes-recipe/image.bbclass
+++ b/meta/classes-recipe/image.bbclass
@@ -42,6 +42,9 @@ IMAGE_FEATURES ?= ""
IMAGE_FEATURES[type] = "list"
IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs
read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password
allow-empty-password allow-root-login serial-autologin-root
post-install-logging overlayfs-etc"
+# Generate snapshot of the package database?
+IMAGE_GEN_PKGDBFS ?= "0"
+
# Generate companion debugfs?
IMAGE_GEN_DEBUGFS ?= "0"
@@ -131,7 +134,8 @@ def rootfs_variables(d):
'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
'IMAGE_LINGUAS_COMPLEMENTARY', 'IMAGE_LOCALES_ARCHIVE',
'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
- 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED',
'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY',
'REPRODUCIBLE_TIMESTAMP_ROOTFS', 'IMAGE_INSTALL_DEBUGFS']
+ 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS', 'IMAGE_GEN_DEBUGFS',
'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY',
'REPRODUCIBLE_TIMESTAMP_ROOTFS',
+ 'IMAGE_INSTALL_DEBUGFS']
variables.extend(rootfs_command_variables(d))
variables.extend(variable_depends(d))
return " ".join(variables)
@@ -337,6 +341,17 @@ python do_image_qa_setscene () {
}
addtask do_image_qa_setscene
+def setup_pkgdbfs_variables(d):
+d.appendVar('IMAGE_ROOTFS', '-pkgdb')
+if d.getVar('IMAGE_LINK_NAME'):
+d.appendVar('IMAGE_LINK_NAME', '-pkgdb')
+d.appendVar('IMAGE_NAME','-pkgdb')
+d.setVar('IMAGE_FSTYPES', 'tar.gz')
+
+python setup_pkgdbfs () {
+setup_pkgdbfs_variables(d)
+}
+
def setup_debugfs_variables(d):
d.appendVar('IMAGE_ROOTFS', '-dbg')
if d.getVar('IMAGE_LINK_NAME'):
@@ -381,6 +396,11 @@ python () {
alltypes = d.getVar('IMAGE_FSTYPES').split()
typedeps = {}
+if d.getVar('IMAGE_GEN_PKGDBFS') == "1":
+pkgdbfs_fstypes = ['tar.gz']
+for t in pkgdbfs_fstypes:
+alltypes.append("pkgdbfs_" + t)
+
if d.getVar('IMAGE_GEN_DEBUGFS') == "1":
debugfs_fstypes = d.getVar('IMAGE_FSTYPES_DEBUGFS').split()
for t in debugfs_fstypes:
@@ -393,6 +413,10 @@ python () {
basetypes[baset]= []
if t not in basetypes[baset]:
basetypes[baset].append(t)
+pkgdb = ""
+if t.startswith("pkgdbfs_"):
+t = t[8:]
+pkgdb = "pkgdbfs_"
debug = ""
if t.startswith("debugfs_"):
t = t[8:]
@@ -401,6 +425,13 @@ python () {
vardeps.add('IMAGE_TYPEDEP:' + t)
if baset not in typedeps:
typedeps[baset] = set()
+deps = [pkgdb + dep for dep in deps]
+for dep in deps:
+if dep not in alltypes:
+alltypes.append(dep)
+_add_type(dep)
+basedep = _image_base_type(dep)
+typedeps[baset].add(basedep)
deps = [debug + dep for dep in deps]
for dep in deps:
if dep not in alltypes:
@@ -419,6 +450,7 @@ python () {
maskedtypes = (d.getVar('IMAGE_TYPES_MASKED') or "").split()
maskedtypes = [dbg + t for t in maskedtypes for dbg in ("", "debugfs_")]
+maskedtypes = [pkgdb + t for t in maskedtypes for pkgdb in ("",
"pkgdbfs_")]
for t in basetypes:
vardeps = set()
@@ -430,6 +462,11 @@ python () {
continue
localdata = bb.data.createCopy(d)
+pkgdb = ""
+if t.startswith("pkgdbfs_"):
+setup_pkgdbf
Re: [OE-core] [PATCH v2 1/3] image.bbclass/rootfs: archive and deploy opkg package database
Hoi Richard,
and thanks for the feedback!
>> archive the package database after the rootfs has been put together as
>> *rootfs-pkdbfs.tar.gz, and put it into the deploy folder.
>>
>> This creates a snapshot of the package mangers state at the point in time
>> when
>> all dependencies have been resolved and installed; which can be used by
>> "follow
>> up" images to be built upon.
>
>I'm torn on this series. On the one hand I can see why it might be
>useful. On the other hand:
>
>* no test cases for it
>* no documentation updates
>* no real indications in the code on what it is doing (no comments)
>
point taken, i'll look into the test framework and add something with V3
and i missed the in-source docs... also sth for V3
i'll redistribute the example/documentation from the cover letter amongst the
meta/conf/documentation.conf for the variables, and in-source comments
any other place in this repo i should be aware off/add documentation to?
>
>It also copies and pastes a lot of the debugfs code and duplicates it
>which makes me wonder if there isn't something better we should be
>doing here.
the copy&paste comes from the similar steps having to be taken at around the
same time.
could you elaborate on "something better"? taking different steps,
de-duplicating code? ...?
>
>There is good info in the 0/3 series email but that will get lost once
>things merge.
good point - i'll incorporate more from the cover-letter into the code itself
with the next version.
>
>> Signed-off-by: Johannes Schneider
>> ---
>> meta/classes-recipe/image.bbclass | 45 ++-
>> meta/classes-recipe/image_types.bbclass | 1 +
>> meta/conf/bitbake.conf| 1 +
>> meta/lib/oe/package_manager/deb/rootfs.py | 1 +
>> meta/lib/oe/package_manager/ipk/rootfs.py | 1 +
>> meta/lib/oe/package_manager/rpm/rootfs.py | 1 +
>> meta/lib/oe/rootfs.py | 35 ++
>> 7 files changed, 83 insertions(+), 2 deletions(-)
>>
>> diff --git a/meta/classes-recipe/image.bbclass
>> b/meta/classes-recipe/image.bbclass
>> index 28be6c6362..c688c39f15 100644
>> --- a/meta/classes-recipe/image.bbclass
>> +++ b/meta/classes-recipe/image.bbclass
>> @@ -42,6 +42,9 @@ IMAGE_FEATURES ?= ""
>> IMAGE_FEATURES[type] = "list"
>> IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs
>> read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password
>> allow-empty-password allow-root-login serial-autologin-root
>> post-install-logging overlayfs-etc"
>>
>> +# Generate snapshot of the package database?
>> +IMAGE_GEN_PKGDBFS ?= "0"
>> +
>> # Generate companion debugfs?
>> IMAGE_GEN_DEBUGFS ?= "0"
>>
>> @@ -131,7 +134,8 @@ def rootfs_variables(d):
>>
>> 'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
>> 'IMAGE_LINGUAS_COMPLEMENTARY', 'IMAGE_LOCALES_ARCHIVE',
>>
>> 'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
>>
>> 'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
>> - 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS',
>> 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY',
>> 'REPRODUCIBLE_TIMESTAMP_ROOTFS', 'IMAGE_INSTALL_DEBUGFS']
>> + 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS',
>> 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR',
>> 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS',
>> + 'IMAGE_INSTALL_DEBUGFS']
>> variables.extend(rootfs_command_variables(d))
>> variables.extend(variable_depends(d))
>> return " ".join(variables)
>> @@ -337,6 +341,19 @@ python do_image_qa_setscene () {
>> }
>> addtask do_image_qa_setscene
>>
>> +def setup_pkgdbfs_variables(d):
>> +d.appendVar('IMAGE_ROOTFS', '-pkgdb')
>> +if d.getVar('IMAGE_LINK_NAME'):
>> +d.appendVar('IMAGE_LINK_NAME', '-pkgdb')
>> +d.appendVar('IMAGE_NAME','-pkgdb')
>> +pkgdbfs_image_fstypes = d.getVar('IMAGE_FSTYPES_PKGDBFS')
>> +if pkgdbfs_image_fstypes:
>> +d.setVar('IMAGE_FSTYPES', pkgdbfs_image_fstypes)
>> +
>> +python setup_pkgdbfs () {
>> +setup_pkgdbfs_variables(d)
>> +}
>> +
>> def setup_debugfs_variables(d):
>> d.appendVar('IMAGE_ROOTFS', '-dbg')
>> if d.getVar('IMAGE_LINK_NAME'):
>> @@ -381,6 +398,11 @@ python () {
>> alltypes = d.getVar('IMAGE_FSTYPES').split()
>> typedeps = {}
>>
>> +if d.getVar('IMAGE_GEN_PKGDBFS') == "1":
>> +pkgdbfs_fstypes = d.getVar('IMAGE_FSTYPES_PKGDBFS').split()
>> +for t in pkgdbfs_fstypes:
>> +alltypes.append("pkgdbfs_" + t)
>> +
>> if d.getVar('IMAGE_GEN_DEBUGFS') == "1":
>>
Re: [OE-core] [PATCH v2 2/3] image.bbclass/rootfs: set&unpack package-database
Hoi Richard,
and thanks for the feedback!
your other mail is still being processed, but to get already back to you on
this one:
>> set the package-database of a "lower image" to unpack and build upon when
>> installing packages for the current image. This way a lean image will be
>> created, which only holds the packages that are not already present in the
>> lower
>> image, that then could be used with overlayfs or systemd-sysext to extend the
>> "lower image" on demand; for development purposes on an RO lower image for
>> example.
>>
>> Signed-off-by: Johannes Schneider
>> ---
>> meta/classes-recipe/image.bbclass | 10 +++-
>> meta/lib/oe/package_manager/deb/rootfs.py | 2 ++
>> meta/lib/oe/package_manager/ipk/rootfs.py | 6 +++--
>> meta/lib/oe/package_manager/rpm/rootfs.py | 7 --
>> meta/lib/oe/rootfs.py | 29 +++
>> 5 files changed, 49 insertions(+), 5 deletions(-)
>>
>> diff --git a/meta/classes-recipe/image.bbclass
>> b/meta/classes-recipe/image.bbclass
>> index c688c39f15..b4a2460187 100644
>> --- a/meta/classes-recipe/image.bbclass
>> +++ b/meta/classes-recipe/image.bbclass
>> @@ -42,8 +42,16 @@ IMAGE_FEATURES ?= ""
>> IMAGE_FEATURES[type] = "list"
>> IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs
>> read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password
>> allow-empty-password allow-root-login serial-autologin-root
>> post-install-logging overlayfs-etc"
>>
>> +# Image layering:
>> +# a "base image" would create a snapshot of the package-database after the
>> +# installation of all packages into the rootfs is done. The next/other image
>> +# "layered on-top" of the former would then import that database and install
>> +# further packages; without reinstalling package dependencies that are
>> already
>> +# installed in the layer below.
>> # Generate snapshot of the package database?
>> IMAGE_GEN_PKGDBFS ?= "0"
>> +# Package-database of the base image, upon which to build up a new
>> image-layer
>> +IMAGE_BASE_PKGDB ?= ""
>>
>> # Generate companion debugfs?
>> IMAGE_GEN_DEBUGFS ?= "0"
>> @@ -134,7 +142,7 @@ def rootfs_variables(d):
>>
>> 'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
>> 'IMAGE_LINGUAS_COMPLEMENTARY', 'IMAGE_LOCALES_ARCHIVE',
>>
>> 'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
>>
>> 'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
>> - 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS',
>> 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR',
>> 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS',
>> + 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS',
>> 'IMAGE_BASE_PKGDB', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED',
>> 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY',
>> 'REPRODUCIBLE_TIMESTAMP_ROOTFS',
>> 'IMAGE_INSTALL_DEBUGFS']
>> variables.extend(rootfs_command_variables(d))
>> variables.extend(variable_depends(d))
>> diff --git a/meta/lib/oe/package_manager/deb/rootfs.py
>> b/meta/lib/oe/package_manager/deb/rootfs.py
>> index 43107c8663..71a21df09b 100644
>> --- a/meta/lib/oe/package_manager/deb/rootfs.py
>> +++ b/meta/lib/oe/package_manager/deb/rootfs.py
>> @@ -152,6 +152,8 @@ class PkgRootfs(DpkgOpkgRootfs):
>>
>> execute_pre_post_process(self.d, deb_pre_process_cmds)
>>
>> +self._unpack_pkg_db_rootfs(['/var/lib/dpkg'])
>> +
>> if self.progress_reporter:
>> self.progress_reporter.next_stage()
>> # Don't support incremental, so skip that
>> diff --git a/meta/lib/oe/package_manager/ipk/rootfs.py
>> b/meta/lib/oe/package_manager/ipk/rootfs.py
>> index 64d9bc7969..408faa8030 100644
>> --- a/meta/lib/oe/package_manager/ipk/rootfs.py
>> +++ b/meta/lib/oe/package_manager/ipk/rootfs.py
>> @@ -276,12 +276,16 @@ class PkgRootfs(DpkgOpkgRootfs):
>> pkgs_to_install = self.manifest.parse_initial_manifest()
>> opkg_pre_process_cmds = self.d.getVar('OPKG_PREPROCESS_COMMANDS')
>> opkg_post_process_cmds = self.d.getVar('OPKG_POSTPROCESS_COMMANDS')
>> +opkg_lib_dir = self.d.getVar('OPKGLIBDIR')
>> +opkg_dir = os.path.join(opkg_lib_dir, 'opkg')
>>
>> # update PM index files
>> self.pm.write_index()
>>
>> execute_pre_post_process(self.d, opkg_pre_process_cmds)
>>
>> +self._unpack_pkg_db_rootfs([opkg_dir])
>> +
>> if self.progress_reporter:
>> self.progress_reporter.next_stage()
>> # Steps are a bit different in order, skip next
>> @@ -317,8 +321,6 @@ class PkgRootfs(
[OE-core] [meta-oe][PATCH] base-files: profile: allow profile.d to set EDITOR
>From 0013c8a6482018d5476e4eb2f4d537c96551e0c6 Mon Sep 17 00:00:00 2001
From: Johannes Schneider
Date: Fri, 13 Oct 2023 08:28:38 +0200
Subject: [PATCH v1] base-files: profile: allow profile.d to set EDITOR
With a profile.d configuration in place that sets the EDITOR variable,
the automatic terminal 'resize' logic would not trigger. Which then
would possibly lead to a 80x24 fallback on the debug serial console.
This can simply be avoided by setting a flag variable when the shell
is first opened, then processing all profile.d includes, trigger the
'resize' depending on the flag and shell-level and finally only set
EDITOR to some default if it is still unset.
Signed-off-by: Johannes Schneider
---
meta/recipes-core/base-files/base-files/profile | 16
1 file changed, 12 insertions(+), 4 deletions(-)
diff --git a/meta/recipes-core/base-files/base-files/profile
b/meta/recipes-core/base-files/base-files/profile
index cc37e1ba77..bded3757cc 100644
--- a/meta/recipes-core/base-files/base-files/profile
+++ b/meta/recipes-core/base-files/base-files/profile
@@ -10,6 +10,12 @@ PATH="/usr/local/bin:/usr/bin:/bin"
# Set the prompt for bash and ash (no other shells known to be in use here)
[ -z "$PS1" ] || PS1='\u@\h:\w\$ '
+# Use the EDITOR not being set as a trigger to call resize later on
+FIRSTTIMESETUP=0
+if [ -z "$EDITOR" ] ; then
+ FIRSTTIMESETUP=1
+fi
+
if [ -d /etc/profile.d ]; then
for i in /etc/profile.d/*.sh; do
if [ -f $i -a -r $i ]; then
@@ -50,17 +56,19 @@ resize() {
}
fi
fi
- # Use the EDITOR not being set as a trigger to call resize
- # and only do this for /dev/tty[A-z] which are typically
+ # only do this for /dev/tty[A-z] which are typically
# serial ports
- if [ -z "$EDITOR" -a "$SHLVL" = 1 ] ; then
+ if [ $FIRSTTIMESETUP -eq 1 -a $SHLVL -eq 1 ] ; then
case $(tty 2>/dev/null) in
/dev/tty[A-z]*) resize >/dev/null;;
esac
fi
fi
-EDITOR="vi"# needed for packages like cron, git-commit
+if [ -z "$EDITOR" ]; then
+ EDITOR="vi" # needed for packages like cron,
git-commit
+fi
+
export PATH PS1 OPIEDIR QPEDIR QTDIR EDITOR TERM
umask 022
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#189058):
https://lists.openembedded.org/g/openembedded-core/message/189058
Mute This Topic: https://lists.openembedded.org/mt/101937238/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [meta-oe][PATCH v4] classes: rootfs-postcommands: autologin root on serial-getty
Hi,
v4 adds a variant for sysv - when/if using util-linux getty, since busybox does
not seem to support an autologin option
also adds some documentation to the bbclass
regards
From: Johannes Schneider
Sent: Tuesday, August 2, 2022 11:40
To: openembedded-core@lists.openembedded.org
Cc: SCHNEIDER Johannes
Subject: [meta-oe][PATCH v4] classes: rootfs-postcommands: autologin root on
serial-getty
when empty-root-password AND serial-autologin-root are part of the
IMAGE_FEATURES, save some of the developers time by not having to type
the (then still sole) 'root' username on the serial console after each
and every reboot
this is done by inserting '--autologin root' into the command line of
the responsible 'getty' service
Signed-off-by: Johannes Schneider
---
meta/classes/core-image.bbclass | 1 +
meta/classes/image.bbclass| 2 +-
meta/classes/rootfs-postcommands.bbclass | 20 +++
.../sysvinit/sysvinit-inittab/start_getty | 3 ++-
4 files changed, 24 insertions(+), 2 deletions(-)
diff --git a/meta/classes/core-image.bbclass b/meta/classes/core-image.bbclass
index 84fd3eeb38..6764035729 100644
--- a/meta/classes/core-image.bbclass
+++ b/meta/classes/core-image.bbclass
@@ -29,6 +29,7 @@
# - allow-empty-password
# - allow-root-login
# - post-install-logging
+# - serial-autologin-root - with 'empty-root-password': autologin 'root' on
the serial console
# - dev-pkgs- development packages (headers, etc.) for all
installed packages in the rootfs
# - dbg-pkgs- debug symbol packages for all installed packages in
the rootfs
# - lic-pkgs- license packages for all installed pacakges in the
rootfs, requires
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 2139a7e576..fe32cdefd5 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -34,7 +34,7 @@ INHIBIT_DEFAULT_DEPS = "1"
# IMAGE_FEATURES may contain any available package group
IMAGE_FEATURES ?= ""
IMAGE_FEATURES[type] = "list"
-IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs
read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password
allow-empty-password allow-root-login post-install-logging overlayfs-etc"
+IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs
read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password
allow-empty-password allow-root-login serial-autologin-root
post-install-logging overlayfs-etc"
# Generate companion debugfs?
IMAGE_GEN_DEBUGFS ?= "0"
diff --git a/meta/classes/rootfs-postcommands.bbclass
b/meta/classes/rootfs-postcommands.bbclass
index a8a952f31d..3dadba6047 100644
--- a/meta/classes/rootfs-postcommands.bbclass
+++ b/meta/classes/rootfs-postcommands.bbclass
@@ -8,6 +8,9 @@ ROOTFS_POSTPROCESS_COMMAND +=
'${@bb.utils.contains_any("IMAGE_FEATURES", [ 'deb
# Allow dropbear/openssh to accept root logins if debug-tweaks or
allow-root-login is enabled
ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains_any("IMAGE_FEATURES", [
'debug-tweaks', 'allow-root-login' ], "ssh_allow_root_login; ", "",d)}'
+# Autologin the root user on the serial console, if empty-root-password and
serial-autologin-root are active
+ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains("IMAGE_FEATURES", [
'empty-root-password', 'serial-autologin-root' ], "serial_autologin_root; ",
"",d)}'
+
# Enable postinst logging if debug-tweaks or post-install-logging is enabled
ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains_any("IMAGE_FEATURES", [
'debug-tweaks', 'post-install-logging' ], "postinst_enable_logging; ", "",d)}'
@@ -196,6 +199,23 @@ ssh_allow_root_login () {
fi
}
+#
+# Autologin the 'root' user on the serial terminal,
+# if empty-root-password' AND 'serial-autologin-root are enabled
+#
+serial_autologin_root () {
+ if ${@bb.utils.contains("DISTRO_FEATURES", "sysvinit", "true", "false",
d)}; then
+ # add autologin option to util-linux getty only
+ sed -i 's/options="/&--autologin root /' \
+ "${IMAGE_ROOTFS}${base_bindir}/start_getty"
+ elif ${@bb.utils.contains("DISTRO_FEATURES", "systemd", "true",
"false", d)}; then
+ if [ -e
${IMAGE_ROOTFS}${systemd_system_unitdir}/serial-getty@.service ]; then
+ sed -i '/^\s*ExecStart\b/ s/getty /&--autologin root /'
\
+
"${IMAGE_ROOTFS}${systemd_system_unitdir}/serial-getty@.service"
+ fi
+ fi
+}
+
python sort_passwd () {
import rootfspostcommands
rootfspostcommands.sort_passwd(d.expand('${IMAGE_ROOTFS}${sysconfdir}'))
diff --git a/meta/recipes-core/sysvinit/sysvinit-inittab/start_getty
b/meta/recipes-core/sysvinit/sysvinit-inittab/start_getty
index 699a1ead1a..7106fb72fb 100644
--- a/meta/recipes-core/sysvinit/sysvinit-inittab/start_getty
+++ b/meta/recipes-core/sysvinit/sysvinit-inittab/sta
[OE-core] [meta-oe][PATCH v4] classes: rootfs-postcommands: autologin root on serial-getty
when empty-root-password AND serial-autologin-root are part of the
IMAGE_FEATURES, save some of the developers time by not having to type
the (then still sole) 'root' username on the serial console after each
and every reboot
this is done by inserting '--autologin root' into the command line of
the responsible 'getty' service
Signed-off-by: Johannes Schneider
---
meta/classes/core-image.bbclass | 1 +
meta/classes/image.bbclass| 2 +-
meta/classes/rootfs-postcommands.bbclass | 20 +++
.../sysvinit/sysvinit-inittab/start_getty | 3 ++-
4 files changed, 24 insertions(+), 2 deletions(-)
diff --git a/meta/classes/core-image.bbclass b/meta/classes/core-image.bbclass
index 84fd3eeb38..6764035729 100644
--- a/meta/classes/core-image.bbclass
+++ b/meta/classes/core-image.bbclass
@@ -29,6 +29,7 @@
# - allow-empty-password
# - allow-root-login
# - post-install-logging
+# - serial-autologin-root - with 'empty-root-password': autologin 'root' on
the serial console
# - dev-pkgs- development packages (headers, etc.) for all
installed packages in the rootfs
# - dbg-pkgs- debug symbol packages for all installed packages in
the rootfs
# - lic-pkgs- license packages for all installed pacakges in the
rootfs, requires
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 2139a7e576..fe32cdefd5 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -34,7 +34,7 @@ INHIBIT_DEFAULT_DEPS = "1"
# IMAGE_FEATURES may contain any available package group
IMAGE_FEATURES ?= ""
IMAGE_FEATURES[type] = "list"
-IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs
read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password
allow-empty-password allow-root-login post-install-logging overlayfs-etc"
+IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs
read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password
allow-empty-password allow-root-login serial-autologin-root
post-install-logging overlayfs-etc"
# Generate companion debugfs?
IMAGE_GEN_DEBUGFS ?= "0"
diff --git a/meta/classes/rootfs-postcommands.bbclass
b/meta/classes/rootfs-postcommands.bbclass
index a8a952f31d..3dadba6047 100644
--- a/meta/classes/rootfs-postcommands.bbclass
+++ b/meta/classes/rootfs-postcommands.bbclass
@@ -8,6 +8,9 @@ ROOTFS_POSTPROCESS_COMMAND +=
'${@bb.utils.contains_any("IMAGE_FEATURES", [ 'deb
# Allow dropbear/openssh to accept root logins if debug-tweaks or
allow-root-login is enabled
ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains_any("IMAGE_FEATURES", [
'debug-tweaks', 'allow-root-login' ], "ssh_allow_root_login; ", "",d)}'
+# Autologin the root user on the serial console, if empty-root-password and
serial-autologin-root are active
+ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains("IMAGE_FEATURES", [
'empty-root-password', 'serial-autologin-root' ], "serial_autologin_root; ",
"",d)}'
+
# Enable postinst logging if debug-tweaks or post-install-logging is enabled
ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains_any("IMAGE_FEATURES", [
'debug-tweaks', 'post-install-logging' ], "postinst_enable_logging; ", "",d)}'
@@ -196,6 +199,23 @@ ssh_allow_root_login () {
fi
}
+#
+# Autologin the 'root' user on the serial terminal,
+# if empty-root-password' AND 'serial-autologin-root are enabled
+#
+serial_autologin_root () {
+ if ${@bb.utils.contains("DISTRO_FEATURES", "sysvinit", "true", "false",
d)}; then
+ # add autologin option to util-linux getty only
+ sed -i 's/options="/&--autologin root /' \
+ "${IMAGE_ROOTFS}${base_bindir}/start_getty"
+ elif ${@bb.utils.contains("DISTRO_FEATURES", "systemd", "true",
"false", d)}; then
+ if [ -e
${IMAGE_ROOTFS}${systemd_system_unitdir}/serial-getty@.service ]; then
+ sed -i '/^\s*ExecStart\b/ s/getty /&--autologin root /'
\
+
"${IMAGE_ROOTFS}${systemd_system_unitdir}/serial-getty@.service"
+ fi
+ fi
+}
+
python sort_passwd () {
import rootfspostcommands
rootfspostcommands.sort_passwd(d.expand('${IMAGE_ROOTFS}${sysconfdir}'))
diff --git a/meta/recipes-core/sysvinit/sysvinit-inittab/start_getty
b/meta/recipes-core/sysvinit/sysvinit-inittab/start_getty
index 699a1ead1a..7106fb72fb 100644
--- a/meta/recipes-core/sysvinit/sysvinit-inittab/start_getty
+++ b/meta/recipes-core/sysvinit/sysvinit-inittab/start_getty
@@ -9,9 +9,10 @@ case $(readlink -f "${getty}") in
if [ -x "/usr/bin/setsid" ] ; then
setsid="/usr/bin/setsid"
fi
+options=""
;;
esac
if [ -e /sys/class/tty/$2 -a -c /dev/$2 ]; then
- ${setsid:-} ${getty} -L $1 $2 $3
+ ${setsid:-} ${getty} ${options:-} -L $1 $2 $3
fi
--
2.25.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
Vie
Re: [OE-core] [meta-oe][PATCH v2] classes: rootfs-postcommands: autologin root on serial-getty
good question, haven't had a sysvinit system in ages... so no easy "let's test
it on this platform" :-s
do you happen to have one?
should the patch include a warning to sysvinit users like "not supported"
"config switch only applies to systemd" ... suggestions? (-:
the only sysvinit+getty reference i see in the code is
./meta/recipes-core/sysvinit/sysvinit-inittab/start_getty
which relies on a symlinked getty - possibly provided by busybox or others?
regards
From: Ross Burton
Sent: Wednesday, July 27, 2022 17:30
To: SCHNEIDER Johannes
Cc: openembedded-core@lists.openembedded.org
Subject: Re: [OE-core] [meta-oe][PATCH v2] classes: rootfs-postcommands:
autologin root on serial-getty
[You don't often get email from ross.bur...@arm.com. Learn why this is
important at https://aka.ms/LearnAboutSenderIdentification ]
This email is not from Hexagon’s Office 365 instance. Please be careful while
clicking links, opening attachments, or replying to this email.
> +serial_autologin_root () {
> + if [ -e ${IMAGE_ROOTFS}${systemd_system_unitdir}/serial-getty@.service
> ]; then
> + sed -i '/^\s*ExecStart\b/ s/getty /&--autologin root /' \
> +
> "${IMAGE_ROOTFS}${systemd_system_unitdir}/serial-getty@.service"
> + fi
> +}
What about images using sysvinit?
Ross
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#168592):
https://lists.openembedded.org/g/openembedded-core/message/168592
Mute This Topic: https://lists.openembedded.org/mt/92623778/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [meta-oe][PATCH v3] classes: rootfs-postcommands: autologin root on serial-getty
Hi, what & where would you like to see (as) additional documentation? (-: regards From: Marta Rybczynska Sent: Wednesday, July 27, 2022 09:27 To: SCHNEIDER Johannes Cc: OE-core Subject: Re: [OE-core] [meta-oe][PATCH v3] classes: rootfs-postcommands: autologin root on serial-getty [You don't often get email from rybczyn...@gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] This email is not from Hexagon’s Office 365 instance. Please be careful while clicking links, opening attachments, or replying to this email. On Wed, Jul 27, 2022 at 5:37 AM Johannes Schneider via lists.openembedded.org wrote: > > when empty-root-password AND serial-autologin-root are part of the > IMAGE_FEATURES, save some of the developers time by not having to type > the (then still sole) 'root' username on the serial consoleafter each > and every reboot > If this change is accepted, it would be good to have it documented really soon. It is likely to break CIs. Regards, Marta -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#168546): https://lists.openembedded.org/g/openembedded-core/message/168546 Mute This Topic: https://lists.openembedded.org/mt/92642961/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [meta-oe][PATCH v3] classes: rootfs-postcommands: autologin root on serial-getty
when empty-root-password AND serial-autologin-root are part of the
IMAGE_FEATURES, save some of the developers time by not having to type
the (then still sole) 'root' username on the serial consoleafter each
and every reboot
this is done by inserting '--autologin root' into the command line of
the responsible 'getty' service
Signed-off-by: Johannes Schneider
---
meta/classes/core-image.bbclass | 1 +
meta/classes/image.bbclass | 2 +-
meta/classes/rootfs-postcommands.bbclass | 15 +++
3 files changed, 17 insertions(+), 1 deletion(-)
diff --git a/meta/classes/core-image.bbclass b/meta/classes/core-image.bbclass
index 84fd3eeb38..6764035729 100644
--- a/meta/classes/core-image.bbclass
+++ b/meta/classes/core-image.bbclass
@@ -29,6 +29,7 @@
# - allow-empty-password
# - allow-root-login
# - post-install-logging
+# - serial-autologin-root - with 'empty-root-password': autologin 'root' on
the serial console
# - dev-pkgs- development packages (headers, etc.) for all
installed packages in the rootfs
# - dbg-pkgs- debug symbol packages for all installed packages in
the rootfs
# - lic-pkgs- license packages for all installed pacakges in the
rootfs, requires
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 2139a7e576..fe32cdefd5 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -34,7 +34,7 @@ INHIBIT_DEFAULT_DEPS = "1"
# IMAGE_FEATURES may contain any available package group
IMAGE_FEATURES ?= ""
IMAGE_FEATURES[type] = "list"
-IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs
read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password
allow-empty-password allow-root-login post-install-logging overlayfs-etc"
+IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs
read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password
allow-empty-password allow-root-login serial-autologin-root
post-install-logging overlayfs-etc"
# Generate companion debugfs?
IMAGE_GEN_DEBUGFS ?= "0"
diff --git a/meta/classes/rootfs-postcommands.bbclass
b/meta/classes/rootfs-postcommands.bbclass
index a8a952f31d..65fe74cc5d 100644
--- a/meta/classes/rootfs-postcommands.bbclass
+++ b/meta/classes/rootfs-postcommands.bbclass
@@ -8,6 +8,9 @@ ROOTFS_POSTPROCESS_COMMAND +=
'${@bb.utils.contains_any("IMAGE_FEATURES", [ 'deb
# Allow dropbear/openssh to accept root logins if debug-tweaks or
allow-root-login is enabled
ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains_any("IMAGE_FEATURES", [
'debug-tweaks', 'allow-root-login' ], "ssh_allow_root_login; ", "",d)}'
+# Autologin the root user on the serial console, if empty-root-password and
serial-autologin-root are active
+ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains("IMAGE_FEATURES", [
'empty-root-password', 'serial-autologin-root' ], "serial_autologin_root; ",
"",d)}'
+
# Enable postinst logging if debug-tweaks or post-install-logging is enabled
ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains_any("IMAGE_FEATURES", [
'debug-tweaks', 'post-install-logging' ], "postinst_enable_logging; ", "",d)}'
@@ -196,6 +199,18 @@ ssh_allow_root_login () {
fi
}
+#
+# Autologin the 'root' user on the serial terminal,
+# if empty-root-password' AND 'serial-autologin-root are enabled
+#
+serial_autologin_root () {
+ if [ -e ${IMAGE_ROOTFS}${systemd_system_unitdir}/serial-getty@.service
]; then
+ sed -i '/^\s*ExecStart\b/ s/getty /&--autologin root /' \
+
"${IMAGE_ROOTFS}${systemd_system_unitdir}/serial-getty@.service"
+ fi
+}
+
+
python sort_passwd () {
import rootfspostcommands
rootfspostcommands.sort_passwd(d.expand('${IMAGE_ROOTFS}${sysconfdir}'))
--
2.25.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#168540):
https://lists.openembedded.org/g/openembedded-core/message/168540
Mute This Topic: https://lists.openembedded.org/mt/92642961/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [meta-oe][PATCH v2] classes: rootfs-postcommands: autologin root on serial-getty
when empty-root-password AND serial-autologin-root are part of the
IMAGE_FEATURES, save some of the developers time by not having to type
the (then still sole) 'root' username on the serial consoleafter each
and every reboot
this is done by inserting '--autologin root' into the command line of
the responsible 'getty' service
Signed-off-by: Johannes Schneider
---
meta/classes/image.bbclass | 2 +-
meta/classes/rootfs-postcommands.bbclass | 15 +++
2 files changed, 16 insertions(+), 1 deletion(-)
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 2139a7e576..fe32cdefd5 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -34,7 +34,7 @@ INHIBIT_DEFAULT_DEPS = "1"
# IMAGE_FEATURES may contain any available package group
IMAGE_FEATURES ?= ""
IMAGE_FEATURES[type] = "list"
-IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs
read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password
allow-empty-password allow-root-login post-install-logging overlayfs-etc"
+IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs
read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password
allow-empty-password allow-root-login serial-autologin-root
post-install-logging overlayfs-etc"
# Generate companion debugfs?
IMAGE_GEN_DEBUGFS ?= "0"
diff --git a/meta/classes/rootfs-postcommands.bbclass
b/meta/classes/rootfs-postcommands.bbclass
index a8a952f31d..e8e9661f58 100644
--- a/meta/classes/rootfs-postcommands.bbclass
+++ b/meta/classes/rootfs-postcommands.bbclass
@@ -8,6 +8,9 @@ ROOTFS_POSTPROCESS_COMMAND +=
'${@bb.utils.contains_any("IMAGE_FEATURES", [ 'deb
# Allow dropbear/openssh to accept root logins if debug-tweaks or
allow-root-login is enabled
ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains_any("IMAGE_FEATURES", [
'debug-tweaks', 'allow-root-login' ], "ssh_allow_root_login; ", "",d)}'
+# Autologin the root user on the serial console, if empty-root-password and
serial-autologin-root are active
+ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains("IMAGE_FEATURES", [
'empty-root-password', 'serial-autologin-root' ], "serial_autologin_root; ",
"",d)}'
+
# Enable postinst logging if debug-tweaks or post-install-logging is enabled
ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains_any("IMAGE_FEATURES", [
'debug-tweaks', 'post-install-logging' ], "postinst_enable_logging; ", "",d)}'
@@ -196,6 +199,18 @@ ssh_allow_root_login () {
fi
}
+#
+# Autologin the 'root' user on the serial terminal,
+# if empty-root-password is enabled
+#
+serial_autologin_root () {
+ if [ -e ${IMAGE_ROOTFS}${systemd_system_unitdir}/serial-getty@.service
]; then
+ sed -i '/^\s*ExecStart\b/ s/getty /&--autologin root /' \
+
"${IMAGE_ROOTFS}${systemd_system_unitdir}/serial-getty@.service"
+ fi
+}
+
+
python sort_passwd () {
import rootfspostcommands
rootfspostcommands.sort_passwd(d.expand('${IMAGE_ROOTFS}${sysconfdir}'))
--
2.25.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#168502):
https://lists.openembedded.org/g/openembedded-core/message/168502
Mute This Topic: https://lists.openembedded.org/mt/92623778/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[OE-core] [PATCH] classes: rootfs-postcommands: autologin root on serial-getty
when debug-tweaks or empty-root-password are part of the
IMAGE_FEATURES, save some of the developers time by not having to type
the (then still sole) 'root' username on the serial consoleafter each
and every reboot
by inserting '--autologin root' into the command line of the
responsible 'getty' service
Signed-off-by: Johannes Schneider
---
meta/classes/rootfs-postcommands.bbclass | 15 +++
1 file changed, 15 insertions(+)
diff --git a/meta/classes/rootfs-postcommands.bbclass
b/meta/classes/rootfs-postcommands.bbclass
index a8a952f31d..39992b101e 100644
--- a/meta/classes/rootfs-postcommands.bbclass
+++ b/meta/classes/rootfs-postcommands.bbclass
@@ -8,6 +8,9 @@ ROOTFS_POSTPROCESS_COMMAND +=
'${@bb.utils.contains_any("IMAGE_FEATURES", [ 'deb
# Allow dropbear/openssh to accept root logins if debug-tweaks or
allow-root-login is enabled
ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains_any("IMAGE_FEATURES", [
'debug-tweaks', 'allow-root-login' ], "ssh_allow_root_login; ", "",d)}'
+# Autologin the root user on the serial console, if debug-tweaks or
empty-root-password are active
+ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains_any("IMAGE_FEATURES", [
'debug-tweaks', 'empty-root-password' ], "serial_autologin_root; ", "",d)}'
+
# Enable postinst logging if debug-tweaks or post-install-logging is enabled
ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains_any("IMAGE_FEATURES", [
'debug-tweaks', 'post-install-logging' ], "postinst_enable_logging; ", "",d)}'
@@ -196,6 +199,18 @@ ssh_allow_root_login () {
fi
}
+#
+# Autologin the 'root' user on the serial terminal,
+# if empty-root-password is enabled
+#
+serial_root_autologin () {
+ if [ -e ${IMAGE_ROOTFS}${systemd_system_unitdir}/serial-getty@.service
]; then
+ sed -i '/^\s*ExecStart\b/ s/getty /&--autologin root /' \
+
"${IMAGE_ROOTFS}${systemd_system_unitdir}/serial-getty@.service"
+ fi
+}
+
+
python sort_passwd () {
import rootfspostcommands
rootfspostcommands.sort_passwd(d.expand('${IMAGE_ROOTFS}${sysconfdir}'))
--
2.25.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#168499):
https://lists.openembedded.org/g/openembedded-core/message/168499
Mute This Topic: https://lists.openembedded.org/mt/92623417/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
