Re: [OE-core] [PATCH] shadow: replace static linking with dynamic libraries in a custom location and bundled with shadow
On Tue, 23 Jan 2024 at 11:13, Alexander Kanavin wrote: > > On Tue, 23 Jan 2024 at 02:13, Dmitry Baryshkov wrote: > > > > On Fri, 19 Jan 2024 at 14:43, Alexander Kanavin > > wrote: > > > > > > Despite our efforts to make static linking work, there have > > > been new reports of bizarre build failures: > > > https://lists.openembedded.org/g/openembedded-core/message/194006 > > > https://lists.openembedded.org/g/openembedded-core/message/193907 > > > > > > This commit changes back to dynamic linking, but places > > > the libraries in a custom location, per RP's suggestion. > > > > > > Signed-off-by: Alexander Kanavin > > > > Alexander, please excuse my ignorance. What was the original issue > > that you observed on your build system? I tried simply reverting the > > offending commit and performing the build. I see libbsd, libattr and > > libmd populated in recipe-sysroot-native for both target and native > > builds that depend on shadow-native (added manual dependency of test > > -native recipe on shadow-native). > > shadow contains executables for user/group management such as useradd. > These executables are used both in regular tasks and in setscene tasks > to create users that a recipe provides (in addition to simply > unpacking the sstate archive). It is those setscene tasks where > providing library dependencies of shadow is problematic. If you delete > tmp/ and run the build again from sstate, you'll see that for those > recipes which add users, shadow-native will be installed but those > three libraries will not. On some systems executing useradd will fall > through to using the build host libraries (which is a ticking time > bomb and can fail later), on others it will fail outright because e.g. > libbsd is not installed on the host. Thanks for the explanation. Indeed, reproduced the issue with these libs missing in sstate. -- With best wishes Dmitry -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#194235): https://lists.openembedded.org/g/openembedded-core/message/194235 Mute This Topic: https://lists.openembedded.org/mt/103828721/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] shadow: replace static linking with dynamic libraries in a custom location and bundled with shadow
On Tue, 23 Jan 2024 at 02:13, Dmitry Baryshkov wrote: > > On Fri, 19 Jan 2024 at 14:43, Alexander Kanavin > wrote: > > > > Despite our efforts to make static linking work, there have > > been new reports of bizarre build failures: > > https://lists.openembedded.org/g/openembedded-core/message/194006 > > https://lists.openembedded.org/g/openembedded-core/message/193907 > > > > This commit changes back to dynamic linking, but places > > the libraries in a custom location, per RP's suggestion. > > > > Signed-off-by: Alexander Kanavin > > Alexander, please excuse my ignorance. What was the original issue > that you observed on your build system? I tried simply reverting the > offending commit and performing the build. I see libbsd, libattr and > libmd populated in recipe-sysroot-native for both target and native > builds that depend on shadow-native (added manual dependency of test > -native recipe on shadow-native). shadow contains executables for user/group management such as useradd. These executables are used both in regular tasks and in setscene tasks to create users that a recipe provides (in addition to simply unpacking the sstate archive). It is those setscene tasks where providing library dependencies of shadow is problematic. If you delete tmp/ and run the build again from sstate, you'll see that for those recipes which add users, shadow-native will be installed but those three libraries will not. On some systems executing useradd will fall through to using the build host libraries (which is a ticking time bomb and can fail later), on others it will fail outright because e.g. libbsd is not installed on the host. Alex -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#194221): https://lists.openembedded.org/g/openembedded-core/message/194221 Mute This Topic: https://lists.openembedded.org/mt/103828721/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] shadow: replace static linking with dynamic libraries in a custom location and bundled with shadow
On Fri, 19 Jan 2024 at 14:43, Alexander Kanavin wrote: > > Despite our efforts to make static linking work, there have > been new reports of bizarre build failures: > https://lists.openembedded.org/g/openembedded-core/message/194006 > https://lists.openembedded.org/g/openembedded-core/message/193907 > > This commit changes back to dynamic linking, but places > the libraries in a custom location, per RP's suggestion. > > Signed-off-by: Alexander Kanavin Alexander, please excuse my ignorance. What was the original issue that you observed on your build system? I tried simply reverting the offending commit and performing the build. I see libbsd, libattr and libmd populated in recipe-sysroot-native for both target and native builds that depend on shadow-native (added manual dependency of test -native recipe on shadow-native). > --- > meta/conf/distro/include/no-static-libs.inc | 5 - > meta/recipes-extended/shadow/shadow.inc | 24 - > 2 files changed, 14 insertions(+), 15 deletions(-) > > diff --git a/meta/conf/distro/include/no-static-libs.inc > b/meta/conf/distro/include/no-static-libs.inc > index 8898d53d756..75359928a14 100644 > --- a/meta/conf/distro/include/no-static-libs.inc > +++ b/meta/conf/distro/include/no-static-libs.inc > @@ -21,11 +21,6 @@ DISABLE_STATIC:pn-libusb1-native = "" > # needed by rust > DISABLE_STATIC:pn-musl = "" > > -# needed by shadow-native to build static executables, particularly useradd > -DISABLE_STATIC:pn-attr-native = "" > -DISABLE_STATIC:pn-libbsd-native = "" > -DISABLE_STATIC:pn-libmd-native = "" > - > EXTRA_OECONF:append = "${DISABLE_STATIC}" > > EXTRA_OECMAKE:append:pn-libical = " -DSHARED_ONLY=True" > diff --git a/meta/recipes-extended/shadow/shadow.inc > b/meta/recipes-extended/shadow/shadow.inc > index 43f456251a5..b8e5e285c45 100644 > --- a/meta/recipes-extended/shadow/shadow.inc > +++ b/meta/recipes-extended/shadow/shadow.inc > @@ -47,16 +47,6 @@ EXTRA_OECONF += "--without-libcrack \ > > CFLAGS:append:libc-musl = " -DLIBBSD_OVERLAY" > > -# Force static linking of utilities so we can use from the sysroot/sstate > for useradd > -# without worrying about the dependency libraries being available > -LDFLAGS:append:class-native = " -no-pie" > -do_compile:prepend:class-native () { > - sed -i -e 's#\(LIBS.*\)-lbsd#\1 ${STAGING_LIBDIR}/libbsd.a > ${STAGING_LIBDIR}/libmd.a#g' \ > - -e 's#\(LIBBSD.*\)-lbsd#\1 ${STAGING_LIBDIR}/libbsd.a > ${STAGING_LIBDIR}/libmd.a#g' \ > - -e 's#\(LIBATTR.*\)-lattr#\1 ${STAGING_LIBDIR}/libattr.a#g' \ > - ${B}/lib/Makefile ${B}/src/Makefile > -} > - > NSCDOPT = "" > NSCDOPT:class-native = "--without-nscd" > NSCDOPT:class-nativesdk = "--without-nscd" > @@ -161,6 +151,20 @@ do_install:append() { > touch ${D}${sysconfdir}/subgid > } > > +# Make executables look for dynamically linked libraries in a custom > location, and install > +# the needed libraries there. That way we can use them from sstate > +# in setscene tasks without worrying about the dependency libraries being > available. > +do_install:append:class-native() { > +binaries=$(find ${D}${base_bindir}/ ${D}${base_sbindir}/ > ${D}${bindir}/ ${D}${sbindir}/ -executable -type f) > +chrpath -k -r ${STAGING_DIR_NATIVE}/lib-shadow-deps $binaries > +mkdir -p ${D}${STAGING_DIR_NATIVE}/lib-shadow-deps/ > +install ${STAGING_LIBDIR_NATIVE}/libattr.so.* > ${STAGING_LIBDIR_NATIVE}/libbsd.so.* ${STAGING_LIBDIR_NATIVE}/libmd.so.* > ${D}${STAGING_DIR_NATIVE}/lib-shadow-deps/ > +install ${D}${libdir}/*.so.* > ${D}${STAGING_DIR_NATIVE}/lib-shadow-deps/ > +} > + > +SYSROOT_DIRS:append:class-native = " ${STAGING_DIR_NATIVE}/lib-shadow-deps/" > +INSANE_SKIP:${PN}:class-native = "already-stripped" > + > PACKAGES =+ "${PN}-base" > FILES:${PN}-base = "\ > ${base_bindir}/login.shadow \ > -- > 2.39.2 > > > > -- With best wishes Dmitry -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#194197): https://lists.openembedded.org/g/openembedded-core/message/194197 Mute This Topic: https://lists.openembedded.org/mt/103828721/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] shadow: replace static linking with dynamic libraries in a custom location and bundled with shadow
I can confirm that this patch fixes the problem reported here: https://lists.openembedded.org/g/openembedded-core/message/193907 We tried it on three sets of 8 builds, and all of them passed successfully: https://tuxapi.tuxsuite.com/v1/groups/tuxsuite/projects/vishal/plans/2bBF73bS9lx2GFzroGyjf8NmGMx https://tuxapi.tuxsuite.com/v1/groups/tuxsuite/projects/vishal/plans/2bChSfD6NDgyJWFfXxMQnpk3hjh https://tuxapi.tuxsuite.com/v1/groups/tuxsuite/projects/vishal/plans/2bCru4v7ocE8yHgmh9GMdb0jDmr On Fri, 19 Jan 2024 at 18:13, Alexander Kanavin wrote: > Despite our efforts to make static linking work, there have > been new reports of bizarre build failures: > https://lists.openembedded.org/g/openembedded-core/message/194006 > https://lists.openembedded.org/g/openembedded-core/message/193907 > > This commit changes back to dynamic linking, but places > the libraries in a custom location, per RP's suggestion. > > Signed-off-by: Alexander Kanavin > --- > meta/conf/distro/include/no-static-libs.inc | 5 - > meta/recipes-extended/shadow/shadow.inc | 24 - > 2 files changed, 14 insertions(+), 15 deletions(-) > > diff --git a/meta/conf/distro/include/no-static-libs.inc > b/meta/conf/distro/include/no-static-libs.inc > index 8898d53d756..75359928a14 100644 > --- a/meta/conf/distro/include/no-static-libs.inc > +++ b/meta/conf/distro/include/no-static-libs.inc > @@ -21,11 +21,6 @@ DISABLE_STATIC:pn-libusb1-native = "" > # needed by rust > DISABLE_STATIC:pn-musl = "" > > -# needed by shadow-native to build static executables, particularly > useradd > -DISABLE_STATIC:pn-attr-native = "" > -DISABLE_STATIC:pn-libbsd-native = "" > -DISABLE_STATIC:pn-libmd-native = "" > - > EXTRA_OECONF:append = "${DISABLE_STATIC}" > > EXTRA_OECMAKE:append:pn-libical = " -DSHARED_ONLY=True" > diff --git a/meta/recipes-extended/shadow/shadow.inc > b/meta/recipes-extended/shadow/shadow.inc > index 43f456251a5..b8e5e285c45 100644 > --- a/meta/recipes-extended/shadow/shadow.inc > +++ b/meta/recipes-extended/shadow/shadow.inc > @@ -47,16 +47,6 @@ EXTRA_OECONF += "--without-libcrack \ > > CFLAGS:append:libc-musl = " -DLIBBSD_OVERLAY" > > -# Force static linking of utilities so we can use from the sysroot/sstate > for useradd > -# without worrying about the dependency libraries being available > -LDFLAGS:append:class-native = " -no-pie" > -do_compile:prepend:class-native () { > - sed -i -e 's#\(LIBS.*\)-lbsd#\1 ${STAGING_LIBDIR}/libbsd.a > ${STAGING_LIBDIR}/libmd.a#g' \ > - -e 's#\(LIBBSD.*\)-lbsd#\1 ${STAGING_LIBDIR}/libbsd.a > ${STAGING_LIBDIR}/libmd.a#g' \ > - -e 's#\(LIBATTR.*\)-lattr#\1 ${STAGING_LIBDIR}/libattr.a#g' > \ > - ${B}/lib/Makefile ${B}/src/Makefile > -} > - > NSCDOPT = "" > NSCDOPT:class-native = "--without-nscd" > NSCDOPT:class-nativesdk = "--without-nscd" > @@ -161,6 +151,20 @@ do_install:append() { > touch ${D}${sysconfdir}/subgid > } > > +# Make executables look for dynamically linked libraries in a custom > location, and install > +# the needed libraries there. That way we can use them from sstate > +# in setscene tasks without worrying about the dependency libraries being > available. > +do_install:append:class-native() { > +binaries=$(find ${D}${base_bindir}/ ${D}${base_sbindir}/ > ${D}${bindir}/ ${D}${sbindir}/ -executable -type f) > +chrpath -k -r ${STAGING_DIR_NATIVE}/lib-shadow-deps $binaries > +mkdir -p ${D}${STAGING_DIR_NATIVE}/lib-shadow-deps/ > +install ${STAGING_LIBDIR_NATIVE}/libattr.so.* > ${STAGING_LIBDIR_NATIVE}/libbsd.so.* ${STAGING_LIBDIR_NATIVE}/libmd.so.* > ${D}${STAGING_DIR_NATIVE}/lib-shadow-deps/ > +install ${D}${libdir}/*.so.* > ${D}${STAGING_DIR_NATIVE}/lib-shadow-deps/ > +} > + > +SYSROOT_DIRS:append:class-native = " > ${STAGING_DIR_NATIVE}/lib-shadow-deps/" > +INSANE_SKIP:${PN}:class-native = "already-stripped" > + > PACKAGES =+ "${PN}-base" > FILES:${PN}-base = "\ > ${base_bindir}/login.shadow \ > -- > 2.39.2 > > > > > -- *Vishal Bhoj* Tech Lead - Microservices CI Team Mob: +91 9886656306 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#194080): https://lists.openembedded.org/g/openembedded-core/message/194080 Mute This Topic: https://lists.openembedded.org/mt/103828721/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-