[OE-core] [PATCH 4/4] gnupg: use native version for signing, rather than one provided by host
Using host gpg has been problematic, and particularly this removes the need to serialize package creation, as long as --auto-expand-secmem is passed to gpg-agent, and gnupg >= 2.2.4 is in use (https://dev.gnupg.org/T3530). Sadly, gpg-agent itself is single-threaded, so in the longer run we might want to seek alternatives: https://lwn.net/Articles/742542/ (a smaller issue is that rpm itself runs the gpg fronted in a serial fashion, which slows down the build in cases of recipes with very large amount of packages, e.g. glibc-locale) Note that sstate signing and verification continues to use host gpg, as depending on native gpg would create circular dependencies. [YOCTO #12022] Signed-off-by: Alexander Kanavin --- meta/classes/sign_package_feed.bbclass | 2 +- meta/classes/sign_rpm.bbclass | 6 +- meta/lib/oe/gpg_sign.py| 8 ++-- meta/recipes-core/meta/signing-keys.bb | 1 + 4 files changed, 9 insertions(+), 8 deletions(-) diff --git a/meta/classes/sign_package_feed.bbclass b/meta/classes/sign_package_feed.bbclass index f03c4802d06..7ff3a35a2fa 100644 --- a/meta/classes/sign_package_feed.bbclass +++ b/meta/classes/sign_package_feed.bbclass @@ -43,4 +43,4 @@ python () { } do_package_index[depends] += "signing-keys:do_deploy" -do_rootfs[depends] += "signing-keys:do_populate_sysroot" +do_rootfs[depends] += "signing-keys:do_populate_sysroot gnupg-native:do_populate_sysroot" diff --git a/meta/classes/sign_rpm.bbclass b/meta/classes/sign_rpm.bbclass index 4961b03618f..64ae7ce30e3 100644 --- a/meta/classes/sign_rpm.bbclass +++ b/meta/classes/sign_rpm.bbclass @@ -68,8 +68,4 @@ python sign_rpm () { do_package_index[depends] += "signing-keys:do_deploy" do_rootfs[depends] += "signing-keys:do_populate_sysroot" -# Newer versions of gpg (at least 2.1.5 and 2.2.1) have issues when signing occurs in parallel -# so unfortunately the signing must be done serially. Once the upstream problem is fixed, -# the following line must be removed otherwise we loose all the intrinsic parallelism from -# bitbake. For more information, check https://bugzilla.yoctoproject.org/show_bug.cgi?id=12022. -do_package_write_rpm[lockfiles] += "${TMPDIR}/gpg.lock" +PACKAGE_WRITE_DEPS += "gnupg-native" diff --git a/meta/lib/oe/gpg_sign.py b/meta/lib/oe/gpg_sign.py index 9cc88f020c1..b17272928fc 100644 --- a/meta/lib/oe/gpg_sign.py +++ b/meta/lib/oe/gpg_sign.py @@ -12,6 +12,7 @@ class LocalSigner(object): self.gpg_path = d.getVar('GPG_PATH') self.gpg_version = self.get_gpg_version() self.rpm_bin = bb.utils.which(os.getenv('PATH'), "rpmsign") +self.gpg_agent_bin = bb.utils.which(os.getenv('PATH'), "gpg-agent") def export_pubkey(self, output_file, keyid, armor=True): """Export GPG public key to a file""" @@ -31,7 +32,7 @@ class LocalSigner(object): """Sign RPM files""" cmd = self.rpm_bin + " --addsign --define '_gpg_name %s' " % keyid -gpg_args = '--no-permission-warning --batch --passphrase=%s' % passphrase +gpg_args = '--no-permission-warning --batch --passphrase=%s --agent-program=%s|--auto-expand-secmem' % (passphrase, self.gpg_agent_bin) if self.gpg_version > (2,1,): gpg_args += ' --pinentry-mode=loopback' cmd += "--define '_gpg_sign_cmd_extra_args %s' " % gpg_args @@ -71,6 +72,9 @@ class LocalSigner(object): if self.gpg_version > (2,1,): cmd += ['--pinentry-mode', 'loopback'] +if self.gpg_agent_bin: +cmd += ["--agent-program=%s|--auto-expand-secmem" % (self.gpg_agent_bin)] + cmd += [input_file] try: @@ -99,7 +103,7 @@ class LocalSigner(object): import subprocess try: ver_str = subprocess.check_output((self.gpg_bin, "--version", "--no-permission-warning")).split()[2].decode("utf-8") -return tuple([int(i) for i in ver_str.split('.')]) +return tuple([int(i) for i in ver_str.split("-")[0].split('.')]) except subprocess.CalledProcessError as e: raise bb.build.FuncFailed("Could not get gpg version: %s" % e) diff --git a/meta/recipes-core/meta/signing-keys.bb b/meta/recipes-core/meta/signing-keys.bb index 2c1cc3845ea..6387d90d474 100644 --- a/meta/recipes-core/meta/signing-keys.bb +++ b/meta/recipes-core/meta/signing-keys.bb @@ -41,6 +41,7 @@ python do_get_public_keys () { } do_get_public_keys[cleandirs] = "${B}" addtask get_public_keys before do_install +do_get_public_keys[depends] += "gnupg-native:do_populate_sysroot" do_install () { if [ -f "${B}/rpm-key" ]; then -- 2.15.1 -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 4/4] gnupg: use native version for signing, rather than one provided by host
Great that you figure out a solution. So I belive we need to revert this commit: commit 043d9ac0ae441e9a7e2ea8934bfc595a03ef9a52 Author: Leonardo Sandoval Date: Mon Sep 25 13:52:59 2017 -0700 sign_rpm.bbclass: force rpm serial signing Newer versions of gpg (at least 2.1.5 and 2.2.1) have issues when signing occurs in parallel so (unfortunately) the signing must be done serially. Once the upstream problem is fixed, this patch must be reverted, otherwise we loose all the intrinsic parallelism from bitbake. [YOCTO #12022] (From OE-Core rev: 5301712f9735fcf8d3dec756772668de930e53fe) On Wed, 10 Jan 2018 14:27:42 +0200 Alexander Kanavin wrote: > Using host gpg has been problematic, and particularly this removes > the need to serialize package creation, as long as --auto-expand-secmem > is passed to gpg-agent, and gnupg >= 2.2.4 is in use > (https://dev.gnupg.org/T3530). > > Sadly, gpg-agent itself is single-threaded, so in the longer run > we might want to seek alternatives: > https://lwn.net/Articles/742542/ > > (a smaller issue is that rpm itself runs the gpg fronted in a serial > fashion, which slows down the build in cases of recipes with very > large amount of packages, e.g. glibc-locale) > > Note that sstate signing and verification continues to use host > gpg, as depending on native gpg would create circular dependencies. > > [YOCTO #12022] > > Signed-off-by: Alexander Kanavin > --- > meta/classes/sign_package_feed.bbclass | 2 +- > meta/classes/sign_rpm.bbclass | 6 +- > meta/lib/oe/gpg_sign.py| 8 ++-- > meta/recipes-core/meta/signing-keys.bb | 1 + > 4 files changed, 9 insertions(+), 8 deletions(-) > > diff --git a/meta/classes/sign_package_feed.bbclass > b/meta/classes/sign_package_feed.bbclass > index f03c4802d06..7ff3a35a2fa 100644 > --- a/meta/classes/sign_package_feed.bbclass > +++ b/meta/classes/sign_package_feed.bbclass > @@ -43,4 +43,4 @@ python () { > } > > do_package_index[depends] += "signing-keys:do_deploy" > -do_rootfs[depends] += "signing-keys:do_populate_sysroot" > +do_rootfs[depends] += "signing-keys:do_populate_sysroot > gnupg-native:do_populate_sysroot" > diff --git a/meta/classes/sign_rpm.bbclass b/meta/classes/sign_rpm.bbclass > index 4961b03618f..64ae7ce30e3 100644 > --- a/meta/classes/sign_rpm.bbclass > +++ b/meta/classes/sign_rpm.bbclass > @@ -68,8 +68,4 @@ python sign_rpm () { > do_package_index[depends] += "signing-keys:do_deploy" > do_rootfs[depends] += "signing-keys:do_populate_sysroot" > > -# Newer versions of gpg (at least 2.1.5 and 2.2.1) have issues when signing > occurs in parallel > -# so unfortunately the signing must be done serially. Once the upstream > problem is fixed, > -# the following line must be removed otherwise we loose all the intrinsic > parallelism from > -# bitbake. For more information, check > https://bugzilla.yoctoproject.org/show_bug.cgi?id=12022. > -do_package_write_rpm[lockfiles] += "${TMPDIR}/gpg.lock" > +PACKAGE_WRITE_DEPS += "gnupg-native" > diff --git a/meta/lib/oe/gpg_sign.py b/meta/lib/oe/gpg_sign.py > index 9cc88f020c1..b17272928fc 100644 > --- a/meta/lib/oe/gpg_sign.py > +++ b/meta/lib/oe/gpg_sign.py > @@ -12,6 +12,7 @@ class LocalSigner(object): > self.gpg_path = d.getVar('GPG_PATH') > self.gpg_version = self.get_gpg_version() > self.rpm_bin = bb.utils.which(os.getenv('PATH'), "rpmsign") > +self.gpg_agent_bin = bb.utils.which(os.getenv('PATH'), "gpg-agent") > > def export_pubkey(self, output_file, keyid, armor=True): > """Export GPG public key to a file""" > @@ -31,7 +32,7 @@ class LocalSigner(object): > """Sign RPM files""" > > cmd = self.rpm_bin + " --addsign --define '_gpg_name %s' " % keyid > -gpg_args = '--no-permission-warning --batch --passphrase=%s' % > passphrase > +gpg_args = '--no-permission-warning --batch --passphrase=%s > --agent-program=%s|--auto-expand-secmem' % (passphrase, self.gpg_agent_bin) > if self.gpg_version > (2,1,): > gpg_args += ' --pinentry-mode=loopback' > cmd += "--define '_gpg_sign_cmd_extra_args %s' " % gpg_args > @@ -71,6 +72,9 @@ class LocalSigner(object): > if self.gpg_version > (2,1,): > cmd += ['--pinentry-mode', 'loopback'] > > +if self.gpg_agent_bin: > +cmd += ["--agent-program=%s|--auto-expand-secmem" % > (self.gpg_agent_bin)] > + > cmd += [input_file] > > try: > @@ -99,7 +103,7 @@ class LocalSigner(object): > import subprocess > try: > ver_str = subprocess.check_output((self.gpg_bin, "--version", > "--no-permission-warning")).split()[2].decode("utf-8") > -return tuple([int(i) for i in ver_str.split('.')]) > +return tuple([int(i) for i in ver_str.split("-")[0].split('.')]) > except subprocess.CalledProcessError as e: > raise b
Re: [OE-core] [PATCH 4/4] gnupg: use native version for signing, rather than one provided by host
On 01/10/2018 05:01 PM, Leonardo Sandoval wrote: Great that you figure out a solution. So I belive we need to revert this commit: commit 043d9ac0ae441e9a7e2ea8934bfc595a03ef9a52 Author: Leonardo Sandoval Date: Mon Sep 25 13:52:59 2017 -0700 sign_rpm.bbclass: force rpm serial signing The revert is already included in the patch... :) Alex -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 4/4] gnupg: use native version for signing, rather than one provided by host
-- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core