Re: Making schleuder build reproducibly
Hi all, On 18-03-13 15:20:48, Georg Faerber wrote: > On 17-10-30 18:21:39, Georg Faerber wrote: > > On 17-06-15 21:19:12, Georg Faerber wrote: > > > I really would like to make the build of schleuder, a gpg enabled > > > mailing list, reproducible. However, I'm a bit lost on my own, > > > that's why I'm searching for input with this mail: > > > > tl;dr: Due to recent changes and fixes upstream, schleuder now does > > build reproducible. For details, please read on. > > Well, this wasn't true back then, however, it looks better now [1]. > This is mainly because of this commit [2]. > > @dkg: I strongly believe that dirmngr still has some problems if invoked > in a chroot without network access, which is why the above commit was > needed. Also, one job recently still failed on the tests.r-b.o infra: > GPG reported "connecting dirmngr failed: IPC connect call failed". > Searching on the Internets leads to [3], however, I'm unsure what to do > regarding this. Any input? One of last tests now also ran into this (which only happened once): Failures: 1) Schleuder::ListBuilder creates a listdir for the list Failure/Error: gpg.generate_key(key_params(list)) GPGME::Error: No agent running # ./lib/schleuder/list_builder.rb:73:in `create_key' # ./lib/schleuder/list_builder.rb:41:in `run' # ./spec/schleuder/unit/list_builder_spec.rb:30:in `block (2 levels) in ' # ./spec/spec_helper.rb:46:in `block (3 levels) in ' # ./spec/spec_helper.rb:45:in `block (2 levels) in ' @dkg: I would still be very happy to get some input on this, to make the build reliably reproducible. Cheers, Georg signature.asc Description: Digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: Making schleuder build reproducibly
Hi all, Update on this: On 17-10-30 18:21:39, Georg Faerber wrote: > On 17-06-15 21:19:12, Georg Faerber wrote: > > I really would like to make the build of schleuder, a gpg enabled > > mailing list, reproducible. However, I'm a bit lost on my own, > > that's why I'm searching for input with this mail: > > tl;dr: Due to recent changes and fixes upstream, schleuder now does > build reproducible. For details, please read on. Well, this wasn't true back then, however, it looks better now [1]. This is mainly because of this commit [2]. @dkg: I strongly believe that dirmngr still has some problems if invoked in a chroot without network access, which is why the above commit was needed. Also, one job recently still failed on the tests.r-b.o infra: GPG reported "connecting dirmngr failed: IPC connect call failed". Searching on the Internets leads to [3], however, I'm unsure what to do regarding this. Any input? Cheers, Georg [1] https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/schleuder.html [2] https://salsa.debian.org/ruby-team/schleuder/commit/b8986fd58c99b900338814c2321bc409bdc39e4b [3] https://michaelheap.com/gpg-connecting-dirmngr-failed-ipc-connect-call-failed/ signature.asc Description: Digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: Making schleuder build reproducibly
On Tue 2017-11-07 17:02:02 +0100, Georg Faerber wrote: > On 17-11-04 16:01:43, Holger Levsen wrote: >> On Mon, Oct 30, 2017 at 06:21:39PM +0100, Georg Faerber wrote: >> > @dkg: It seems, there is still a bug / race in dirmngr, which leads to >> > errors like "can't connect to '127.0.0.1': no IP address for host" and >> > in turn "marking host '127.0.0.1' as dead". See the attached debug log for >> > details, the log was taken on October 1st with dirmrngr out of unstable. >> > I'm happy to debug this further, if needed. >> >> indeed, random success+failure is visible for 3.2.1-1 on armhf: >> >> https://tests.reproducible-builds.org/debian/rb-pkg/buster/armhf/schleuder.html > > I'm actually unsure if these failures are caused by dirmngr, or if it's, > more likely, failing ruby code. I believe it's likely that your local keyserver isn't listening properly, but dirmngr's errors/warnings are certainly confusing. I've started a thread over on gnupg-devel to try to clean up the dirmngr error messages at least. If you can replicate that sequence of log messages (in particular, the "Unknown host" weirdness) with a minimal example on some system configuration, it'd be great to follow up over there. I've been unable to replicate it myself with dirmngr 2.2.2-1. --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: Making schleuder build reproducibly
On 2017-11-04, Holger Levsen wrote: > On Mon, Oct 30, 2017 at 06:21:39PM +0100, Georg Faerber wrote: >> @dkg: It seems, there is still a bug / race in dirmngr, which leads to >> errors like "can't connect to '127.0.0.1': no IP address for host" and >> in turn "marking host '127.0.0.1' as dead". See the attached debug log for >> details, the log was taken on October 1st with dirmrngr out of unstable. >> I'm happy to debug this further, if needed. > > indeed, random success+failure is visible for 3.2.1-1 on armhf: > > https://tests.reproducible-builds.org/debian/rb-pkg/buster/armhf/schleuder.html While there are some successes, they seem to be the rare minority, if you click on filter by test-history for armhf on the upper left. The last armhf build failure was due to unreachable local keyserver. The armhf builders all have firewalls, but that wouldn't explain why it sometimes succeeds and usually fails, and I didn't set up anything specifically firewalling localhost... live well, vagrant ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: Making schleuder build reproducibly
On Tue, Nov 07, 2017 at 05:02:02PM +0100, Georg Faerber wrote: > Is it possible to get temporary access to a armhf dev machine, to debug > this further? some raspi2 or 3 should do ;) -- cheers, Holger signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: Making schleuder build reproducibly
Hi Holger, On 17-11-04 16:01:43, Holger Levsen wrote: > On Mon, Oct 30, 2017 at 06:21:39PM +0100, Georg Faerber wrote: > > @dkg: It seems, there is still a bug / race in dirmngr, which leads to > > errors like "can't connect to '127.0.0.1': no IP address for host" and > > in turn "marking host '127.0.0.1' as dead". See the attached debug log for > > details, the log was taken on October 1st with dirmrngr out of unstable. > > I'm happy to debug this further, if needed. > > indeed, random success+failure is visible for 3.2.1-1 on armhf: > > https://tests.reproducible-builds.org/debian/rb-pkg/buster/armhf/schleuder.html I'm actually unsure if these failures are caused by dirmngr, or if it's, more likely, failing ruby code. Is it possible to get temporary access to a armhf dev machine, to debug this further? Thanks and cheers, Georg signature.asc Description: Digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: Making schleuder build reproducibly
Hi Georg, thanks for your summary and the work leading to it! On Mon, Oct 30, 2017 at 06:21:39PM +0100, Georg Faerber wrote: > @dkg: It seems, there is still a bug / race in dirmngr, which leads to > errors like "can't connect to '127.0.0.1': no IP address for host" and > in turn "marking host '127.0.0.1' as dead". See the attached debug log for > details, the log was taken on October 1st with dirmrngr out of unstable. > I'm happy to debug this further, if needed. indeed, random success+failure is visible for 3.2.1-1 on armhf: https://tests.reproducible-builds.org/debian/rb-pkg/buster/armhf/schleuder.html -- cheers, Holger signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: Making schleuder build reproducibly
Hi all, On 17-06-15 21:19:12, Georg Faerber wrote: > I really would like to make the build of schleuder, a gpg enabled > mailing list, reproducible. However, I'm a bit lost on my own, that's > why I'm searching for input with this mail: tl;dr: Due to recent changes and fixes upstream, schleuder now does build reproducible. For details, please read on. -- Well, it took me quite a while and some time, but it was worth it, it seems [1]. Here's a short summary of the changes: - Sometimes, gpg-agent gets stuck, killing it before and after running each test helps [2]. - Some tests expected specific dates (like 2016-12-06) to be returned by gpg. However, these dates are non deterministic, they depend on locale settings. Quoting a comment in the corresponding upstream merge request: "Oh my... I wondered before why gpg doesn't show timezone-information anywhere but assumed that they just don't store it or hide it because it doesn't add actual information. TIL gpg knows about timezones and also converts dates but doesn't tell about it... So actually in order to compare a key-date I have to remember (or guess) which timezone it was created in, and calulate the time-difference to my local timezone, in order to guess if a off-by-one-day-mismatch is maybe not an actual mismatch? I'm shaking my head." Therefore, we've relaxed the expected output, to not match specific dates, but a specific format, like -MM-DD. [3] - Some tests check if getting and / or refreshing keys via the keyserver mechanism works. For this to work, a local keyserver is started, which mocks the behaviour of the keyservers on the Internets. In the past, we just started the keyserver before running the test and assumed it to work. Sometimes this failed, because the keyserver took longer to start, leading to failed tests, because it was in fact unreachable. This was solved via introducing a check for the keyserver to be up, before running the test. [4] - I've never encountered the following during the "official" tests run by jenkins, but I've run into this three times while using reprotest. I'm not sure if this is a bug, race or a feature, but during the second build, the build dir was read only. In the past, the database in which schleuder stores list information was just below the build dir, read only in these cases, leading to a hanging test suite. This was solved via introducing erb code parsing in the schleuder config [5], and using this feature via env vars in Debian to move the database dir below /tmp/ [6]. @dkg: It seems, there is still a bug / race in dirmngr, which leads to errors like "can't connect to '127.0.0.1': no IP address for host" and in turn "marking host '127.0.0.1' as dead". See the attached debug log for details, the log was taken on October 1st with dirmrngr out of unstable. I'm happy to debug this further, if needed. That's all for now! Thanks for this initiative and the work all of you're putting into it, highly appreciated! Cheers, Georg [1] https://tests.reproducible-builds.org/debian/history/schleuder.html [2] https://0xacab.org/schleuder/schleuder/commit/8ecd8a4cdc76bc8fcdee54397fd0928d338146c1 [3] https://0xacab.org/schleuder/schleuder/commit/e52b6851b30d2578d2eb6c451425549a630d2ba6 [4] https://0xacab.org/schleuder/schleuder/commit/5acd2df470e420ef54ac1bf1193638ec8a23bbba [5] https://0xacab.org/schleuder/schleuder/commit/d188224ce60f8850001825dc94a7d1fa470342ae [6] https://anonscm.debian.org/cgit/pkg-ruby-extras/schleuder.git/commit/?id=f1cafc243766ecd0ad3ec84d8b1adf26ef2e0c66 2017-10-01 06:16:42 dirmngr[32131] listening on socket '/tmp/schleuder-test/example.org/list62/S.dirmngr' 2017-10-01 06:16:42 dirmngr[32132.0] permanently loaded certificates: 149 2017-10-01 06:16:42 dirmngr[32132.0] runtime cached certificates: 0 2017-10-01 06:16:42 dirmngr[32132.0]trusted certificates: 149 (148,0,0,1) 2017-10-01 06:16:42 dirmngr[32132.0] failed to open cache dir file '/tmp/schleuder-test/example.org/list62/crls.d/DIR.txt': No such file or directory 2017-10-01 06:16:42 dirmngr[32132.0] creating directory '/tmp/schleuder-test/example.org/list62/crls.d' 2017-10-01 06:16:42 dirmngr[32132.0] new cache dir file '/tmp/schleuder-test/example.org/list62/crls.d/DIR.txt' created 2017-10-01 06:16:42 dirmngr[32132.6] handler for fd 6 started 2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 -> # Home: /tmp/schleuder-test/example.org/list62 2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 -> # Config: /tmp/schleuder-test/example.org/list62/dirmngr.conf 2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 -> OK Dirmngr 2.2.1 at your service 2017-10-01 06:16:42 dirmngr[32132.6] connection from process 32128 (0:0) 2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 <- GETINFO version 2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 -> D 2.2.1 2017-10-01 06:16:42 dirmngr[32132.6] DBG: chan_6 -> OK 2017-10-01 06:16:42 dirmngr[32132.6] DBG:
Re: Making schleuder build reproducibly
Hi Georg-- On Thu 2017-06-15 21:19:12 +0200, Georg Faerber wrote: > I really would like to make the build of schleuder, a gpg enabled > mailing list, reproducible. However, I'm a bit lost on my own, that's > why I'm searching for input with this mail: > > Some of the upstream provided tests check if importing a key from a > keyserver work. For this, a keyserver is started, bound to localhost to > mock the behaviour of the keyservers out there. This seems to work > sometimes [1], sometimes it fails, leading to > > [2] gpg: error retrieving 'ad...@example.org' via keyserver: No keyserver > available > > or > > [2] gpg: connecting dirmngr at > '/tmp/schleuder-test/example.org/list4/S.dirmngr' failed: IPC connect call > failed Hm, these seem like they could be failures of either: * dirmngr * the test/demo keyserver have you tried instrumenting either of these components of the test suite to log more data so that you can see why they're failing? for dimrngr, you'd do something like adding the following to dirmngr.conf in the example homedir: debug-level guru log-file /tmp/schleuder-test/example.org/list4/dirmngr.log debug 0x verbose and then in the event of a test failure, dump the contents of dirmngr.log I'm not sure how the test/demo keyserver is currently implemented so i'm not sure how to get better diagnostics from it. hth, --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Making schleuder build reproducibly
Dear reproducible builds, I really would like to make the build of schleuder, a gpg enabled mailing list, reproducible. However, I'm a bit lost on my own, that's why I'm searching for input with this mail: Some of the upstream provided tests check if importing a key from a keyserver work. For this, a keyserver is started, bound to localhost to mock the behaviour of the keyservers out there. This seems to work sometimes [1], sometimes it fails, leading to [2] gpg: error retrieving 'ad...@example.org' via keyserver: No keyserver available or [2] gpg: connecting dirmngr at '/tmp/schleuder-test/example.org/list4/S.dirmngr' failed: IPC connect call failed Thanks for any pointers and the work you're doing! Cheers, Georg [1] https://tests.reproducible-builds.org/debian/rbuild/unstable/armhf/schleuder_3.1.0-2.rbuild.log [2] https://tests.reproducible-builds.org/debian/rbuild/unstable/i386/schleuder_3.1.0-2.rbuild.log [3] https://tests.reproducible-builds.org/debian/rbuild/unstable/amd64/schleuder_3.1.0-2.rbuild.log signature.asc Description: Digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
