bug#44559: gnutls 3.6.12 fails to build: FAIL: status-request-revoked
Hi, Ludovic Courtès writes: > Hi, > > Maxim Cournoyer skribis: > >> I lost track of the details -- is this problem still likely to bite us >> in the future, or was something committed to mitigate against it? > > “Time bombs” are the worst of non-reproducibility bugs because (1) we > don’t have workarounds other than changing the system time on the > machine that performs the build, and (2) we currently lack a way to > detect them. I see. Thanks for explaining! > In this thread, Chris and I proposed addressing #2 by offering ways to > automatically or manually run builds “in the future”¹. We could actually > similarly address #1 by running builds “in the past”. > > In any case, the first step is probably to offer a VM service for Guix > System, similar to childhurd, with an option to specify the starting > time. > > So we’re not done yet, but the discussion has drifted beyond the > original GnuTLS bug report, so perhaps we can close it and take it > elsewhere? I think so! Few seems to visit these years old reports. If there's something to brainstorm about, I'd rather see it happen in a new thread on guix-devel. Thanks for the reply! Closing. Maxim
bug#44559: gnutls 3.6.12 fails to build: FAIL: status-request-revoked
Hi, Maxim Cournoyer skribis: > I lost track of the details -- is this problem still likely to bite us > in the future, or was something committed to mitigate against it? “Time bombs” are the worst of non-reproducibility bugs because (1) we don’t have workarounds other than changing the system time on the machine that performs the build, and (2) we currently lack a way to detect them. In this thread, Chris and I proposed addressing #2 by offering ways to automatically or manually run builds “in the future”¹. We could actually similarly address #1 by running builds “in the past”. In any case, the first step is probably to offer a VM service for Guix System, similar to childhurd, with an option to specify the starting time. So we’re not done yet, but the discussion has drifted beyond the original GnuTLS bug report, so perhaps we can close it and take it elsewhere? Thanks, Ludo’. ¹ QEMU can be passed an option like “-rtc base=2020-01-01”.
bug#44559: gnutls 3.6.12 fails to build: FAIL: status-request-revoked
Hi everyone, Ludovic Courtès writes: > Hi, > > Christopher Baines skribis: > >> Ludovic Courtès writes: >> >>> Perhaps we could start by testing this hypothesis on a separate build >>> farm. Chris, Mathieu, WDYT? >> >> I'm currently thinking about attempting these kind of things (testing >> building derivations under different conditions) through the agent tags >> in the Guix Build Coordinator. >> >> I haven't used this functionality yet, but it's mostly implemented. The >> idea is that agents have tags, that describe various attributes that are >> important (time=normal, time=future, maybe for example), and builds can >> also be targeted at specific agents by tagging the builds with those >> same tags. > > Sounds nice! Also varying kernels I guess. > >> Where I'm going with this is that I'm not sure a separate build farm is >> needed, it would be good to just incorperate this in to the build farm >> used for testing patches and non-master branches. > > Sure. For the build-in-the-future thing, I think we could just do that > by default; what I meant is that we just need to double-check beforehand > that nothing breaks badly. I lost track of the details -- is this problem still likely to bite us in the future, or was something committed to mitigate against it? Thanks, Maxim
bug#44559:
Ludovic Courtès writes: > Hi, > > Christopher Baines skribis: > >> Ludovic Courtès writes: >> >>> Perhaps we could start by testing this hypothesis on a separate build >>> farm. Chris, Mathieu, WDYT? >> >> I'm currently thinking about attempting these kind of things (testing >> building derivations under different conditions) through the agent tags >> in the Guix Build Coordinator. >> >> I haven't used this functionality yet, but it's mostly implemented. The >> idea is that agents have tags, that describe various attributes that are >> important (time=normal, time=future, maybe for example), and builds can >> also be targeted at specific agents by tagging the builds with those >> same tags. > > Sounds nice! Also varying kernels I guess. Yeah, there's a whole range of variations it would be nice to methodically test against (filesystems, Linux versions, system time, one vs many cores, ...). >> Where I'm going with this is that I'm not sure a separate build farm is >> needed, it would be good to just incorperate this in to the build farm >> used for testing patches and non-master branches. > > Sure. For the build-in-the-future thing, I think we could just do that > by default; what I meant is that we just need to double-check beforehand > that nothing breaks badly. Ah, I guess that might work, if I have some time, I'll have a look in to making the necessary machine changes. signature.asc Description: PGP signature
bug#44559:
Hi, Christopher Baines skribis: > Ludovic Courtès writes: > >> Perhaps we could start by testing this hypothesis on a separate build >> farm. Chris, Mathieu, WDYT? > > I'm currently thinking about attempting these kind of things (testing > building derivations under different conditions) through the agent tags > in the Guix Build Coordinator. > > I haven't used this functionality yet, but it's mostly implemented. The > idea is that agents have tags, that describe various attributes that are > important (time=normal, time=future, maybe for example), and builds can > also be targeted at specific agents by tagging the builds with those > same tags. Sounds nice! Also varying kernels I guess. > Where I'm going with this is that I'm not sure a separate build farm is > needed, it would be good to just incorperate this in to the build farm > used for testing patches and non-master branches. Sure. For the build-in-the-future thing, I think we could just do that by default; what I meant is that we just need to double-check beforehand that nothing breaks badly. Thanks, Ludo’.
bug#44559:
Ludovic Courtès writes: > Perhaps we could start by testing this hypothesis on a separate build > farm. Chris, Mathieu, WDYT? I'm currently thinking about attempting these kind of things (testing building derivations under different conditions) through the agent tags in the Guix Build Coordinator. I haven't used this functionality yet, but it's mostly implemented. The idea is that agents have tags, that describe various attributes that are important (time=normal, time=future, maybe for example), and builds can also be targeted at specific agents by tagging the builds with those same tags. Where I'm going with this is that I'm not sure a separate build farm is needed, it would be good to just incorperate this in to the build farm used for testing patches and non-master branches. signature.asc Description: PGP signature
bug#44559: Detecting “expiring” builds
It occurred to me that, just like we have childhurds, we could provide a service that sets up a sub-Guix System running in a VM with its clock set years ahead, and you would offload to that. That’s not perfect, but it’s a rather easy addition. Another option would be to have built-in support in the daemon. If you turn on some option, it would transparently run derivation builds in qemu-user (does that support changing the system date?) or similar, but that’s more work. Ludo’.
bug#44559:
Hi,
Maxime Devos skribis:
> On Fri, 2021-02-19 at 16:33 +0100, Ludovic Courtès wrote:
>> [...]
>> Longer-term, we need to find a way to address or avoid this issue. A
>> brute-force approach would be to have the build machines at ci.guix run
>> with a clock ten years ahead. That should generally be fine since the
>> only place where timestamps matter are unmodified upstream tarballs. In
>> all other cases, mtime is set to 1.
>
> Alternatively, could the build container be adjusted to always begin at
> 1970-01-01, using ‘time namespaces’?
>
> Linux: https://lwn.net/Articles/766089/
Unfortunately, time namespaces are just for CLOCK_{MONOTONIC,BOOTTIME},
which I think is of little use here:
https://issues.guix.gnu.org/44559#3
> Also, is there any particular reason to set the clock only ten years ahead,
> and not, say, a millenia or two? Some possible reasons:
>
> * year 2038,2446 problem: the ext2 and ext4 filesystems have a restricted
> date range
> * year 2038 problem:
> https://www.gnu.org/software/hurd/gnumach-doc/Host-Interface.html#Host-Interface
>
> IMO, the year 2038 problem is a bug and affected packages should simply be
> fixed.
> But perhaps reality is a little more complicated.
Yeah, one problem at a time. :-)
Setting it 10 years ahead would cache the kind of issue we’re talking
about, while not opening the Y2038 can of worms. I think we need to try
that out and see how it goes.
Ludo’.
bug#44559:
Hi Guix! Thanks to all of you for your thoughtful replies! On Feb 19, 2021, at 10:33 AM, Ludovic Courtès wrote: > I agree it’s a problem, and yes, it would probably be a good idea to > release 1.2.1 with the upgraded GnuTLS we now have in ‘master’. I’m very heartened by your affirmation of the project’s support of bootstrappability and building from source. :-) In addition, I think it would be good to make sure that the package transformation options are powerful enough to allow users to sidestep these problems in their own workflow and decrease the pressure on maintainers. On Feb 19, 2021, at 10:33 AM, Ludovic Courtès wrote: > ‘--without-tests’ should work, but you need to pass the right version > number I guess? Oh! That may be the case. I am using `guix time-machine` however, and that does not yet have the `--without-tests` flag, I have opened bug#46650 so that we can discuss that issue there. On Feb 19, 2021, at 1:32 PM, Maxime Devos wrote: > Alternatively, could the build container be adjusted to always begin at > 1970-01-01, using ‘time namespaces’? Unfortunately, as Ludovic mentioned earlier in this thread, time_namespaces(7) is only for CLOCK_MONOTONIC and. CLOCK_BOOTTIME. :-( Carl Dong cont...@carldong.me "I fight for the users" > On Feb 19, 2021, at 10:33 AM, Ludovic Courtès wrote: > > Hi Carl, > > Carl Dong skribis: > >> As bitcoin core begins the planning to officially transition to Guix-based >> releases, I've had many community members build guix v1.2.0 from source and >> afterward attempt `--bootstrap --no-substitutes` builds. As you may imagine, >> they are getting stuck on this gnutls problem and cannot proceed further. > > Yeah. :-/ > >> I'm wondering: >> >> 1. Is there a workaround that does not involve changing the system time? We >> have attempted several flags: >> 1. --with-graft=gnutls=gnutls@3.6.14 >> 2. --without-tests=gnutls >> 3. --with-input=gnutls=gnutls@3.6.14 >> These attempts all failed to work around this bug, and I’m curious as >> to why that would be. My guess would be that when we do `--bootstrap`, Guix >> bootstraps itself first without taking into account these flags? > > ‘--without-tests’ should work, but you need to pass the right version > number I guess? > >> 2. Since bootstrappability is one of the core tenets of Guix, might it be >> appropriate to cut a v1.2.1 release with this problem (and any other >> potential bootstrap problems) fixed? (Happy to discuss in separate thread if >> more appropriate) > > I agree it’s a problem, and yes, it would probably be a good idea to > release 1.2.1 with the upgraded GnuTLS we now have in ‘master’. > > Longer-term, we need to find a way to address or avoid this issue. A > brute-force approach would be to have the build machines at ci.guix run > with a clock ten years ahead. That should generally be fine since the > only place where timestamps matter are unmodified upstream tarballs. In > all other cases, mtime is set to 1. > > Perhaps we could start by testing this hypothesis on a separate build > farm. Chris, Mathieu, WDYT? > > Thanks, > Ludo’.
bug#44559:
Hi Guix, On Fri, 2021-02-19 at 16:33 +0100, Ludovic Courtès wrote: > [...] > Longer-term, we need to find a way to address or avoid this issue. A > brute-force approach would be to have the build machines at ci.guix run > with a clock ten years ahead. That should generally be fine since the > only place where timestamps matter are unmodified upstream tarballs. In > all other cases, mtime is set to 1. Alternatively, could the build container be adjusted to always begin at 1970-01-01, using ‘time namespaces’? Linux: https://lwn.net/Articles/766089/ Hurd analogue: https://www.gnu.org/software/hurd/gnumach-doc/Host-Interface.html#Host-Interface (Of course, someone needs to find the time to write the patches first. Maybe I'll have a try at it eventually, but probably not anytime soon.) Also, is there any particular reason to set the clock only ten years ahead, and not, say, a millenia or two? Some possible reasons: * year 2038,2446 problem: the ext2 and ext4 filesystems have a restricted date range * year 2038 problem: https://www.gnu.org/software/hurd/gnumach-doc/Host-Interface.html#Host-Interface IMO, the year 2038 problem is a bug and affected packages should simply be fixed. But perhaps reality is a little more complicated. Greetings, Maxime signature.asc Description: This is a digitally signed message part
bug#44559:
Hi Carl, Carl Dong skribis: > As bitcoin core begins the planning to officially transition to Guix-based > releases, I've had many community members build guix v1.2.0 from source and > afterward attempt `--bootstrap --no-substitutes` builds. As you may imagine, > they are getting stuck on this gnutls problem and cannot proceed further. Yeah. :-/ > I'm wondering: > > 1. Is there a workaround that does not involve changing the system time? We > have attempted several flags: > 1. --with-graft=gnutls=gnutls@3.6.14 > 2. --without-tests=gnutls > 3. --with-input=gnutls=gnutls@3.6.14 > These attempts all failed to work around this bug, and I’m curious as > to why that would be. My guess would be that when we do `--bootstrap`, Guix > bootstraps itself first without taking into account these flags? ‘--without-tests’ should work, but you need to pass the right version number I guess? > 2. Since bootstrappability is one of the core tenets of Guix, might it be > appropriate to cut a v1.2.1 release with this problem (and any other > potential bootstrap problems) fixed? (Happy to discuss in separate thread if > more appropriate) I agree it’s a problem, and yes, it would probably be a good idea to release 1.2.1 with the upgraded GnuTLS we now have in ‘master’. Longer-term, we need to find a way to address or avoid this issue. A brute-force approach would be to have the build machines at ci.guix run with a clock ten years ahead. That should generally be fine since the only place where timestamps matter are unmodified upstream tarballs. In all other cases, mtime is set to 1. Perhaps we could start by testing this hypothesis on a separate build farm. Chris, Mathieu, WDYT? Thanks, Ludo’.
bug#44559:
On Tue, Feb 16, 2021 at 04:00:11PM -0500, Carl Dong wrote: > Hi all, > > As bitcoin core begins the planning to officially transition to Guix-based > releases, I've had many community members build guix v1.2.0 from source and > afterward attempt `--bootstrap --no-substitutes` builds. As you may imagine, > they are getting stuck on this gnutls problem and cannot proceed further. > > I'm wondering: > > 1. Is there a workaround that does not involve changing the system time? We > have attempted several flags: > 1. --with-graft=gnutls=gnutls@3.6.14 > 2. --without-tests=gnutls > 3. --with-input=gnutls=gnutls@3.6.14 > These attempts all failed to work around this bug, and I’m curious as > to why that would be. My guess would be that when we do `--bootstrap`, Guix > bootstraps itself first without taking into account these flags? > > 2. Since bootstrappability is one of the core tenets of Guix, might it be > appropriate to cut a v1.2.1 release with this problem (and any other > potential bootstrap problems) fixed? (Happy to discuss in separate thread if > more appropriate) You should see what the Guix maintainers say about this. My personal opinion is that you should fork Guix your use case. If you are building from the bootstrap, there is little added cost to making minor adjustments like disabling this test. You can periodically re-sync your fork with GNU Guix as convenient. And it's probably more in tune with your threat model. [0] This problem of "expiring software" has occurred several times in Guix's history and I'm sure it will happen again. In general, users are expected to use substitutes to work around it. They are no worse off than with traditional binary distros in that case. [0] Savannah is great but lacking the resources to devote to security.
bug#44559:
Hi all, As bitcoin core begins the planning to officially transition to Guix-based releases, I've had many community members build guix v1.2.0 from source and afterward attempt `--bootstrap --no-substitutes` builds. As you may imagine, they are getting stuck on this gnutls problem and cannot proceed further. I'm wondering: 1. Is there a workaround that does not involve changing the system time? We have attempted several flags: 1. --with-graft=gnutls=gnutls@3.6.14 2. --without-tests=gnutls 3. --with-input=gnutls=gnutls@3.6.14 These attempts all failed to work around this bug, and I’m curious as to why that would be. My guess would be that when we do `--bootstrap`, Guix bootstraps itself first without taking into account these flags? 2. Since bootstrappability is one of the core tenets of Guix, might it be appropriate to cut a v1.2.1 release with this problem (and any other potential bootstrap problems) fixed? (Happy to discuss in separate thread if more appropriate) Cheers, Carl Dong cont...@carldong.me "I fight for the users"
bug#44559: Solution
I created a procedure to work around the build failure and enable a successful build: # when gnutls-3.6.12 build breaks you need to do: # run the following as root # turn off networking ip link set enp0s25 down # Fixup the time so that the build will succeed timedatectl set-ntp false timedatectl set-time '2020-10-01' # Try to build but it will absolutely fail by lack of source if you # don't enable networking or because you enable networking. # But turn on networking when specified below guix time-machine --commit=94585fffb23079fe71110e2bf99782eb4ccfa12b -- # build --no-grafts --check guile3.0-gnutls # after it finishes building # /gnu/store/vhphki5sg9xkdhh2pbc8gi6vhpfzryf0-gnutls-3.6.12.drv # and starts building # /gnu/store/akm0wl58avib46g3d9razlfzfgfg8m6m-python-3.8.2.drv... # but before it begins building # /gnu/store/bja7gqzxr62a0akid0rpzmynzy78nkwg-zstd-1.4.4.tar.gz.drv. # Fix the time and turn on networking as it has additional things to # download. # specifically # https://github.com/facebook/zstd/releases/download/v1.4.4/zstd-1.4.4.tar.gz # and substitutes for some reason # failing to do so will result in you needing to repeat the above steps # again. timedatectl set-time '$current_date' timedatectl set-ntp true # turn on networking ip link set enp0s25 up # it'll fail building because the time is current again # But that is fine, we now will not need networking for this build cycle # and thus the altered time will be fine and the build will be # successful this time # turn off networking ip link set enp0s25 down # Fixup the time so that the build will succeed timedatectl set-ntp false timedatectl set-time '2020-10-01' #guix build gnutls@3.6.12 finally guix time-machine --commit=94585fffb23079fe71110e2bf99782eb4ccfa12b -- # build --no-grafts --check guile3.0-gnutls # wait until it completes. # Then we can put the system back in a working state timedatectl set-time '$current_date' timedatectl set-ntp true # turn on networking ip link set enp0s25 up
bug#44559: gnutls 3.6.12 fails to build: FAIL: status-request-revoked
Hi, Marius Bakke skribis: > Ludovic Courtès writes: > >> The question for us becomes how to ensure long-term reproducibility in >> the presence of such bugs. >> >> In this case, I think the only solution would be to change the system >> clock when one rebuilds GnuTLS (or to use ‘--without-tests=gnutls’, but >> you end up with different derivations, which is not necessarily >> desirable). >> >> Thoughts? > > There is a related bug report here: > > https://issues.guix.gnu.org/39310 > > Perhaps we could make a "--with-system-clock" option for 'guix build' > that instructs the daemon to fake the system time? How would it fake it though? There are time_namespaces(7), but it’s only for CLOCK_MONOTONIC and CLOCK_BOOTTIME. LD_PRELOAD like ‘datefudge’ does is probably not a viable option. Ludo’.
bug#44559: gnutls 3.6.12 fails to build: FAIL: status-request-revoked
Ludovic Courtès writes: > The question for us becomes how to ensure long-term reproducibility in > the presence of such bugs. > > In this case, I think the only solution would be to change the system > clock when one rebuilds GnuTLS (or to use ‘--without-tests=gnutls’, but > you end up with different derivations, which is not necessarily > desirable). > > Thoughts? There is a related bug report here: https://issues.guix.gnu.org/39310 Perhaps we could make a "--with-system-clock" option for 'guix build' that instructs the daemon to fake the system time? signature.asc Description: PGP signature
bug#44559: gnutls 3.6.12 fails to build: FAIL: status-request-revoked
Hi, Christopher Baines skribis: > I found this when trying to build guile3.0-gnutls: > > guix time-machine --commit=94585fffb23079fe71110e2bf99782eb4ccfa12b -- > build --no-grafts --check guile3.0-gnutls > > > FAIL: status-request-revoked > > > trying NORMAL:-VERS-ALL:+VERS-TLS1.2 > received status request > received status request > cert_verify_callback:263: certificate verify status doesn't match: 100402 != > 22FAIL status-request-revoked (exit status: 1) This was fixed upstream between 3.6.12 and 3.6.14 with this patch by Bernhard (it’s a small world!): --8<---cut here---start->8--- commit ed208fe55f31478732fd6cc394f9576b315a42cd Author: Bernhard M. Wiedemann Date: Sun Apr 5 15:09:57 2020 +0200 tests: Fix status-request-revoked after 2020-10-24 included certs expire 2020-10-24 so this test fails after that date. Fixes #967 This patch was done while working on reproducible builds for openSUSE. Signed-off-by: Bernhard M. Wiedemann --8<---cut here---end--->8--- The question for us becomes how to ensure long-term reproducibility in the presence of such bugs. In this case, I think the only solution would be to change the system clock when one rebuilds GnuTLS (or to use ‘--without-tests=gnutls’, but you end up with different derivations, which is not necessarily desirable). Thoughts? Ludo’.
bug#44559: gnutls 3.6.12 fails to build: FAIL: status-request-revoked
I found this when trying to build guile3.0-gnutls: guix time-machine --commit=94585fffb23079fe71110e2bf99782eb4ccfa12b -- build --no-grafts --check guile3.0-gnutls FAIL: status-request-revoked trying NORMAL:-VERS-ALL:+VERS-TLS1.2 received status request received status request cert_verify_callback:263: certificate verify status doesn't match: 100402 != 22FAIL status-request-revoked (exit status: 1) signature.asc Description: PGP signature
