Re: Corrupt/missing store items
On Tue, 22 Jun 2021, Ricardo Wurmus wrote: Jack Hill writes: What happens if you try “guix build --check --no-grafts /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv” or similar? The database probably states that /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16 has already been built, so with “--check” you might be able to make it build it again. `guix gc --verify=repair` fixed the problem, so I won't be able to test this (unless this recurs, but I hope it doesn't). So Guix think's it's sucessfully built gnutls, but it's still not present in the store. I've also tried copying the store items from other hosts. No luck so far… How did that fail? I didn't capture verbose output, but I believe nothing was sent because the problematic host reported it already had that item. How could the directory have disappeared? Guix appears to be very surprised that the directory is missing. Apologies for leaving out more details about this host earlier. The filesystem is btrfs and its a virtual machine running in someone else's VM cluster. I had to reset and boot into older generations a few times before I noticed the problem because of a misconfiguration I made. Some of these older generation involved downgrading the kernel from 5.12 to 5.11. I suppose that could have caused file system corruption, although I would not have expected the store to be written to at the time of the resets. Best, Jack
Re: Corrupt/missing store items
Jack Hill writes: Given what i know about the problem: substitute operations are failing because gnutls is not available, I tried: jackhill@kalessin ~$ guix build --substitute-urls=http://ci.guix.gnu.org /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv /gnu/store/2vjql2vd6srqxhr0r9xnhclqgc5kxlh1-gnutls-3.6.16-debug /gnu/store/ads39f4czri8s1k43qd0qpxi6dr3w0zn-gnutls-3.6.16-doc /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16 jackhill@kalessin ~$ ls -l /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16 ls: cannot access '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16': No such file or directory What happens if you try “guix build --check --no-grafts /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv” or similar? The database probably states that /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16 has already been built, so with “--check” you might be able to make it build it again. So Guix think's it's sucessfully built gnutls, but it's still not present in the store. I've also tried copying the store items from other hosts. No luck so far… How did that fail? How could the directory have disappeared? Guix appears to be very surprised that the directory is missing. -- Ricardo
Re: Corrupt/missing store items
On Tue, 22 Jun 2021, Jack Hill wrote: On Tue, 22 Jun 2021, Christopher Baines wrote: Jack Hill writes: Given what i know about the problem: substitute operations are failing because gnutls is not available, I tried: jackhill@kalessin ~$ guix build --substitute-urls=http://ci.guix.gnu.org /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv /gnu/store/2vjql2vd6srqxhr0r9xnhclqgc5kxlh1-gnutls-3.6.16-debug /gnu/store/ads39f4czri8s1k43qd0qpxi6dr3w0zn-gnutls-3.6.16-doc /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16 jackhill@kalessin ~$ ls -l /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16 ls: cannot access '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16': No such file or directory So Guix think's it's sucessfully built gnutls, but it's still not present in the store. I've also tried copying the store items from other hosts. No luck so far… Does guix gc --verify=repair say anything? It fails, becasue without gnutls my substitute servers can't be contacted. I also can't reconfigure with http:// substitute servers. Unfortunatley, gc doesn't accept a substitute-urls option. Ok, I started the daemon by hand with http-only substitute servers, and --verify=repair appears to have solved my problem as far as I can tell so far. Thanks to you and nckx! Jack
Re: Corrupt/missing store items
On Tue, 22 Jun 2021, Christopher Baines wrote: Jack Hill writes: Given what i know about the problem: substitute operations are failing because gnutls is not available, I tried: jackhill@kalessin ~$ guix build --substitute-urls=http://ci.guix.gnu.org /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv /gnu/store/2vjql2vd6srqxhr0r9xnhclqgc5kxlh1-gnutls-3.6.16-debug /gnu/store/ads39f4czri8s1k43qd0qpxi6dr3w0zn-gnutls-3.6.16-doc /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16 jackhill@kalessin ~$ ls -l /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16 ls: cannot access '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16': No such file or directory So Guix think's it's sucessfully built gnutls, but it's still not present in the store. I've also tried copying the store items from other hosts. No luck so far… Does guix gc --verify=repair say anything? It fails, becasue without gnutls my substitute servers can't be contacted. I also can't reconfigure with http:// substitute servers. Unfortunatley, gc doesn't accept a substitute-urls option. jackhill@kalessin ~$ sudo guix gc --verify=repair reading the store... checking path existence... path `/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16' disappeared, but it still has valid referrers! substitute: updating substitutes from 'https://ci.guix.gnu.org'... 0.0%Backtrace: substitute: In ice-9/boot-9.scm: substitute: 1752:10 16 (with-exception-handler _ _ #:unwind? _ # _) substitute: In unknown file: substitute: 15 (apply-smob/0 #) substitute: In ice-9/boot-9.scm: substitute: 724:2 14 (call-with-prompt _ _ #) substitute: In ice-9/eval.scm: substitute: 619:8 13 (_ #(#(#))) substitute: In guix/ui.scm: substitute: 2147:12 12 (run-guix-command _ . _) substitute: In ice-9/boot-9.scm: substitute: 1752:10 11 (with-exception-handler _ _ #:unwind? _ # _) substitute: 1752:10 10 (with-exception-handler _ _ #:unwind? _ # _) substitute: In guix/scripts/substitute.scm: substitute:757:18 9 (_) substitute:348:26 8 (process-query # _ #:cache-urls _ #:acl _) substitute: In guix/substitutes.scm: substitute:358:27 7 (lookup-narinfos/diverse _ _ # …) substitute:315:31 6 (lookup-narinfos "https://ci.guix.gnu.org; _ # _ # _) substitute:238:26 5 (fetch-narinfos _ _ #:open-connection _ # _) substitute: In ice-9/boot-9.scm: substitute: 1685:16 4 (raise-exception _ #:continuable? _) substitute: 1685:16 3 (raise-exception _ #:continuable? _) substitute: 1780:13 2 (_ #< components: (#<> #<…>) substitute: 1685:16 1 (raise-exception _ #:continuable? _) substitute: 1685:16 0 (raise-exception _ #:continuable? _) substitute: substitute: ice-9/boot-9.scm:1685:16: In procedure raise-exception: substitute: no code for module (gnutls) warning: `/gnu/store/3qbmgxrgml2bbi2hzg61nq0amyvg9sgq-guix-1.3.0-4.4985a42/bin/guix substitute' died unexpectedly Thanks! Jack
Re: Corrupt/missing store items
Jack Hill writes: > Given what i know about the problem: substitute operations are failing > because gnutls is not available, I tried: > > jackhill@kalessin ~$ guix build --substitute-urls=http://ci.guix.gnu.org > /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv > /gnu/store/2vjql2vd6srqxhr0r9xnhclqgc5kxlh1-gnutls-3.6.16-debug > /gnu/store/ads39f4czri8s1k43qd0qpxi6dr3w0zn-gnutls-3.6.16-doc > /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16 > jackhill@kalessin ~$ ls -l > /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16 > ls: cannot access > '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16': No such file or > directory > > So Guix think's it's sucessfully built gnutls, but it's still not > present in the store. I've also tried copying the store items from > other hosts. No luck so far… Does guix gc --verify=repair say anything? signature.asc Description: PGP signature
Re: Corrupt/missing store items
Given what i know about the problem: substitute operations are failing because gnutls is not available, I tried: jackhill@kalessin ~$ guix build --substitute-urls=http://ci.guix.gnu.org /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv /gnu/store/2vjql2vd6srqxhr0r9xnhclqgc5kxlh1-gnutls-3.6.16-debug /gnu/store/ads39f4czri8s1k43qd0qpxi6dr3w0zn-gnutls-3.6.16-doc /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16 jackhill@kalessin ~$ ls -l /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16 ls: cannot access '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16': No such file or directory So Guix think's it's sucessfully built gnutls, but it's still not present in the store. I've also tried copying the store items from other hosts. No luck so far…
Corrupt/missing store items
Hi Guix, I could use some help figuring out what's happened to my store and how to recover. I first noticed the problem when trying to check if a package build reproducably. Unfortunately, I did not capture the output from that, but did capture subsequent commands. It seems as if the metadata has gotten out of sync with the store: jackhill@kalessin ~$ guix gc --verify reading the store... checking path existence... path `/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16' disappeared, but it still has valid referrers! jackhill@kalessin ~$ guix build /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv The following graft will be made: /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv applying 1 graft for /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv ... grafting '/gnu/store/199npi1hcv7zn0r19vl29np6ccshii4p-gnutls-3.6.16-debug' -> '/gnu/store/2vjql2vd6srqxhr0r9xnhclqgc5kxlh1-gnutls-3.6.16-debug'... grafting '/gnu/store/8ixa3p7hwb26warjinffcrvzl064wbcg-gnutls-3.6.16-doc' -> '/gnu/store/ads39f4czri8s1k43qd0qpxi6dr3w0zn-gnutls-3.6.16-doc'... grafting '/gnu/store/akc7l65z459pnifrr6bcm97cjvmpvp9k-gnutls-3.6.16' -> '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16'... updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/bin/certtool' updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/bin/gnutls-cli' updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/bin/gnutls-cli-debug' updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/bin/gnutls-serv' updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/bin/ocsptool' updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/bin/psktool' updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/bin/srptool' updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/lib/guile/3.0/extensions/guile-gnutls-v-2.so.0.0.0' updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/lib/guile/3.0/site-ccache/gnutls.go' updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/lib/guile/3.0/site-ccache/gnutls/extra.go' updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/lib/libgnutls.so.30.28.2' updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/lib/libgnutlsxx.so.28.1.0' successfully built /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv /gnu/store/2vjql2vd6srqxhr0r9xnhclqgc5kxlh1-gnutls-3.6.16-debug /gnu/store/ads39f4czri8s1k43qd0qpxi6dr3w0zn-gnutls-3.6.16-doc /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16 jackhill@kalessin ~$ guix gc --verify reading the store... checking path existence... path `/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16' disappeared, but it still has valid referrers! jackhill@kalessin ~$ ls -l /gnu/store/^C ^C^C jackhill@kalessin ~$ ls -l /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16 ls: cannot access '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16': No such file or directory Then, I ran `guix gc` to collect all the garbage. Trying to rebuild the gnutls derivation results in: jackhill@kalessin ~$ guix build /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv 18.6 MB will be downloaded: /gnu/store/2wrp7x9aclqsapm58dz5i654qds8nbb8-guile-2.0.14 /gnu/store/mz5fvdfks10rmwxf29n95bp9bim6wq7g-nettle-3.5.1 /gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31 /gnu/store/s3dcqzwqaakv1yx37by9chksdbkgih17-glibc-2.31-static /gnu/store/199npi1hcv7zn0r19vl29np6ccshii4p-gnutls-3.6.16-debug /gnu/store/8ixa3p7hwb26warjinffcrvzl064wbcg-gnutls-3.6.16-doc /gnu/store/akc7l65z459pnifrr6bcm97cjvmpvp9k-gnutls-3.6.16 The following graft will be made: /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv substituting /gnu/store/2wrp7x9aclqsapm58dz5i654qds8nbb8-guile-2.0.14... Backtrace: In ice-9/boot-9.scm: 1752:10 19 (with-exception-handler _ _ #:unwind? _ # _) 1752:10 18 (with-exception-handler _ _ #:unwind? _ # _) In guix/scripts/substitute.scm: 771:15 17 (_) In ice-9/exceptions.scm: 406:15 16 (process-substitution # _ "/gnu/store/…" …) In ice-9/boot-9.scm: 1752:10 15 (with-exception-handler _ _ #:unwind? _ # _) In guix/scripts/substitute.scm: 479:9 14 (_) In ice-9/boot-9.scm: 1747:15 13 (with-exception-handler # …) In guix/scripts/substitute.scm: 408:14 12 (_) 382:28 11 (_ #< scheme: https userinfo: #f host: "ci.guix.g…> …) In guix/build/download.scm: 453:4 10 (open-connection-for-uri _ #:timeout _ # _) 251:2 9 (tls-wrap # "ci.guix.gnu.org" # …) 164:15 8
