Re: GitLab to plans to delete dormant projects

[zimoun]

Hi,

I guess, Gitlab means the instance gitlab.com, right?


On sam., 06 août 2022 at 09:08, Olivier Dion via "Development of GNU Guix and 
the GNU System distribution."  wrote:

> Many packages origin in Guix use an url to a GitLab project.  What are
> the consequence of such deletion on Guix reproducibility?  Will it
> affects the time-machine?

As explained by others, thanks to Software Heritage, the time-machine
should not be impacted when Gitlab.com would stop to serve some source.


First, Guix is able to automatically fallback to SWH when upstream
source are unavailable.  Considering substitutes is turned on, fetching
respects this order:

 1. try with Guix build farms
 2. try upstream defined by origin
 3. try SWH
 4. try other “webarchives“


Second, the coverage by SWH depends on the kind of origin (url-fetch,
git-fetch, etc.) because it is not the same entry point (for SWH).

On a side note, SWH ingests many forges using what they call a ‘loader’
[1].  For example, their Git loader ingests an instance of a Gitlab
forge, e.g., gitlab.com; but many others too as gitlab.inria.fr or
gitlab.freedesktop.org or gitlab.gnome.org etc.

It exists a (rudimentary) ‘nixguix’ loader [2]. ;-)  This loader reads
the file ’sources.json’ [3] and then SWH ingests all the tarball
archives.

Moreover, “guix lint -c archival” allows to send a save request to SWH
but this request is only for Git origin.


In summary, it is highly probable that the source code is in SWH.


Third, the issue: being able to later fetch back from SWH using the
(meta-)information we have now.  Other said, the fallback requires an
unique identifier.  The net: this identifier needs to be compatible with
SWH, which provides their own–named swh-id.

The Git commit hash is compatible.  But the checksum is not.  That’s why
the Guix project currently maintains a map (named Disarchive) from the
checksum to swh-id allowing to rebuild the expected source code from the
data stored in SWH.

Well, many Guix packages use a string Git tag for referring.  It can
lead to issues, as in-place replacements.  SWH regularly crawls, ingests
and build “snapshots” (history of history) but there is no guarantee
that the Guix origin is well-covered – aside Guix is currently not able
to manage these snapshots. :-)

And today, the main weakness is about Subversion or CVS.  Some packages
– deep in the dependency graph – are svn-fetch or cvs-fetch.  And there
is no robust fallback mechanism, AFAIK.


In summary, the time-machine may or may not work.  The main factor when
it fails is about the availability of the substitute (from Guix build
farm).  Other said, older the time-machine jump is, and higher the
probability of the failure becomes.


Back to Gitlab.com.  Using Guix 8f0d45c from July, 18th let “guix repl”;
code attached below.


--8<---cut here---start->8---
$ guix repl
GNU Guile 3.0.8
Copyright (C) 1995-2021 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
scheme@(guix-user)> (load "from-gitlab-dot-com.scm")
scheme@(guix-user)> (length packages-on-gitlab.com)
$1 = 223

scheme@(guix-user)> (length git-references-on-gitlab.com)
$2 = 213

scheme@(guix-user)> ,pp tarballs-from-gitlab.com
$3 = (#
 #
 #
 #
 #
 #
 #
 #
 #
 #)

scheme@(guix-user)> ,pp recursive-packages-on-gitlab.com
$4 = (#
 #)
--8<---cut here---end--->8---


It means that the string “gitlab.com“ appears in the origin of 223
packages and 213 of those are git-fetch.  Others said, 10 packages are
using url-fetch with tarballs generated by Gitlab.com.

Only 2 packages are recursive git-reference, therefore badly covered.
Guix is currently not able to fully save them in SWH.  Moreover, fetch
back the data from SWH works but the not the resulting checksum; which
defeats the fallback.  See .


--8<---cut here---start->8---
scheme@(guix-user)> (length archived-packages-on-swh)
$5 = 202

scheme@(guix-user)> ,pp missing-packages
$6 = (#
 #
 #
 #
 #
 #
 #
 #
 #
 #
 #
 #
 #
 #
 #
 #
 #
 #
 #
 #
 #)
--8<---cut here---end--->8---

Not that bad. :-)

Note that the 2 packages using recursive checkouts are not missing; the
data is in SWH but the checksum hits bug#48540.


Ok, let save these missing packages.

--8<---cut here---start->8---
$ for p in tint2 surfraw remmina libsequoia zn-poly ecl-cl-utilities 
cl-utilities sbcl-cl-utilities openrgb guile-ac-d-bus guile-goblins graphviz 
komikku bitcoin-unlimited fulcrum flowee kicad kicad-footprints kicad-symbols 
emacs-execline python-pyodbc-c; do guix lint -c archival $p ;done

gnu/packages/xdisorg.scm:1848:12: tint2@0.14.6: Disarchive entry refers to 
non-exis

Re: GitLab to plans to delete dormant projects

[Ludovic Courtès]

Hi,

Olivier Dion via "Development of GNU Guix and the GNU System
distribution."  skribis:

> On Sat, 06 Aug 2022, Julien Lepiller  wrote:
>> Our build farms need those sources, so they keep them in cache. If you
>> need a source, you can always substitute from the build farms if the
>> origin disappeared (that's actually the default and you don't even
>> need to trust the build farm for that to work).
>
> Does the cache as a time to live?  For example, would a source from 2020 in
> 20 years still be available on the build farms?  Or would the build
> farms make a request to Software Heritage?

Our build farms have limited capacity, so they probably won’t keep
everything forever, but that’s the mission of Software Heritage (SWH).

Guix automatically falls back to SWH for some time now, and SWH
archives all of gitlab.com, so I’m not very concerned:

  
https://guix.gnu.org/en/blog/2019/connecting-reproducible-deployment-to-a-long-term-source-code-archive/

A shortcoming of our code, as Maxime mentions, is that it doesn’t
correctly handle retrieval of recursive checkouts from SWH.  That’s a
bug to fix, and there’s another one in this area:

  https://issues.guix.gnu.org/48540

> Okay great then.  Maybe it would be a good idea to test this by
> simulating the deletion of a Gitlab repo.  Better find it out now than
> when it's too late.

Other than checking whether ‘guix lint -c archival PKG’ complains, one
can run:

  guix build -S PKG --check

HTH!

Ludo’.

Re: GitLab to plans to delete dormant projects

[John Kehayias]

Hi all,

--- Original Message ---
On Saturday, August 6th, 2022 at 9:08 AM, Olivier Dion via "Development of GNU 
Guix and the GNU System distribution."  wrote:

>
>
> Hi,
>
> Following this article https://lwn.net/Articles/903858/, GitLab is
> planning to start deleting project that were idle for > 12 months.
>
> Many packages origin in Guix use an url to a GitLab project. What are
> the consequence of such deletion on Guix reproducibility? Will it
> affects the time-machine?
>

I think having good backup and archival plans are great, so not to dissuade 
anyone on this, but as an update here looks like GitLab has walked back on this:

https://www.theregister.com/2022/08/05/gitlab_reverses_deletion_policy/

John

Re: GitLab to plans to delete dormant projects

[Development of GNU Guix and the GNU System distribution.]

On Sat, 06 Aug 2022, Julien Lepiller  wrote:
> Our build farms need those sources, so they keep them in cache. If you
> need a source, you can always substitute from the build farms if the
> origin disappeared (that's actually the default and you don't even
> need to trust the build farm for that to work).

Does the cache as a time to live?  For example, would a source from 2020 in
20 years still be available on the build farms?  Or would the build
farms make a request to Software Heritage?

> Another fallback option when substitution is not possible is to get
> the source from Software Heritage. They keep an archive of almost
> everything. To do that, they have listers that help tgem find sources
> fsom different sesvices. They have a lister for GitLab, and even one
> for Guix. Also, as part of guix lint, a request is sert to swh if the
> oriqin is not yet archived.
>
> Hopefully that means our origins are saved by Software Heritage, so we
> can transparently fall back to them.

Okay great then.  Maybe it would be a good idea to test this by
simulating the deletion of a Gitlab repo.  Better find it out now than
when it's too late.

-- 
Olivier Dion
oldiob.dev

Re: GitLab to plans to delete dormant projects

[Julien Lepiller]

Our build farms need those sources, so they keep them in cache. If you need a 
source, you can always substitute from the build farms if the origin 
disappeared (that's actually the default and you don't even need to trust the 
build farm for that to work).

Another fallback option when substitution is not possible is to get the source 
from Software Heritage. They keep an archive of almost everything. To do that, 
they have listers that help tgem find sources fsom different sesvices. They 
have a lister for GitLab, and even one for Guix. Also, as part of guix lint, a 
request is sert to swh if the oriqin is not yet archived.

Hopefully that means our origins are saved by Software Heritage, so we can 
transparently fall back to them.

Le 6 août 2022 15:08:21 GMT+02:00, "Olivier Dion via Development of GNU Guix 
and the GNU System distribution."  a écrit :
>Hi,
>
>Following this article , GitLab is
>planning to start deleting project that were idle for > 12 months.
>
>Many packages origin in Guix use an url to a GitLab project.  What are
>the consequence of such deletion on Guix reproducibility?  Will it
>affects the time-machine?
>
>-- 
>Olivier Dion
>oldiob.dev
>
>

Re: GitLab to plans to delete dormant projects

[Maxime Devos]

On 06-08-2022 15:08, Olivier Dion via Development of GNU Guix and the 
GNU System distribution. wrote:

Hi,

Following this article, GitLab is
planning to start deleting project that were idle for > 12 months.

Many packages origin in Guix use an url to a GitLab project.  What are
the consequence of such deletion on Guix reproducibility?  Will it
affects the time-machine?


software heritage should avoid some problems, but from what I've heard 
it doesn't support all edge cases yet (something about recursive 
checkouts?).


I think it would be a good idea to make some Guile script to find all 
GitLab git checkouts in Guix and run the swh linter on them to make sure 
they are archived.


Additionally, it would be nice to support multiple URLs as fallbacks in 
the 'origin' record for git-fetch (like we have for url-fetch) to avoid 
the three points of failures (SWH and the copy at the substitute 
servers) in the fallback mechanism.


Greetings,
Maxime.



OpenPGP_0x49E3EE22191725EE.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

GitLab to plans to delete dormant projects

[Development of GNU Guix and the GNU System distribution.]

Hi,

Following this article , GitLab is
planning to start deleting project that were idle for > 12 months.

Many packages origin in Guix use an url to a GitLab project.  What are
the consequence of such deletion on Guix reproducibility?  Will it
affects the time-machine?

-- 
Olivier Dion
oldiob.dev