Re: GitLab to plans to delete dormant projects

[Ludovic Courtès]

Hi,

Olivier Dion via "Development of GNU Guix and the GNU System
distribution."  skribis:

> On Sat, 06 Aug 2022, Julien Lepiller  wrote:
>> Our build farms need those sources, so they keep them in cache. If you
>> need a source, you can always substitute from the build farms if the
>> origin disappeared (that's actually the default and you don't even
>> need to trust the build farm for that to work).
>
> Does the cache as a time to live?  For example, would a source from 2020 in
> 20 years still be available on the build farms?  Or would the build
> farms make a request to Software Heritage?

Our build farms have limited capacity, so they probably won’t keep
everything forever, but that’s the mission of Software Heritage (SWH).

Guix automatically falls back to SWH for some time now, and SWH
archives all of gitlab.com, so I’m not very concerned:

  
https://guix.gnu.org/en/blog/2019/connecting-reproducible-deployment-to-a-long-term-source-code-archive/

A shortcoming of our code, as Maxime mentions, is that it doesn’t
correctly handle retrieval of recursive checkouts from SWH.  That’s a
bug to fix, and there’s another one in this area:

  https://issues.guix.gnu.org/48540

> Okay great then.  Maybe it would be a good idea to test this by
> simulating the deletion of a Gitlab repo.  Better find it out now than
> when it's too late.

Other than checking whether ‘guix lint -c archival PKG’ complains, one
can run:

  guix build -S PKG --check

HTH!

Ludo’.

Re: GitLab to plans to delete dormant projects

[John Kehayias]

Hi all,

--- Original Message ---
On Saturday, August 6th, 2022 at 9:08 AM, Olivier Dion via "Development of GNU 
Guix and the GNU System distribution."  wrote:

>
>
> Hi,
>
> Following this article https://lwn.net/Articles/903858/, GitLab is
> planning to start deleting project that were idle for > 12 months.
>
> Many packages origin in Guix use an url to a GitLab project. What are
> the consequence of such deletion on Guix reproducibility? Will it
> affects the time-machine?
>

I think having good backup and archival plans are great, so not to dissuade 
anyone on this, but as an update here looks like GitLab has walked back on this:

https://www.theregister.com/2022/08/05/gitlab_reverses_deletion_policy/

John

Re: GitLab to plans to delete dormant projects

[Development of GNU Guix and the GNU System distribution.]

On Sat, 06 Aug 2022, Julien Lepiller  wrote:
> Our build farms need those sources, so they keep them in cache. If you
> need a source, you can always substitute from the build farms if the
> origin disappeared (that's actually the default and you don't even
> need to trust the build farm for that to work).

Does the cache as a time to live?  For example, would a source from 2020 in
20 years still be available on the build farms?  Or would the build
farms make a request to Software Heritage?

> Another fallback option when substitution is not possible is to get
> the source from Software Heritage. They keep an archive of almost
> everything. To do that, they have listers that help tgem find sources
> fsom different sesvices. They have a lister for GitLab, and even one
> for Guix. Also, as part of guix lint, a request is sert to swh if the
> oriqin is not yet archived.
>
> Hopefully that means our origins are saved by Software Heritage, so we
> can transparently fall back to them.

Okay great then.  Maybe it would be a good idea to test this by
simulating the deletion of a Gitlab repo.  Better find it out now than
when it's too late.

-- 
Olivier Dion
oldiob.dev

Re: GitLab to plans to delete dormant projects

[Julien Lepiller]

Our build farms need those sources, so they keep them in cache. If you need a 
source, you can always substitute from the build farms if the origin 
disappeared (that's actually the default and you don't even need to trust the 
build farm for that to work).

Another fallback option when substitution is not possible is to get the source 
from Software Heritage. They keep an archive of almost everything. To do that, 
they have listers that help tgem find sources fsom different sesvices. They 
have a lister for GitLab, and even one for Guix. Also, as part of guix lint, a 
request is sert to swh if the oriqin is not yet archived.

Hopefully that means our origins are saved by Software Heritage, so we can 
transparently fall back to them.

Le 6 août 2022 15:08:21 GMT+02:00, "Olivier Dion via Development of GNU Guix 
and the GNU System distribution."  a écrit :
>Hi,
>
>Following this article , GitLab is
>planning to start deleting project that were idle for > 12 months.
>
>Many packages origin in Guix use an url to a GitLab project.  What are
>the consequence of such deletion on Guix reproducibility?  Will it
>affects the time-machine?
>
>-- 
>Olivier Dion
>oldiob.dev
>
>

Re: GitLab to plans to delete dormant projects

[Maxime Devos]

On 06-08-2022 15:08, Olivier Dion via Development of GNU Guix and the 
GNU System distribution. wrote:

Hi,

Following this article, GitLab is
planning to start deleting project that were idle for > 12 months.

Many packages origin in Guix use an url to a GitLab project.  What are
the consequence of such deletion on Guix reproducibility?  Will it
affects the time-machine?


software heritage should avoid some problems, but from what I've heard 
it doesn't support all edge cases yet (something about recursive 
checkouts?).


I think it would be a good idea to make some Guile script to find all 
GitLab git checkouts in Guix and run the swh linter on them to make sure 
they are archived.


Additionally, it would be nice to support multiple URLs as fallbacks in 
the 'origin' record for git-fetch (like we have for url-fetch) to avoid 
the three points of failures (SWH and the copy at the substitute 
servers) in the fallback mechanism.


Greetings,
Maxime.



OpenPGP_0x49E3EE22191725EE.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature