Dear, Currently "guix pull" from Savannah and issues can arise. As we recently experimented. Tobias and Ricardo recently discussed how to mirror the repo. IMHO, it is a good idea to mirror but not a good idea to locate it on Ricardo infrastructure, again. :-)
Well, I propose to see if we can mirror on sourceware.org which are GNU friends. ;-) I took the liberty to contact them on IRC. See below. What I see is: we still "guix pull" from Savannah as usual. Then if Savannah is down, we catch the error and we try (transparently for the user) the sourceware url. As a first contact, they agree on the principle. Even they ask numbers about traffic etc. :-) What do you think? All the best, simon -- [Fri Dec 13 2019] <zimoun> I am working with Guix and we discussed mirroring the GNU Guix repo on sourceware. [17:56] <zimoun> The best before doing plan is to ask if it should be possible, on the principle. [17:57] <fche> is this a source code repo or somethign else? <fche> (sorry, not familiar) <cgf> Also, what would be the reason for a mirror on sourceware if you already have a repo? [17:58] <zimoun> To be more concrete, and if you are not familiar with Guix, there is command named "guix pull" which basically "git pull". Currently, the main repo is on Savannah. <zimoun> And everything is fine with Savannah. :-) <zimoun> However, when Savannah is down; which happens, really rarely, but happens, then all Guix is kind-of broken. [17:59] <fche> so how big is this repo, what form is it, ? [18:00] <zimoun> Broken means that the users cannot update their system. And it is not related to development but related on normal usage. Well, "guix pull" is kind-of equivalent to "apt-get update". <fche> accessed via git or what? <fche> we're not set up to host large binaries e.g. [18:01] <zimoun> Yes, via Git. And it is only for source. <cgf> Maybe hold off on this until new sourceware is online? <zimoun> The mirror of https://git.savannah.gnu.org/cgit/guix.git/ <zimoun> And if it is too big, maybe even only the master branch [18:02] <fche> hm, I can't git-clone that bad boy here <fche> ah never mind starting [18:03] <zimoun> What I have in mind, and I speak for myself and not on the behalf of the Guix project. <fche> not that big apparently, some 170MB of content, that's not a problem <zimoun> sorry, my browser returns me the wrong URL [18:04] <zimoun> https://git.savannah.gnu.org/cgit/guix.git/ <zimoun> https://git.savannah.gnu.org/git/guix.git <zimoun> <fche> yeah <fche> do you have a personal sourceware presence already? <zimoun> Just to be on the same wavelength, currently, each time an user run "guix pull" then they "git pull" from https://git.savannah.gnu.org/git/guix.git. However, when Savannah is down which rarely happens, everybody is annoyed. So I would like that "guix pull" catch the error and then try to use a mirror. Which means, that sourceware.org/your-convention-guix will be the default mirror. So the traffic pull will be very low (when Savannah is [18:11] <zimoun> down or unreachable) and the traffic push a bit more. <mjw> guix is awesome! But savannah often is overloaded :{ <zimoun> No I do not have a personnal sourceware presence already. [18:12] * fche has no objection <fche> how do you plan to make the local git repo a fresh enough mirror? <zimoun> Then, I can ask to Savannah admin to have a hook to automatically mirror to sourceware <fche> or you can run a cron job on sourceware to auto git-pull periodically [18:13] <zimoun> yes, I have not think too much about the details. Because I would like first if it was even possible. :-) <fche> do you expect the repo to seriously grow over time ? [18:14] <zimoun> No. But I can do some stats with Git to answer you more precisely. Currently, Guix is around 13000 commits per years. However, I do not know how many addition lines it represents [18:16] <mjw> It guix.git is ~124MB currently. It normally sees multiple commits a day, but most of them are fairly small (a couple of lines to a handful of files). [18:18] <fche> I mean it's not going to grow from 120MB to 1GB or 10G in the forseeable future? <fche> those are the levels at which we might get concerned <zimoun> Who knows ;-) But I do not think so. And in case, you can say: stop, it is too much for our infrastructure. [18:20] <zimoun> Well, if we have a first agreement on an hostage (mirror), can I save this conversation and post it on the mailing list guix-devel. Collect feedback; especially from the maintainers. Then come back to you with more concrete details. [18:21] <fche> 'forseeable' doesn't ask for infinite prophecy powers :) <zimoun> :-) <fche> yeah, I think we can do it. <zimoun> thank you for your help. <segher> but how much load on sourceware will this cause? <zimoun> do you want to be CC of the email? <mjw> guix is very much pure GNU, and very much sources should produce reproducible builds. You can expect it to stay text based forever and never see any binary blobs. <fche> segher, it's a git repo similar in size to valgrind [18:22] <zimoun> Almost no load on sourceware. As I tried to explain above. It would be a feedback when Savannah is down -- which happens rarely from my experience. <segher> fche: no... how many pulls per seconds are asked from it [18:23] <segher> zimoun: that doesn't change the worst case at all <zimoun> segher: I have no idea. I think it is hard to count. <fche> segher, I expect nothing noticeable, but zimoun will let us know if there's reason to worry [18:24] <fche> ie a few downloads a day is nothing; a hundred git updates an hour is nothing <segher> and a simple gcc pull is almost 20m <mjw> zimoun, I can also provide a backup mirror. Not as fancy as sourceware, but if you need something to fallback on. [18:25] <zimoun> segher: yes, it is just a first contact to see if we can agree. I totally understand your limitation. <segher> a fresh clone, i mean <fche> the gcc git repo is weird/svny and much larger <segher> yes <segher> my point is that we already are overloaded [18:26] <fche> not with git clones <segher> the server, in general <mjw> zimoun, it would go under https://code.wildebeest.org/git/mirror/ (I cannot guarantee an SLA though, but I think neither can sourceware) <fche> and definitely not with git updates <segher> if it is actually light weight, then who cares ;-) [18:27] <cgf> overloaded sourceware is why I suggested waiting until the new server comes online. <segher> right <cgf> There's also the issue of outdated git. [18:28] <segher> it already is super annoying (to GCC) that half of the time (or whatever; it feels very often) the problem is caused by cygwin, not by us <cgf> Probably more like 70% of the time. :-) [18:29] <zimoun> mjw: thank you. And thank you for the hard work you did on the gnu list moderation. <segher> cgf: it feels that way, yes <mjw> zimoun, haha, yeah about that... o boy. <cgf> I may be wrong but I don't think there are any other projects like cygwin on sourceware. cygwin is a source code repository (lightweight) and a binary distribution mechanism (imo, heavy weight) [18:30] <cgf> but I digress... <zimoun> mjw: as you are also on #guix maybe you can suggest the idea of wildebeest. Did you subscribe to the guix-devel mailing list? [18:31] <segher> cgf: right, and sharing resources with a project with different usage patterns of those resources is a bit problematic <fche> segher, let us know of problems when they appear, and we'll keep tweaking things [18:32] <fche> there are a lot of knobs to turn <segher> fche: i can tell you things were bad yesterday [18:33] <segher> not too useful, eh :-) <zimoun> My point coming here and asking is: establish a first contact to see if we can go further together. Or if it is already a no go. Because I understand you need numbers and details. And I do not have them, right now. But I would like to open the discussion on this topic of mirroring. <fche> segher, exactly :) <mjw> And on Monday gcc will switch to git too! [18:34] <fche> zimoun, we can put something like that up in a matter of minutes <segher> (gcc bugzilla gave ISEs most of the time) <mjw> well as a mirror and then Dec 31! finally git all the way! <fche> mjw, that should help our loads, svn is much less efficient than git <zimoun> fche: thank you for the offer but I am not maintainer and I speak on my own voice. I would like to hear the civodul/Ludo's opinion :-) [18:35] <mjw> civodul did suggest sourceware too, I did tell him to join here. But does he listen to me... [18:36] <zimoun> mjw: yeah! I know... that's why I am doing ;-) <zimoun> but you know, french diplomacy :-) [18:37]