Re: [Guix-europe-sac] Shutting down qa.guix?
Andreas Enge writes: > Am Sat, Dec 09, 2023 at 11:54:59AM +0100 schrieb Ludovic Courtès: >> I think this underlines a collective failure to get our act together. > > indeed, and besides what Simon mentioned about the bank situation I think > there was a certain lack of consistency between deciding on the technical > and on the financial questions. And of course the lack of urgency, since > anyway things continued thanks to Chris... So thank you for all you have > done, Chris, and thank you for taking action now to force us to get our act > together! Indeed QA has become a central part of our project infrastructure. > > I suggest the following, which resolves the lockstep between technology and > finance: > - Decide that the funds at the FSF pay for the hosting in its current form. > Ideally move the billing to Guix Foundation, and then, as we use to do > for bayfront, periodically ask the FSF to reimburse the hosting cost. > I think we have an informal consensus for this, and just require a formal > vote at the Guix spending committee and at the Guix Foundation SAC. > - Reimburse Chris for the costs incurred for hosting before this move. > As Simon has said, we have a vote for this already, but could also > start over with the exact amount once the first point is handled, so > that Chris does not pay for future hosting any more. > > Then in a separate step, other people can discuss about potential > technical changes to the hosting situation. It would probably be good > to have a small group of people, including Chris at least for a > transitory period, who take care of the sysadmin part. > > Any thoughts on this proposal? Sounds good to me. signature.asc Description: PGP signature
Re: [Guix-europe-sac] Shutting down qa.guix?
Hello, Am Sat, Dec 09, 2023 at 11:54:59AM +0100 schrieb Ludovic Courtès: > I think this underlines a collective failure to get our act together. indeed, and besides what Simon mentioned about the bank situation I think there was a certain lack of consistency between deciding on the technical and on the financial questions. And of course the lack of urgency, since anyway things continued thanks to Chris... So thank you for all you have done, Chris, and thank you for taking action now to force us to get our act together! Indeed QA has become a central part of our project infrastructure. I suggest the following, which resolves the lockstep between technology and finance: - Decide that the funds at the FSF pay for the hosting in its current form. Ideally move the billing to Guix Foundation, and then, as we use to do for bayfront, periodically ask the FSF to reimburse the hosting cost. I think we have an informal consensus for this, and just require a formal vote at the Guix spending committee and at the Guix Foundation SAC. - Reimburse Chris for the costs incurred for hosting before this move. As Simon has said, we have a vote for this already, but could also start over with the exact amount once the first point is handled, so that Chris does not pay for future hosting any more. Then in a separate step, other people can discuss about potential technical changes to the hosting situation. It would probably be good to have a small group of people, including Chris at least for a transitory period, who take care of the sysadmin part. Any thoughts on this proposal? Andreas
RFC (was Re: Shutting down qa.guix?)
Hi Ludo, On sam., 09 déc. 2023 at 11:54, Ludovic Courtès wrote: > I know this has been discussed several times and it remains unclear to > me why as a project we never managed to move forward—maybe the comfort > of the status quo? It would help if the project would have a process for such move. Why not some Request-For-Comment? Request-For-Comment process: concrete implementation Simon Tournier Tue, 31 Oct 2023 12:14:42 +0100 id:87h6m7yrfh@gmail.com https://lists.gnu.org/archive/html/guix-devel/2023-10 https://yhetil.org/guix/87h6m7yrfh@gmail.com Cheers, simon
Re: Shutting down qa.guix?
Hi everyone, Forgive me for cutting in a conversation like this. I believe qa.guix can still be salvaged: I can host it per gratis, at least temporarily. I am the admin of repo.jing.rocks, and I own the hardware. The servers are in my bedroom/living room. Adding qa.guix would cost nothing for me. > As for system administration, is there documentation that people willing to help could look at? Very concrete things like: what services are running on which machines, what do I do if one of them is stuck or if I get this error message, etc. Yes, docs are important, and I couldn't find one. What kind of spec does qa.guix require to keep operational? I see in the threads: > Storage is also an issue as beid currently is working with 340G of total storage and that's almost full [..] I can spare a few terabytes of spinning rust storage. How about RAM and CPU? Does 64 vcpus (AMD EPYC 7773X) and 128GB RAM sound enough? If anyone wants to know to full spec, I can write it later. Please note that the hardware I own requires non-free firmware, will this be a problem? Also, I'm not a good programmer, I can't maintain any package. I'm still learning Guile syntax and how to adapt to Guix. Someone else would have to work with me to keep the whole system going. Will this be an issue? For the stability of repo.jing.rocks, please refer to [1] and [2]: [1] https://mirror-master.debian.org/status/mirror-info/repo.jing.rocks.html [2] https://archlinux.org/mirrors/jing.rocks/ -- Jing Luo About me: https://jing.rocks/about/ PGP Fingerprint: 4E09 8D19 00AA 3F72 1899 2614 09B3 316E 13A1 1EFC OpenPGP_signature.asc Description: OpenPGP digital signature
Re: Shutting down qa.guix?
Maxim Cournoyer writes: > Hi, > > Christopher Baines writes: > >> Tobias Geerinckx-Rice writes: >> >>> Christopher Baines 写道： it's not the most cost effective setup >>> >>> Has this been explained in more detail before? >> >> Probably not, beid is currently a CPX51 Hetzner cloud server costing >> €65.33 a month. This has been useful as it's enabled scaling the >> resources dynamically, but it would be possible to reduce the costs and >> still have sufficient RAM/disk space by using a Hetzner server auction >> machine for example. >> >> It's not all about cost though, given the data service is one of the >> slow points of QA, if we want QA to get faster at giving feedback, it's >> probably important to not try and cut costs on this part of the system. > > Isn't QA mostly slow because of the lack of x86 build machines? Does > the head node needs to be powerful itself? What kind of resources does > it likes having the most? CPU? RAM? Storage? There are two key bottlenecks, processing the revisions in the data service, then the build coordinator performing the builds. For the data service, lots of RAM helps as computing and building the derivations for Guix (similar to pull, time-machine, ...) is quite expensive in CPU and RAM. Also computing all the derivations for each revision takes a lot of RAM. Storage is also an issue as beid currently is working with 340G of total storage and that's almost full, and this doesn't leave any space for maintenance. More storage means being able to store data about more patch series at once. For the build coordinator, the machine doesn't need to be powerful, it has quite low requirements. While bayfront's storage isn't particularly fast, it's more than sufficient in terms of hardware. More build machines, including x86 ones would speed up the test results for patches and branches though. signature.asc Description: PGP signature
Re: Shutting down qa.guix?
Hi, On sam., 09 déc. 2023 at 11:54, Ludovic Courtès wrote: > I think this underlines a collective failure to get our act together. I do not consider a collective failure considering the payment for the service. About the maintenance of such service, that’s another question, IMHO. :-) > Anyway, would it be possible for you to transfer billing of the hardware > (Hetzner?) to Guix Foundation? Does Guix Foundation know what it would > cost them? Just to put context about these questions for transferring billing: 1. The initial discussion happened on the private mailing list guix-sysadmin. Board of Guix Foundation had been CC on July 6th. 2. Chris provided billing information on July 14th. Then Andreas answered that it is sustainable for Guix Foundation, potentially with the help of Guix Spending Committee. 3. On July 13th, the Solidary Administration Council of Guix Foundation had been solicited; as specified by Guix Foundation statutes. 4. On September 12th, the SAC voted yes about supporting the financial cost of QA. Moreover, since it is a recurrent cost, some means for collecting money had also been discussed. 5. Then what has been unexpected is the energy that the Board of Guix Foundation invested in resolving an unrelated but blocking situation with our bank. It probably slowed down all the operational part. Well, from my understanding of the Guix Foundation side, all is ready for transferring the current billing and, again from my understanding of the situation, what is missing is the effective operational transfer. Now, some relationships with our bank are smoothed… somehow. :-) > The “spending committee” (Tobias, Ricardo, and myself), which oversees > expenditure from the funds held at the FSF, can also be in the loop to > provide additional financial support. Since it is recurrent cost, yes it would nice to discuss between Guix Foundation’s SAC and Spending Committee how to secure the financial support of a recurrent cost. > PS: You’ve done amazing work over the years. As a contributor, I find > it super reassuring to know that you’re always available and > focused, committed to improving patch workflows for many years. > It’s been a long road already and I admire this level of commitment > and hard work. I still remember the first time we met with Chris, it was back on December 2018, right before Reproducible Build Summit in Paris. Chris already presented what could be done for improving the patch workflows. It’s inspiring to see such commitment and hard work over the years. Cheers, simon
Re: Shutting down qa.guix?
Hi, Christopher Baines writes: > Tobias Geerinckx-Rice writes: > >> Christopher Baines 写道： >>> it's not the most cost effective setup >> >> Has this been explained in more detail before? > > Probably not, beid is currently a CPX51 Hetzner cloud server costing > €65.33 a month. This has been useful as it's enabled scaling the > resources dynamically, but it would be possible to reduce the costs and > still have sufficient RAM/disk space by using a Hetzner server auction > machine for example. > > It's not all about cost though, given the data service is one of the > slow points of QA, if we want QA to get faster at giving feedback, it's > probably important to not try and cut costs on this part of the system. Isn't QA mostly slow because of the lack of x86 build machines? Does the head node needs to be powerful itself? What kind of resources does it likes having the most? CPU? RAM? Storage? -- Thanks, Maxim
Re: Shutting down qa.guix?
Tobias Geerinckx-Rice writes: > Christopher Baines 写道： >> it's not the most cost effective setup > > Has this been explained in more detail before? Probably not, beid is currently a CPX51 Hetzner cloud server costing €65.33 a month. This has been useful as it's enabled scaling the resources dynamically, but it would be possible to reduce the costs and still have sufficient RAM/disk space by using a Hetzner server auction machine for example. It's not all about cost though, given the data service is one of the slow points of QA, if we want QA to get faster at giving feedback, it's probably important to not try and cut costs on this part of the system. signature.asc Description: PGP signature
Re: Shutting down qa.guix?
Hi Chris, I agree that Guix should step in to maintain the level of service that QA currently offers, by paying for hosting and sharing responsibility for system administration. Whether the software's maintained or improved is something over which we've historically had very poor control. Things go awry when we pretend otherwise. Christopher Baines 写道： it's not the most cost effective setup Has this been explained in more detail before? I also think that fundamentally I may think that processes and tooling to make changes is more important than others regard it to be. The disproportionate amount of effort you've put in, mostly unaided, implies as much. (Proportionally disproportional thanks for that, by the way.) Both more comments to provide some context for the configuration. and making some high level architecture diagrams for QA and the bordeaux build farm would be very much appreciated! As for monitoring and responding to problems, that's a bit more complicated, but in most cases a herd restart of the relevant bit will temporarily resolve the issue. Chuffed that the Guix Data Service embodies the same debugging philosophy as the other Guix infrastructure. Kind regards, T G-R signature.asc Description: PGP signature
Re: Shutting down qa.guix?
Ludovic Courtès writes: > Hello, > > Christopher Baines skribis: > >> I am still planning to shutdown data.qa.guix.gnu.org and >> QA which depends on it within the next couple of weeks. I do hope it can >> return some point though, and hopefully sooner rather than later. >> >> On this like most decisions I'm indecisive, I could try and keep the >> current server going, but it's not the most cost effective setup and >> it's very low on disk space. I could replace the server with some >> slightly better setup, but this would still mean I'm managing a key part >> of the infrastructure, which is something I'm trying to move away >> from. There was some discussion of the project taking over the hosting, >> and maybe that will happen at some point, but it hasn't happened yet. So >> while not having qa.guix.gnu.org for a time isn't ideal, I'm still going >> with this approach. > > I think this underlines a collective failure to get our act together. > > I believe there’s consensus that qa.guix is useful and has been a boost > for reviewers and contributors; we’d probably all want it to provide > quicker feedback, which is a sign of success: we’ve come to rely on it. > > I know this has been discussed several times and it remains unclear to > me why as a project we never managed to move forward—maybe the comfort > of the status quo? In addition, it's also unclear to me who should be making decisions on things like this. I also think that fundamentally I may think that processes and tooling to make changes is more important than others regard it to be. While it has no inherent value to users, personally I see it as so much more important than actual Guix features or packages since the value to users comes through Guix getting better faster, because of the increased pace of changes and reduced number of regressions. > Anyway, would it be possible for you to transfer billing of the hardware > (Hetzner?) to Guix Foundation? Does Guix Foundation know what it would > cost them? I believe so, at least I think that's possible. The costs have also been discussed previously. > The “spending committee” (Tobias, Ricardo, and myself), which oversees > expenditure from the funds held at the FSF, can also be in the loop to > provide additional financial support. > > As for system administration, is there documentation that people willing > to help could look at? Very concrete things like: what services are > running on which machines, what do I do if one of them is stuck or if I > get this error message, etc. The configuration for beid, the machine that runs data.qa.guix.gnu.org and Patchwork is in maintenance.git. It could probably use some more comments to provide some context for the configuration. There's also probably a benefit from making some high level architecture diagrams for QA and the bordeaux build farm, and I can try and make a start on these. As for monitoring and responding to problems, that's a bit more complicated, but in most cases a herd restart of the relevant bit will temporarily resolve the issue. I'm still working on mitigating some of the underlying problems that cause things to break. signature.asc Description: PGP signature
Shutting down qa.guix?
Hello, Christopher Baines skribis: > I am still planning to shutdown data.qa.guix.gnu.org and > QA which depends on it within the next couple of weeks. I do hope it can > return some point though, and hopefully sooner rather than later. > > On this like most decisions I'm indecisive, I could try and keep the > current server going, but it's not the most cost effective setup and > it's very low on disk space. I could replace the server with some > slightly better setup, but this would still mean I'm managing a key part > of the infrastructure, which is something I'm trying to move away > from. There was some discussion of the project taking over the hosting, > and maybe that will happen at some point, but it hasn't happened yet. So > while not having qa.guix.gnu.org for a time isn't ideal, I'm still going > with this approach. I think this underlines a collective failure to get our act together. I believe there’s consensus that qa.guix is useful and has been a boost for reviewers and contributors; we’d probably all want it to provide quicker feedback, which is a sign of success: we’ve come to rely on it. I know this has been discussed several times and it remains unclear to me why as a project we never managed to move forward—maybe the comfort of the status quo? Anyway, would it be possible for you to transfer billing of the hardware (Hetzner?) to Guix Foundation? Does Guix Foundation know what it would cost them? The “spending committee” (Tobias, Ricardo, and myself), which oversees expenditure from the funds held at the FSF, can also be in the loop to provide additional financial support. As for system administration, is there documentation that people willing to help could look at? Very concrete things like: what services are running on which machines, what do I do if one of them is stuck or if I get this error message, etc. Thanks, Ludo’. PS: You’ve done amazing work over the years. As a contributor, I find it super reassuring to know that you’re always available and focused, committed to improving patch workflows for many years. It’s been a long road already and I admire this level of commitment and hard work.
