Hello Guix! This week, the node behind {ci,issues,disarchive}.guix.gnu.org and guix.gnu.org was down twice for a few hours—nothing terrible in the end, but it reminded us that, even though Guix doesn’t rely on any particular machine, we can definitely feel the inconvenience when it’s down.
We were unlucky enough that it happened days after the other build farm, bordeaux.guix.gnu.org, ran out of disk space and had its CI stopped, right before the big merge—so it doesn’t have substitutes for current master. While discussing this on IRC the other day, we thought that perhaps it was time to have an infrastructure hackathon. How about Tuesday, Dec. 21st? (Probably with a followup in January.) Here are tasks that were brought up: • Set up a backup server for berlin.guix.gnu.org, the head node of the ci.guix.gnu.org, possibly moving some services such as the web site there. • Add DNS redundancy for guix.gnu.org so it can point to one of two hosts (need to figure out certbot challenges so both machines can update their certificates). • Set up status.guix.gnu.org with sysadmin status updates (possibly using Prometheus?). • Come up with a plan to add disks to the RAID array on bayfront, the head node of bordeaux.guix.gnu.org. • Work on a plan to back up the Disarchive database currently on berlin.guix. • Work on a plan to mirror nars from ci.guix and bordeaux.guix, using plain rsync or <https://git.cbaines.net/guix/nar-herder/about/>. • Have a documented procedure to set up substitute mirrors, such as the one in .cn (I can’t find the URL), ideally with plain rsync access. Am I forgetting something? Some of these tasks require root or physical access for the final steps, but most of them are about (1) coming up with a plan, and (2) adjusting the system configuration at <https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/>. Anyone with sysadmin experience to share and Guix System knowledge can join! We’ll communicate over #guix on irc.libera.chat. Who’s in? :-) Cheers, Ludo’.