Christopher Baines <[email protected]> writes: > Andreas Enge <[email protected]> writes: > >> for a day or two I have not seen any progress on the branch on QA, and >> the build farm seems to be busy with other stuff. I have restarted the >> build coordinator, bffe and qa-frontpage on bayfront, and the data >> service for QA. So far to no avail, although the builds for this branch >> are marked as scheduled: >> >> https://data.qa.guix.gnu.org/revision/4af027127a87f508237bd40a9f683644c79e9f58/builds >> >> Chris, do you see what is going wrong? > > It's not immediately clear. > > Regarding the build coordinator and bffe, I don't know of any issues > with them, so if they're running but you think something might be up, > let me know instead of restarting them, as I'll try and investigate > further. > > Unfortunately the same can't be said for the data service currently, it > keeps running out of file descriptors, potentially due to the same > problem that's been affecting Cuirass. There's an open Pull Request that > I think might help: https://codeberg.org/guix/guix/pulls/11744 > > The QA Frontpage is still pending at least partial replacement, so no > harm in restarting it, and sometimes it can also be quite helpful. > > One action that can get builds going is running the following command on > bayfront: > > guix-build-coordinator setup-failures clear > > I think when the data service instances are low on file descriptors, > that can lead to more errors and setup failures from builds. You can see > the setup failures here https://bordeaux.guix.gnu.org/setup-failures but > I'll look at putting a link to that on the activity page. I've already > got it highlighting issues with agents and that's really helpful. > > The most likely issue for what you're seeing is that the builds are > blocked, but for some reason the data service hasn't kept it's tracking > of that up to date. If the builds for the branch have seemingly stopped, > you can check this by going to the builds page on the revision, > filtering by scheduled and clicking through to the builds. If it says > blocked there, but it's in the "Unknown" column on the qa-frontpage, > then it's probably a data service issue.
The QA Frontpage is showing unknown builds for x86_64-linux, but there's only builds for 3 packages running currently and nothing in the plans. There's only a few setup failures and the only one related to world-rebuild is a julia package. So something looks off. Going to the builds list and filtering by scheduled gives: https://data.qa.guix.gnu.org/revision/b211abcf010337fc1915af0cb623007709d75b1c/builds?build_status=scheduled&system=x86_64-linux&target=none&limit_results=&all_results=on The first build listed is this one: https://data.qa.guix.gnu.org/build-server/2/build?build_server_build_id=e5131f3d-a12b-40fc-92e7-a9b6c51b4151 It's not showing as blocked in the data service at least. Clicking through to the bffe and expanding out the inputs section shows some scheduled inputs: https://bordeaux.guix.gnu.org/build/e5131f3d-a12b-40fc-92e7-a9b6c51b4151 If you click on the build for the first scheduled (blue) input, and keep doing that until you find a build with no scheduled inputs, you find the blockage. This build for slurm can't start because mariadb has failed to build: https://bordeaux.guix.gnu.org/build/97f96879-8f39-4ab9-bc78-e71ff7e71760 Clicking back to the derivation in the data service, it shows no builds for mariadb: https://data.qa.guix.gnu.org/gnu/store/lzzmmb1q7pqk1s85wimrqnw60v7hj2w2-mariadb-10.11.14.drv The derivation is different, but the outputs are the same. The data service tries to keep track of builds by output, but maybe it lost track of the world-rebuild branch between when these builds failed and now. I'll have a look at tweaking the data deletion code in the data service so this is less likely to happen.
signature.asc
Description: PGP signature
