Hello Guix, Cuirass has stopped processing (old) jobs for aarch64 and x86_64. After digging through the database it's because (db-get-pending-build ...) returns a build that is missing from the Jobs table:
WITH pending_dependencies AS (SELECT Builds.id, count(dep.id) as deps FROM Builds LEFT JOIN BuildDependencies as bd ON bd.source = Builds.id LEFT JOIN Builds AS dep ON bd.target = dep.id AND dep.status != 0 WHERE Builds.status = -2 AND Builds.system = 'x86_64-linux' GROUP BY builds.id ORDER BY Builds.priority ASC, Builds.timestamp DESC) SELECT id FROM pending_dependencies where deps = 0 limit 1; id -------- 335212 However: select * from jobs where build = 335212; name | evaluation | build | status | system ------+------------+-------+--------+-------- (0 rows) For clarity: select id,derivation,evaluation,job_name,nix_name,status from builds where id = 335212; id | derivation | evaluation | job_name | nix_name | status --------+-------------------------------------------------------------------+------------+-----------------------+-------------------+-------- 335212 | /gnu/store/yzgcza0nijnp79mzz878q9a61p6jykgh-perftest-4.5-0.20.drv | 103435 | perftest.x86_64-linux | perftest-4.5-0.20 | -2 The derivation is also missing from the Outputs table, which causes the monster query in (db-get-builds ...), which is what workers call to fetch the next job, to return nothing. 335212 belongs to evaluation 103435 according to the above query, but does not show up here: https://ci.guix.gnu.org/eval/103435?all=&paginate=0 The build id sequence appears to belong to this evaluation: https://ci.guix.gnu.org/eval/103436?all=&paginate=0 (notice how it has 335211 and 335213). I'm not sure how to recover from this. Either manually create the entries in Jobs and Outputs, or delete the offending Builds entry? The 335212 build is for x86_64-linux, we have the same problem with 335087 (also perftest) on aarch64. i686-linux and powerpc64le-linux is fine. Ideas?