On 3/1/25 07:19, Stefan Hajnoczi wrote:
Hi,
On February 26th GitLab CI started failing many jobs because they
could not be scheduled. I've been unable to merge pull requests
because the CI is not working.
Here is an example failed job:
https://gitlab.com/qemu-project/qemu/-/jobs/9281757413
Hi Stefan,
until February 26th the Digital Ocean runners were not enabled; I tried
enabling them (which is what caused the issue) to start gauging how much
credit we would need to be able to move from Azure to DO for CI. I
posted a note on IRC, I'm sorry if you missed that.
The cache PVC appears to be a manual addition made to the running
cluster but not committed to qemu.git. I don't understand why the
problems only started surfacing now. Maybe a recent .gitlab-ci.d/
change changed how the timeout behaves or maybe the gitlab-runner
configuration that enables the cache PVC simply wasn't picked up by
the gitlab-runner Pod until February 26th?
Almost: the cache is not used on Azure, which is why it works.
In the short term I made a manual edit to the ConfigMap removing
gitlab-cache-pvc (but I didn't delete the PVC resource itself). Jobs
are at least running now, although they may take longer due to the
lack of cache.
Ok, thanks for debugging that. I think what you did is right, and the
caching setup should be tested more on a secondary cluster.
(As to the DO credits numbers, the cost of the k8s cluster is about
$75/month, and since we were granted $2000 in credits we have only
$1100/year to spend on the actual jobs. The plan is to check on the
credits left at the end of March and bring our estimates to DO's open
source program manager).
Paolo
In the long term maybe we should deploy minio
(https://github.com/minio/minio) or another Kubernetes S3-like service
so gitlab-runner can properly use a global cache without ReadWriteOnce
limitations?
Since I don't know the details of how the Digital Ocean Kubernetes
cluster was configured for gitlab-runner I don't want to make too many
changes without your input. Please let me know what you think.
Stefan
