ok. I did what I could with the resources we have right now to improve things on the s390x builders.
1. I noticed that we had the kvm instances oversubscribed on cpus (the host has 32, we had 42 used). So, I lowered all the kvm builders to 3 vcpus from 4. (Those are 15-24). 2. I moved the varnish package cache from 07 (a z/vm guest) to 24 (a kvm guest). I have noticed the z/vm ones (thats 01-14) seem to suffer from slowdowns or high io wait more under high load and/or over a long time. Hopefully moving that to a more stable instance will help with lots of issues people have seen with not being able to download or the like. 3. I switched the cache model on the kvm ones to unsafe, which we had already used on a number of other builders. I think the worst that can happen here is that the vm becomes corrupt if it's abruptly shutdown or killed, but thats fine, we can just spin up a new one. If a build gets messed up, koji would just restart it again on another vm, etc. 4. There was a misconfiguration in kojid where if the cache was not answering it tried directly, but it was trying the wrong url. I have corrected this, so if the primary cache is down it should fall back to trying directly on it's own. 5. I've updated and rebooted them all. They seem to behave much better after reboots and then slowly get slower over time. ;( I've been watching it for the last hour or so and so far 0 failures that I can attribute to s390x cache or builder infra. Hopefully that should make things more stable. If you see problems cropping up again, please do open an infra ticket and we can see what futher we can do. :) kevin
signature.asc
Description: PGP signature
_______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
