GitHub user parshimers created a discussion: Stuck merge queue postmortem
Hi folks, In case you weren't aware, for the past 36 or so hours, no merges were going into the main repo (or any other repo). The reason for this, is because we require every change to go through the merge queue on GitHub before being merged to main. The merge queue timeout is set to 45 minutes. That means if the Actions required for a PR to be merged from the queue aren't started before that, the PR will be ejected from the merge queue. When you combine that limit with the runner starvation in ASF, it manifests into the situation we had, where no change can be merged because it is not assigned a runner in time, and is simpy ejected. This is compounded by the fact that nobody is permitted to bypass the merge queue. It's a very deep hole that we were in. Humbedooh in Infra was able to (hopefully) teleport us out of it by bypassing the merge queue to commit https://github.com/apache/texera/commit/2407d79ee7a94ac92c65c720254566c25f69dd8a on our behalf. This workaround simply raises the time limit. Again we received community help on this part, Andrew Lamb of DataFusion saw the situation we were in and pointed out they had done this to help their quandry. Now we come to the part where we need to decide on what to do to stop this from happening. Clearly what happened is not acceptable. There are situations where code needs to be urgently merged (e.g. CVEs, etc.) and having this happen during one of those fires would be a disaster. Therefore we need a "break glass" kind of relief valve. I think there are 2 things we can do to stop this situation from happening. One, is to merge Yicong's change to allow the self-hosted runners to run most of our jobs. That will be the primary fix. The second, is to allow some subset of folks in the project the privilege to bypass the merge queue or other restrictions, in an emergency. I wasn't sure if this was possible, but after discussions with Humbedooh of ASF Infra it seems like it is. If we are all agreed on both points, I would like to start a discussion within the PPMC on private@ as to who should be granted this authority for emergency situations, like we experienced. Looking forward to hearing everyone's thoughts. GitHub link: https://github.com/apache/texera/discussions/9104 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
