Looking more in depth into the failures (in particular for PR-6880 [1]) I found the exact root cause leading to the ChannelClosedException errors that we see when Jenkins loses contact with agent pods.
Turns out on Sep 22 someone added a CPU-based autoscaler directly on the GCE Managed Instance Group (MIG) backing the slave-pool9 node pool. The autoscaling mode was set to ON basically bringing nodes up and down based on CPU usage. The problem is that this GCE-level autoscaler doesn't know anything about Kubernetes or pods. It just looks at VM CPU utilization. When CPU drops below 60%, which happens all the time during I/O-heavy build/test phases, it deletes VMs on the spot without draining pods first. The GKE cluster autoscaler was already enabled on the node pool and handles scale-down properly (checks pod scheduling, drains gracefully, etc.) but the MIG autoscaler bypasses all of that. For PR-6880 specifically, at 06:26 UTC on Oct 16 the MIG autoscaler saw average CPU at 40.6% and decided to scale down from 6 to 5 nodes. It deleted a VM and killed all 5 Jenkins agent pods running on it. Within two minutes the CPU was back above 80% and the autoscaler started scaling up again. This scale-down/scale-up thrashing has been going on all day with nodes getting created and destroyed within minutes. I just changed the MIG autoscaler mode from ON to ONLY_SCALE_OUT to prevent pods getting killed while running. I am hoping that scale-down will be handled by the GKE cluster autoscaler as before. We will continue monitoring but if you notice other issues let us know. Best, Stamatis [1] https://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-6880/2/pipeline/115/ On Fri, Oct 16, 2026 at 9:32 AM Stamatis Zampetakis <[email protected]> wrote: > > The problem is still there even after merging HIVE-30128 so we will > keep looking. Will update this thread once I have further information. > > Best, > Stamatis > > On Thu, Oct 15, 2026 at 6:36 PM Stamatis Zampetakis <[email protected]> > wrote: > > > > My analysis of the problem in the past few days also points to the GKE > > autoscaler bringing down nodes with scheduled pods. HIVE-30128 is a > > promising fix so I merged it a few minutes ago. Let's continue > > monitoring the situation to see if additional actions are needed. > > > > For the record, there has been a Kubernetes upgrade on September 20 > > which may have brought new changes in the autoscaler logic aggravating > > the issue. > > > > $ gcloud logging read > > 'protoPayload.methodName="google.container.internal.ClusterManagerInternal.PatchCluster" > > AND protoPayload.metadata.operationType="UPGRADE_MASTER" AND > > timestamp>="2026-08-20T00:00:00Z"' --limit=10 --format=json | jq '.[] > > |{timestamp:.timestamp,newversion:.protoPayload.metadata.currentMasterVersion, > > oldversion:.protoPayload.metadata.previousMasterVersion}' > > { > > "timestamp": "2026-09-27T23:15:18.476357688Z", > > "newversion": "1.35.6-gke.1250000", > > "oldversion": "1.35.6-gke.1250000" > > } > > { > > "timestamp": "2026-09-20T23:23:51.890372730Z", > > "newversion": "1.35.6-gke.1250000", > > "oldversion": "1.34.9-gke.1655001" > > } > > > > There were ChannelClosedException issues appearing on master around > > September 20 [1] but also before [2, 3] so it's hard to fully blame > > just the upgrade. > > > > Best, > > Stamatis > > > > [1] https://ci.hive.apache.org/job/hive-precommit/job/master/3094/ > > [2] https://ci.hive.apache.org/job/hive-precommit/job/master/3063/ > > [3] https://ci.hive.apache.org/job/hive-precommit/job/master/3006/ > > > > On Wed, Oct 14, 2026 at 6:20 PM Raghav Aggarwal > > <[email protected]> wrote: > > > > > > Hi Stamatis, > > > > > > This issue has been occurring for over three weeks, so it is not a recent > > > problem. I have observed it multiple times across my pull requests. > > > > > > For reference: > > > > > > - A fix attempt appears to be in progress (HIVE-30128): > > > https://github.com/apache/hive/pull/6843 > > > - A related PR merged 19 days ago demonstrates this pod failure issue: > > > https://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-6750 > > > > > > Regards, > > > Raghav Aggarwal > > > > > > > > > On Wed, Oct 14, 2026 at 5:19 PM Stamatis Zampetakis <[email protected]> > > > wrote: > > >> > > >> Small correction: The PR 6644 and 6872 advance normally. However, in > > >> PR#6815 the ChannelClosedException popped up but the underlying pods > > >> are long gone so don't really know what happened. I will check a bit > > >> further to see if I can determine something. > > >> > > >> [2026-10-14T10:07:50.257Z] Cannot contact > > >> hive-precommit-pr-6815-5-9327p-g96pl-71l7f: > > >> hudson.remoting.ChannelClosedException: Channel > > >> "hudson.remoting.Channel@5ca733ed:JNLP4-connect connection from > > >> 10.0.13.8/10.0.13.8:39556": Remote call on JNLP4-connect connection > > >> from 10.0.13.8/10.0.13.8:39556 failed. The channel is closing down or > > >> has closed down > > >> > > >> Best, > > >> Stamatis > > >> > > >> On Wed, Oct 14, 2026 at 12:36 PM Stamatis Zampetakis > > >> <[email protected]> wrote: > > >> > > > >> > Hi Konstantin, > > >> > > > >> > I just checked the Google Cloud incidents and it appears that there > > >> > was a problem affecting VMs in us-central1 where our pods are running. > > >> > The incident timeline: > > >> > * Started on October 13, 2026 at 3:19:12 PM UTC+1 > > >> > * Resolved on October 13, 2026 at 6:35:59 PM UTC+1 > > >> > > > >> > At the moment, I see 61 pods up and running normally for the following > > >> > PRs: 6644, 6815, 6872. Let's see if the problem persists to > > >> > investigate further. > > >> > > > >> > Best, > > >> > Stamatis > > >> > > > >> > On Wed, Oct 14, 2026 at 12:48 AM Konstantin Bereznyakov > > >> > <[email protected]> wrote: > > >> > > > > >> > > Hi all, > > >> > > > > >> > > I investigated repeated CI failures on HIVE-30037 (PR #6842 > > >> > > (https://github.com/apache/hive/pull/6842)) and found matching > > >> > > agent/pod failures on unrelated PRs. > > >> > > > > >> > > All times below are October 13, UTC: > > >> > > > > >> > > #6842 build #12: split-09 pod failed at 09:24; postprocessing raised > > >> > > AgentOfflineException. Archived test reports contain zero > > >> > > failures/errors, including the new branch tests. > > >> > > PR #6861 (https://github.com/apache/hive/pull/6861), build #8: > > >> > > split-15 pod failed at 16:58. > > >> > > PR #6871 (https://github.com/apache/hive/pull/6871), build #3: six > > >> > > pods failed between 18:58 and 19:17. > > >> > > #6842 build #13: pod failed at 19:28 during workspace upload, after > > >> > > successful compilation and before testing. > > >> > > > > >> > > The logs consistently show ChannelClosedException, “Pod just failed,” > > >> > > and agent deletion. This appears to be a shared infrastructure issue. > > >> > > > > >> > > Could someone with CI access check the Kubernetes pod/node events? I > > >> > > have saved all four console logs. > > >> > > > > >> > > Thanks, > > >> > > Konstantin > > > > > > > > > > > > -- > > > Thanks, > > > Raghav Aggarwal
