The problem is still there even after merging HIVE-30128 so we will keep looking. Will update this thread once I have further information.
Best, Stamatis On Thu, Oct 15, 2026 at 6:36 PM Stamatis Zampetakis <[email protected]> wrote: > > My analysis of the problem in the past few days also points to the GKE > autoscaler bringing down nodes with scheduled pods. HIVE-30128 is a > promising fix so I merged it a few minutes ago. Let's continue > monitoring the situation to see if additional actions are needed. > > For the record, there has been a Kubernetes upgrade on September 20 > which may have brought new changes in the autoscaler logic aggravating > the issue. > > $ gcloud logging read > 'protoPayload.methodName="google.container.internal.ClusterManagerInternal.PatchCluster" > AND protoPayload.metadata.operationType="UPGRADE_MASTER" AND > timestamp>="2026-08-20T00:00:00Z"' --limit=10 --format=json | jq '.[] > |{timestamp:.timestamp,newversion:.protoPayload.metadata.currentMasterVersion, > oldversion:.protoPayload.metadata.previousMasterVersion}' > { > "timestamp": "2026-09-27T23:15:18.476357688Z", > "newversion": "1.35.6-gke.1250000", > "oldversion": "1.35.6-gke.1250000" > } > { > "timestamp": "2026-09-20T23:23:51.890372730Z", > "newversion": "1.35.6-gke.1250000", > "oldversion": "1.34.9-gke.1655001" > } > > There were ChannelClosedException issues appearing on master around > September 20 [1] but also before [2, 3] so it's hard to fully blame > just the upgrade. > > Best, > Stamatis > > [1] https://ci.hive.apache.org/job/hive-precommit/job/master/3094/ > [2] https://ci.hive.apache.org/job/hive-precommit/job/master/3063/ > [3] https://ci.hive.apache.org/job/hive-precommit/job/master/3006/ > > On Wed, Oct 14, 2026 at 6:20 PM Raghav Aggarwal > <[email protected]> wrote: > > > > Hi Stamatis, > > > > This issue has been occurring for over three weeks, so it is not a recent > > problem. I have observed it multiple times across my pull requests. > > > > For reference: > > > > - A fix attempt appears to be in progress (HIVE-30128): > > https://github.com/apache/hive/pull/6843 > > - A related PR merged 19 days ago demonstrates this pod failure issue: > > https://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-6750 > > > > Regards, > > Raghav Aggarwal > > > > > > On Wed, Oct 14, 2026 at 5:19 PM Stamatis Zampetakis <[email protected]> > > wrote: > >> > >> Small correction: The PR 6644 and 6872 advance normally. However, in > >> PR#6815 the ChannelClosedException popped up but the underlying pods > >> are long gone so don't really know what happened. I will check a bit > >> further to see if I can determine something. > >> > >> [2026-10-14T10:07:50.257Z] Cannot contact > >> hive-precommit-pr-6815-5-9327p-g96pl-71l7f: > >> hudson.remoting.ChannelClosedException: Channel > >> "hudson.remoting.Channel@5ca733ed:JNLP4-connect connection from > >> 10.0.13.8/10.0.13.8:39556": Remote call on JNLP4-connect connection > >> from 10.0.13.8/10.0.13.8:39556 failed. The channel is closing down or > >> has closed down > >> > >> Best, > >> Stamatis > >> > >> On Wed, Oct 14, 2026 at 12:36 PM Stamatis Zampetakis <[email protected]> > >> wrote: > >> > > >> > Hi Konstantin, > >> > > >> > I just checked the Google Cloud incidents and it appears that there > >> > was a problem affecting VMs in us-central1 where our pods are running. > >> > The incident timeline: > >> > * Started on October 13, 2026 at 3:19:12 PM UTC+1 > >> > * Resolved on October 13, 2026 at 6:35:59 PM UTC+1 > >> > > >> > At the moment, I see 61 pods up and running normally for the following > >> > PRs: 6644, 6815, 6872. Let's see if the problem persists to > >> > investigate further. > >> > > >> > Best, > >> > Stamatis > >> > > >> > On Wed, Oct 14, 2026 at 12:48 AM Konstantin Bereznyakov > >> > <[email protected]> wrote: > >> > > > >> > > Hi all, > >> > > > >> > > I investigated repeated CI failures on HIVE-30037 (PR #6842 > >> > > (https://github.com/apache/hive/pull/6842)) and found matching > >> > > agent/pod failures on unrelated PRs. > >> > > > >> > > All times below are October 13, UTC: > >> > > > >> > > #6842 build #12: split-09 pod failed at 09:24; postprocessing raised > >> > > AgentOfflineException. Archived test reports contain zero > >> > > failures/errors, including the new branch tests. > >> > > PR #6861 (https://github.com/apache/hive/pull/6861), build #8: > >> > > split-15 pod failed at 16:58. > >> > > PR #6871 (https://github.com/apache/hive/pull/6871), build #3: six > >> > > pods failed between 18:58 and 19:17. > >> > > #6842 build #13: pod failed at 19:28 during workspace upload, after > >> > > successful compilation and before testing. > >> > > > >> > > The logs consistently show ChannelClosedException, “Pod just failed,” > >> > > and agent deletion. This appears to be a shared infrastructure issue. > >> > > > >> > > Could someone with CI access check the Kubernetes pod/node events? I > >> > > have saved all four console logs. > >> > > > >> > > Thanks, > >> > > Konstantin > > > > > > > > -- > > Thanks, > > Raghav Aggarwal
