pmcquighan-camus opened a new issue, #76871: URL: https://github.com/apache/airflow/issues/76871
### Under which category would you file this issue? Providers ### Apache Airflow version 3.4.0 ### What happened and how to reproduce it? I wasn't sure if it's better to create a new issue or re-open https://github.com/apache/airflow/issues/63838 which was not fully resolved by the proposed fix https://github.com/apache/airflow/pull/63915. The underlying issue is still that the hook is waiting for a condition that will never occur. Specifically it is looking for "succeeded pods" [here](https://github.com/apache/airflow/blob/providers-google/22.8.0/providers/google/src/airflow/providers/google/cloud/triggers/kubernetes_engine.py#L362) and if that list is different than the initial pod that it knew of, then the hook would check that for XCom values. However, that can *never* occur since the pod is only considered "Succeeded" if all containers in the pod are completed, and the `airflow-xcom-sidecar` container will not terminate until the [extract_xcom](https://github.com/apache/airflow/blob/providers-google/22.8.0/providers/google/src/airflow/providers/google/cloud/triggers/kubernetes_engine.py#L396) method is called which sends it the SIGINT. Here is a snippet of the task logs: ``` [2026-11-10 14:37:17] INFO - Found matching pod job-k8s-output-zn4ofy9n-trj98 with labels {'airflow_kpo_in_cluster': 'None', 'airflow_version': '3.4.0', 'batch.kubernetes.io/controller-uid': '19b01c5b-816b-4c79-a7e3-a51c6a4829ce', 'batch.kubernetes.io/job-name': 'job-k8s-output-zn4ofy9n', 'controller-uid': '19b01c5b-816b-4c79-a7e3-a51c6a4829ce', 'dag_id': 'k8s-test', 'job-name': 'job-k8s-output-zn4ofy9n', 'kubernetes_pod_operator': 'True', 'run_id': 'manual__2026-11-10T193706.9565410000-ad7d048ad', 'task_id': 'k8s_output', 'try_number': '1'} [2026-11-10 14:37:17] INFO - Pausing task as DEFERRED. [2026-11-10 14:37:19] INFO - trigger k8s-test/manual__2026-11-10T19:37:06.956541+00:00/k8s_output/-1/1 (ID 520) starting [2026-11-10 14:37:20] INFO - No succeeded pods found yet for job job-k8s-output-zn4ofy9n, retrying in 10.0 seconds... [2026-11-10 14:37:30] INFO - No succeeded pods found yet for job job-k8s-output-zn4ofy9n, retrying in 10.0 seconds... [2026-11-10 14:37:40] INFO - No succeeded pods found yet for job job-k8s-output-zn4ofy9n, retrying in 10.0 seconds... [2026-11-10 14:37:50] INFO - No succeeded pods found yet for job job-k8s-output-zn4ofy9n, retrying in 10.0 seconds... [2026-11-10 14:38:00] INFO - No succeeded pods found yet for job job-k8s-output-zn4ofy9n, retrying in 10.0 seconds... [2026-11-10 14:38:10] INFO - No succeeded pods found yet for job job-k8s-output-zn4ofy9n, retrying in 10.0 seconds... [2026-11-10 14:38:21] INFO - No succeeded pods found yet for job job-k8s-output-zn4ofy9n, retrying in 10.0 seconds... [2026-11-10 14:38:31] INFO - No succeeded pods found yet for job job-k8s-output-zn4ofy9n, retrying in 10.0 seconds... [2026-11-10 14:38:41] INFO - No succeeded pods found yet for job job-k8s-output-zn4ofy9n, retrying in 10.0 seconds... [2026-11-10 14:38:51] INFO - No succeeded pods found yet for job job-k8s-output-zn4ofy9n, retrying in 10.0 seconds... [2026-11-10 14:39:01] WARNING - No succeeded pods found for job job-k8s-output-zn4ofy9n after timeout, falling back to original pod list [2026-11-10 14:39:01] INFO - Requesting status for the job 'job-k8s-output-zn4ofy9n' [2026-11-10 14:39:01] INFO - The job 'job-k8s-output-zn4ofy9n' is incomplete. Sleeping for 10 sec. [2026-11-10 14:39:11] INFO - Requesting status for the job 'job-k8s-output-zn4ofy9n' [2026-11-10 14:39:11] INFO - The job 'job-k8s-output-zn4ofy9n' is incomplete. Sleeping for 10 sec. [2026-11-10 14:39:21] INFO - Requesting status for the job 'job-k8s-output-zn4ofy9n' [2026-11-10 14:39:21] INFO - The job 'job-k8s-output-zn4ofy9n' is incomplete. Sleeping for 10 sec. [2026-11-10 14:39:31] INFO - Requesting status for the job 'job-k8s-output-zn4ofy9n' [2026-11-10 14:39:31] INFO - The job 'job-k8s-output-zn4ofy9n' is incomplete. Sleeping for 10 sec. [2026-11-10 14:39:41] INFO - Requesting status for the job 'job-k8s-output-zn4ofy9n' [2026-11-10 14:39:41] INFO - The job 'job-k8s-output-zn4ofy9n' is incomplete. Sleeping for 10 sec. [2026-11-10 14:39:51] INFO - Requesting status for the job 'job-k8s-output-zn4ofy9n' [2026-11-10 14:39:52] INFO - The job 'job-k8s-output-zn4ofy9n' is incomplete. Sleeping for 10 sec. [2026-11-10 14:40:02] INFO - Requesting status for the job 'job-k8s-output-zn4ofy9n' .... ``` Eventually there is a timeout as the condition never occurs. The pod for the job in k8s will sit there with the main container exited and the `airflow-xcom-sidecar` idling indefinitely. ### What you think should happen instead? Ideally, the airflow hook should have seen that the job had a new pod created and then waited for that pod to complete and successfully extracted the XCom data. If the hook had waited on the correct pod, then the xcom sidecar would have been terminated properly. ### Operating System _No response_ ### Deployment None ### Apache Airflow Provider(s) google ### Versions of Apache Airflow Providers apache-airflow-providers-cncf-kubernetes==10.25.0 apache-airflow-providers-google==22.8.0 plus all the other standard providers with airflow 3.4.0, these are just the two most relevant ones. ### Official Helm Chart version Not Applicable ### Kubernetes Version _No response_ ### Helm Chart configuration _No response_ ### Docker Image customizations Not applicable ### Anything else? Replacing L353-398 with: ```python # Re-discover pods for this job to handle retry scenarios # where the original pod failed and a new pod was created label_selector = f"batch.kubernetes.io/job-name={self.job_name}" xcom_results = [] completed = False while not completed: pods = await self.hook.list_pods( namespace=self.pod_namespace, label_selector=label_selector, ) self.pod_names = [p.metadata.name for p in pods] self.log.info(f"found pods: {self.pod_names}") try: for pod_name in self.pod_names: pod = await self.hook.get_pod(name=pod_name, namespace=self.pod_namespace) await self.hook.wait_until_container_complete( name=pod_name, namespace=self.pod_namespace, container_name=self.base_container_name, poll_interval=self.poll_interval, ) self.log.info("Checking if xcom sidecar container is started.") await self.hook.wait_until_container_started( name=pod_name, namespace=self.pod_namespace, container_name=PodDefaults.SIDECAR_CONTAINER_NAME, poll_interval=self.poll_interval, ) self.log.info("Extracting result from xcom sidecar container.") loop = asyncio.get_running_loop() xcom_result = await loop.run_in_executor(None, self.pod_manager.extract_xcom, pod) xcom_results.append(xcom_result) completed = True except async_client.ApiException as e: if str(e.status) == "404": self.log.warning( "Pod was deleted while waiting for completion, retrying" ) else: raise job: V1Job = await self.hook.wait_until_job_complete( name=self.job_name, namespace=self.job_namespace, poll_interval=self.poll_interval ) ``` seems to work correctly, but i'm not positive it's the best approach. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
