anantauprety opened a new issue, #76325:
URL: https://github.com/apache/airflow/issues/76325
### Under which category would you file this issue?
Providers
### Apache Airflow version
3.3.2
### What happened and how to reproduce it?
With `deferrable=True`, `wait_until_job_complete=True` and `get_logs=True`,
a Job whose container exits non-zero fails the task with `Job failed with
error: BackoffLimitExceeded`. The task log contains none of the container's
output, so the cause of the failure isn't visible in Airflow.
In `KubernetesJobOperator.execute_complete`, the error check raises before
the log-writing block runs:
```python
try:
ti = context["ti"]
ti.xcom_push(key="job", value=event["job"])
if event["status"] == "error":
raise AirflowException(event["message"]) # raised here on failure
if self.get_logs: # so this never runs for
a failed Job
for pod_name in event.get("pod_names") or []:
...
self._write_logs(pods_by_name[pod_name])
...
finally:
self._cleanup_monitoring_pods_from_dict(...) # then deletes the pod
(on_finish_action=delete_pod)
```
Logs are written only when the Job succeeds. When it fails,
`on_finish_action="delete_pod"` deletes the pod in the `finally` block, so its
output can't be read afterwards either.
A second, related gap: `pod_names` holds only the pods found when the task
deferred. Pods the Job controller creates later for `backoffLimit` retries are
never logged or cleaned up.
To reproduce:
```python
KubernetesJobOperator(
task_id="fail",
image="busybox",
cmds=["sh", "-c", "echo 'the real error'; exit 1"],
backoff_limit=1,
wait_until_job_complete=True,
deferrable=True,
get_logs=True,
)
```
The task fails with `BackoffLimitExceeded`, and `the real error` doesn't
appear in the task log.
### What you think should happen instead?
When `get_logs=True`, write the pod logs before raising on `status ==
"error"`, and include every pod that belongs to the Job. For example, look them
up by the `batch.kubernetes.io/job-name=<job>` label rather than only the pods
in `event["pod_names"]`. A failure while reading logs should be logged as a
warning and should not replace the Job's own error.
### Operating System
Debian (Astro Runtime image)
### Deployment
Astronomer
### Apache Airflow Provider(s)
cncf-kubernetes
### Versions of Apache Airflow Providers
apache-airflow-providers-cncf-kubernetes==10.23.0 (the code is unchanged in
10.25.0 and on `main`)
### Official Helm Chart version
Not Applicable
### Kubernetes Version
v1.35.8 (EKS)
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
This happens every time a Job fails. Issue #42144 also covered deferred-mode
log retrieval, but for a different cause (`pod_name` was `None` in the event),
and it was closed as stale. We work around this one by overriding
`execute_complete` in a subclass to write the logs of all the Job's pods before
calling `super()`.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
