ChickenBenny opened a new issue, #75557:
URL: https://github.com/apache/airflow/issues/75557
### Under which category would you file this issue?
Task SDK
### Apache Airflow version
3.4.0b2
### What happened and how to reproduce it?
`supervise_task` reports a launch failure to the API server and the task log
only when the coordinator raises `TaskLaunchError`.
`SubprocessCoordinator.execute_task` raises it only for native Dag files. These
failures raise other exceptions:
- `@task.stub` task whose artifact is missing: `_resolve_artifact_bundle`
and `_build_execute_task_command` raise `FileNotFoundError` (e.g. no JAR in the
`task_handler_bundle_name` bundle).
- Runtime exits or times out before connecting (bad Java classpath, Go
executable needing a newer glibc, missing Node module, ...):
`_accept_connections` raises `RuntimeError` / `TimeoutError`. This applies to
native Dags too.
The exception reaches the executor before `task_instances.start()`, so:
- the try has no task log;
- the task is failed through the scheduler's "state mismatch" path
(`_process_executor_events`), which looks like an executor problem, and is
retried even for a configuration error;
- the runtime's stdout/stderr, already read into `drained` by
`_accept_connections`, is discarded. On a successful start it would have been
forwarded to the task log by `_register_pipe_readers(data=drained)`, so the
actual error (e.g. `Error: Could not find or load main class ...`) ends up in
no log at all.
**Reproduction**
A script in Breeze calls the real `supervise_task` with a `JavaCoordinator`,
mocking only the API client (`create_autospec(Client)`), `_is_native_dag_file`
(returns `False`, i.e. a stub task) and `_resolve_artifact_bundle` (a temp
directory):
- Case A: the bundle directory holds no JAR.
- Case B: the runtime command is `sh -c "echo 'Error: Could not find or load
main class com.example.MyTasks' >&2; exit 1"`.
- Case C (control): the same failure raised as `TaskLaunchError`, as the
native Dag path does.
```
===== Case A: no JAR in the task_handler_bundle_name bundle =====
supervise_task raised FileNotFoundError: cannot find a JAR with Main-Class
metadata in /tmp/tmppcubeudw
task_instances.start called: False
task_instances.finish called: False
task_instances.retry called: False
task log size: 0 bytes
===== Case B: runtime exits before connecting =====
[info ] Starting subprocess [coordinators.subprocess]
loc=_subprocess.py:372 pid=21
supervise_task raised RuntimeError: process exited with 1 before connecting
task_instances.start called: False
task_instances.finish called: False
task_instances.retry called: False
task log size: 0 bytes
runtime stderr in task log: False
===== Case C (control): the same failure raised as TaskLaunchError =====
[warning ] Cannot launch the task's runtime [supervisor] reason='cannot
find a JAR with Main-Class metadata in /bundle'
supervise_task returned normally
task_instances.start called: True
task_instances.retry called: True
task log size: 231 bytes
task log:
{"level":"error","event":"Cannot launch the task's runtime","reason":"cannot
find a JAR with Main-Class metadata in /bundle","logger":"task"}
```
In case B, the `Error: Could not find or load main class ...` line printed
by the runtime appears nowhere in the output.
**What a user sees**
Cases A and B follow the executor and scheduler code paths (not checked in a
running UI):
| Where | A: artifact missing | B: runtime exits early | C:
`TaskLaunchError` |
|---|---|---|---|
| Try reached RUNNING | No | No | Yes |
| Task log | Empty | Empty | Has the reason |
| Audit log | `state mismatch`: "Executor ... reported ... finished with
state failed, but the task instance's state attribute is queued" | Same |
Nothing unusual |
| Retries | Retried, though a configuration error cannot succeed | Retried |
Follows `TaskLaunchError.retryable` |
| Worker log | `FileNotFoundError` with the path | Only `process exited with
1 before connecting` | Has the reason |
| Runtime's own error output | n/a (runtime not started) | Nowhere | n/a
(runtime not started) |
### What you think should happen instead?
Raise these failures as `TaskLaunchError`, so the task is started, fails
with the reason in its task log, and the runtime's captured output is included.
A missing artifact should not be retryable; a runtime that exits or times out
could be.
### Operating System
_No response_
### Deployment
None
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
_No response_
### Official Helm Chart version
Not Applicable
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
Related: #74036 introduced `TaskLaunchError` for native Dag files; #73970
moved artifacts into Dag bundles, which makes the missing-artifact case likely
during upgrades.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
