ChickenBenny opened a new pull request, #75561: URL: https://github.com/apache/airflow/pull/75561
closes: #75557 --- `supervise_task` fails a task whose runtime cannot launch, with the reason in its task log, only when the coordinator raises `TaskLaunchError`. `SubprocessCoordinator` raised it only for a native Dag file that is missing or cannot run. Any other failure before the runtime connected reached the executor before the task instance was started: a `@task.stub` artifact that cannot be found, a runtime that cannot be executed (`java` not installed, wrong architecture), an unsupported supervisor schema version, or a runtime that exits or times out before connecting. The try then had no task log, the task was failed through the scheduler's "state mismatch" path, and the runtime's output, already read by `_accept_connections`, was dropped. A JVM that printed `Error: Could not find or load main class ...` left that line in no log at all. Rather than catching each of these where it is raised, `SubprocessCoordinator` now raises any failure before the task instance is started as `TaskLaunchError`, through one context manager, `_raise_as_launch_error`, around two blocks: - in `execute_task`, resolving the Dag bundle and building the command; - in `_PopenActivitySubprocess.start`, starting the runtime until it connects. `_on_child_started`, which starts the task instance, and `wait()` stay outside, because the supervisor starts the task instance itself when it handles a `TaskLaunchError`. A `TaskLaunchError` raised inside, such as a routing mistake that is not retryable, is kept as it is. The blocks are only indented, so the diff reads best with whitespace ignored. `_accept_connections` adds the runtime's output to its error: the last 4 KiB of stdout and stderr, including output still buffered in the sockets, read without blocking because a runtime can exit before its last lines are read. The new failures are retryable, like a native Dag file that is missing or cannot run: an artifact can appear once a bundle finishes syncing, and a runtime that timed out on a busy worker can start on the next try. Catching `Exception` also reports a coordinator bug as a launch failure; its traceback is then in the task log rather than in no log. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
