aaron-y-chen opened a new pull request, #74292: URL: https://github.com/apache/airflow/pull/74292
When testing `LocalExecutor` process termination under multiprocessing `spawn`, each spawned worker process boots a clean Python interpreter and re-imports the Airflow application stack before entering the workload and writing the marker file. On the Python 3.14 core test jobs in CI, process startup and module imports take between 8 and 10 seconds. The previously hardcoded 10-second polling deadline caused intermittent timeouts in canary runs, such as [run 37321253685, job 111809020070](https://github.com/apache/airflow/actions/runs/37321253685/job/111809020070): ```text FAILED airflow-core/tests/unit/executors/test_local_executor.py::TestLocalExecutorBookkeeping::test_actual_worker_death_after_start_releases_slot[callback-spawn] - assert 369.138999549 < 369.129381766 + where 369.138999549 = <built-in function monotonic>() + where <built-in function monotonic> = time.monotonic ``` In that exact test execution: - `test_actual_worker_death_after_start_releases_slot[task-spawn]`: 7.87s (passed) - `test_actual_worker_death_after_start_releases_slot[connection-spawn]`: 9.62s (passed) - `test_actual_worker_death_after_start_releases_slot[callback-spawn]`: 10.05s (timed out against deadline = start_monotonic + 10) The Sqlite job of the same run ([job 111809020108](https://github.com/apache/airflow/actions/runs/37321253685/job/111809020108)) failed the same way on a different variant: `task-spawn` timed out at 10.06s, while `callback-spawn` and `connection-spawn` passed at 7.83s and 8.92s. This PR: - Increases the polling deadline from 10s to 30s with an informative assertion error message. - Adds an early liveness check (`assert any(proc.is_alive() for proc in executor.workers.values())`) so that if a worker process unexpectedly crashes during startup, the test fails immediately with the workers' exit codes rather than waiting for the timeout. - Increases the test `@pytest.mark.execution_timeout` to 60s so it stays above the 30s polling deadline plus teardown, and the `proc.join()` timeout after `SIGKILL` to 5s for headroom on busy runners. --- ##### Was generative AI tooling used to co-author this PR? <!-- If generative AI tooling has been used in the process of authoring this PR, please change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by". --> - [x] Yes (please specify the tool below) Generated-by: [Claude Opus-5.5] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
