seanghaeli opened a new pull request, #68865: URL: https://github.com/apache/airflow/pull/68865
The in-process Execution API owns a background asyncio event loop + daemon thread. After #68840, their cleanup is registered with `weakref.finalize` keyed on the `InProcessExecutionAPI` **instance**. But callers build a sync `Client` from `InProcessExecutionAPI().transport` and discard the factory object, so the instance is garbage-collected while the transport is still in use. The finalizer then stops the loop, and every subsequent request hangs on the dead loop — surfacing as `Timeout` failures across the Dag-processor and triggerer in-process API tests, currently red on `main` (~24/31 serialization-core canary jobs: `Error while closing in-process execution API lifespan -> TimeoutError`). Fix: key the finalizer on the returned **WSGI transport** (which the `Client` holds) instead of the throwaway factory instance, so loop/thread/lifespan teardown happens when the transport is collected, not the factory. **Verified locally** (breeze, py3.10): `test_processor.py::TestDagFileProcessor::test_top_level_variable_set` and `test_top_level_variable_access_not_found` now pass (`2 passed in 27.37s`); both previously hung to timeout. This is the same root cause and approach as @potiuk's #68855 — opening this as a ready-to-merge equivalent since `main` is red for all PRs and #68855 is currently blocked on an unrelated CI issue. Happy to close this in favor of #68855 if that lands first; whichever unblocks `main` fastest. related: #68840, #68855 --- ##### Was generative AI tooling used to co-author this PR? - [x] Yes — Claude Code (Opus 4.8) Generated-by: Claude Code (Opus 4.8) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
