goingforstudying-ctrl opened a new pull request, #71995: URL: https://github.com/apache/airflow/pull/71995
Been digging into #71707 — the case where a task subprocess hangs forever with every thread stuck at `ssl.py:440` building fresh `SSLContext`s. The mechanism there is that `ActivitySubprocess.start()` does a bare `os.fork()` on Linux, and if a sibling thread in the supervisor (OTel exporter, google-auth refresh threads, an OpenLineage listener...) happens to be inside OpenSSL's global lock at that exact moment, the child inherits the held lock with no owner left to release it. Every later `SSLContext` construction in that child then blocks forever. What surprised me is that `[core] execute_tasks_new_python_interpreter=True` doesn't actually protect you from this. The executor honors the knob when launching the *supervisor* (e.g. celery's `_execute_in_subprocess`), but the supervisor then bare-forks the task runner regardless — the config stops one fork short of the process that actually runs user code. The Edge worker hit the same class of problem and got the config honored for its launch path in #65943; this does the equivalent for the Task SDK's own fork. The fork+exec machinery itself already exists and isn't macOS-specific (it's what `_should_use_exec()` gates), so the change is small: when the config is set and we're launching the real task entry point, take the exec path too. Test stub targets keep bare fork as before, and the DAG processor / triggerer paths are untouched since the option is task-scoped. Verified on Linux: with the config set the task child comes up as a fresh interpreter (`_child_exec_main` rehydrating over the dup2'd FDs, supervisor comms working end to end), `ruff`/`mypy` are clean, and the existing supervisor test suite passes (the handful of `test_exit_by_signal`/remote-logging errors fail identically on a clean checkout — missing optional deps locally, unrelated). Added selection tests for default / config-set / fork-unsafe-platform / stub-target combinations. closes: #71707 --- ##### Was generative AI tooling used to co-author this PR? - [ ] Yes (please specify the tool below) --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
