potiuk commented on PR #73626:
URL: https://github.com/apache/airflow/pull/73626#issuecomment-6647016724
Thanks for working on this. I'm closing this PR because it attributes the
hang to the wrong lock, so the fix and its tests are built around a different
problem from the one reported in #73583.
The PR blames `_service_instance_id_lock`, which only matters if another
thread holds it at the moment of the fork. #73583 reports a fully reproducible
hang, and its py-spy stacks stop in `submit`, not in the detector.
The likelier cause is at-fork hook ordering on `_global_shutdown_lock` in
`concurrent.futures.thread`. That module holds the lock across `fork` and
releases it in its own `after_in_child` hook, and hooks run in registration
order:
1. With `[traces] otel_service` set, `_get_backcompat_config` builds a
static `Resource({...})`, so `TracerProvider` registers its fork hook before
`concurrent.futures.thread` is imported.
2. The first later `ThreadPoolExecutor` use imports that module, so its hook
runs after the provider's.
3. In the child, the provider's handler reaches `submit` while the lock is
still held, and deadlocks.
I reproduced this on `main` through `configure_otel` (SDK 1.45.1, Python
3.13, macOS):
| Setup | Fork result |
|---|---|
| No `otel_service` | Fine |
| `otel_service` set | Fine |
| `otel_service` set, then one executor use before `fork` | Hangs every
time, with `_service_instance_id_lock` free |
This has not been confirmed on the reporter's setup (RHEL, Python 3.12, SDK
1.44.0).
The tests here hold `_service_instance_id_lock`, so they don't cover that
path. Even for that race, metrics stay exposed: `stats.py` resets the backend
after every fork, so the child's first metric goes through `Resource.create()`
and the same detector. A child that inherits the lock locked hangs there.
#73583 stays open. A new PR is welcome if it fixes the ordering problem
directly, with a regression test that sets `otel_service`, uses an executor and
then forks.
This comment was drafted with AI assistance and may contain mistakes. If the
analysis is wrong, please say so here and a maintainer will take another look.
---
Drafted-by: Claude Code (Opus 5.5); reviewed by @potiuk before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
