0BVer opened a new pull request, #77283:
URL: https://github.com/apache/airflow/pull/77283

   Forked child processes hang forever when traces are enabled and the 
deprecated `[traces] otel_service` option is set (and neither 
`OTEL_SERVICE_NAME` nor `OTEL_RESOURCE_ATTRIBUTES` is). In Airflow this shows 
up on Linux, where the supervisor starts Dag processor and task processes with 
a bare `os.fork()` (on macOS, 3.4 and later use `posix_spawn`, which runs no 
at-fork hooks), as Dag processor parsing processes killed by 
`dag_file_processor_timeout` and as hung task processes. I see the same hang on 
3.3.2 and 3.4.0 with `[traces] otel_service` set, in Dag processor parsing 
processes and LocalExecutor task processes, and the code that builds the static 
`Resource` is identical in both versions.
   
   #73583 reports a hang on 3.3.2 and its py-spy stack matches the path 
described below, but the issue does not say whether `[traces] otel_service` is 
set on the reporter's setup, so this is `related:` and not `closes:`.
   
   related: #73583
   
   ## Cause
   
   Since opentelemetry-sdk 1.44, `TracerProvider` (and `MeterProvider`, 
`LoggerProvider`) registers an `os.register_at_fork(after_in_child=...)` 
handler. In the child it refreshes process-dependent resource attributes 
through `get_aggregated_resources()`, which calls 
`ThreadPoolExecutor.submit()`. `submit()` takes 
`concurrent.futures.thread._global_shutdown_lock`.
   
   `concurrent.futures.thread` is imported lazily. When it is imported it 
registers its own fork hooks: `before` acquires `_global_shutdown_lock` and 
`after_in_child` resets it. `after_in_child` hooks run in registration order, 
so if the module is first imported after the provider was created, the 
provider's handler runs in the child while the lock is still in the held state 
inherited from the parent, and `submit()` blocks forever.
   
   With `[traces] otel_service` set, `_get_backcompat_config` passed a static 
`Resource({...})` to `TracerProvider`. The provider only calls 
`Resource.create()` itself when it gets no resource 
(`opentelemetry/sdk/trace/__init__.py:1271`, SDK 1.45.2), and 
`Resource.create()` is what imports `concurrent.futures.thread`. So with a 
static resource nothing has imported the module when the provider registers its 
handler (`:1296`). Without `otel_service`, `Resource.create()` runs first and 
the order is safe.
   
   When the late import happens:
   
   - In a real Airflow process, `import airflow` runs `configure_otel()` 
(through `settings.initialize()`) and later imports `concurrent.futures.thread` 
on its own: on the 3.3.1 release image the first import comes from 
`airflow.sdk.io` through `fsspec`. So with `[traces] otel_service` set, a plain 
fork hangs with every exporter, with no executor use by application code.
   - With `configure_otel()` alone it depends on the exporter. With the default 
`otlp` (gRPC) exporter, the exporter module imports `opentelemetry.sdk._logs`, 
whose `ReadWriteLogRecord` default value `Resource.create({})` runs at import 
time, inside `configure_otel()` after the provider was created, so every fork 
hangs. `import grpc` itself does not import `concurrent.futures.thread`. With 
the `console` or `otlp_proto_http` exporter, forks hang once anything in the 
parent process touches `concurrent.futures.ThreadPoolExecutor` before forking 
(attribute access is enough); if nothing does, the child imports the module 
fresh inside the provider's handler and does not hang. The review comment on 
the closed #73626 reported, through `configure_otel()` alone, that 
`otel_service` alone is fine and that one executor use before the fork hangs. 
It does not name the exporter; with the default `otlp` exporter I see a hang 
without any executor use.
   
   The hang is in `ThreadPoolExecutor.submit()` on `_global_shutdown_lock`. 
`_service_instance_id_lock`, which #73626 focused on, is not held in the child 
(observed `locked() == False`), and the stack never reaches the detector that 
takes it.
   
   SDK versions before 1.44 have no `TracerProvider` fork handler and are not 
affected. The ordering sensitivity inside the SDK is tracked in 
open-telemetry/opentelemetry-python#5789 with a proposed fix in 
open-telemetry/opentelemetry-python#5790 (both still open).
   
   <details>
   <summary>Stdlib-only reproducer of the mechanism</summary>
   
   No OpenTelemetry involved: a fork hook that submits to a 
`ThreadPoolExecutor` hangs when it was registered before 
`concurrent.futures.thread` was imported, and does not when the import came 
first.
   
   ```python
   import concurrent.futures, os, signal, sys, time
   
   
   def provider_like_after_in_child():
       # Same shape as the SDK's TracerProvider fork handler.
       with concurrent.futures.ThreadPoolExecutor(1) as executor:
           executor.submit(lambda: None).result()
   
   
   if sys.argv[1] == "import-first":
       concurrent.futures.ThreadPoolExecutor  # imports 
concurrent.futures.thread, its fork hooks go first
   os.register_at_fork(after_in_child=provider_like_after_in_child)
   concurrent.futures.ThreadPoolExecutor  # otherwise imported only here, after 
the hook above
   
   pid = os.fork()
   if pid == 0:
       os._exit(0)
   for _ in range(50):
       if os.waitpid(pid, os.WNOHANG)[0]:
           print(sys.argv[1], "-> child exited: OK")
           break
       time.sleep(0.1)
   else:
       os.kill(pid, signal.SIGKILL)
       os.waitpid(pid, 0)
       print(sys.argv[1], "-> child HUNG for 5s, killed")
   ```
   
   `python repro.py import-late` hangs and `python repro.py import-first` 
exits, on Python 3.10 to 3.14.
   
   </details>
   
   <details>
   <summary>Reproducer with opentelemetry-sdk only (no Airflow)</summary>
   
   ```python
   import os, signal, sys, time
   from opentelemetry.sdk.resources import Resource
   from opentelemetry.sdk.trace import TracerProvider
   
   attrs = {"service.name": "my-service"}
   resource = Resource(attrs) if sys.argv[1] == "static" else 
Resource.create(attrs)
   provider = TracerProvider(resource=resource)  # SDK >= 1.44 registers an 
after_in_child hook here
   import opentelemetry.sdk._logs  # what the OTLP gRPC exporter imports: first 
import of concurrent.futures.thread
   
   pid = os.fork()
   if pid == 0:
       os._exit(0)
   for _ in range(50):
       if os.waitpid(pid, os.WNOHANG)[0]:
           print(sys.argv[1], "-> child exited: OK")
           break
       time.sleep(0.1)
   else:
       os.kill(pid, signal.SIGKILL)
       os.waitpid(pid, 0)
       print(sys.argv[1], "-> child HUNG for 5s, killed")
   ```
   
   `python repro.py static` hangs and `python repro.py create` exits on 
opentelemetry-sdk 1.44.0 and 1.45.2 (Python 3.10 to 3.14). Both exit on 1.43.0. 
Keep the `provider` variable: the SDK's fork hook holds only a weak reference 
to it.
   
   </details>
   
   <details>
   <summary>At-fork hook order, default gRPC exporter (Python 3.12.8, SDK 
1.45.2)</summary>
   
   Hooks registered through `os.register_at_fork`, in registration order, with 
`[traces] otel_service` set:
   
   | # | Before (static `Resource`) | Slots | After (`Resource.create`) | Slots 
|
   |---|---|---|---|---|
   | 1 | `threading` | after_in_child | `threading` | after_in_child |
   | 2 | `random` | after_in_child | `random` | after_in_child |
   | 3 | `logging` | before, after_in_parent, after_in_child | `logging` | 
before, after_in_parent, after_in_child |
   | 4 | **`TracerProvider.__init__`** | after_in_child | 
**`concurrent.futures.thread`** (`:41`) | before, after_in_parent, 
after_in_child |
   | 5 | `asyncio/events.py` | after_in_child | `concurrent.futures.thread` 
(`:44`) | after_in_child |
   | 6 | **`concurrent.futures.thread`** (`:41`) | before, after_in_parent, 
after_in_child | **`TracerProvider.__init__`** | after_in_child |
   | 7 | `concurrent.futures.thread` (`:44`) | after_in_child | 
`asyncio/events.py` | after_in_child |
   | 8 | `BatchProcessor` | after_in_child | `BatchProcessor` | after_in_child |
   
   (The `:44` hook exists from Python 3.12.)
   
   - Before: the parent's `before` hooks run in reverse (thread hook acquires 
`_global_shutdown_lock`). In the child, hooks 1 to 4 run in order and the child 
blocks inside hook 4, so hook 6 (`_global_shutdown_lock._at_fork_reinit`) never 
runs. `_global_shutdown_lock.locked()` was `True` before every hook up to the 
provider's.
   - After: the thread module's `_at_fork_reinit` runs as hook 4, the lock is 
free when the provider's handler runs as hook 6, and the child returns.
   
   Stack of the hung child (`faulthandler` on `SIGUSR1`):
   
   ```
     File ".../concurrent/futures/thread.py", line 166 in submit
     File ".../opentelemetry/sdk/resources/__init__.py", line 752 in 
get_aggregated_resources
     File ".../opentelemetry/sdk/resources/__init__.py", line 726 in 
_get_process_dependent_resource
     File ".../opentelemetry/sdk/trace/__init__.py", line 1300 in _handle_fork
     File ".../opentelemetry/sdk/trace/__init__.py", line 1294 in 
_after_in_child
   ```
   
   `thread.py:166` is `with self._shutdown_lock, _global_shutdown_lock:`. 
`self._shutdown_lock` belongs to an executor created inside the child, so only 
`_global_shutdown_lock` can be held. The reporter's py-spy stack in #73583 has 
the same frames.
   
   </details>
   
   <details>
   <summary>Fork test matrix (126 runs)</summary>
   
   `buggy` is `main` with `otel_service` set, `fixed` is this change with 
`otel_service` set, `noservice` is without `otel_service` (the code path is the 
same on `main` and with this change; these runs used this change's module). 
`+exec` is one `ThreadPoolExecutor` use in the parent before `fork()`. Every 
cell is one fresh interpreter with a 4 second child timeout, macOS arm64, 
`opentelemetry-sdk` and the `otlp` exporters at the same version.
   
   | Python / SDK | Exporter | buggy | buggy +exec | fixed | fixed +exec | 
noservice | noservice +exec |
   |---|---|---|---|---|---|---|---|
   | 3.10 to 3.14 / 1.45.2, 3.12 / 1.44.0 | otlp (gRPC) | HANG | HANG | OK | OK 
| OK | OK |
   | 3.10 to 3.14 / 1.45.2, 3.12 / 1.44.0 | otlp_proto_http | OK | HANG | OK | 
OK | OK | OK |
   | 3.10 to 3.14 / 1.45.2, 3.12 / 1.44.0 | console | OK | HANG | OK | OK | OK 
| OK |
   | 3.12 / 1.43.0 | all three | OK | OK | OK | OK | OK | OK |
   
   All 24 hangs are `buggy` runs on SDK 1.44 or later, and every one ends in 
`submit()` in the stack dump. There were no hangs in the 42 `fixed` runs, the 
42 `noservice` runs and the 6 `buggy` runs on SDK 1.43.0. Airflow requires 
Python 3.11 or newer, so 3.10 only shows the mechanism of the standalone module.
   
   </details>
   
   <details>
   <summary>Real Airflow process (released 3.3.1 image)</summary>
   
   `apache/airflow:3.3.1` image (Linux aarch64, Python 3.13.14, 
opentelemetry-sdk 1.44.0), `AIRFLOW__TRACES__OTEL_ON=True`, an unreachable 
local OTLP endpoint. The script does `import airflow` and then a bare 
`os.fork()`, with a 20 second child timeout.
   
   | Setup | Fork result |
   |---|---|
   | `[traces] otel_service` set, exporter `otlp`, `console` or 
`otlp_proto_http`, no executor use | HANG (all three) |
   | `[traces] otel_service` set, `console`, plus a `ThreadPoolExecutor` 
attribute access | HANG |
   | only `OTEL_SERVICE_NAME`; `otel_service` and `OTEL_SERVICE_NAME` both set; 
no service; traces off | OK |
   | `[traces] otel_service` set, with `Resource({...})` changed to 
`Resource.create({...})` in the installed copy (`otlp` three times, `console` 
and `otlp_proto_http` with the executor access) | OK |
   
   `airflow dag-processor --num-runs 2` on the same image (sqlite, 
`dag_file_processor_timeout=15`, one trivial Dag file):
   
   - Stock with `[traces] otel_service` set: both parsing processes are killed 
by the processor timeout and the Dag is not registered.
   - Stock with only `OTEL_SERVICE_NAME`, or with the one-line change applied 
to the installed copy: no timeout, the Dag is registered.
   
   This is the 3.3.1 release with the one-line change applied to its installed 
copy, not this branch built under breeze.
   
   </details>
   
   ## Fix
   
   `_get_backcompat_config` now builds the resource with 
`Resource.create({"service.name": ...})` instead of `Resource({"service.name": 
...})`. `Resource.create()` runs in Airflow code before the provider is 
constructed, so `concurrent.futures.thread` registers its fork hooks first, 
which is what the path without `otel_service` already does. This does not 
depend on the order of calls inside `TracerProvider.__init__`. The existing 
guard is unchanged: the option is only used when neither `OTEL_SERVICE_NAME` 
nor `OTEL_RESOURCE_ATTRIBUTES` is set. A short comment records why plain 
`Resource(...)` must not be used here. A bare `import 
concurrent.futures.thread` before constructing the provider would fix the 
ordering too, without changing the resource attributes; `Resource.create()` was 
chosen so the resource is built exactly as on the path without `otel_service`.
   
   Limits of this fix:
   
   - It relies on `Resource.create()` importing `concurrent.futures.thread`. 
That holds in every release that has the fork handler: I ran 1.44.0 and 1.45.2 
and read the source of 1.45.0 and 1.45.1. The fix proposed in 
open-telemetry/opentelemetry-python#5790 runs the process-dependent detectors 
synchronously in fork handlers, so `Resource.create()` itself would still use 
the thread pool.
   - Other code in the same process (local settings, plugins, instrumentation) 
that builds a provider with a static `Resource` before that module is imported 
can still hit the same ordering problem. Only an SDK fix closes that.
   - Airflow's metrics logger is not affected as written: `otel_logger.py` 
calls `Resource.create(...)` before constructing `MeterProvider`.
   
   ## Behaviour notes
   
   - `service.name` is unchanged (`my-service` before and after for 
`otel_service = my-service`).
   - The resource is now built the same way as in a setup without 
`otel_service`, so besides `service.name` it carries the SDK default attributes 
(`telemetry.sdk.*`, `service.instance.id`) and honors 
`OTEL_EXPERIMENTAL_RESOURCE_DETECTORS`. Before, it had `service.name` only.
   - No environment variable is set, so child processes and the metrics logger 
(which reads `OTEL_SERVICE_NAME` before `[metrics] otel_service`) are not 
affected.
   
   ## Regression test
   
   
`TestConfigureOtelServiceName.test_fork_after_executor_use_does_not_deadlock` 
in `shared/observability/tests/observability/test_traces.py` runs a fresh 
interpreter (pytest has already imported `concurrent.futures.thread`, so the 
import order cannot be reproduced in-process). It asserts that 
`concurrent.futures.thread` is not imported yet, calls `configure_otel` with 
`otel_service` set (console exporter, no network), uses a `ThreadPoolExecutor` 
once, then forks. The parent waits for the child with a 10 second `SIGALRM` 
timeout and kills a child that does not return. The test asserts a zero exit 
code and `service.name == "my-service"`.
   
   The test fails on `main` without this change (`forked child deadlocked` 
after 10 seconds) and passes with it. It can only fail on opentelemetry-sdk 
1.44 or later, because older versions have no provider fork handler.
   
   ## Verification
   
   Python 3.13.1, opentelemetry-sdk 1.45.2, macOS.
   
   - `uv run --exact --group dev pytest -n auto` in `shared/observability`: 197 
passed. The same command in every `shared/*` distribution (the loop that 
`breeze verify` lists): all pass.
   - `breeze testing core-tests` as listed by `breeze verify` (Python 3.11): 
the tests that need no database pass (5755 passed), and the database tests on 
sqlite pass for all six test types (API, Always, CLI, Core, Other, 
Serialization).
   - `uv run --project task-sdk pytest 
task-sdk/tests/task_sdk/execution_time/test_tracing.py`: 5 passed.
   - `breeze run pytest airflow-core/tests/unit/observability`: 8 passed.
   - `prek run --from-ref origin/main --stage pre-commit` (including 
`mypy-shared-observability`, ruff and ruff-format) and the manual stage: all 
passed.
   - The fork matrix and reproducers above, run on macOS in isolated 
virtualenvs outside the repository with the exporters `otlp`, `otlp_proto_http` 
and `console`, Python 3.10 to 3.14 and SDK 1.43.0, 1.44.0 and 1.45.2.
   - The regression test on Linux in the Airflow CI image (Python 3.11): it 
fails without the change (`forked child deadlocked`) and passes with it.
   - Equivalent stdlib and SDK-only control scripts on Linux in containers 
(Python 3.12 with SDK 1.43.0, 1.44.0 and 1.45.2; Python 3.13 with 1.43.0 and 
1.45.2): same results as on macOS.
   - A real Dag processor and a bare fork after `import airflow` on the 
released `apache/airflow:3.3.1` image, with and without the one-line change 
applied to the installed copy (see the details above).
   
   Not run with this change: the OTel docker-compose integration tests (left to 
CI), this branch as a real Airflow process under breeze, LocalExecutor and 
Celery workers, and x86_64. SDK 1.45.0 and 1.45.1 were only checked by reading 
the source, and the standalone matrix has no `OTEL_SERVICE_NAME` variant (the 
real-image run does).
   
   On my own deployment (Python 3.13, SDK 1.45.2, OTLP gRPC exporter, `[traces] 
otel_service` set) the forked child hangs without this change, and it does not 
when only `OTEL_SERVICE_NAME` is set.
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes — Claude Code (Sonnet 5.5, with Opus and Fable subagents for 
verification and review)
   
   Generated-by: Claude Code (Sonnet 5.5, with Opus and Fable subagents for 
verification and review) following [the 
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   
   ---
   
   * Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
 for more information. Note: commit author/co-author name and email in commits 
become permanently public when merged.
   * For fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   * When adding dependency, check compliance with the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   * For significant user-facing changes create newsfragment: 
`{pr_number}.significant.rst`, in 
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
 You can add this file in a follow-up commit after the PR is created so you 
know the PR number.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to