0BVer opened a new pull request, #77283:
URL: https://github.com/apache/airflow/pull/77283
Forked child processes hang forever when traces are enabled and the
deprecated `[traces] otel_service` option is set (and neither
`OTEL_SERVICE_NAME` nor `OTEL_RESOURCE_ATTRIBUTES` is). In Airflow this shows
up on Linux, where the supervisor starts Dag processor and task processes with
a bare `os.fork()` (on macOS, 3.4 and later use `posix_spawn`, which runs no
at-fork hooks), as Dag processor parsing processes killed by
`dag_file_processor_timeout` and as hung task processes. I see the same hang on
3.3.2 and 3.4.0 with `[traces] otel_service` set, in Dag processor parsing
processes and LocalExecutor task processes, and the code that builds the static
`Resource` is identical in both versions.
#73583 reports a hang on 3.3.2 and its py-spy stack matches the path
described below, but the issue does not say whether `[traces] otel_service` is
set on the reporter's setup, so this is `related:` and not `closes:`.
related: #73583
## Cause
Since opentelemetry-sdk 1.44, `TracerProvider` (and `MeterProvider`,
`LoggerProvider`) registers an `os.register_at_fork(after_in_child=...)`
handler. In the child it refreshes process-dependent resource attributes
through `get_aggregated_resources()`, which calls
`ThreadPoolExecutor.submit()`. `submit()` takes
`concurrent.futures.thread._global_shutdown_lock`.
`concurrent.futures.thread` is imported lazily. When it is imported it
registers its own fork hooks: `before` acquires `_global_shutdown_lock` and
`after_in_child` resets it. `after_in_child` hooks run in registration order,
so if the module is first imported after the provider was created, the
provider's handler runs in the child while the lock is still in the held state
inherited from the parent, and `submit()` blocks forever.
With `[traces] otel_service` set, `_get_backcompat_config` passed a static
`Resource({...})` to `TracerProvider`. The provider only calls
`Resource.create()` itself when it gets no resource
(`opentelemetry/sdk/trace/__init__.py:1271`, SDK 1.45.2), and
`Resource.create()` is what imports `concurrent.futures.thread`. So with a
static resource nothing has imported the module when the provider registers its
handler (`:1296`). Without `otel_service`, `Resource.create()` runs first and
the order is safe.
When the late import happens:
- In a real Airflow process, `import airflow` runs `configure_otel()`
(through `settings.initialize()`) and later imports `concurrent.futures.thread`
on its own: on the 3.3.1 release image the first import comes from
`airflow.sdk.io` through `fsspec`. So with `[traces] otel_service` set, a plain
fork hangs with every exporter, with no executor use by application code.
- With `configure_otel()` alone it depends on the exporter. With the default
`otlp` (gRPC) exporter, the exporter module imports `opentelemetry.sdk._logs`,
whose `ReadWriteLogRecord` default value `Resource.create({})` runs at import
time, inside `configure_otel()` after the provider was created, so every fork
hangs. `import grpc` itself does not import `concurrent.futures.thread`. With
the `console` or `otlp_proto_http` exporter, forks hang once anything in the
parent process touches `concurrent.futures.ThreadPoolExecutor` before forking
(attribute access is enough); if nothing does, the child imports the module
fresh inside the provider's handler and does not hang. The review comment on
the closed #73626 reported, through `configure_otel()` alone, that
`otel_service` alone is fine and that one executor use before the fork hangs.
It does not name the exporter; with the default `otlp` exporter I see a hang
without any executor use.
The hang is in `ThreadPoolExecutor.submit()` on `_global_shutdown_lock`.
`_service_instance_id_lock`, which #73626 focused on, is not held in the child
(observed `locked() == False`), and the stack never reaches the detector that
takes it.
SDK versions before 1.44 have no `TracerProvider` fork handler and are not
affected. The ordering sensitivity inside the SDK is tracked in
open-telemetry/opentelemetry-python#5789 with a proposed fix in
open-telemetry/opentelemetry-python#5790 (both still open).
<details>
<summary>Stdlib-only reproducer of the mechanism</summary>
No OpenTelemetry involved: a fork hook that submits to a
`ThreadPoolExecutor` hangs when it was registered before
`concurrent.futures.thread` was imported, and does not when the import came
first.
```python
import concurrent.futures, os, signal, sys, time
def provider_like_after_in_child():
# Same shape as the SDK's TracerProvider fork handler.
with concurrent.futures.ThreadPoolExecutor(1) as executor:
executor.submit(lambda: None).result()
if sys.argv[1] == "import-first":
concurrent.futures.ThreadPoolExecutor # imports
concurrent.futures.thread, its fork hooks go first
os.register_at_fork(after_in_child=provider_like_after_in_child)
concurrent.futures.ThreadPoolExecutor # otherwise imported only here, after
the hook above
pid = os.fork()
if pid == 0:
os._exit(0)
for _ in range(50):
if os.waitpid(pid, os.WNOHANG)[0]:
print(sys.argv[1], "-> child exited: OK")
break
time.sleep(0.1)
else:
os.kill(pid, signal.SIGKILL)
os.waitpid(pid, 0)
print(sys.argv[1], "-> child HUNG for 5s, killed")
```
`python repro.py import-late` hangs and `python repro.py import-first`
exits, on Python 3.10 to 3.14.
</details>
<details>
<summary>Reproducer with opentelemetry-sdk only (no Airflow)</summary>
```python
import os, signal, sys, time
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
attrs = {"service.name": "my-service"}
resource = Resource(attrs) if sys.argv[1] == "static" else
Resource.create(attrs)
provider = TracerProvider(resource=resource) # SDK >= 1.44 registers an
after_in_child hook here
import opentelemetry.sdk._logs # what the OTLP gRPC exporter imports: first
import of concurrent.futures.thread
pid = os.fork()
if pid == 0:
os._exit(0)
for _ in range(50):
if os.waitpid(pid, os.WNOHANG)[0]:
print(sys.argv[1], "-> child exited: OK")
break
time.sleep(0.1)
else:
os.kill(pid, signal.SIGKILL)
os.waitpid(pid, 0)
print(sys.argv[1], "-> child HUNG for 5s, killed")
```
`python repro.py static` hangs and `python repro.py create` exits on
opentelemetry-sdk 1.44.0 and 1.45.2 (Python 3.10 to 3.14). Both exit on 1.43.0.
Keep the `provider` variable: the SDK's fork hook holds only a weak reference
to it.
</details>
<details>
<summary>At-fork hook order, default gRPC exporter (Python 3.12.8, SDK
1.45.2)</summary>
Hooks registered through `os.register_at_fork`, in registration order, with
`[traces] otel_service` set:
| # | Before (static `Resource`) | Slots | After (`Resource.create`) | Slots
|
|---|---|---|---|---|
| 1 | `threading` | after_in_child | `threading` | after_in_child |
| 2 | `random` | after_in_child | `random` | after_in_child |
| 3 | `logging` | before, after_in_parent, after_in_child | `logging` |
before, after_in_parent, after_in_child |
| 4 | **`TracerProvider.__init__`** | after_in_child |
**`concurrent.futures.thread`** (`:41`) | before, after_in_parent,
after_in_child |
| 5 | `asyncio/events.py` | after_in_child | `concurrent.futures.thread`
(`:44`) | after_in_child |
| 6 | **`concurrent.futures.thread`** (`:41`) | before, after_in_parent,
after_in_child | **`TracerProvider.__init__`** | after_in_child |
| 7 | `concurrent.futures.thread` (`:44`) | after_in_child |
`asyncio/events.py` | after_in_child |
| 8 | `BatchProcessor` | after_in_child | `BatchProcessor` | after_in_child |
(The `:44` hook exists from Python 3.12.)
- Before: the parent's `before` hooks run in reverse (thread hook acquires
`_global_shutdown_lock`). In the child, hooks 1 to 4 run in order and the child
blocks inside hook 4, so hook 6 (`_global_shutdown_lock._at_fork_reinit`) never
runs. `_global_shutdown_lock.locked()` was `True` before every hook up to the
provider's.
- After: the thread module's `_at_fork_reinit` runs as hook 4, the lock is
free when the provider's handler runs as hook 6, and the child returns.
Stack of the hung child (`faulthandler` on `SIGUSR1`):
```
File ".../concurrent/futures/thread.py", line 166 in submit
File ".../opentelemetry/sdk/resources/__init__.py", line 752 in
get_aggregated_resources
File ".../opentelemetry/sdk/resources/__init__.py", line 726 in
_get_process_dependent_resource
File ".../opentelemetry/sdk/trace/__init__.py", line 1300 in _handle_fork
File ".../opentelemetry/sdk/trace/__init__.py", line 1294 in
_after_in_child
```
`thread.py:166` is `with self._shutdown_lock, _global_shutdown_lock:`.
`self._shutdown_lock` belongs to an executor created inside the child, so only
`_global_shutdown_lock` can be held. The reporter's py-spy stack in #73583 has
the same frames.
</details>
<details>
<summary>Fork test matrix (126 runs)</summary>
`buggy` is `main` with `otel_service` set, `fixed` is this change with
`otel_service` set, `noservice` is without `otel_service` (the code path is the
same on `main` and with this change; these runs used this change's module).
`+exec` is one `ThreadPoolExecutor` use in the parent before `fork()`. Every
cell is one fresh interpreter with a 4 second child timeout, macOS arm64,
`opentelemetry-sdk` and the `otlp` exporters at the same version.
| Python / SDK | Exporter | buggy | buggy +exec | fixed | fixed +exec |
noservice | noservice +exec |
|---|---|---|---|---|---|---|---|
| 3.10 to 3.14 / 1.45.2, 3.12 / 1.44.0 | otlp (gRPC) | HANG | HANG | OK | OK
| OK | OK |
| 3.10 to 3.14 / 1.45.2, 3.12 / 1.44.0 | otlp_proto_http | OK | HANG | OK |
OK | OK | OK |
| 3.10 to 3.14 / 1.45.2, 3.12 / 1.44.0 | console | OK | HANG | OK | OK | OK
| OK |
| 3.12 / 1.43.0 | all three | OK | OK | OK | OK | OK | OK |
All 24 hangs are `buggy` runs on SDK 1.44 or later, and every one ends in
`submit()` in the stack dump. There were no hangs in the 42 `fixed` runs, the
42 `noservice` runs and the 6 `buggy` runs on SDK 1.43.0. Airflow requires
Python 3.11 or newer, so 3.10 only shows the mechanism of the standalone module.
</details>
<details>
<summary>Real Airflow process (released 3.3.1 image)</summary>
`apache/airflow:3.3.1` image (Linux aarch64, Python 3.13.14,
opentelemetry-sdk 1.44.0), `AIRFLOW__TRACES__OTEL_ON=True`, an unreachable
local OTLP endpoint. The script does `import airflow` and then a bare
`os.fork()`, with a 20 second child timeout.
| Setup | Fork result |
|---|---|
| `[traces] otel_service` set, exporter `otlp`, `console` or
`otlp_proto_http`, no executor use | HANG (all three) |
| `[traces] otel_service` set, `console`, plus a `ThreadPoolExecutor`
attribute access | HANG |
| only `OTEL_SERVICE_NAME`; `otel_service` and `OTEL_SERVICE_NAME` both set;
no service; traces off | OK |
| `[traces] otel_service` set, with `Resource({...})` changed to
`Resource.create({...})` in the installed copy (`otlp` three times, `console`
and `otlp_proto_http` with the executor access) | OK |
`airflow dag-processor --num-runs 2` on the same image (sqlite,
`dag_file_processor_timeout=15`, one trivial Dag file):
- Stock with `[traces] otel_service` set: both parsing processes are killed
by the processor timeout and the Dag is not registered.
- Stock with only `OTEL_SERVICE_NAME`, or with the one-line change applied
to the installed copy: no timeout, the Dag is registered.
This is the 3.3.1 release with the one-line change applied to its installed
copy, not this branch built under breeze.
</details>
## Fix
`_get_backcompat_config` now builds the resource with
`Resource.create({"service.name": ...})` instead of `Resource({"service.name":
...})`. `Resource.create()` runs in Airflow code before the provider is
constructed, so `concurrent.futures.thread` registers its fork hooks first,
which is what the path without `otel_service` already does. This does not
depend on the order of calls inside `TracerProvider.__init__`. The existing
guard is unchanged: the option is only used when neither `OTEL_SERVICE_NAME`
nor `OTEL_RESOURCE_ATTRIBUTES` is set. A short comment records why plain
`Resource(...)` must not be used here. A bare `import
concurrent.futures.thread` before constructing the provider would fix the
ordering too, without changing the resource attributes; `Resource.create()` was
chosen so the resource is built exactly as on the path without `otel_service`.
Limits of this fix:
- It relies on `Resource.create()` importing `concurrent.futures.thread`.
That holds in every release that has the fork handler: I ran 1.44.0 and 1.45.2
and read the source of 1.45.0 and 1.45.1. The fix proposed in
open-telemetry/opentelemetry-python#5790 runs the process-dependent detectors
synchronously in fork handlers, so `Resource.create()` itself would still use
the thread pool.
- Other code in the same process (local settings, plugins, instrumentation)
that builds a provider with a static `Resource` before that module is imported
can still hit the same ordering problem. Only an SDK fix closes that.
- Airflow's metrics logger is not affected as written: `otel_logger.py`
calls `Resource.create(...)` before constructing `MeterProvider`.
## Behaviour notes
- `service.name` is unchanged (`my-service` before and after for
`otel_service = my-service`).
- The resource is now built the same way as in a setup without
`otel_service`, so besides `service.name` it carries the SDK default attributes
(`telemetry.sdk.*`, `service.instance.id`) and honors
`OTEL_EXPERIMENTAL_RESOURCE_DETECTORS`. Before, it had `service.name` only.
- No environment variable is set, so child processes and the metrics logger
(which reads `OTEL_SERVICE_NAME` before `[metrics] otel_service`) are not
affected.
## Regression test
`TestConfigureOtelServiceName.test_fork_after_executor_use_does_not_deadlock`
in `shared/observability/tests/observability/test_traces.py` runs a fresh
interpreter (pytest has already imported `concurrent.futures.thread`, so the
import order cannot be reproduced in-process). It asserts that
`concurrent.futures.thread` is not imported yet, calls `configure_otel` with
`otel_service` set (console exporter, no network), uses a `ThreadPoolExecutor`
once, then forks. The parent waits for the child with a 10 second `SIGALRM`
timeout and kills a child that does not return. The test asserts a zero exit
code and `service.name == "my-service"`.
The test fails on `main` without this change (`forked child deadlocked`
after 10 seconds) and passes with it. It can only fail on opentelemetry-sdk
1.44 or later, because older versions have no provider fork handler.
## Verification
Python 3.13.1, opentelemetry-sdk 1.45.2, macOS.
- `uv run --exact --group dev pytest -n auto` in `shared/observability`: 197
passed. The same command in every `shared/*` distribution (the loop that
`breeze verify` lists): all pass.
- `breeze testing core-tests` as listed by `breeze verify` (Python 3.11):
the tests that need no database pass (5755 passed), and the database tests on
sqlite pass for all six test types (API, Always, CLI, Core, Other,
Serialization).
- `uv run --project task-sdk pytest
task-sdk/tests/task_sdk/execution_time/test_tracing.py`: 5 passed.
- `breeze run pytest airflow-core/tests/unit/observability`: 8 passed.
- `prek run --from-ref origin/main --stage pre-commit` (including
`mypy-shared-observability`, ruff and ruff-format) and the manual stage: all
passed.
- The fork matrix and reproducers above, run on macOS in isolated
virtualenvs outside the repository with the exporters `otlp`, `otlp_proto_http`
and `console`, Python 3.10 to 3.14 and SDK 1.43.0, 1.44.0 and 1.45.2.
- The regression test on Linux in the Airflow CI image (Python 3.11): it
fails without the change (`forked child deadlocked`) and passes with it.
- Equivalent stdlib and SDK-only control scripts on Linux in containers
(Python 3.12 with SDK 1.43.0, 1.44.0 and 1.45.2; Python 3.13 with 1.43.0 and
1.45.2): same results as on macOS.
- A real Dag processor and a bare fork after `import airflow` on the
released `apache/airflow:3.3.1` image, with and without the one-line change
applied to the installed copy (see the details above).
Not run with this change: the OTel docker-compose integration tests (left to
CI), this branch as a real Airflow process under breeze, LocalExecutor and
Celery workers, and x86_64. SDK 1.45.0 and 1.45.1 were only checked by reading
the source, and the standalone matrix has no `OTEL_SERVICE_NAME` variant (the
real-image run does).
On my own deployment (Python 3.13, SDK 1.45.2, OTLP gRPC exporter, `[traces]
otel_service` set) the forked child hangs without this change, and it does not
when only `OTEL_SERVICE_NAME` is set.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Sonnet 5.5, with Opus and Fable subagents for
verification and review)
Generated-by: Claude Code (Sonnet 5.5, with Opus and Fable subagents for
verification and review) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]