banana-creator-dev opened a new issue, #73147:
URL: https://github.com/apache/airflow/issues/73147
### Apache Airflow version
3.3.1
### If "Other Airflow 2/3 version" selected, which one?
_No response_
### What happened?
With OTel metrics enabled, the scheduler exports **two copies of its
metrics**: one from the main process, which behaves correctly, and one from a
forked child that never advances. The child's copy is republished on every
export interval, indefinitely, holding the counter values it inherited at the
moment of the fork.
The scheduler container runs the main process plus four forked children:
```
401440 ppid=401414 threads=18 elapsed=2488 airflow scheduler <- main
402459 ppid=401440 threads=17 elapsed=2473 airflow scheduler
402492 ppid=401440 threads=18 elapsed=2473 airflow scheduler
402524 ppid=401440 threads=15 elapsed=2473 airflow scheduler
407463 ppid=401440 threads=17 elapsed=2240 airflow scheduler <- exports
the frozen copy
```
The three children forked at startup are quiet. The one forked ~4 minutes
later — after the parent had already been counting — inherited a non-zero
accumulation and exports it forever:
```
airflow_scheduler_heartbeat{service_instance_id="airflow-scheduler-6545f66d48-tqgjv"}
277 (climbing)
airflow_scheduler_heartbeat{service_instance_id="c674b1e6-8df6-4fb1-ba05-161e0ce45725"}
44 (frozen, last sample 8s old)
```
Seven of the ten live `airflow_scheduler_*` series are duplicated this way,
and each scheduler restart mints another frozen set.
This looks like the other side of the fix for #64690. `get_otel_logger()`
now resets the SDK's `Once()` guard before `set_meter_provider()`:
```python
_metrics_internal._METER_PROVIDER_SET_ONCE._done = False
_metrics_internal._METER_PROVIDER = None
```
That gives a forked child a working provider where before it had a dead one.
Children that never emit anything again still get a live
`PeriodicExportingMetricReader` and keep flushing the instrument state they
inherited.
Two consequences:
1. **Duplicated series.** Anything that sums or averages across instances
double-counts while both copies are live.
2. **Per-series alerting on a heartbeat is impossible.**
`rate(airflow_scheduler_heartbeat[5m]) == 0` is the natural liveness check, and
it fires forever on the frozen copy while the scheduler is perfectly healthy.
The only workaround is to aggregate first — `sum(rate(...)) == 0` — which
silently gives up the ability to alert on one unhealthy instance among several.
### Why the two copies have different identities
Worth noting for anyone reproducing on an older SDK. `opentelemetry-sdk`
1.44 added `ServiceInstanceIdResourceDetector`, whose docstring says:
> because this detector is process dependent, providers refresh it
automatically after a fork and merge the newly detected value on top of the
existing resource. This means that if a user explicitly sets
`service.instance.id` (e.g. via OTEL_RESOURCE_ATTRIBUTES or
Resource.create(attributes=...)), that value will be overwritten with a newly
generated UUID the next time the process forks.
So on 1.44 the child's resource carries a fresh UUID and the frozen copy
lands in its own series, which is how this became visible at all. The child's
environment is correct — `Resource.create()` run by hand inside the same
container returns the pod name — the UUID comes from the post-fork refresh.
On earlier SDKs the child would carry the *same* `service.instance.id` as
the parent, so the two copies collide on one series and the frozen values
interleave with the real ones instead of sitting beside them. That is worse and
harder to notice.
### What you think should happen instead?
A forked child that does not emit metrics should not export. Stopping the
inherited metric reader in the child — for example from an
`os.register_at_fork(after_in_child=...)` hook, or by shutting the provider
down before re-initialising — would leave one copy of each series and make a
per-instance heartbeat check work again.
### How to reproduce
1. Airflow 3.3.1, `opentelemetry-sdk` 1.44.0, KubernetesExecutor.
2. Enable OTel metrics: `AIRFLOW__METRICS__OTEL_ON=True`, plus
`OTEL_METRICS_EXPORTER=otlp`, `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`,
`OTEL_EXPORTER_OTLP_METRICS_PROTOCOL=http/protobuf`,
`OTEL_METRIC_EXPORT_INTERVAL=30000`.
3. Let the scheduler run for ten minutes or so, long enough for it to fork a
child after the heartbeat has been counting.
4. Query `airflow_scheduler_heartbeat`. Two series appear: one climbing, one
frozen and still being re-sent.
`ps -eo pid,ppid,etimes,args` inside the scheduler container shows the
child, and its elapsed time matches the moment the frozen series first appears.
### Operating System
Ubuntu 24.04.4 LTS
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
KubernetesExecutor, one scheduler replica, k3s 1.36. Metrics go to an
OpenTelemetry Collector on each node over OTLP HTTP and on to Prometheus by
remote write.
### Anything else?
Happens on every scheduler start. The number of frozen series grows by one
set per restart.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
