dstandish opened a new pull request, #71804: URL: https://github.com/apache/airflow/pull/71804
A fork hands the child the parent's `MeterProvider`, and the OTel SDK registers `register_at_fork(after_in_child=...)` for every `PeriodicExportingMetricReader`, so the child restarts the exporter thread behind that inherited pipeline. Nothing in the child records to it, so it republishes the totals the parent held at the instant of the fork — once per export interval, for as long as the child lives. A consumer then sees two writers on one cumulative series: one climbing, one frozen. Airflow forks constantly, and long-lived children make that permanent rather than momentary: LocalExecutor pool workers, the OpenLineage dag-state-change `ProcessPoolExecutor` and the scheduler's log and health-check servers all fork from a scheduler whose pipeline is already live, then outlive many export cycles. Setting the reader's shutdown event alone is not enough: the ticker publishes one last collection on its way out. Dropping the collect callback keeps that final pass from carrying the parent's totals with it. A child that emits metrics of its own still builds its own pipeline in `get_otel_logger()`, so this costs it nothing — the test covers both. The pipeline built from `OTEL_CONFIG_FILE` is deliberately left alone: declarative configuration is the sole source of SDK construction there, so a child has no way to rebuild what it would lose. Reproduction, against the console exporter with a 200 ms interval — a child that records nothing and lives 1 second: ``` export from parent pid=27208: scheduler.counter=7 export from CHILD pid=27209: scheduler.counter=7 export from parent pid=27208: scheduler.counter=7 export from CHILD pid=27209: scheduler.counter=7 export from parent pid=27208: scheduler.counter=8 <- only the parent ever moves ``` After the change the child exports nothing. The new test asserts that directly, by redirecting the child's fd 1 to a file so its exports can be told apart from the parent's. related: #71800 — that PR fixes the neighbouring case where a *replaced* provider keeps exporting within one process. This one is about inheritance across `fork()`; the two are independent and touch different code paths in the same module. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5) Generated-by: Claude Code (Opus 5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
