1fanwang opened a new pull request, #71564:
URL: https://github.com/apache/airflow/pull/71564
Running schedulers in high availability is the recommended production setup,
but every Airflow process reports the same `service.name`. Nothing
distinguishes one scheduler's telemetry from another's, so all replicas publish
to the same series and the backend keeps whichever export arrived last.
It shows up most clearly on gauges. Each scheduler samples the metadata
database on its own loop, so `pool.open_slots` and its siblings settle on an
arbitrary replica's sample and look like they are flapping. Operators
reasonably read that as a broken metric.
OpenTelemetry already solves this with `service.instance.id`, and Airflow
already honours it — the meter provider is built with `Resource.create()`,
which merges `OTEL_RESOURCE_ATTRIBUTES`. No code change is needed. But the
metrics docs never mention high availability at all, so there is nothing to
lead an operator to it.
This adds a short section to the OpenTelemetry docs covering why replicas
collide, the environment variable that separates them, aggregating across
replicas at query time, the `instance` label mapping for Prometheus-compatible
backends, and the fact that StatsD has no equivalent.
# Testing Done
The documented mechanism was run rather than assumed.
**1. Airflow really does merge the variable.** This is the claim the whole
section rests on:
```console
$ OTEL_RESOURCE_ATTRIBUTES="service.instance.id=scheduler-a" python -c "
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
r = Resource.create(attributes={SERVICE_NAME: 'airflow'})
for k, v in sorted(r.attributes.items()): print(f'{k} = {v}')"
service.instance.id = scheduler-a
service.name = airflow
telemetry.sdk.language = python
telemetry.sdk.name = opentelemetry
telemetry.sdk.version = 1.44.0
```
`Resource.create()` there is the same call the meter provider makes in
`otel_logger.py`.
**2. It actually separates the replicas.** Two `SafeOtelLogger` instances
built the way `get_otel_logger()` builds one, differing only in
`service.instance.id`, each reporting a different `open_slots` for the same
pool:
<details><summary>Script and output</summary>
```python
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import InMemoryMetricReader
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from airflow_shared.observability.metrics.otel_logger import SafeOtelLogger
def build_scheduler(instance_id):
resource = Resource.create(
attributes={SERVICE_NAME: "airflow", "service.instance.id":
instance_id}
)
reader = InMemoryMetricReader()
return
SafeOtelLogger(otel_provider=MeterProvider(metric_readers=[reader],
resource=resource)), reader
tags = {"pool_name": "default_pool"}
schedulers = [build_scheduler("scheduler-a"), build_scheduler("scheduler-b")]
for (scheduler, _), open_slots in zip(schedulers, (128, 126)):
scheduler.gauge("pool.open_slots", open_slots, tags=tags)
for _, reader in schedulers:
rm = reader.get_metrics_data().resource_metrics[0]
instance = rm.resource.attributes["service.instance.id"]
for metric in rm.scope_metrics[0].metrics:
point = next(iter(metric.data.data_points))
print(f"{metric.name}{{service.instance.id={instance}, {tags}}} =
{point.value}")
```
```
airflow.pool.open_slots{service.instance.id=scheduler-a, {'pool_name':
'default_pool'}} = 128
airflow.pool.open_slots{service.instance.id=scheduler-b, {'pool_name':
'default_pool'}} = 126
```
Both samples survive as separate series, so `min by (pool_name)` returns 126
rather than an arbitrary one of the two. Without the attribute the second
export overwrites the first.
</details>
**3. The page itself.** Both linked specs return `200`. `prek run --files
airflow-core/docs/.../metrics.rst` passes, including the RST and codespell
hooks. Parsing the file with docutils reports no structural problems — the only
messages are the `envvar`, `doc` and `ref` roles at lines 45, 54 and 294, which
are Sphinx roles that predate this change. I could not run the full `breeze
build-docs` render locally, since it wanted an interactive CI image rebuild; CI
covers it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
