xBis7 commented on code in PR #71564:
URL: https://github.com/apache/airflow/pull/71564#discussion_r3796251773
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -98,6 +98,42 @@ Add the Collector details to your configuration file e.g.
``airflow.cfg``
`SDK environment variable documentation
<https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader>`_
for more information.
+Replicated components
+---------------------
+
+Every Airflow process reports the same ``service.name``, so a deployment
running more than one
Review Comment:
`service.name` and `service.instance.id` aren't the same. So for example, if
I have this in my docker-compose file
```yaml
airflow-scheduler:
<<: *airflow-common
command: scheduler
environment:
<<: *airflow-common-env
OTEL_SERVICE_NAME: airflow-scheduler
OTEL_RESOURCE_ATTRIBUTES: service.instance.id=airflow-scheduler-1
...
airflow-scheduler-2:
<<: *airflow-common
command: scheduler
environment:
<<: *airflow-common-env
OTEL_SERVICE_NAME: airflow-scheduler
OTEL_RESOURCE_ATTRIBUTES: service.instance.id=airflow-scheduler-2
...
```
Then in prometheus I would get
```
airflow_scheduler_critical_section_duration{
exported_instance="airflow-scheduler-1", exported_job="airflow-scheduler",
instance="otel-collector:8889", job="otel-collector",
otel_scope_name="airflow.metrics.otel_logger"
}
```
and
```
airflow_scheduler_critical_section_duration{
exported_instance="airflow-scheduler-2", exported_job="airflow-scheduler",
instance="otel-collector:8889", job="otel-collector",
otel_scope_name="airflow.metrics.otel_logger"
}
```
I think it's worth distinguishing and mentioning both.
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -98,6 +98,42 @@ Add the Collector details to your configuration file e.g.
``airflow.cfg``
`SDK environment variable documentation
<https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader>`_
for more information.
+Replicated components
Review Comment:
This is confusing. There isn't actual replication because we don't have a
leader-follower relationship between the schedulers. They are both independent.
It doesn't necessarily have to do with scheduler HA, because there is value
in distinguishing between different components such as worker and scheduler as
well.
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -98,6 +98,42 @@ Add the Collector details to your configuration file e.g.
``airflow.cfg``
`SDK environment variable documentation
<https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader>`_
for more information.
+Replicated components
+---------------------
+
+Every Airflow process reports the same ``service.name``, so a deployment
running more than one
+replica of a component — schedulers in high availability, several triggerers
or Dag processors —
+sends telemetry the backend cannot attribute to an individual process.
+
+Gauges are where this shows up in the data. Each scheduler samples the
metadata database on its
Review Comment:
> Gauges are where this shows up in the data.
We should be more generic and describe what will happen if they don't use
distinct service ids.
We should say that the OpenTelemetry service name defaults to `Airflow` if
unset and explain when it's good to set it to something else. Also explain why
and when to set the `service.instance.id`.
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -98,6 +98,42 @@ Add the Collector details to your configuration file e.g.
``airflow.cfg``
`SDK environment variable documentation
<https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader>`_
for more information.
+Replicated components
+---------------------
+
+Every Airflow process reports the same ``service.name``, so a deployment
running more than one
+replica of a component — schedulers in high availability, several triggerers
or Dag processors —
+sends telemetry the backend cannot attribute to an individual process.
+
+Gauges are where this shows up in the data. Each scheduler samples the
metadata database on its
+own loop and exports a value for the same series, so ``pool.open_slots`` and
its siblings keep
+whichever export arrived last and appear to flap between replicas' samples.
+
+Give each replica a unique ``service.instance.id`` so its samples form their
own series:
Review Comment:
You use the word `replica` and then refer to a scheduler, triggerer, dag
processor, which is inaccurate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
