xBis7 commented on code in PR #71564:
URL: https://github.com/apache/airflow/pull/71564#discussion_r3804766046
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -53,6 +53,12 @@ custom StatsD client. This module must be available on your
:envvar:`PYTHONPATH`
See :doc:`../modules_management` for details on how Python and Airflow manage
modules.
+.. note::
+
+ StatsD has no resource concept, so metrics cannot be attributed to the
process that
+ produced them. Use OpenTelemetry if you need that — see
Review Comment:
Should we also say that in scheduler HA it's recommended to use OTel
otherwise, you end up with the values from the scheduler that exports last?
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -98,6 +104,51 @@ Add the Collector details to your configuration file e.g.
``airflow.cfg``
`SDK environment variable documentation
<https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader>`_
for more information.
+.. _identifying-components-and-their-instances:
+
+Identifying components and their instances
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+OpenTelemetry labels each metric with the resource that produced it. Two
resource attributes
+decide how much of a deployment can be told apart:
+
+``service.name``
+ Which component reported the metric. It defaults to ``airflow`` for every
Airflow process, so
+ a scheduler, a triggerer and a worker arrive under one name. Set it per
component to attribute
+ a metric to the kind of process that produced it.
+
+``service.instance.id``
+ Which process of that component reported the metric. It is unset by
default, so processes
+ running the same component are indistinguishable. Set it per process to
attribute a metric to
Review Comment:
```suggestion
running the same component (e.g. 2+ schedulers) are indistinguishable.
Set it per process to attribute a metric to
```
##########
airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -98,6 +104,51 @@ Add the Collector details to your configuration file e.g.
``airflow.cfg``
`SDK environment variable documentation
<https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader>`_
for more information.
+.. _identifying-components-and-their-instances:
+
+Identifying components and their instances
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+OpenTelemetry labels each metric with the resource that produced it. Two
resource attributes
+decide how much of a deployment can be told apart:
+
+``service.name``
+ Which component reported the metric. It defaults to ``airflow`` for every
Airflow process, so
+ a scheduler, a triggerer and a worker arrive under one name. Set it per
component to attribute
+ a metric to the kind of process that produced it.
+
+``service.instance.id``
+ Which process of that component reported the metric. It is unset by
default, so processes
+ running the same component are indistinguishable. Set it per process to
attribute a metric to
+ one of them.
+
+Airflow reads ``service.name`` from ``OTEL_SERVICE_NAME``, and every other
resource attribute from
+``OTEL_RESOURCE_ATTRIBUTES``:
+
+.. code-block:: bash
+
+ # on one of the schedulers
+ export OTEL_SERVICE_NAME="airflow-scheduler"
+ export OTEL_RESOURCE_ATTRIBUTES="service.instance.id=$(hostname)"
+
+Processes that share a resource also share a series, and the backend keeps
whichever export
+arrived last. Where several processes run the same component this loses data
rather than
Review Comment:
> Where several processes run the same component this loses data rather
This is confusing. Maybe you meant `when`.
```suggestion
arrived last. When several processes run the same component, data are lost
instead of aggregated.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
