GitHub user apoorva-01 added a comment to the discussion: How to define custom
metrics
Yeah, that's basically the supported way. In Airflow 3 the metrics client moved
to the task SDK, and the import you wrote is the right one:
```python
from airflow.sdk.observability import stats
stats.gauge("my_service.queue_depth", 42)
stats.incr("my_service.processed")
stats.decr("my_service.in_flight")
stats.timing("my_service.batch_ms", 1234) # ms or a timedelta
with stats.timer("my_service.batch"): # times the block
...
```
`from airflow.stats import Stats` still works but it's deprecated now (it just
warns and forwards to the same place), so the `airflow.sdk.observability` path
is the one to use going forward.
The one gotcha that trips people up: these calls are no-ops unless you actually
have a metrics backend turned on. Out of the box nothing is emitting, so your
gauge goes nowhere and it looks broken. Turn one on in config:
```ini
[metrics]
statsd_on = True
statsd_host = localhost
statsd_port = 8125
statsd_prefix = airflow
```
or the OpenTelemetry equivalent with `otel_on = True`.
Also watch `[metrics] metrics_allow_list` / `metrics_block_list` if you've set
them. Allow-list is prefix-matched, so if it's configured and your
`my_service.*` prefix isn't in it, the custom metric gets filtered out silently
(and if you set an allow-list, the block-list is ignored entirely). Signatures
if you need the rest: `incr/decr(stat, count=None, rate=None, *, tags=...)`,
`gauge(stat, value, *, tags=...)`, all take a `tags` dict for dimensional
metrics.
Agree a dedicated "Custom Metrics" section in metrics.rst would help, this
comes up a lot.
GitHub link:
https://github.com/apache/airflow/discussions/69096#discussioncomment-17523512
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
