jayachandrakasarla opened a new pull request, #69449:
URL: https://github.com/apache/airflow/pull/69449
Closes #66792
The API server emits process-level metrics but nothing per-route, so
operators can't see request rate, latency, or error rate for individual
endpoints.
This adds an ASGI middleware that records a counter and a latency histogram
per completed request, tagged with the templated route, HTTP method, and
status. The templated route (/api/v2/dags/{dag_id}/dagRuns) is used as the tag
so per-dag_id requests don't explode metric cardinality.
Metrics go through the existing Stats backend, so they work with StatsD,
Datadog, and OpenTelemetry. The middleware is only wired in when a metrics
backend is enabled, so deployments without one doesn't export these.
Metrics from `/metrics` endpoint of otel collector running on port 28889
```
# HELP airflow_api_server_request_count_total
# TYPE airflow_api_server_request_count_total counter
airflow_api_server_request_count_total{instance="a682a7ed-927f-438d-bf78-e0693f498dab",job="test",method="GET",otel_scope_name="airflow._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",route="/api/v2/assets/events",status="200"}
2
airflow_api_server_request_count_total{instance="a682a7ed-927f-438d-bf78-e0693f498dab",job="test",method="GET",otel_scope_name="airflow._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",route="/api/v2/auth/login",status="307"}
1
airflow_api_server_request_count_total{instance="a682a7ed-927f-438d-bf78-e0693f498dab",job="test",method="GET",otel_scope_name="airflow._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",route="/api/v2/dags/{dag_id}/dagRuns",status="200"}
2
airflow_api_server_request_count_total{instance="a682a7ed-927f-438d-bf78-e0693f498dab",job="test",method="GET",otel_scope_name="airflow._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",route="/api/v2/dags/{dag_id}/dagRuns",status="403"}
1
airflow_api_server_request_count_total{instance="a682a7ed-927f-438d-bf78-e0693f498dab",job="test",method="GET",otel_scope_name="airflow._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",route="/api/v2/dags/{dag_id}/dagRuns/{dag_run_id}/hitlDetails",status="200"}
2
airflow_api_server_request_count_total{instance="a682a7ed-927f-438d-bf78-e0693f498dab",job="test",method="GET",otel_scope_name="airflow._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",route="/api/v2/importErrors",status="200"}
2
airflow_api_server_request_count_total{instance="a682a7ed-927f-438d-bf78-e0693f498dab",job="test",method="GET",otel_scope_name="airflow._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",route="/api/v2/monitor/health",status="200"}
2
airflow_api_server_request_count_total{instance="a682a7ed-927f-438d-bf78-e0693f498dab",job="test",method="GET",otel_scope_name="airflow._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",route="/api/v2/plugins",status="200"}
2
airflow_api_server_request_count_total{instance="a682a7ed-927f-438d-bf78-e0693f498dab",job="test",method="GET",otel_scope_name="airflow._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",route="/api/v2/plugins/importErrors",status="200"}
2
airflow_api_server_request_count_total{instance="a682a7ed-927f-438d-bf78-e0693f498dab",job="test",method="GET",otel_scope_name="airflow._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",route="/api/v2/pools",status="200"}
2
airflow_api_server_request_count_total{instance="a682a7ed-927f-438d-bf78-e0693f498dab",job="test",method="GET",otel_scope_name="airflow._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",route="/api/v2/version",status="200"}
4
```
Screenshots of PromQL queries:
<img width="1503" height="758" alt="image"
src="https://github.com/user-attachments/assets/ac111578-b233-40ef-bc66-c4dbed1e946c";
/>
<img width="1505" height="697" alt="image"
src="https://github.com/user-attachments/assets/feee5721-f9c8-48e0-81d5-6aaa52055064";
/>
<img width="1504" height="763" alt="image"
src="https://github.com/user-attachments/assets/1a9bed76-dbd8-49c2-9f20-038bb0e0b5fc";
/>
And here's the screenshot of custom metrics dashboard on Grafana:
<img width="1498" height="753" alt="image"
src="https://github.com/user-attachments/assets/ff964c33-b7fa-4346-9f59-1872ae1f559d";
/>
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (Claude Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
