Hey all,
I'd like to propose extending the provider discovery mechanism to support
metrics backends, following the same pattern we already use for executors,
auth-managers, and secrets-backends.
MOTIVATION
Today, Airflow's metrics backends (StatsD, Datadog, OTel) are hardcoded in
core. Adding a new backend requires changes to airflow-core itself which
creates an unnecessary coupling. A CloudWatch metrics backend, for example,
belongs in the Amazon provider package alongside the other AWS integrations,
but today there's no way to ship one without modifying core.
We already solved this problem for executors, auth-managers, and
secrets-backends. Metrics backends are one of the few remaining categories
that still require core changes to extend.
PROPOSAL
1.Add metrics backend discovery following the established provider extension
pattern:
New `metrics-backends` key in `provider.yaml` so providers can declare their
backend class(es) exactly how they currently declare executors, auth-managers,
etc:
```
metrics-backends:
airflow.providers.amazon.aws.metrics.cloudwatch_logger.SafeCloudWatchLogger
```
2, Add a new `_discover_metrics_backends()` method in `providers_manager.py`,
following the same pattern as `_discover_executors()`,
`_discover_auth_managers()`, etc.
3. Add a new config option: `[metrics] metrics_backend` to accept an import
path. If set, then load the backend from the provider. If unset, existing
boolean flags (statsd_on, otel_on) continue to work unchanged.
SCOPE
- Fully backward compatible; zero changes to existing config or behavior
- Existing statsd_on / otel_on flags all work exactly as before
- No new user-facing concepts; just "providers can now supply metrics backends
too"
OUT OF SCOPE
- Not moving existing backends out of core. StatsD, Datadog, and OTel stay
where they are for now. This just opens the door for providers to add new ones.
Moving existing backends to providers is a separate, future conversation.
- Not changing the StatsLogger protocol. The protocol (incr, decr, gauge,
timing, timer) is unchanged. Provider backends implement the same interface.
- This doesn't feellike an AIP-level change to me. It follows an existing,
well-established pattern with no new architectural concepts, no breaking
changes, and minimal code. I believe a DISCUSS thread is sufficient.
PERFORMANCE
Discovery happens once at startup during the existing providers_manager scan
(adds one dict key lookup per installed provider to an already-running loop).
The backend is instantiated once via the factory. After that, Stats.incr()
hits the instantiated backend directly with zero per-call overhead. No
different from how executors or auth-managers are discovered and loaded today.
IMMEDIATE USE CASE
I'm building a SafeCloudWatchLogger for the Amazon provider that sends Airflow
metrics to CloudWatch via EMF (Embedded Metric Format). This eliminates the
need for a CloudWatch Agent / StatsD sidecar for AWS users. This discovery
mechanism is the prerequisite; without it, there's no way for a provider to
register a metrics backend.
TIMELINE
I'd like to target this for Airflow 3.4. The core change is small and
non-breaking. I'm happy to have the PR ready for review once there's community
consensus.
QUESTIONS
1. Does anyone see a reason this needs a full AIP rather than a DISCUSS + PR?
2. Any concerns with the approach beyond what I've described?
- ferruzzi
