I'm not sure why that justifies an AIP. If I understand your concern correctly, I could add a CloudWatch metrics backend beside Otel and StatsD in core without an AIP? But that would just add more coupling instead of making steps towards fixing it. Putting an AWS metrics backend in the AWS provider package is logical, follows existing patterns, and is one more step towards decoupling the optional/feature/provider portions of the code from the core product code.
After these proposed changes, the existing backends can be migrated to their own provider packages, I just don't plan on arguing to make three new provider packages as part of this project. - ferruzzi ________________________________ From: Elad Kalif <[email protected]> Sent: Wednesday, July 8, 2026 4:14 PM To: [email protected] <[email protected]> Subject: RE: [EXT] [DISCUSS] Provider-supplied metrics backends via provider.yaml discovery CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque. To my presepctive this needs AIP. I say this specificly because you mentioned the existed metric backends won't be migrated. There might be edge cases with other backends that not apply to the new AWS metric backend. Also, from a user point of view this is going to be extremely confusing. Do we expect users do jungle between docs in core to providers? I am not sure that it's smart to do this just for a new aws metric backend. I'd like to give the example of SMTP which is a very "minor" feature. Users are very confused with the half mogration. Some in core. Some in providers. We delay fixes to core because we want to move to providers. This is uncomfortable situation. I'd ask for an AIP. Most of the content can be what you wrote here but with additional information into: 1. Why are we sure users won't be confused with why some are in providers and some are not. 2. Why are we sure there is no expexted issues with the xurrent existed mechanism. 3. How would we deliever a fix given the mix of core and providers? I am not sure we can easily explain to users why datadog needs core fix while cloudwatch not. I mean, we have a datadog provider that is very confusing. בתאריך יום ה׳, 9 ביולי 2026, 01:59, מאת Ferruzzi, Dennis < [email protected]>: > Hey all, > > I'd like to propose extending the provider discovery mechanism to support > metrics backends, following the same pattern we already use for executors, > auth-managers, and secrets-backends. > > MOTIVATION > > Today, Airflow's metrics backends (StatsD, Datadog, OTel) are hardcoded in > core. Adding a new backend requires changes to airflow-core itself which > creates an unnecessary coupling. A CloudWatch metrics backend, for > example, belongs in the Amazon provider package alongside the other AWS > integrations, but today there's no way to ship one without modifying core. > > We already solved this problem for executors, auth-managers, and > secrets-backends. Metrics backends are one of the few remaining categories > that still require core changes to extend. > > PROPOSAL > > 1.Add metrics backend discovery following the established provider > extension pattern: > New `metrics-backends` key in `provider.yaml` so providers can declare > their backend class(es) exactly how they currently declare executors, > auth-managers, etc: > > ``` > metrics-backends: > > airflow.providers.amazon.aws.metrics.cloudwatch_logger.SafeCloudWatchLogger > ``` > > 2, Add a new `_discover_metrics_backends()` method in > `providers_manager.py`, following the same pattern as > `_discover_executors()`, `_discover_auth_managers()`, etc. > > 3. Add a new config option: `[metrics] metrics_backend` to accept an > import path. If set, then load the backend from the provider. If unset, > existing boolean flags (statsd_on, otel_on) continue to work unchanged. > > SCOPE > > - Fully backward compatible; zero changes to existing config or behavior > - Existing statsd_on / otel_on flags all work exactly as before > - No new user-facing concepts; just "providers can now supply metrics > backends too" > > OUT OF SCOPE > > - Not moving existing backends out of core. StatsD, Datadog, and OTel > stay where they are for now. This just opens the door for providers to add > new ones. Moving existing backends to providers is a separate, future > conversation. > - Not changing the StatsLogger protocol. The protocol (incr, decr, gauge, > timing, timer) is unchanged. Provider backends implement the same interface. > - This doesn't feellike an AIP-level change to me. It follows an > existing, well-established pattern with no new architectural concepts, no > breaking changes, and minimal code. I believe a DISCUSS thread is > sufficient. > > PERFORMANCE > > Discovery happens once at startup during the existing providers_manager > scan (adds one dict key lookup per installed provider to an already-running > loop). The backend is instantiated once via the factory. After that, > Stats.incr() hits the instantiated backend directly with zero per-call > overhead. No different from how executors or auth-managers are discovered > and loaded today. > > IMMEDIATE USE CASE > > I'm building a SafeCloudWatchLogger for the Amazon provider that sends > Airflow metrics to CloudWatch via EMF (Embedded Metric Format). This > eliminates the need for a CloudWatch Agent / StatsD sidecar for AWS users. > This discovery mechanism is the prerequisite; without it, there's no way > for a provider to register a metrics backend. > > TIMELINE > > I'd like to target this for Airflow 3.4. The core change is small and > non-breaking. I'm happy to have the PR ready for review once there's > community consensus. > > QUESTIONS > > 1. Does anyone see a reason this needs a full AIP rather than a DISCUSS + > PR? > 2. Any concerns with the approach beyond what I've described? > > - ferruzzi > >
