The behavior should stay the same and we can document how people can create custom metrics, just like with traces
https://github.com/apache/airflow/blob/main/airflow-core/docs/administration-and-deployment/logging-monitoring/traces.rst#adding-custom-spans-in-tasks With moving `Stats` under `_shared` and then the addition of the `DualStatsManager` for handling the legacy metrics, things got too complicated. I think that the best approach is to remove the `DualStatsManager` entirely and move the logic under `Stats` but with the metaclass, it will be tricky and eventually hard to maintain. Things need to get simplified to accomodate for all the new changes. But the behavior should stay the same. On Wed, Mar 18, 2026 at 7:27 PM Jarek Potiuk <[email protected]> wrote: > Also - from me - no `+1` is needed, but I strongly support this direction > with strong arguments. > > I personally introduced the "Public Interface of Airflow" (I had to > convince many maintainers that it was necessary). In airflow 2.6.0: > > https://airflow.apache.org/docs/apache-airflow/2.6.0/public-airflow-interface.html > . > The discussion about introducing it (November 2022) is here: > https://lists.apache.org/thread/1by8ko8jrrp1xwxt5781bwn2tokxjodl (very > long, but you can take a look what were the arguments and reasoning there). > > I started the discussion by explaining the basis of Hyrum's Law and SemVer > there. And explained that in a sufficiently complex system, everything is > breaking someone's workflow (Hyrum's law). Also, when you look at SemVer - > this is an intention (even if intentions are not well documented), not a > fact-based approach. > > I think what really matters are our intentions—whether something is a > public interface—not whether technically we are removing class or method > that has no "_" in front. While "_" is a clear sign that something is not > intended to be public, lack of such underscore does not automatically mean > something is public. Additionally - as you can read in that discussion - > when evaluating whether a change is breaking or not we should assess the > risk whether a change will break someone's workflow or not. > > Adding the page was a very deliberate move on my part. This was exactly > what I did to find it useful today. All 3.5 years of having and updating > the page, were **precisely** for this LAZY CONSENSUS thread (and similar > that follow). > > And I plan to conduct a detailed review and update here at the devlist > before the 3.2 release. One reason for introducing that page was to allow > us, several years later, to see whether we communicated our intention to > make something public or not. This basically follows this FAQ from SemVer > https://semver.org/#documenting-the-entire-public-api-is-too-much-work > > > Documenting the entire public API is too much work! > > > > It is your responsibility as a professional developer to properly > document software that is intended for use by others. Managing software > complexity is a hugely important part of keeping a project efficient, and > that’s hard to do if nobody knows how to use your software, > or what > methods are safe to call. In the long run, Semantic Versioning, and the > insistence on a well defined public API can keep everyone and everything > running smoothly. > > > > It turns out it's not "too much work." We did it. We communicated during > the last 3.5 years. Our users had a chance to adapt and learn from our > communications. If they didn't, this is entirely their problem. > > We did *NOT* communicate our intention to make Stats public. You can take a > look at that page and it's history. I looked at all "public interface" > versions, and Stats are not mentioned. So we are clear to remove it > according to SemVer and it's NOT a breaking change according to SemVer. > > The risk of removing it is very small - especially if we explain what to do > if someone used them in release notes as a "significant change" > description. And we are planning to as far as I know. So risk is not an > issue IMHO. > > I am 100% for removal. > > J. > > > > On Wed, Mar 18, 2026 at 5:39 PM Ferruzzi, Dennis <[email protected]> > wrote: > > > I know we don't have to plus-one a lazy consensus, but maybe I can > provide > > a little context since I did the bulk of the work on the OTel metrics and > > creating that particular mess. > > > > FWIW, I've definitely had people ask if it's possible to ad custom > metrics > > in their Dags. If you had asked me ten minutes ago if the process to do > it > > was in our official docs, I'd have said yes... but I just looked and > > couldn't find it. So I guess I lean toward saying it isn't documented, > > therefore it isn't public interface. > > > > I would support this with the caveat that it's still possible for the > user > > to add custom metrics to their Dags should they choose to do so. > > ________________________________ > > From: Daniel Standish via dev <[email protected]> > > Sent: Wednesday, March 18, 2026 7:05 AM > > To: [email protected] <[email protected]> > > Cc: Daniel Standish <[email protected]> > > Subject: RE: [EXT] [LAZY CONSENSUS] Remove the airflow.metrics and > > airflow.stats modules > > > > CAUTION: This email originated from outside of the organization. Do not > > click links or open attachments unless you can confirm the sender and > know > > the content is safe. > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > pouvez > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain > que > > le contenu ne présente aucun risque. > > > > > > > > To try to emphasize some points: > > > > * airflow.stats and airflow.metrics have been in there since 2.7 > > * as of now they are removed in main, and these removals will be > reflected > > in 3.2 > > * this is ok if we agree that these were not user-public interface (i.e. > > people are not using them in their dags) > > * a (silent) vote of +1 in this lazy consensus means, yes we can remove > (do > > nothing) > > * a vote of -1 means, we can't just remove we have to keep the interfaces > > around where they were, so we have to add backcompat shims > > > > This is the same vote we had for traces (added 3.0), but now for > > airflow.stats and airflow.metrics added (2.7) > > > > > > On Wed, Mar 18, 2026 at 3:23 AM Christos Bisias <[email protected]> > > wrote: > > > > > Hello everyone, > > > > > > For accessing metrics and traces, there were metaclass interfaces, > called > > > `Stats` and `Trace` respectively. Each interface was reading the config > > > and then initializing the instance of the appropriate implementation. > > Each > > > metaclass was acting like a singleton cache and in all consecutive > calls > > > after the first one, it would always access the same existing instance. > > > > > > Daniel Standish has been leading an effort to simplify the OTel traces > > > code. He was able to remove the `Trace` interface and add a single > > > configuration when initializing all the airflow settings in core. > > > > > > Traces were simpler because they have a single supported implementation > > > while metrics have multiple. But after discussions, we think that we > can > > > follow a similar approach and remove the stats interface. The reason is > > > that there is too much complexity that increases with every new > addition > > or > > > refactoring. It's going to be easier to maintain if simplified. > > > > > > The question is whether `airflow.metrics` and `airflow.stats` are > public > > > interfaces. They have been around since airflow 2.7 but as far as I > know, > > > no one from the community has talked about adding user-defined metrics > > from > > > under dags. Additionally, there are no docs saying how users can do it. > > The > > > only public info is for enabling the project's internal metrics. > > > > > > I'm considering these modules as private and I'm assuming that users > > aren't > > > aware of them or using them. I think they are safe to remove. > > > > > > I would like to call for a lazy consensus vote which will run for 72 > > hours. > > > If no one votes -1 during the voting period, the vote will be > considered > > > passed. > > > > > > Thank you! > > > > > > Regards, > > > Christos > > > > > >
