Hey Elad. I think (and let me repeat that) - nobody suggests that we ALWAYS remove some deprecations after 6 months. The idea - as I see it - is completely reversed. As I understand current idea is that we say "By default we do not introduce any breaking changes (i.e. we do not remove deprecations) for 6 months from the moment we introduced last breaking changes. This means that we will have have (again by default) at most two breaking changes in each provider per year). Of course this is just "default" and we can do deprecations more - or less - often if we see we want to make exception, but we need to have a very good reason for it. But I do not think anyone is putting an expiry date and will "force" removal of deprecations at specific date. Introducing breaking change in provider means that there are SOME deprecations removed - maybe all, maybe just a few. I think the clue of this change is to introduce more-or-less stable cadence "generally speaking we make a breaking change in provider not more often than every 6 months".
It's very similar to our rule that we relase providers more-or-less every 2 weeks. Sometimes we do it less often, sometimes we don't release provider even if we could (for example we did that with FAB) and sometimes we do ad-hoc release because we see a need for it. Is this correct rephrasing of the proposal - Eugen and Michał? Do I understand it correctly ? J. On Tue, Mar 5, 2024 at 2:23 PM Elad Kalif <elad...@apache.org> wrote: > > Eugen, about your two items: > - What should be used instead > - The date after which the method/parameter/operator will be removed > > > The first is already included today in all deprecation warnings. > The second is a big -1 from me. We normally say that it will be removed in > the next major release. Sometimes we enforce it and sometimes we don't. > That is our choice. > I said it before and I will say it again. Time based deprecation is not the > way to do it. Sometimes the author of a PR and the reviewers are not fully > aware of the blast radius of a specific deprecation. > In Airflow any committer can merge any PR - I do not think it's right to > put on the shoulders of the committer the decision of when something is to > be removed before we even accept the deprecation. > > Michał, The policy we have now is a shared governance model which means > that Google can decide and act on deprecating code and also on when to > remove code. > We as maintainers, and specifically I as release manager for provider will > try to accomodate Google decision on this as Google knows best what is > right for their customers. > In the meantime, Google provider has many deprecation warnings some of them > are very old - I didn't see Google start a cleanup deprecation process. > If you think this is the right time then feel free to raise PRs for this. > This is why I say that policy may be redundant - you can do it at will. > > On Tue, Mar 5, 2024 at 12:49 PM Eugen Kosteev <eu...@kosteev.com> wrote: > > > Hi. > > > > Thanks for feedback and discussion. > > Let me summarise what actually I want to propose (because there were a lot > > in my initial email). > > > > My proposal is, in case of deprecation, to emit messages in the specific > > format, example of the message: > > “”” > > The “given” parameter is deprecated and will be removed after dd.mm.yyyy. > > Please use “this” parameter instead. > > “”” > > The format of the warning message may vary, but it has to contain: > > - What should be used instead > > - The date after which the method/parameter/operator will be removed > > > > Everything else, such as decommission/removal only once bumping major > > version, cadence of releases, etc. is not something that I propose to > > change. > > > > The idea in general is to make Airflow users a convenient way to adjust > > their DAGs to deprecations, thus: > > - what to use instead > > - when they need to update > > should help them. > > > > If we have a date in the deprecation message, it means that the removal > > (actual decommission) will happen in one of the major releases after this > > date. > > > > What do you think about this? > > > > > > On Mon, Mar 4, 2024 at 7:03 PM Michał Modras > > <michalmod...@google.com.invalid> wrote: > > > > > First, I'd like to say that I support Eugen's proposal and I agree with > > the > > > enhancements suggested by Jarek. > > > > > > I'm a bit confused about Elad's point 3 - are you suggesting having a > > > global policy for all providers, or that we should not codify our > > approach > > > at all since different providers have different requirements, instead of > > > having a policy per provider? > > > > > > I would be happy for this proposal (with some enhancements as needed) to > > be > > > adopted by all providers, or we can keep it locally to Google provider. > > > > > > On Thu, Feb 29, 2024 at 12:47 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > > > One more thing. This one is essentially impossible (Hyrum's law > > explains > > > > that very well https://www.hyrumslaw.com/: > > > > > > > > > The change is considered to be a breaking (in the context of google > > > > providers package) if a DAG that was working before, stops to work > > after > > > > the change. > > > > > > > > I propose to change it to: > > > > > > > > > The change is considered to be a breaking if a DAG using the > > providers > > > > (in the > > > > way they were intended to be used), that was working before, stops to > > > work > > > > after > > > > the change. > > > > > > > > It is absolutely not possible to make sure that every single DAG > > written > > > by > > > > someone will continue working after any change. Literally. > > > > > > > > For example if someone extends BigQueryOperator and calls > > > > (self._this_very_much_internal_method()` in his operator's new > > execute(), > > > > and we rename the method, their DAG will stop working. Is it breaking ? > > > No. > > > > Is the user using it with the intention how it should be used? > > Absolutely > > > > not. > > > > > > > > SemVer - is not a promise things won't break, it's a promise that our > > > > intention is that things won't break. But whether the way our users use > > > it > > > > will make it break or not, that's another story. > > > > > > > > J > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Feb 29, 2024 at 12:38 PM Jarek Potiuk <ja...@potiuk.com> > > wrote: > > > > > > > > > BTW. Another example of an exception would be a security fix. We > > > **migh** > > > > > find a security issue that will require a breaking change. But then > > > > again - > > > > > the breaking change might **only** cover that security fix - we do > > not > > > > have > > > > > to remove all the deprecations in this case (but we could if we > > decide > > > > so - > > > > > very much depending on nature of those deprecations) > > > > > > > > > > On Thu, Feb 29, 2024 at 12:36 PM Jarek Potiuk <ja...@potiuk.com> > > > wrote: > > > > > > > > > >> I think it's a good idea to codify the rules, indeed what Elad > > > > mentioned, > > > > >> we **mostly** follow a very similar approach, but it's not written > > > > >> anywhere. > > > > >> > > > > >> > > > > >> Re: points of Elad: > > > > >> > > > > >> 1. I am ok with 6 months, but I agree here we should not have > > "strict" > > > > >> limits. It's fine to have some default (i.e. accumulate deprecating > > > > changes > > > > >> for some time - max. 6 months) - but I think the policy should have > > an > > > > >> option to apply exceptions - for example when we have a major change > > > in > > > > >> many libraries and we need to make some breaking changes (ads case) > > we > > > > >> shall do it (but in this case we don't have to immediately remove > > all > > > > >> deprecations - they can wait for the regular (say 6 months) breaking > > > > change > > > > >> that will be coming. The users can - if needed - even in new > > versions > > > of > > > > >> Airflow - downgrade to previous provider versions in case they have > > > > problem > > > > >> with it. > > > > >> > > > > >> I like the idea of having some time (6 months) where we can very > > > safely > > > > >> say "OK, enough is enough, lets remove deprecations" - and this > > > > proposal > > > > >> addresses it nicely. Currently it's purely subjective judgment. One > > > > small > > > > >> thing here to clarify for "accumulating" deprecations - I think we > > > > should > > > > >> make sure the deprecation is present in already released > > > > minor/patchlevel > > > > >> version to be able to remove it (so even if it is deprecated 2 > > months > > > > ago > > > > >> but we had release with the deprecation, it's ok to remove it in the > > > > next > > > > >> breaking change. > > > > >> > > > > >> 2. Agree (and again here "strict" is not good I think). I'd treat > > > that 6 > > > > >> months as a baseline. Before that - we should have very good reason > > to > > > > make > > > > >> a breaking change (and we might decide to do it partially only) - > > > after > > > > - > > > > >> we are free to remove things that have been already announced as > > > > deprecated > > > > >> and we should consider making breaking change whenever we release a > > > new > > > > >> version for whatever reason (but again - not mandatory). I's just > > an > > > > >> indication that tells us "Last breaking release was 6 months ago, > > it's > > > > now > > > > >> OK to remove all the deprecations without having an extremely good > > > > reason". > > > > >> > > > > >> Very much agree with 3. -> this should be the same as for other > > > > >> providers. We should adopt the same rule for all of them. And if we > > > > apply > > > > >> the adjustments above - we should - I think - be ok to apply it to > > any > > > > >> provider I think. > > > > >> > > > > >> J > > > > >> > > > > >> > > > > >> > > > > >> On Thu, Feb 29, 2024 at 12:20 PM Elad Kalif <elad...@apache.org> > > > wrote: > > > > >> > > > > >>> This is not too much different than what we already do. > > > > >>> We deprecate then we remove. We almost never create breaking change > > > > >>> without > > > > >>> deprecating first. > > > > >>> > > > > >>> I'd like to raise a few notes: > > > > >>> > > > > >>> 1. Major releases can happen frequently. I see no problem with it. > > > This > > > > >>> is > > > > >>> more a question of what are the changes. If google has 3 major > > > releases > > > > >>> one > > > > >>> with breaking change to ads, another to leveldb and 3rd to cloud > > > these > > > > >>> are > > > > >>> not related one to another and unlikely that there is 1 user that > > > uses > > > > >>> all > > > > >>> of these together. We create major releases when needed and we > > keep a > > > > >>> close > > > > >>> eye and challenge why a change must be a breaking change. If > > possible > > > > we > > > > >>> will always prefer to be backward compatible. > > > > >>> 2. Time based deprecations are not a good approach. Some changes > > > > require > > > > >>> much more time to be acted upon and others may be trivial. > > > > >>> As a rule we almost never cut a breaking change release just to > > > remove > > > > >>> deprecations. > > > > >>> 3. Setting a policy per provider is not the right way to go. We > > have > > > a > > > > >>> shared governance model for provider handling and I as release > > > manager > > > > >>> will > > > > >>> do what I can to accommodate requests from companies who > > participate > > > in > > > > >>> managing the provider. Which means that if Google wants to remove > > > > >>> deprecations and ask for a breaking change release most likely we > > > will > > > > >>> accommodate their request. My point of view is that the company > > knows > > > > >>> what > > > > >>> is best for their own users. I don't think that what is right for > > > > Google > > > > >>> must bind AWS or Microsoft. 1 policy goes against the shared > > > > >>> governance idea. > > > > >>> > > > > >>> > > > > >>> > > > > >>> On Thu, Feb 29, 2024 at 11:04 AM Eugen Kosteev <eu...@kosteev.com> > > > > >>> wrote: > > > > >>> > > > > >>> > Hi. > > > > >>> > > > > > >>> > I would like to discuss/propose a deprecation policy for > > > > >>> > apache-airflow-providers-google package, mostly for deprecating > > > > Airflow > > > > >>> > operators (but not limited to it). > > > > >>> > > > > > >>> > *Some background:* > > > > >>> > Airflow google provider package (as most of other provider > > packages > > > > in > > > > >>> > Airflow) has a tendency to constantly evolve. > > > > >>> > There are cases when its features or operators/hooks have to be > > > > >>> deprecated. > > > > >>> > The typical example of such a case is “refactoring” which leads > > to > > > > the > > > > >>> > renaming of operators, parameters or public methods. Such > > > > refactorings > > > > >>> > maybe consequence of: > > > > >>> > - using a new dependent library (that causes some methods to be > > > > >>> deprecated) > > > > >>> > - using a new version of API (that causes some parameters to be > > > > >>> deprecated) > > > > >>> > - restructuring/migration of the operators (that causes some > > > > operators > > > > >>> to > > > > >>> > be deprecated) > > > > >>> > Given the need of a “deprecation” procedure, it looks essential > > to > > > > >>> > establish policy around it to have users of Airflow google > > provider > > > > >>> package > > > > >>> > clearly understand when and how to adapt their DAGs. > > > > >>> > > > > > >>> > *Preambula (versioning of the package):* > > > > >>> > As mentioned in “Airflow’s release process and version policy” ( > > > > >>> > > > > > >>> > > > > > >>> > > > > > > > > > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#airflow-s-release-process-and-version-policy > > > > >>> > ) > > > > >>> > google provider package (and others) should follow SemVer, > > meaning > > > > >>> that any > > > > >>> > breaking changes should be released together with bumping major > > > > >>> version of > > > > >>> > the package. > > > > >>> > The change is considered to be a breaking (in the context of > > google > > > > >>> > providers package) if a DAG that was working before, stops to > > work > > > > >>> after > > > > >>> > the change. > > > > >>> > > > > > >>> > *My proposal (for discussion):* > > > > >>> > The entire procedure of deprecating (either method, parameter or > > > > >>> operator) > > > > >>> > consists of two steps: > > > > >>> > - Emission of the deprecation warning message > > > > >>> > Example of the message: > > > > >>> > “”” > > > > >>> > The “given” parameter is deprecated and will be removed after > > > > >>> dd.mm.yyyy. > > > > >>> > Please use “this” parameter instead. > > > > >>> > “”” > > > > >>> > - Once the date of the deprecated method/parameter/operator is > > > > passed, > > > > >>> it > > > > >>> > can be removed (together with bumping major version of the > > package) > > > > >>> > > > > > >>> > The format of the warning message may vary, but it has to > > contain: > > > > >>> > - What should be used instead > > > > >>> > - The date after which the method/parameter/operator will be > > > removed > > > > >>> > > > > > >>> > *Additional considerations:* > > > > >>> > - Bumping a major version of the Airflow google provider package > > > > >>> shouldn’t > > > > >>> > happen very frequently (presumably not more frequently than 6 > > > > months), > > > > >>> > therefore when it happens it may accumulate removals of something > > > > that > > > > >>> was > > > > >>> > communicated to be removed even a couple of months ago. (related > > > > >>> discussion > > > > >>> > https://github.com/apache/airflow/blob/main/PROVIDERS.rst) > > > > >>> > Example: if today is 2025-12-20 and we are releasing new major > > > > version > > > > >>> of > > > > >>> > the package, we can/should remove all deprecated > > > > >>> > methods/parameters/operators which have date prior to 2025-12-20 > > - > > > it > > > > >>> can > > > > >>> > be November 2025, October 2025 or even earlier, since previous > > bump > > > > of > > > > >>> > major version happened e.g. in summer of 2025. > > > > >>> > - By default all deprecations should allow a 6 months time period > > > > until > > > > >>> > they will be removed and not available. This period will give > > > Airflow > > > > >>> users > > > > >>> > enough time and flexibility to update their DAGs before actual > > > > removal > > > > >>> > happens. > > > > >>> > On a case by case basis this period can be adjusted given > > specific > > > > >>> > circumstances (e.g. in case deprecation is because of underlying > > > API > > > > >>> sunset > > > > >>> > which can happen earlier than in 6 months). > > > > >>> > > > > > >>> > Apart from deprecation warning messages that will be printed as > > > logs, > > > > >>> this > > > > >>> > information will be put as well in: > > > > >>> > - Airflow documentation > > > > >>> > - Release notes of Airflow google provider package > > > > >>> > > > > > >>> > Please, let me know what you think about this. > > > > >>> > > > > > >>> > -- > > > > >>> > Eugene > > > > >>> > > > > > >>> > > > > >> > > > > > > > > > > > > > -- > > Eugene > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
