First, I'd like to say that I support Eugen's proposal and I agree with the enhancements suggested by Jarek.
I'm a bit confused about Elad's point 3 - are you suggesting having a global policy for all providers, or that we should not codify our approach at all since different providers have different requirements, instead of having a policy per provider? I would be happy for this proposal (with some enhancements as needed) to be adopted by all providers, or we can keep it locally to Google provider. On Thu, Feb 29, 2024 at 12:47 PM Jarek Potiuk <ja...@potiuk.com> wrote: > One more thing. This one is essentially impossible (Hyrum's law explains > that very well https://www.hyrumslaw.com/: > > > The change is considered to be a breaking (in the context of google > providers package) if a DAG that was working before, stops to work after > the change. > > I propose to change it to: > > > The change is considered to be a breaking if a DAG using the providers > (in the > way they were intended to be used), that was working before, stops to work > after > the change. > > It is absolutely not possible to make sure that every single DAG written by > someone will continue working after any change. Literally. > > For example if someone extends BigQueryOperator and calls > (self._this_very_much_internal_method()` in his operator's new execute(), > and we rename the method, their DAG will stop working. Is it breaking ? No. > Is the user using it with the intention how it should be used? Absolutely > not. > > SemVer - is not a promise things won't break, it's a promise that our > intention is that things won't break. But whether the way our users use it > will make it break or not, that's another story. > > J > > > > > > > On Thu, Feb 29, 2024 at 12:38 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > BTW. Another example of an exception would be a security fix. We **migh** > > find a security issue that will require a breaking change. But then > again - > > the breaking change might **only** cover that security fix - we do not > have > > to remove all the deprecations in this case (but we could if we decide > so - > > very much depending on nature of those deprecations) > > > > On Thu, Feb 29, 2024 at 12:36 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > >> I think it's a good idea to codify the rules, indeed what Elad > mentioned, > >> we **mostly** follow a very similar approach, but it's not written > >> anywhere. > >> > >> > >> Re: points of Elad: > >> > >> 1. I am ok with 6 months, but I agree here we should not have "strict" > >> limits. It's fine to have some default (i.e. accumulate deprecating > changes > >> for some time - max. 6 months) - but I think the policy should have an > >> option to apply exceptions - for example when we have a major change in > >> many libraries and we need to make some breaking changes (ads case) we > >> shall do it (but in this case we don't have to immediately remove all > >> deprecations - they can wait for the regular (say 6 months) breaking > change > >> that will be coming. The users can - if needed - even in new versions of > >> Airflow - downgrade to previous provider versions in case they have > problem > >> with it. > >> > >> I like the idea of having some time (6 months) where we can very safely > >> say "OK, enough is enough, lets remove deprecations" - and this > proposal > >> addresses it nicely. Currently it's purely subjective judgment. One > small > >> thing here to clarify for "accumulating" deprecations - I think we > should > >> make sure the deprecation is present in already released > minor/patchlevel > >> version to be able to remove it (so even if it is deprecated 2 months > ago > >> but we had release with the deprecation, it's ok to remove it in the > next > >> breaking change. > >> > >> 2. Agree (and again here "strict" is not good I think). I'd treat that 6 > >> months as a baseline. Before that - we should have very good reason to > make > >> a breaking change (and we might decide to do it partially only) - after > - > >> we are free to remove things that have been already announced as > deprecated > >> and we should consider making breaking change whenever we release a new > >> version for whatever reason (but again - not mandatory). I's just an > >> indication that tells us "Last breaking release was 6 months ago, it's > now > >> OK to remove all the deprecations without having an extremely good > reason". > >> > >> Very much agree with 3. -> this should be the same as for other > >> providers. We should adopt the same rule for all of them. And if we > apply > >> the adjustments above - we should - I think - be ok to apply it to any > >> provider I think. > >> > >> J > >> > >> > >> > >> On Thu, Feb 29, 2024 at 12:20 PM Elad Kalif <elad...@apache.org> wrote: > >> > >>> This is not too much different than what we already do. > >>> We deprecate then we remove. We almost never create breaking change > >>> without > >>> deprecating first. > >>> > >>> I'd like to raise a few notes: > >>> > >>> 1. Major releases can happen frequently. I see no problem with it. This > >>> is > >>> more a question of what are the changes. If google has 3 major releases > >>> one > >>> with breaking change to ads, another to leveldb and 3rd to cloud these > >>> are > >>> not related one to another and unlikely that there is 1 user that uses > >>> all > >>> of these together. We create major releases when needed and we keep a > >>> close > >>> eye and challenge why a change must be a breaking change. If possible > we > >>> will always prefer to be backward compatible. > >>> 2. Time based deprecations are not a good approach. Some changes > require > >>> much more time to be acted upon and others may be trivial. > >>> As a rule we almost never cut a breaking change release just to remove > >>> deprecations. > >>> 3. Setting a policy per provider is not the right way to go. We have a > >>> shared governance model for provider handling and I as release manager > >>> will > >>> do what I can to accommodate requests from companies who participate in > >>> managing the provider. Which means that if Google wants to remove > >>> deprecations and ask for a breaking change release most likely we will > >>> accommodate their request. My point of view is that the company knows > >>> what > >>> is best for their own users. I don't think that what is right for > Google > >>> must bind AWS or Microsoft. 1 policy goes against the shared > >>> governance idea. > >>> > >>> > >>> > >>> On Thu, Feb 29, 2024 at 11:04 AM Eugen Kosteev <eu...@kosteev.com> > >>> wrote: > >>> > >>> > Hi. > >>> > > >>> > I would like to discuss/propose a deprecation policy for > >>> > apache-airflow-providers-google package, mostly for deprecating > Airflow > >>> > operators (but not limited to it). > >>> > > >>> > *Some background:* > >>> > Airflow google provider package (as most of other provider packages > in > >>> > Airflow) has a tendency to constantly evolve. > >>> > There are cases when its features or operators/hooks have to be > >>> deprecated. > >>> > The typical example of such a case is “refactoring” which leads to > the > >>> > renaming of operators, parameters or public methods. Such > refactorings > >>> > maybe consequence of: > >>> > - using a new dependent library (that causes some methods to be > >>> deprecated) > >>> > - using a new version of API (that causes some parameters to be > >>> deprecated) > >>> > - restructuring/migration of the operators (that causes some > operators > >>> to > >>> > be deprecated) > >>> > Given the need of a “deprecation” procedure, it looks essential to > >>> > establish policy around it to have users of Airflow google provider > >>> package > >>> > clearly understand when and how to adapt their DAGs. > >>> > > >>> > *Preambula (versioning of the package):* > >>> > As mentioned in “Airflow’s release process and version policy” ( > >>> > > >>> > > >>> > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#airflow-s-release-process-and-version-policy > >>> > ) > >>> > google provider package (and others) should follow SemVer, meaning > >>> that any > >>> > breaking changes should be released together with bumping major > >>> version of > >>> > the package. > >>> > The change is considered to be a breaking (in the context of google > >>> > providers package) if a DAG that was working before, stops to work > >>> after > >>> > the change. > >>> > > >>> > *My proposal (for discussion):* > >>> > The entire procedure of deprecating (either method, parameter or > >>> operator) > >>> > consists of two steps: > >>> > - Emission of the deprecation warning message > >>> > Example of the message: > >>> > “”” > >>> > The “given” parameter is deprecated and will be removed after > >>> dd.mm.yyyy. > >>> > Please use “this” parameter instead. > >>> > “”” > >>> > - Once the date of the deprecated method/parameter/operator is > passed, > >>> it > >>> > can be removed (together with bumping major version of the package) > >>> > > >>> > The format of the warning message may vary, but it has to contain: > >>> > - What should be used instead > >>> > - The date after which the method/parameter/operator will be removed > >>> > > >>> > *Additional considerations:* > >>> > - Bumping a major version of the Airflow google provider package > >>> shouldn’t > >>> > happen very frequently (presumably not more frequently than 6 > months), > >>> > therefore when it happens it may accumulate removals of something > that > >>> was > >>> > communicated to be removed even a couple of months ago. (related > >>> discussion > >>> > https://github.com/apache/airflow/blob/main/PROVIDERS.rst) > >>> > Example: if today is 2025-12-20 and we are releasing new major > version > >>> of > >>> > the package, we can/should remove all deprecated > >>> > methods/parameters/operators which have date prior to 2025-12-20 - it > >>> can > >>> > be November 2025, October 2025 or even earlier, since previous bump > of > >>> > major version happened e.g. in summer of 2025. > >>> > - By default all deprecations should allow a 6 months time period > until > >>> > they will be removed and not available. This period will give Airflow > >>> users > >>> > enough time and flexibility to update their DAGs before actual > removal > >>> > happens. > >>> > On a case by case basis this period can be adjusted given specific > >>> > circumstances (e.g. in case deprecation is because of underlying API > >>> sunset > >>> > which can happen earlier than in 6 months). > >>> > > >>> > Apart from deprecation warning messages that will be printed as logs, > >>> this > >>> > information will be put as well in: > >>> > - Airflow documentation > >>> > - Release notes of Airflow google provider package > >>> > > >>> > Please, let me know what you think about this. > >>> > > >>> > -- > >>> > Eugene > >>> > > >>> > >> >
