Just one comment. It might be nit-picking and a question of wording. I don't think we can say that `we guarantee` something. The whole ASF licence is very explicit that we don't guarantee basically anything and users are solely responsible for assessing their risks and we take no responsibility whatsoever for anything. See section 7. of https://www.apache.org/licenses/LICENSE-2.0
We can at most (as a PMC) express our intentions to do something. There are legitimate cases where our intentions cannot be kept and we will have to make tough decisions, bite the bullet and release breaking changes. I don't think we can make `absolute` promises. But we can do some safeguarding here, for example we could add a need to cast a vote or run `lazy consensus`. That would be pretty acceptable and the safeguard will be "serious" enough to make sure that yes, indeed we are going to keep to our intentions. But again - this is nit-picking for the wording possibly. Just ane side note - in case the reason about breaking compatibility would be about critical security issue causing it, the vote will have to be kept on private@a.o with PMC's only, but this is also quite acceptable and security is one of the very few exceptions were votes (including voting on release) can be held in private@a.a.o and results announced after the release. J. On Fri, Mar 8, 2024 at 11:58 PM Michał Modras <michalmod...@google.com.invalid> wrote: > > To Jarek's phrasing - I think it is mostly right. The way I think about it: > when deprecating we guarantee that for 6 months we will not do breaking > changes, but after that period, on an occasion of releasing a major > provider package version (which is probably breaking anyway) we can do > cleanup and remove the old deprecated code. > > Elad - we are planning to do the cleanups, hence we wanted to put the > policy in place for the Google provider, so we constraint ourselves and our > users know what to expect, as opposed to doing things ad hoc and at will. > In this sense the policy is redundant - we could not surface it, and > internally follow these rules to the same effect, but we prefer to be > transparent and inform the users of how the deprecation process will look. > The goal of the policy is to inform as opposed to enforce anything stricter > that we would like to do anyway. > > On Wed, Mar 6, 2024 at 5:14 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > > Hey Elad. I think (and let me repeat that) - nobody suggests that we > > ALWAYS remove some deprecations after 6 months. The idea - as I see > > it - is completely reversed. As I understand current idea is that we > > say "By default we do not introduce any breaking changes (i.e. we do > > not remove deprecations) for 6 months from the moment we introduced > > last breaking changes. This means that we will have have (again by > > default) at most two breaking changes in each provider per year). Of > > course this is just "default" and we can do deprecations more - or > > less - often if we see we want to make exception, but we need to have > > a very good reason for it. But I do not think anyone is putting an > > expiry date and will "force" removal of deprecations at specific date. > > Introducing breaking change in provider means that there are SOME > > deprecations removed - maybe all, maybe just a few. I think the clue > > of this change is to introduce more-or-less stable cadence "generally > > speaking we make a breaking change in provider not more often than > > every 6 months". > > > > It's very similar to our rule that we relase providers more-or-less > > every 2 weeks. Sometimes we do it less often, sometimes we don't > > release provider even if we could (for example we did that with FAB) > > and sometimes we do ad-hoc release because we see a need for it. > > > > Is this correct rephrasing of the proposal - Eugen and Michał? Do I > > understand it correctly ? > > > > J. > > > > On Tue, Mar 5, 2024 at 2:23 PM Elad Kalif <elad...@apache.org> wrote: > > > > > > Eugen, about your two items: > > > - What should be used instead > > > - The date after which the method/parameter/operator will be removed > > > > > > > > > The first is already included today in all deprecation warnings. > > > The second is a big -1 from me. We normally say that it will be removed > > in > > > the next major release. Sometimes we enforce it and sometimes we don't. > > > That is our choice. > > > I said it before and I will say it again. Time based deprecation is not > > the > > > way to do it. Sometimes the author of a PR and the reviewers are not > > fully > > > aware of the blast radius of a specific deprecation. > > > In Airflow any committer can merge any PR - I do not think it's right to > > > put on the shoulders of the committer the decision of when something is > > to > > > be removed before we even accept the deprecation. > > > > > > Michał, The policy we have now is a shared governance model which means > > > that Google can decide and act on deprecating code and also on when to > > > remove code. > > > We as maintainers, and specifically I as release manager for provider > > will > > > try to accomodate Google decision on this as Google knows best what is > > > right for their customers. > > > In the meantime, Google provider has many deprecation warnings some of > > them > > > are very old - I didn't see Google start a cleanup deprecation process. > > > If you think this is the right time then feel free to raise PRs for this. > > > This is why I say that policy may be redundant - you can do it at will. > > > > > > On Tue, Mar 5, 2024 at 12:49 PM Eugen Kosteev <eu...@kosteev.com> wrote: > > > > > > > Hi. > > > > > > > > Thanks for feedback and discussion. > > > > Let me summarise what actually I want to propose (because there were a > > lot > > > > in my initial email). > > > > > > > > My proposal is, in case of deprecation, to emit messages in the > > specific > > > > format, example of the message: > > > > “”” > > > > The “given” parameter is deprecated and will be removed after > > dd.mm.yyyy. > > > > Please use “this” parameter instead. > > > > “”” > > > > The format of the warning message may vary, but it has to contain: > > > > - What should be used instead > > > > - The date after which the method/parameter/operator will be removed > > > > > > > > Everything else, such as decommission/removal only once bumping major > > > > version, cadence of releases, etc. is not something that I propose to > > > > change. > > > > > > > > The idea in general is to make Airflow users a convenient way to adjust > > > > their DAGs to deprecations, thus: > > > > - what to use instead > > > > - when they need to update > > > > should help them. > > > > > > > > If we have a date in the deprecation message, it means that the removal > > > > (actual decommission) will happen in one of the major releases after > > this > > > > date. > > > > > > > > What do you think about this? > > > > > > > > > > > > On Mon, Mar 4, 2024 at 7:03 PM Michał Modras > > > > <michalmod...@google.com.invalid> wrote: > > > > > > > > > First, I'd like to say that I support Eugen's proposal and I agree > > with > > > > the > > > > > enhancements suggested by Jarek. > > > > > > > > > > I'm a bit confused about Elad's point 3 - are you suggesting having a > > > > > global policy for all providers, or that we should not codify our > > > > approach > > > > > at all since different providers have different requirements, > > instead of > > > > > having a policy per provider? > > > > > > > > > > I would be happy for this proposal (with some enhancements as > > needed) to > > > > be > > > > > adopted by all providers, or we can keep it locally to Google > > provider. > > > > > > > > > > On Thu, Feb 29, 2024 at 12:47 PM Jarek Potiuk <ja...@potiuk.com> > > wrote: > > > > > > > > > > > One more thing. This one is essentially impossible (Hyrum's law > > > > explains > > > > > > that very well https://www.hyrumslaw.com/: > > > > > > > > > > > > > The change is considered to be a breaking (in the context of > > google > > > > > > providers package) if a DAG that was working before, stops to work > > > > after > > > > > > the change. > > > > > > > > > > > > I propose to change it to: > > > > > > > > > > > > > The change is considered to be a breaking if a DAG using the > > > > providers > > > > > > (in the > > > > > > way they were intended to be used), that was working before, stops > > to > > > > > work > > > > > > after > > > > > > the change. > > > > > > > > > > > > It is absolutely not possible to make sure that every single DAG > > > > written > > > > > by > > > > > > someone will continue working after any change. Literally. > > > > > > > > > > > > For example if someone extends BigQueryOperator and calls > > > > > > (self._this_very_much_internal_method()` in his operator's new > > > > execute(), > > > > > > and we rename the method, their DAG will stop working. Is it > > breaking ? > > > > > No. > > > > > > Is the user using it with the intention how it should be used? > > > > Absolutely > > > > > > not. > > > > > > > > > > > > SemVer - is not a promise things won't break, it's a promise that > > our > > > > > > intention is that things won't break. But whether the way our > > users use > > > > > it > > > > > > will make it break or not, that's another story. > > > > > > > > > > > > J > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Feb 29, 2024 at 12:38 PM Jarek Potiuk <ja...@potiuk.com> > > > > wrote: > > > > > > > > > > > > > BTW. Another example of an exception would be a security fix. We > > > > > **migh** > > > > > > > find a security issue that will require a breaking change. But > > then > > > > > > again - > > > > > > > the breaking change might **only** cover that security fix - we > > do > > > > not > > > > > > have > > > > > > > to remove all the deprecations in this case (but we could if we > > > > decide > > > > > > so - > > > > > > > very much depending on nature of those deprecations) > > > > > > > > > > > > > > On Thu, Feb 29, 2024 at 12:36 PM Jarek Potiuk <ja...@potiuk.com> > > > > > wrote: > > > > > > > > > > > > > >> I think it's a good idea to codify the rules, indeed what Elad > > > > > > mentioned, > > > > > > >> we **mostly** follow a very similar approach, but it's not > > written > > > > > > >> anywhere. > > > > > > >> > > > > > > >> > > > > > > >> Re: points of Elad: > > > > > > >> > > > > > > >> 1. I am ok with 6 months, but I agree here we should not have > > > > "strict" > > > > > > >> limits. It's fine to have some default (i.e. accumulate > > deprecating > > > > > > changes > > > > > > >> for some time - max. 6 months) - but I think the policy should > > have > > > > an > > > > > > >> option to apply exceptions - for example when we have a major > > change > > > > > in > > > > > > >> many libraries and we need to make some breaking changes (ads > > case) > > > > we > > > > > > >> shall do it (but in this case we don't have to immediately > > remove > > > > all > > > > > > >> deprecations - they can wait for the regular (say 6 months) > > breaking > > > > > > change > > > > > > >> that will be coming. The users can - if needed - even in new > > > > versions > > > > > of > > > > > > >> Airflow - downgrade to previous provider versions in case they > > have > > > > > > problem > > > > > > >> with it. > > > > > > >> > > > > > > >> I like the idea of having some time (6 months) where we can very > > > > > safely > > > > > > >> say "OK, enough is enough, lets remove deprecations" - and this > > > > > > proposal > > > > > > >> addresses it nicely. Currently it's purely subjective judgment. > > One > > > > > > small > > > > > > >> thing here to clarify for "accumulating" deprecations - I think > > we > > > > > > should > > > > > > >> make sure the deprecation is present in already released > > > > > > minor/patchlevel > > > > > > >> version to be able to remove it (so even if it is deprecated 2 > > > > months > > > > > > ago > > > > > > >> but we had release with the deprecation, it's ok to remove it > > in the > > > > > > next > > > > > > >> breaking change. > > > > > > >> > > > > > > >> 2. Agree (and again here "strict" is not good I think). I'd > > treat > > > > > that 6 > > > > > > >> months as a baseline. Before that - we should have very good > > reason > > > > to > > > > > > make > > > > > > >> a breaking change (and we might decide to do it partially only) > > - > > > > > after > > > > > > - > > > > > > >> we are free to remove things that have been already announced as > > > > > > deprecated > > > > > > >> and we should consider making breaking change whenever we > > release a > > > > > new > > > > > > >> version for whatever reason (but again - not mandatory). I's > > just > > > > an > > > > > > >> indication that tells us "Last breaking release was 6 months > > ago, > > > > it's > > > > > > now > > > > > > >> OK to remove all the deprecations without having an extremely > > good > > > > > > reason". > > > > > > >> > > > > > > >> Very much agree with 3. -> this should be the same as for other > > > > > > >> providers. We should adopt the same rule for all of them. And > > if we > > > > > > apply > > > > > > >> the adjustments above - we should - I think - be ok to apply it > > to > > > > any > > > > > > >> provider I think. > > > > > > >> > > > > > > >> J > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> On Thu, Feb 29, 2024 at 12:20 PM Elad Kalif <elad...@apache.org > > > > > > > > wrote: > > > > > > >> > > > > > > >>> This is not too much different than what we already do. > > > > > > >>> We deprecate then we remove. We almost never create breaking > > change > > > > > > >>> without > > > > > > >>> deprecating first. > > > > > > >>> > > > > > > >>> I'd like to raise a few notes: > > > > > > >>> > > > > > > >>> 1. Major releases can happen frequently. I see no problem with > > it. > > > > > This > > > > > > >>> is > > > > > > >>> more a question of what are the changes. If google has 3 major > > > > > releases > > > > > > >>> one > > > > > > >>> with breaking change to ads, another to leveldb and 3rd to > > cloud > > > > > these > > > > > > >>> are > > > > > > >>> not related one to another and unlikely that there is 1 user > > that > > > > > uses > > > > > > >>> all > > > > > > >>> of these together. We create major releases when needed and we > > > > keep a > > > > > > >>> close > > > > > > >>> eye and challenge why a change must be a breaking change. If > > > > possible > > > > > > we > > > > > > >>> will always prefer to be backward compatible. > > > > > > >>> 2. Time based deprecations are not a good approach. Some > > changes > > > > > > require > > > > > > >>> much more time to be acted upon and others may be trivial. > > > > > > >>> As a rule we almost never cut a breaking change release just to > > > > > remove > > > > > > >>> deprecations. > > > > > > >>> 3. Setting a policy per provider is not the right way to go. We > > > > have > > > > > a > > > > > > >>> shared governance model for provider handling and I as release > > > > > manager > > > > > > >>> will > > > > > > >>> do what I can to accommodate requests from companies who > > > > participate > > > > > in > > > > > > >>> managing the provider. Which means that if Google wants to > > remove > > > > > > >>> deprecations and ask for a breaking change release most likely > > we > > > > > will > > > > > > >>> accommodate their request. My point of view is that the company > > > > knows > > > > > > >>> what > > > > > > >>> is best for their own users. I don't think that what is right > > for > > > > > > Google > > > > > > >>> must bind AWS or Microsoft. 1 policy goes against the shared > > > > > > >>> governance idea. > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> On Thu, Feb 29, 2024 at 11:04 AM Eugen Kosteev < > > eu...@kosteev.com> > > > > > > >>> wrote: > > > > > > >>> > > > > > > >>> > Hi. > > > > > > >>> > > > > > > > >>> > I would like to discuss/propose a deprecation policy for > > > > > > >>> > apache-airflow-providers-google package, mostly for > > deprecating > > > > > > Airflow > > > > > > >>> > operators (but not limited to it). > > > > > > >>> > > > > > > > >>> > *Some background:* > > > > > > >>> > Airflow google provider package (as most of other provider > > > > packages > > > > > > in > > > > > > >>> > Airflow) has a tendency to constantly evolve. > > > > > > >>> > There are cases when its features or operators/hooks have to > > be > > > > > > >>> deprecated. > > > > > > >>> > The typical example of such a case is “refactoring” which > > leads > > > > to > > > > > > the > > > > > > >>> > renaming of operators, parameters or public methods. Such > > > > > > refactorings > > > > > > >>> > maybe consequence of: > > > > > > >>> > - using a new dependent library (that causes some methods to > > be > > > > > > >>> deprecated) > > > > > > >>> > - using a new version of API (that causes some parameters to > > be > > > > > > >>> deprecated) > > > > > > >>> > - restructuring/migration of the operators (that causes some > > > > > > operators > > > > > > >>> to > > > > > > >>> > be deprecated) > > > > > > >>> > Given the need of a “deprecation” procedure, it looks > > essential > > > > to > > > > > > >>> > establish policy around it to have users of Airflow google > > > > provider > > > > > > >>> package > > > > > > >>> > clearly understand when and how to adapt their DAGs. > > > > > > >>> > > > > > > > >>> > *Preambula (versioning of the package):* > > > > > > >>> > As mentioned in “Airflow’s release process and version > > policy” ( > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > > > > > > > > > > > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#airflow-s-release-process-and-version-policy > > > > > > >>> > ) > > > > > > >>> > google provider package (and others) should follow SemVer, > > > > meaning > > > > > > >>> that any > > > > > > >>> > breaking changes should be released together with bumping > > major > > > > > > >>> version of > > > > > > >>> > the package. > > > > > > >>> > The change is considered to be a breaking (in the context of > > > > google > > > > > > >>> > providers package) if a DAG that was working before, stops to > > > > work > > > > > > >>> after > > > > > > >>> > the change. > > > > > > >>> > > > > > > > >>> > *My proposal (for discussion):* > > > > > > >>> > The entire procedure of deprecating (either method, > > parameter or > > > > > > >>> operator) > > > > > > >>> > consists of two steps: > > > > > > >>> > - Emission of the deprecation warning message > > > > > > >>> > Example of the message: > > > > > > >>> > “”” > > > > > > >>> > The “given” parameter is deprecated and will be removed after > > > > > > >>> dd.mm.yyyy. > > > > > > >>> > Please use “this” parameter instead. > > > > > > >>> > “”” > > > > > > >>> > - Once the date of the deprecated method/parameter/operator > > is > > > > > > passed, > > > > > > >>> it > > > > > > >>> > can be removed (together with bumping major version of the > > > > package) > > > > > > >>> > > > > > > > >>> > The format of the warning message may vary, but it has to > > > > contain: > > > > > > >>> > - What should be used instead > > > > > > >>> > - The date after which the method/parameter/operator will be > > > > > removed > > > > > > >>> > > > > > > > >>> > *Additional considerations:* > > > > > > >>> > - Bumping a major version of the Airflow google provider > > package > > > > > > >>> shouldn’t > > > > > > >>> > happen very frequently (presumably not more frequently than 6 > > > > > > months), > > > > > > >>> > therefore when it happens it may accumulate removals of > > something > > > > > > that > > > > > > >>> was > > > > > > >>> > communicated to be removed even a couple of months ago. > > (related > > > > > > >>> discussion > > > > > > >>> > https://github.com/apache/airflow/blob/main/PROVIDERS.rst) > > > > > > >>> > Example: if today is 2025-12-20 and we are releasing new > > major > > > > > > version > > > > > > >>> of > > > > > > >>> > the package, we can/should remove all deprecated > > > > > > >>> > methods/parameters/operators which have date prior to > > 2025-12-20 > > > > - > > > > > it > > > > > > >>> can > > > > > > >>> > be November 2025, October 2025 or even earlier, since > > previous > > > > bump > > > > > > of > > > > > > >>> > major version happened e.g. in summer of 2025. > > > > > > >>> > - By default all deprecations should allow a 6 months time > > period > > > > > > until > > > > > > >>> > they will be removed and not available. This period will give > > > > > Airflow > > > > > > >>> users > > > > > > >>> > enough time and flexibility to update their DAGs before > > actual > > > > > > removal > > > > > > >>> > happens. > > > > > > >>> > On a case by case basis this period can be adjusted given > > > > specific > > > > > > >>> > circumstances (e.g. in case deprecation is because of > > underlying > > > > > API > > > > > > >>> sunset > > > > > > >>> > which can happen earlier than in 6 months). > > > > > > >>> > > > > > > > >>> > Apart from deprecation warning messages that will be printed > > as > > > > > logs, > > > > > > >>> this > > > > > > >>> > information will be put as well in: > > > > > > >>> > - Airflow documentation > > > > > > >>> > - Release notes of Airflow google provider package > > > > > > >>> > > > > > > > >>> > Please, let me know what you think about this. > > > > > > >>> > > > > > > > >>> > -- > > > > > > >>> > Eugene > > > > > > >>> > > > > > > > >>> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Eugene > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
