[VOTE] Release Airflow 2.3.1 from 2.3.1rc1

Hey fellow Airflowers, I have cut Airflow 2.3.1rc1. This email is calling a vote on the release, which will last at least 72 hours, from Saturday, May 21, 2022, at 10:40 pm UTC until Tuesday, May 24, 2022, at 10:40 pm UTC

Re: [DISCUSS] - Policy for removing deprecated code from providers

> > It just means that ANY major release can remove deprecated code regardless > of when we introduced the deprecation. Precisely, yes. On Sat, 21 May 2022 at 20:23, Elad Kalif wrote: > rethinking about this I do support Ash's approach. > > > Following the above. Are you proposing we make it a

Re: [DISCUSS] - Policy for removing deprecated code from providers

rethinking about this I do support Ash's approach. > Following the above. Are you proposing we make it as a rule? The rule is that we can. It doesn't mean that we will every time. It just means that ANY major release can remove deprecated code regardless of when we introduced the deprecation. The

Re: [DISCUSS] - Policy for removing deprecated code from providers

I don't think *one size fits all approach* will work with Providers. The wait for X months without a major version won't be possible to achieve in cases where we need to bump dependencies of providers that break things too. (example Google provider 3.0.0 -> 4.0.0) If users were caught by surprise

Re: [DISCUSS] - Policy for removing deprecated code from providers

And about this > By "policy for policy's sake" this is exactly what I mean - we have so much > work down already it feels nigh on impossible to know everything. BTW. I see it differently. We have policies, precisely because we are not able to follow everything, so we write policies that anyone w

Re: [DISCUSS] - Policy for removing deprecated code from providers

https://github.com/semver/semver/blob/master/semver.md. Nothing says when you have to make breeding changes of course. But it's a sign to the users that something has broken. If you want to wait six months feel free. But I don't think that should be our policy. I the policy to be "we release u

Re: [DISCUSS] - Policy for removing deprecated code from providers

> Will min Airflow version aside: we agreed we're going to follow SemVer, so I > say we stick to that. X+1.0.0 (when ever we choose to release it) is when we > remove the deprecated code. That's not entirely correct. Unfortunately semver.org is down now so I cannot check my authoritative source

Re: [DISCUSS] - Policy for removing deprecated code from providers

Ah right okay. By "policy for policy's sake" this is exactly what I mean - we have so much work down already it feels nigh on impossible to know everything. Will min Airflow version aside: we agreed we're going to follow SemVer, so I say we stick to that. X+1.0.0 (when ever we choose to release

Re: [DISCUSS] - Policy for removing deprecated code from providers

> This is exactly what we had discussed while separating out the providers from > the core Airflow repo. > The same principle that we apply for bumping dependencies for Airflow's deps > should apply to provider deps which is "we only ever bump the minimum version > of a dep if we really need to"

Re: [DISCUSS] - Policy for removing deprecated code from providers

> I'd also say we should keep the min Airflow version as low as possible for > providers, and only update it when we actually want to require a new feature > in a specific provider. Are you saying we should periodically just up the min > core requirement even if it's not required? This is a pol

Re: [DISCUSS] - Policy for removing deprecated code from providers

> > I'd also say we should keep the min Airflow version as low as possible for > providers, and only update it when we actually want to require a new > feature in a specific provider. Are you saying we should periodically just > up the min core requirement even if it's not required? That for me i

Re: [DISCUSS] - Policy for removing deprecated code from providers

Oh okay. Will then I disagree with your proposal quite strongly: providers should be run like separate projects with next to nothing tied to core. So long as X.*.* issues a warning then X+1.0.0 can delete that code, even if it's released the next week - it's not like we do this very often anywa

Re: [DISCUSS] - Policy for removing deprecated code from providers

Both have nothing to do with Providers. -> Both have nothing to do with Core. On Sat, May 21, 2022 at 6:48 PM Ash Berlin-Taylor wrote: > > > - a deprecation could be announced in providers' version X, should not be > > removed in X+1 version and deprecation could be removed in X+2 version > > a

Re: [DISCUSS] - Policy for removing deprecated code from providers

Yes. It is in providers. And it's about Google not Salesforce (that one has not yet been released). Both have nothing to do with Providers. But my proposal is to couple deprecation removal with the "min-version" of Airflow. The "min-version" is the moment where people will have to upgrade Airflow

Re: [DISCUSS] - Policy for removing deprecated code from providers

> - a deprecation could be announced in providers' version X, should not be > removed in X+1 version and deprecation could be removed in X+2 version > assuming "deprecation notice period" passes. This is no longer SemVer then. On 21 May 2022 15:52:12 BST, Rafal Biegacz wrote: >Elad - thank yo

Re: [DISCUSS] - Policy for removing deprecated code from providers

I think your are talking about a different case to Elad. The impetus for this discussion was deprecated operator for Tableau in the Salesforce provider. So nothing there is to do with Airflow core version, just purely on the providers side. -ash On 21 May 2022 17:01:01 BST, Jarek Potiuk wrot

Re: [DISCUSS] - Policy for removing deprecated code from providers

Tl;DR; My proposal for policy is very simple: let's remove deprecations, if they were introduced in the version of Airflow that we do not support any more in providers. The next round of providers will have "Airflow >= 2.2". So if we added a deprecation in 2.1, it means that it is safe to remove th

Re: Airflow RBAC vs. DAGs modifying Airflow DB content...

Rafael, I think the key component here is your CI/CD pipeline and PR standards. I limit who can approve PRs, who can approve a release (not always the same people), and require vulnerability scanning on all containers and the DAGs within those containers before a PR and release to prod are appr

Re: [DISCUSSION] AIP-49 OpenTelemetry Support for Apache Airflow

On Wed, 18 May 2022 at 16:44, Howard Yoo wrote: > 2. So, the reason why I ended up implementing span_json was that between the > scheduler who submits the tasks to be processed, and the worker that needs to > pick them up from the queue (which is implemented in meta database of > airflow) - nee