RE: [VOTE] Release Airflow 2.9.1 from 2.9.1rc2

[Scheffler Jens (XC-AS/EAE-ADA-T)]

Good morning,

I re-tested 2.9.1rc2 and also after deployment, code diff checks - no problems 
found.

+1 (non binding)

Mit freundlichen Grüßen / Best regards

Jens Scheffler

Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T)
Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen | GERMANY | 
www.bosch.com
Tel. +49 711 811-91508 | Mobil +49 160 90417410 | jens.scheff...@de.bosch.com

Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000;
Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer; 
Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. Markus 
Forschner, 
Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert

-Original Message-
From: Hussein Awala  
Sent: Sunday, May 5, 2024 7:54 PM
To: dev@airflow.apache.org
Subject: Re: [VOTE] Release Airflow 2.9.1 from 2.9.1rc2

+1 (binding)

On Sunday, May 5, 2024, Jed Cunningham  wrote:

> +1 (binding)
>
> Checked reproducibility, signatures, checksums, licences. Used it with 
> the helm chart with a few different configs.
>

Re: [VOTE] Release Airflow 2.9.1 from 2.9.1rc2

[Hussein Awala]

+1 (binding)

On Sunday, May 5, 2024, Jed Cunningham  wrote:

> +1 (binding)
>
> Checked reproducibility, signatures, checksums, licences. Used it with the
> helm chart with a few different configs.
>

Re: [VOTE] Release Airflow 2.9.1 from 2.9.1rc2

[Jed Cunningham]

+1 (binding)

Checked reproducibility, signatures, checksums, licences. Used it with the
helm chart with a few different configs.

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Scheffler Jens (XC-AS/EAE-ADA-T)]

Thanks for the document write-up, Kaxil. I assume this is mostly a vision 
statement.

Looking forward for a larger addendum where we can collect things that we all 
can vote and agree on as targets.

As I started earlier with a confluence page and it seems this is not accessible 
to all, shall we convert this to a Google Doc for better collaboration and item 
collection?

Sent from Outlook for iOS

From: Vikram Koka 
Sent: Sunday, May 5, 2024 3:34:33 AM
To: dev@airflow.apache.org 
Subject: Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic 
(Airflow 3) approach

Thank you for your feedback, Bolke and Andrey!

Bolke,
I have replied to some of your comments in the doc.
I will provide a detailed write up on the "Interactive DAG run" (or
synchronous DAG run) capability, which has generated some early questions.
I had intended to get an AIP published for that as a follow-up, but I
believe that a simpler write up would be useful ahead of the AIP.

Andrey,
You raise an interesting point.

As part of the Airflow 2.0 release, we as a community had decided to
strictly adhere to Semver as detailed in the document you referenced. We
also consciously split out the "Core Airflow" releases from the "Provider"
releases at that time. We had a clear expectation then for the cadence of
both minor and patch releases, which we have generally adhered to since
then.

Personally, I am more concerned about our Provider releases right now, as
compared to the cadence of our major releases. I believe that one of the
proposed changes in the Airflow 3 document i.e. the clear separation for
Task Execution will help here, but more may be needed.

Definitely interested in more feedback on this as well.

Vikram


On Sat, May 4, 2024 at 10:57 AM Andrey Anshin 
wrote:

> I would like to propose to change (at least discuss) release policy around
> the Major version of Airflow.
>
> Right now it is described as "These releases do not happen with any regular
> interval or on any predictable schedule." :
>
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D=0
>
> So maybe it is time to make it schedulable, e.g. one per two years or so.
> This one could help us to avoid such a discussion in the future, like "We
> don't know when Airflow 4 is coming.". At the moment when the new major
> version will be released new features wouldn't be added in the old major
> version, however we would support bug / security for a while, e.g. 1 year
> for bug fixes, 3 years for security fixes with a total 5 year lifecycle per
> a major version. These just are approximate time periods for a definition
> of current period, bugfix period and security fix period.
>
> In contributors' perspective it helps with dropping the deprecated stuff
> which resolves some old problem: we have to support everything including
> deprecated stuff and without schedulable lifecycle for the deprecated stuff
> it could be showstopper for the new feature, because sometimes it hard to
> support two different approaches for long period of time with no hope that
> it will happen soon. For some fundamental stuff which do not require a lot
> things time to support we could postponed removal for next after the next
> release, e.g. deprecate in Airflow 3, but remove it in Airflow 5
>
> In the user perspective, they have at least bug fix support for a while, if
> someone want to use legacy version it their choice, however no new
> features, no new version of providers (after one year)
>
>
> 
> Best Wishes
> *Andrey Anshin*
>
>
>
> On Sat, 4 May 2024 at 19:17, Bolke de Bruin  wrote:
>
> > I have left several comments :-). And on interactive dag runs even after
> > the explanation of Vikram I still don't have a clue what we want to
> > accomplish there :-P.
> >
> > I would like to see a mantra or team for Airflow 3. That helps nudging
> > people in the same direction. Suggestions in the comments.
> >
> > Bolke
> > Sent from my iPhone
> >
> > > On 4 May 2024, at 01:14, Vikram Koka 
> > wrote:
> > >
> > > Good point Jed.
> > > I responded back to your comment in the doc as well and very open to
> > > changing the term in the doc.
> > >
> > > Used the term "interactive DAG run" as the ability to invoke or
> trigger a
> > > DAG run through the API, with the expectation of getting back a result
> > > immediately. An alternate term could be a "synchronous DAG run".
> > >
> > > Regardless, this is a significant change so a good term to 

Re: [DISCUSS] simplifying try_number handling

[Scheffler Jens (XC-AS/EAE-ADA-T)]

I aleasy had a hard time understanding this, thought it is a feature which I 
did not understand. So +1 (binding) from my side for cleaning up!

Sent from Outlook for iOS

From: Brent Bovenzi 
Sent: Friday, May 3, 2024 4:43:43 PM
To: dev@airflow.apache.org 
Subject: Re: [DISCUSS] simplifying try_number handling

+1
Pumped to remove confusion around tries

On Fri, May 3, 2024 at 5:01 AM Wei Lee  wrote:

> Thanks, Daniel! +1 for this one. This was confusing when I worked on the
> starting from triggerer stuff.
>
> Best,
> Wei
>
>
> > On May 3, 2024, at 11:59 AM, Amogh Desai 
> wrote:
> >
> > Looks good to me.
> >
> > Personally I never ran into any issues with this so far but I agree with
> > the issues it solves.
> > Thanks & Regards,
> > Amogh Desai
> >
> >
> > On Fri, May 3, 2024 at 2:50 AM Vincent Beck  wrote:
> >
> >> I am all +1 on this one. This thing gave me headaches when working on
> >> AIP-44 and I could not understand the difference between the private
> >> "_try_number" and the public "try_number". Thanks for simplifying it!
> >>
> >> This is obviously assuming it does not break anything I am not aware of
> :)
> >>
> >> On 2024/05/02 19:37:32 Daniel Standish wrote:
> >>> TLDR
> >>> * changing handling of try_number in
> >>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F39336=05%7C02%7CJens.Scheffler%40de.bosch.com%7C2dd9210e80024a4fbdc708dc6b7f7949%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638503442435313477%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C=zXJ9XFLQLVv3SWFELfSJ1gywceTUwchLmerly0uGhTg%3D=0
> >>> * no more private attr
> >>> * no more getter that changes value based on state of task
> >>> * no more decrementing
> >>> * try number now only handled by scheduler
> >>> * hope that sounds good to all of you
> >>>
> >>> For more detail read on...
> >>>
> >>> In 
> >>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F39336=05%7C02%7CJens.Scheffler%40de.bosch.com%7C2dd9210e80024a4fbdc708dc6b7f7949%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638503442435323497%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C=QeWCDz8wBLu66BjxgBl4WhZErjISJCy1sliA0KZR%2Bng%3D=0
> >>>  I am doing some work
> to
> >>> resolve some longstanding pain and frustration caused by try_number.
> >>>
> >>> The way we handle try_number has for quite some time been messy and
> >>> problematic.
> >>>
> >>> For example, if you access `ti.try_number` and then change the state to
> >> or
> >>> from RUNNING, you will get a different value if you access it again!
> >>>
> >>> And the responsibility for managing this number has been distributed
> >>> throughout the codebase.  For example the task itself always increments
> >>> when it starts running.  But then if it defers or reschedules itself,
> it
> >>> decrements it back down so that when it runs again and naively
> >> increments,
> >>> then it will be right again.
> >>>
> >>> Recently more issues have become visible as I have worked with AIP-44
> >>> because for example pydantic does not like private attrs and it's just
> >>> awkward to know *what value to use* when serializing it when the TI
> will
> >>> give you a different answer depending on the state of the task!
> >>>
> >>> And there's yet another edge case being solved in this community PR
> >>>  >.
> >>> And then when we start looking at try history and AIP-64, it also
> >> forces a
> >>> look at this.
> >>>
> >>> So it all sounds bad and indeed it is bad but I think I have a
> solution.
> >>>
> >>> What I do is, just have the scheduler increment try_number at the
> moment
> >>> when it schedules the task.  It alone will have the responsibility for
> >>> incrementing try_number.  And no where will it ever be decremented.  It
> >>> will not be incremented when resuming after deferral or reschedule.
> And
> >>> that's about all there is to it.
> >>>
> >>> I've tested it out and it works.  But I'm working through many test
> >>> failures that need to be resolved (there's lots of asserts re
> >> try_number).
> >>>
> >>> One small thing I just want to point out is that if a user were
> >> previously
> >>> to be doing `task.run()` sort of manually without the task having been
> >>> scheduled by the scheduler, well now their 

Re: [VOTE] Airflow Providers prepared on May 01, 2024

[rom sharon]

+1 (non-binding)

‫בתאריך שבת, 4 במאי 2024 ב-22:43 מאת ‪Kaxil Naik‬‏ <‪kaxiln...@gmail.com
‬‏>:‬

> +1 binding for all except Pinecone one that Ankit is solving.
>
>
>
> On Sat, 4 May 2024 at 18:02, Andrey Anshin 
> wrote:
>
> > +1 (binding)
> >
> > Check licences, signatures, checksums, and changes
> >
> >
> > On Fri, 3 May 2024 at 15:58, Pankaj Koti  > .invalid>
> > wrote:
> >
> > > +1 (non-binding)
> > >
> > > Tested my set of changes.
> > >
> > > Best regards,
> > >
> > > *Pankaj Koti*
> > > Senior Software Engineer (Airflow OSS Engineering team)
> > > Location: Pune, Maharashtra, India
> > > Timezone: Indian Standard Time (IST)
> > >
> > >
> > > On Fri, May 3, 2024 at 4:25 PM Hussein Awala  wrote:
> > >
> > > > +1 (binding) checked licences, checksums, signatures and sources, and
> > ran
> > > > some testing dags for cncf.kuberenetes and amazon providers.
> > > >
> > > > On Fri, May 3, 2024 at 9:29 AM Wei Lee  wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > Best,
> > > > > Wei
> > > > >
> > > > > > On May 3, 2024, at 2:53 PM, Pankaj Singh <
> ags.pankaj1...@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > On Thu, May 2, 2024 at 6:03 PM Elad Kalif 
> > > wrote:
> > > > > >
> > > > > >> I will exclude pinecone from this release.
> > > > > >> Please continue voting excluding pinecone provider
> > > > > >>
> > > > > >> On Thu, May 2, 2024 at 3:19 PM Ankit Chaurasia <
> > sunank...@gmail.com
> > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >>> -1 non-binding for Pinecone: 2.0.0rc1
> > > > > >>>
> > > > > >>> There is an issue with Pinecone: 2.0.0rc1 with the system test.
> > > This
> > > > > bug
> > > > > >> is
> > > > > >>> introduced as part of the following PR Pinecone provider
> support
> > > for
> > > > > >>> pinecone-client>=3 (#37307): @rawwar
> > > > > >>> 
> > > > > >>>
> > > > > >>> PR #39365   >should
> > > fix
> > > > > it.
> > > > > >>>
> > > > > >>> *Ankit Chaurasia*
> > > > > >>> HomePage  |  LinkedIn
> > > > > >>> 
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>> On Thu, May 2, 2024 at 10:47 AM Amogh Desai <
> > > > amoghdesai@gmail.com>
> > > > > >>> wrote:
> > > > > >>>
> > > > >  +1 non binding
> > > > > 
> > > > >  Installed the tarball and ran some example DAGs for hive and
> > cncf
> > > > > >>> provider,
> > > > >  works as expected.
> > > > > 
> > > > >  Thanks & Regards,
> > > > >  Amogh Desai
> > > > > 
> > > > > 
> > > > >  On Wed, May 1, 2024 at 10:49 PM Vincent Beck <
> > vincb...@apache.org
> > > >
> > > > > >>> wrote:
> > > > > 
> > > > > > +1 non binding. All AWS system tests are working successfully
> > > > against
> > > > > > apache-airflow-providers-amazon==8.21.0rc1. You can see the
> > > results
> > > > > >>> here:
> > > > > >
> > > > > 
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://aws-mwaa.github.io/#/open-source/system-tests/version/fe4605a10e26f1b8a180979ba5765d1cb7fb0111_8.21.0rc1.html
> > > > >  .
> > > > > > The only failure (example_bedrock) is a known issue in the
> test
> > > > > >> itself
> > > > >  and
> > > > > > is currently being worked on.
> > > > > >
> > > > > > On 2024/05/01 13:03:07 Elad Kalif wrote:
> > > > > >> Hey all,
> > > > > >>
> > > > > >> I have just cut the new wave Airflow Providers packages.
> This
> > > > email
> > > > > >>> is
> > > > > >> calling a vote on the release,
> > > > > >> which will last for 72 hours - which means that it will end
> on
> > > May
> > > > > >>> 04,
> > > > > > 2024
> > > > > >> 13:00 PM UTC and until 3 binding +1 votes have been
> received.
> > > > > >>
> > > > > >>
> > > > > >> Consider this my (binding) +1.
> > > > > >>
> > > > > >> *Note some of the providers are rc2 and some rc1.*
> > > > > >>
> > > > > >> Airflow Providers are available at:
> > > > > >> https://dist.apache.org/repos/dist/dev/airflow/providers/
> > > > > >>
> > > > > >> *apache-airflow-providers--*.tar.gz* are the
> binary
> > > > > >> Python "sdist" release - they are also official "sources"
> for
> > > the
> > > > > > provider
> > > > > >> packages.
> > > > > >>
> > > > > >> *apache_airflow_providers_-*.whl are the binary
> > > > > >> Python "wheel" release.
> > > > > >>
> > > > > >> The test procedure for PMC members is described in
> > > > > >>
> > > > > >
> > > > > 
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members
> > > > > >>
> > > > > >> The test procedure for and Contributors who would like to
> test
> >