Re: What goes into the next release (was [VOTE] Release Airflow 2.8.1 from 2.8.1rc1)

[Jarek Potiuk]

I think if anything - that should be a separate page in our GitHub -
which will be a) easier to do b) maybe indeed makes sense c) this is where
we put "developer" docs.

I think we agreed already quite some time ago that "airflow.apache.org" is
generally for the users and all the docs for contributors/maintainers
should go as .md/.rst files in our repository, but not into the airflow
website.

Yeah - I think it might be worth it if we keep it separate from README :).
I think we had far too long README and (as mentioned above) over the last
year or so it got shorter and shorter, while it gained only the "super
important" information IMHO. So yeah having a separate FAQ taking my mail
as context might be a good idea.

I will open PR for that and if others agree it's a good idea (and help with
reviewing it, we can merge it).

J.







On Thu, Jan 18, 2024 at 5:29 AM Amogh Desai 
wrote:

> Thanks @Jarek Potiuk !
>
> I vaguely knew the process but not in such detail, thanks for putting it
> together in this email.
> I will visit the document
> https://github.com/apache/airflow?tab=readme-ov-file#what-goes-into-the-next-release
> and if I find any clarifications, I will send it across as a PR.
>
> One suggestion: what if we put it together in a document and put it on the
> airflow website under
> https://airflow.apache.org/community/ where we mention "how we release"?
>
> Might be a bit too much given the context people will have while
> visiting the airflow website..
>
> Thanks & Regards,
> Amogh Desai
>
> On Wed, Jan 17, 2024 at 6:13 PM Jarek Potiuk  wrote:
>
>> I separate it out, because it seems that despite the efforts to explain
>> and
>> document how our releases work It's not clear even for the PMC chair, so
>> likely it warrants a separate thread - also it will be easier to find it
>> in
>> the archives this way.
>>
>> I think this is an important topic that all maintainers should be aware
>> of,
>> so let me take a task to explain it in a longer email (and separate
>> thread).
>>
>> I created it in a form of FAQ, because it seems that similar questions
>> might be asked by others.
>>
>> *Do we have a process defined here?*
>>
>> Answering Bolke's question - who was apparently confused what our process
>> is:
>>
>> > I see that some improvements to Airflow.io (two weeks ago) were not
>> included and some provider updates neither.  Haven't checked anything else
>> yet.
>>
>> Apparently there is some confusion about the process, but yes we have a
>> well defined and well tested (pretty much 4 years now) process that we
>> follow., We follow it since I remember actually - it's been also done the
>> same in 1.10 - but with some variations, Likely we do it the same way
>> since
>> the beginning of 2.0, but it has been refined and improved over time - by
>> those who volunteered their time in the release process (a lot of ad-hoc
>> discussion have been traditionally happening in #release-management slack
>> channel) and as of few months ago we even documented it (It was in
>> November
>> 2023) - with this PR https://github.com/apache/airflow/pull/35245
>>
>> It is currently described in a prominent place in our most important (and
>> over the last year or so the README, we made the README pretty short and
>> contains only super-important information on how Airflow is developed)
>> README.md under *"What goes into the next release?"* chapter:
>>
>>
>> https://github.com/apache/airflow?tab=readme-ov-file#what-goes-into-the-next-release
>> .
>>
>> *How does the selection process for cherry-picking work?*
>>
>> In short (and this is the most important thing that every maintainer
>> should
>> be aware of): *those maintainers who think that issue should be included
>> should mark it with the next (in this case 2.8.1) milestone*. It's up to
>> individual maintainers who want to include certain changes to take care
>> about it and mark the issues they think are bug fixes, to go into the next
>> release
>>
>> This is the only thing that the maintainer has to do to get the PR
>> proposed
>> to be considered in the next patchlevel release. Sometimes - if
>> controversial - maintainers discuss the proposals in #release-management
>> channel, sometimes in #development or in the PR itself (especially if the
>> release manager decides to not include it and changes the milestone (and
>> explains why).
>>
>> *What's the release manager's role ?*
>>
>> Release manager's job is purely mechanical (as also mandated by the Apache
>> Software Foundation release manager role description) to assess
>> cherry-pick
>> ability of those changes. Release manager - at the sole discretion and
>> individual decision (this is the only place in the whole ASF setup where a
>> single person has such power to make individual decisions) can reject some
>> of those who other maintainers think should be included. But the release
>> manager on his own does not make proposals on what should be included.
>>
>> *Is this process following the ASF 

Re: [ANNOUNCE] New PMC member: Andrey Anshin (taragolis)

[Pankaj Koti]

Many congratulations Andrey (taragolis :) ) Very very well deserved 


Best regards,

*Pankaj Koti*
Senior Software Engineer (Airflow OSS Engineering team)
Location: Pune, Maharashtra, India
Timezone: Indian Standard Time (IST)
Phone: +91 9730079985


On Tue, Jan 16, 2024 at 11:03 PM Mehta, Shubham 
wrote:

> Congratulations, Andrey!!
>
> On 2024-01-16, 6:38 AM, "Josh Fell"  LID> wrote:
>
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
> le contenu ne présente aucun risque.
>
>
>
>
>
>
> Congrats Andrey! Very well deserved indeed!
>
>
> On Mon, Jan 15, 2024 at 7:04 PM Wei Lee  weilee...@gmail.com>> wrote:
>
>
> > Congrats Andrey! 
> >
> > Best,
> > Wei
> >
> > > On Jan 16, 2024, at 4:06 AM, Bishundeo, Rajeshwar
> > mailto:rbish...@amazon.com.inva>LID> wrote:
> > >
> > > Congrats Andrey!! Well deserved!!
> > >
> > > -- Rajesh
> > >
> > >
> > >
> > >
> > >
> > >
> > > On 2024-01-15, 3:01 PM, "Oliveira, Niko"  
> > >LID>
> wrote:
> > >
> > >
> > > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you can confirm the sender and
> know
> > the content is safe.
> > >
> > >
> > >
> > >
> > >
> > >
> > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> > externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si
> vous
> > ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
> > certain que le contenu ne présente aucun risque.
> > >
> > >
> > >
> > >
> > >
> > >
> > > Congrats Andrey!
> > >
> > >
> > > 
> > > From: Pankaj Singh  ags.pankaj1...@gmail.com>  > ags.pankaj1...@gmail.com >>
> > > Sent: Monday, January 15, 2024 11:11:01 AM
> > > To: dev@airflow.apache.org   dev@airflow.apache.org >
> > > Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [ANNOUNCE] New PMC member:
> > Andrey Anshin (taragolis)
> > >
> > >
> > > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you can confirm the sender and
> know
> > the content is safe.
> > >
> > >
> > >
> > >
> > >
> > >
> > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> > externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si
> vous
> > ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
> > certain que le contenu ne présente aucun risque.
> > >
> > >
> > >
> > >
> > >
> > >
> > > Congrats Andrey 
> > >
> > >
> > > On Tue, Jan 16, 2024 at 12:30 AM Aritra Basu  
> > >>
> > > wrote:
> > >
> > >
> > >> Congrats Andrey!!
> > >>
> > >> --
> > >> Regards,
> > >> Aritra Basu
> > >>
> > >> On Tue, Jan 16, 2024, 12:24 AM Vincent Beck  
> > >> wrote:
> > >>
> > >>> Congrats Andrey!
> > >>>
> > >>> On 2024/01/15 18:46:32 ambika garg wrote:
> >  Congrats Andrey!!
> > 
> >  On Mon, Jan 15, 2024 at 1:44 PM Amogh Desai <
> amoghdesai@gmail.com 
> > >>
> >  wrote:
> > 
> > > Congrats Andrey!!
> > >
> > > On Tue, 16 Jan 2024 at 12:08 AM, Hussein Awala  
> > >>
> > >>> wrote:
> > >
> > >> Congratulations Andrey, very well deserved!
> > >>
> > >> On Mon 15 Jan 2024 at 19:35, Jarek Potiuk  
> > >>
> > >> wrote:
> > >>
> > >>> I have the pleasure to announce that The Project Management
> > >>> Committee
> > >>> (PMC) for Apache Airflow has invited Andrey Anshin (taragolis) to
> > > become
> > >>> Apache
> > >>> Airflow PMC Member and we are pleased to announce that he has
> > >>> kindly
> > >>> accepted it.
> > >>>
> > >>> Being a PMC member enables assistance with the management and to
> > >>> guide
> > >>> the direction of the project.
> > >>>
> > >>> Congratulations Andrey, I'd say it's been very well deserved !.
> > >>>
> > >>> Regards,
> > >>> Jarek on behalf of the Apache Airflow PMC
> > >>>
> > >>
> > >
> > 
> > >>>
> > >>> 

Re: [DISCUSSION] Adding support for Qdrant provider

[Anush Shetty]

Hey guys. Just pushed the integration tests for the Qdrant provider with
the CI configurations as requested and some refactoring for the Qdrant hook
to remove redundancy.

On Tue, 16 Jan 2024 at 19:58, Jarek Potiuk  wrote:

> > self hosted image and the cloud offering, comprehensive integration tests
> with the
> Docker image should suffice.
>
> I am fine with that.
>
> On Tue, Jan 16, 2024 at 3:00 PM Anush Shetty 
> wrote:
>
> > P.S. The updates to Qdrant and the clients are backwards compatible.
> > Further reducing any maintenance overhead.
> >
> > On Tue, 16 Jan, 2024, 7:29 pm Anush Shetty, 
> > wrote:
> >
> > > We'd gladly add the integration tests along with the mock tests that
> are
> > > currently in place and since there's no difference in running a self
> > hosted
> > > image and the cloud offering, comprehensive integration tests with the
> > > Docker image should suffice.
> > >
> > > As you said, would appreciate any thoughts from the community.
> > >
> > > On Tue, 16 Jan, 2024, 7:11 pm Jarek Potiuk,  wrote:
> > >
> > >> BTW: Dashboard links here:
> > >>
> > >>
> >
> https://airflow.apache.org/ecosystem/#airflow-provider-system-test-dashboards
> > >>
> > >> On Tue, Jan 16, 2024 at 2:39 PM Jarek Potiuk 
> wrote:
> > >>
> > >> > I'd love to hear from others in the community who already use Qdrant
> > >> what
> > >> > they think :) ?
> > >> >
> > >> > Few comments to Anush:
> > >> >
> > >> > I did a bit of review of the links and did some usual research.
> > >> >
> > >> > 1) Re: requirements it does not introduce any big issues. Urllib3 <
> 2
> > >> is a
> > >> > bit strange (but we are anyhow limited by botocore now, so not a big
> > >> issue,
> > >> > I hope it can be removed in the future.
> > >> >
> > >> > Requires-Dist: fastembed (==0.1.1) ; (python_version < "3.12") and
> > >> (extra
> > >> > == "fastembed")
> > >> > Requires-Dist: grpcio (>=1.41.0)
> > >> > Requires-Dist: grpcio-tools (>=1.41.0)
> > >> > Requires-Dist: httpx[http2] (>=0.14.0)
> > >> > Requires-Dist: numpy (<1.21) ; python_version < "3.8"
> > >> > Requires-Dist: numpy (>=1.21) ; python_version >= "3.8" and
> > >> python_version
> > >> > < "3.12"
> > >> > Requires-Dist: numpy (>=1.26) ; python_version >= "3.12"
> > >> > Requires-Dist: portalocker (>=2.7.0,<3.0.0)
> > >> > Requires-Dist: pydantic (>=1.10.8)
> > >> > Requires-Dist: urllib3 (>=1.26.14,<2.0.0)
> > >> >
> > >> > 2) Open source version seems to be fully supported and alive.  This
> > >> looks
> > >> > pretty cool after looking at the information provided. The code is
> > small
> > >> > and literally calling the library QdrantClient, so it does not seem
> > like
> > >> > something that might require a lot of maintenance,
> > >> >
> > >> > My concerns are with testability and future-proof maintenance. This
> > is a
> > >> > fast-pacing area. There will be breaking changes.  Yes. There are
> unit
> > >> > tests and system tests there. But we have no time/possibility to run
> > our
> > >> > tests against real quadrant serve and especially against one run in
> > the
> > >> > cloud "by hand".
> > >> >
> > >> > So, two points:
> > >> >
> > >> > 1) Open-source version: Similar to Kafka provider - seems Qdrant
> has a
> > >> > nicely dockerized version that can be installed from officially
> > released
> > >> > images (https://qdrant.tech/documentation/quick-start/) - seems
> like
> > >> > perfect candidate to run integration tests with it on our CI. If
> that
> > is
> > >> > there, this means that we can both - easily make sure it continues
> to
> > >> work,
> > >> > but also - equally easily bump the version of Qudrant when new
> > >> major/minor
> > >> > release is out and have our tests run automatically in our CI. And
> it
> > >> will
> > >> > nicely run in Breeze with `breeze --integration qdrant` when someone
> > >> wants
> > >> > to run the integration tests locally: See
> > >> >
> > >>
> >
> https://github.com/apache/airflow/tree/main/tests/integration/providers/apache/kafka
> > >> > and
> > >> >
> > >>
> >
> https://github.com/apache/airflow/blob/main/scripts/ci/docker-compose/integration-kafka.yml
> > >> > - I think that shoudl be condition of approving it
> > >> >
> > >> > 2) Cloud version: It would also help if you could (especially if you
> > >> want
> > >> > to run the system tests against your cloud) that you get similar
> > >> dashboards
> > >> > as we have for Amazon and other LLM providers (maintained by
> > Astronomer)
> > >> > which would show the status of system tests you run with main
> version.
> > >> >
> > >> > Are you ok with extending the PR and adding integration tests and
> > >> > committing to maintaining such a dashboard?
> > >> >
> > >> > If there are voices from the community "yeah it's useful" - and the
> > >> points
> > >> > 1) and 2) are addressed, I am quite positive about accepting the
> > >> provider :)
> > >> >
> > >> > J
> > >> >
> > >> >
> > >> >
> > >> > On Tue, Jan 16, 2024 at 1:41 PM Anush Shetty <
> anush.she...@qdrant.com
> > 

Re: [VOTE] Release Airflow 2.8.1 from 2.8.1rc1

[Phani Kumar]

+1 non binding

On Thu, Jan 18, 2024 at 10:30 AM Amogh Desai 
wrote:

> +1 non binding
>
> Installed the RC with helm charts, tried a couple of installation
> configurations
> and ran a few examples, not seeing any regression.
>
> Thanks & Regards,
> Amogh Desai
>
> On Thu, Jan 18, 2024 at 12:16 AM Jed Cunningham 
> wrote:
>
> > +1 (binding)
> >
> > Checked signatures, checksums, licences. Used it with the helm chart
> with a
> > few different configs
> >
>

Re: [VOTE] Release Airflow 2.8.1 from 2.8.1rc1

[Amogh Desai]

+1 non binding

Installed the RC with helm charts, tried a couple of installation
configurations
and ran a few examples, not seeing any regression.

Thanks & Regards,
Amogh Desai

On Thu, Jan 18, 2024 at 12:16 AM Jed Cunningham 
wrote:

> +1 (binding)
>
> Checked signatures, checksums, licences. Used it with the helm chart with a
> few different configs
>

Re: What goes into the next release (was [VOTE] Release Airflow 2.8.1 from 2.8.1rc1)

[Amogh Desai]

Thanks @Jarek Potiuk !

I vaguely knew the process but not in such detail, thanks for putting it
together in this email.
I will visit the document
https://github.com/apache/airflow?tab=readme-ov-file#what-goes-into-the-next-release
and if I find any clarifications, I will send it across as a PR.

One suggestion: what if we put it together in a document and put it on the
airflow website under
https://airflow.apache.org/community/ where we mention "how we release"?

Might be a bit too much given the context people will have while
visiting the airflow website..

Thanks & Regards,
Amogh Desai

On Wed, Jan 17, 2024 at 6:13 PM Jarek Potiuk  wrote:

> I separate it out, because it seems that despite the efforts to explain and
> document how our releases work It's not clear even for the PMC chair, so
> likely it warrants a separate thread - also it will be easier to find it in
> the archives this way.
>
> I think this is an important topic that all maintainers should be aware of,
> so let me take a task to explain it in a longer email (and separate
> thread).
>
> I created it in a form of FAQ, because it seems that similar questions
> might be asked by others.
>
> *Do we have a process defined here?*
>
> Answering Bolke's question - who was apparently confused what our process
> is:
>
> > I see that some improvements to Airflow.io (two weeks ago) were not
> included and some provider updates neither.  Haven't checked anything else
> yet.
>
> Apparently there is some confusion about the process, but yes we have a
> well defined and well tested (pretty much 4 years now) process that we
> follow., We follow it since I remember actually - it's been also done the
> same in 1.10 - but with some variations, Likely we do it the same way since
> the beginning of 2.0, but it has been refined and improved over time - by
> those who volunteered their time in the release process (a lot of ad-hoc
> discussion have been traditionally happening in #release-management slack
> channel) and as of few months ago we even documented it (It was in November
> 2023) - with this PR https://github.com/apache/airflow/pull/35245
>
> It is currently described in a prominent place in our most important (and
> over the last year or so the README, we made the README pretty short and
> contains only super-important information on how Airflow is developed)
> README.md under *"What goes into the next release?"* chapter:
>
>
> https://github.com/apache/airflow?tab=readme-ov-file#what-goes-into-the-next-release
> .
>
> *How does the selection process for cherry-picking work?*
>
> In short (and this is the most important thing that every maintainer should
> be aware of): *those maintainers who think that issue should be included
> should mark it with the next (in this case 2.8.1) milestone*. It's up to
> individual maintainers who want to include certain changes to take care
> about it and mark the issues they think are bug fixes, to go into the next
> release
>
> This is the only thing that the maintainer has to do to get the PR proposed
> to be considered in the next patchlevel release. Sometimes - if
> controversial - maintainers discuss the proposals in #release-management
> channel, sometimes in #development or in the PR itself (especially if the
> release manager decides to not include it and changes the milestone (and
> explains why).
>
> *What's the release manager's role ?*
>
> Release manager's job is purely mechanical (as also mandated by the Apache
> Software Foundation release manager role description) to assess cherry-pick
> ability of those changes. Release manager - at the sole discretion and
> individual decision (this is the only place in the whole ASF setup where a
> single person has such power to make individual decisions) can reject some
> of those who other maintainers think should be included. But the release
> manager on his own does not make proposals on what should be included.
>
> *Is this process following the ASF rules?*
>
> I believe yes, The release manager's role is nicely described here:
> https://infra.apache.org/release-publishing.html#releasemanager. And there
> is a far more complete description here that describes the whole process
> https://www.apache.org/legal/release-policy.html#management - also
> mentioning that it's the PMC's responsibility (and particularly PMC
> chair's) to adhere to the process.
>
> *What's the role of individual maintainers?*
>
> The role of maintainers (collectively) to propose things for the next
> release. In our case it happens with setting the milestone on a PR.
>
> *When proposed PRs are rejected?*
>
> There are various reasons to reject those - if too complex to cherry-pick
> or when the release manager assesses it's a new feature, not a bugfix.
> Essentially (according to semver) when it comes to the user-facing changes,
> the PATCHLEVEL release should contain only bugfixes. and may contain docs
> changes if they are fixing/improving docs (not about new features) and also
> 

Re: [VOTE] Release Airflow 2.8.1 from 2.8.1rc1

[Jed Cunningham]

+1 (binding)

Checked signatures, checksums, licences. Used it with the helm chart with a
few different configs

Re: [VOTE] Release Airflow 2.8.1 from 2.8.1rc1

[Ephraim Anierobi]

Hi Bolke,

Thanks for checking up on this release.
The changes in Airflow.io are mostly improvement changes and this is a
patch release. In patch releases, we don't include improvement changes and
new features. We only include bug fixes, doc changes, and miscellaneous
changes that are not core to airflow but necessary for the bug fixes to
work properly.

I think those other bigger changes were bug fixes. Here's documentation on
what we include in a release:
https://github.com/apache/airflow?tab=readme-ov-file#what-goes-into-the-next-release

Regards
- Ephraim

On Wed, 17 Jan 2024 at 09:03, Bolke de Bruin  wrote:

> Just checking:
>
> I see that some improvements to Airflow.io (two weeks ago) were not
> included and some provider updates neither.  Haven't checked anything else
> yet.
>
> Is that intentional? Ie. is that the purpose of this release. Other
> big(ger) and more recent changes have been included hence asking.
>
> B.
>
> Sent from my iPhone
>
> > On 16 Jan 2024, at 20:18, Jarek Potiuk  wrote:
> >
> > +1 (binding): Checked all my changes, I ran airflow in a few
> combinations
> > (MySQL / Postgres Local/Celery executor. It looks and works well - run a
> > few dags and navigated through a number of screens. Checked licences,
> > signatures, checksums, performed a "reproducible build" check and it
> worked
> > (with a small glitch explained below).
> >
> > BTW. The new "hatchling-built" package looks good and it nicely installs
> > airflow + extras (and it has a far better and cleaner set of extras -
> > finally you can reproducibly install airflow with `all` extra if you
> > ***REALLY*** want :).
> >
> > Reproducibility (also for other PMC members doing the check): I found a
> > small glitch in the "reproducible" part of verifying the packages. While
> > wheel and sdist packages are exactly the same binary-wise, the
> > source-tarball was binarry different for me. I decompressed it and
> compared
> > the content - they are identical - but there is one difference which I
> > overlooked - the group permissions for files in Ephraim's tarball are
> > different from mine. I have totally forgotten about the fact that umask
> > might set different group/other permissions when checking out the files
> > from git. Fix will be coming shortly - in the meantime I recommend anyone
> > who will be doing the comparison to uncompress both tarballs and compare
> > the contents with `diff -r`.
> >
> >
> >
> >> On Tue, Jan 16, 2024 at 11:30 AM Ephraim Anierobi <
> >> ephraimanier...@apache.org> wrote:
> >>
> >> Hey fellow Airflowers,
> >>
> >> I have cut Airflow 2.8.1rc1. This email is calling a vote on the
> release,
> >> which will last at least 72 hours, from Tuesday, January 16, 2024 at
> 10:30
> >> am UTC
> >> until Friday, January 19, 2024, at 10:30 am UTC
> >> <
> >>
> https://www.timeanddate.com/worldclock/fixedtime.html?msg=8=20240119T1030=1440
> >>> ,
> >> and until 3 binding +1 votes have been received.
> >>
> >> Status of testing of the release is kept at
> >> https://github.com/apache/airflow/issues/36808
> >>
> >> Consider this my (binding) +1.
> >>
> >> Airflow 2.8.1rc1 is available at:
> >> https://dist.apache.org/repos/dist/dev/airflow/2.8.1rc1/
> >>
> >> *apache-airflow-2.8.1-source.tar.gz* is a source release that comes with
> >> INSTALL instructions.
> >> *apache-airflow-2.8.1.tar.gz* is the binary Python "sdist" release.
> >> *apache_airflow-2.8.1-py3-none-any.whl* is the binary Python wheel
> "binary"
> >> release.
> >>
> >> Public keys are available at:
> >> https://dist.apache.org/repos/dist/release/airflow/KEYS
> >>
> >> Please vote accordingly:
> >>
> >> [ ] +1 approve
> >> [ ] +0 no opinion
> >> [ ] -1 disapprove with the reason
> >>
> >> Only votes from PMC members are binding, but all members of the
> community
> >> are encouraged to test the release and vote with "(non-binding)".
> >>
> >> The test procedure for PMC members is described in:
> >>
> >>
> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-pmc-members
> >>
> >> The test procedure for and Contributors who would like to test this RC
> is
> >> described in:
> >>
> >>
> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-contributors
> >>
> >>
> >> Please note that the version number excludes the `rcX` string, so it's
> now
> >> simply 2.8.1. This will allow us to rename the artifact without
> modifying
> >> the artifact checksums when we actually release.
> >>
> >> Release Notes:
> >> https://github.com/apache/airflow/blob/2.8.1rc1/RELEASE_NOTES.rst
> >>
> >> Changes since 2.8.0:
> >>
> >> *Significant Changes*
> >>
> >> *Target version for core dependency ``pendulum`` package set to 3
> >> (#36281).*
> >> Support for pendulum 2.1.2 will be saved for a while, presumably until
> the
> >> next feature version of Airflow.
> >> It is advised to upgrade user code to use pendulum 3 as soon as
> possible.
> >>
> >> *Airflow 

What goes into the next release (was [VOTE] Release Airflow 2.8.1 from 2.8.1rc1)

[Jarek Potiuk]

I separate it out, because it seems that despite the efforts to explain and
document how our releases work It's not clear even for the PMC chair, so
likely it warrants a separate thread - also it will be easier to find it in
the archives this way.

I think this is an important topic that all maintainers should be aware of,
so let me take a task to explain it in a longer email (and separate thread).

I created it in a form of FAQ, because it seems that similar questions
might be asked by others.

*Do we have a process defined here?*

Answering Bolke's question - who was apparently confused what our process
is:

> I see that some improvements to Airflow.io (two weeks ago) were not
included and some provider updates neither.  Haven't checked anything else
yet.

Apparently there is some confusion about the process, but yes we have a
well defined and well tested (pretty much 4 years now) process that we
follow., We follow it since I remember actually - it's been also done the
same in 1.10 - but with some variations, Likely we do it the same way since
the beginning of 2.0, but it has been refined and improved over time - by
those who volunteered their time in the release process (a lot of ad-hoc
discussion have been traditionally happening in #release-management slack
channel) and as of few months ago we even documented it (It was in November
2023) - with this PR https://github.com/apache/airflow/pull/35245

It is currently described in a prominent place in our most important (and
over the last year or so the README, we made the README pretty short and
contains only super-important information on how Airflow is developed)
README.md under *"What goes into the next release?"* chapter:

https://github.com/apache/airflow?tab=readme-ov-file#what-goes-into-the-next-release
.

*How does the selection process for cherry-picking work?*

In short (and this is the most important thing that every maintainer should
be aware of): *those maintainers who think that issue should be included
should mark it with the next (in this case 2.8.1) milestone*. It's up to
individual maintainers who want to include certain changes to take care
about it and mark the issues they think are bug fixes, to go into the next
release

This is the only thing that the maintainer has to do to get the PR proposed
to be considered in the next patchlevel release. Sometimes - if
controversial - maintainers discuss the proposals in #release-management
channel, sometimes in #development or in the PR itself (especially if the
release manager decides to not include it and changes the milestone (and
explains why).

*What's the release manager's role ?*

Release manager's job is purely mechanical (as also mandated by the Apache
Software Foundation release manager role description) to assess cherry-pick
ability of those changes. Release manager - at the sole discretion and
individual decision (this is the only place in the whole ASF setup where a
single person has such power to make individual decisions) can reject some
of those who other maintainers think should be included. But the release
manager on his own does not make proposals on what should be included.

*Is this process following the ASF rules?*

I believe yes, The release manager's role is nicely described here:
https://infra.apache.org/release-publishing.html#releasemanager. And there
is a far more complete description here that describes the whole process
https://www.apache.org/legal/release-policy.html#management - also
mentioning that it's the PMC's responsibility (and particularly PMC
chair's) to adhere to the process.

*What's the role of individual maintainers?*

The role of maintainers (collectively) to propose things for the next
release. In our case it happens with setting the milestone on a PR.

*When proposed PRs are rejected?*

There are various reasons to reject those - if too complex to cherry-pick
or when the release manager assesses it's a new feature, not a bugfix.
Essentially (according to semver) when it comes to the user-facing changes,
the PATCHLEVEL release should contain only bugfixes. and may contain docs
changes if they are fixing/improving docs (not about new features) and also
environment/build script changes (so non-user-facing changes) as they are
pretty much always needed to keep the things nicely building - those are
usually skipped from the changelog as non-user facing).

*Why are provider changes not cherry-picked?*

In our case - basically none of the provider changes are cherry-picked -
unless they are needed to make the builds work well (sometimes happen).
Providers are ALWAYS released from the latest main code, not from the
v2-8-stable branch. In fact all the tests and ci checks for providers are
skipped in the non-main (v2* branches). So yes - not seeing provider
changes cherry-picked is absolutely expected.

*Do we skip over some changes when releasing a patchlevel release? What's
the purpose of patch-level releases?*

Answering Bolke's question:

> Is that 

Re: [VOTE] Release Airflow 2.8.1 from 2.8.1rc1

[Bolke de Bruin]

Just checking: 

I see that some improvements to Airflow.io (two weeks ago) were not included 
and some provider updates neither.  Haven't checked anything else yet.

Is that intentional? Ie. is that the purpose of this release. Other big(ger) 
and more recent changes have been included hence asking.

B.

Sent from my iPhone

> On 16 Jan 2024, at 20:18, Jarek Potiuk  wrote:
> 
> +1 (binding): Checked all my changes, I ran airflow in a few combinations
> (MySQL / Postgres Local/Celery executor. It looks and works well - run a
> few dags and navigated through a number of screens. Checked licences,
> signatures, checksums, performed a "reproducible build" check and it worked
> (with a small glitch explained below).
> 
> BTW. The new "hatchling-built" package looks good and it nicely installs
> airflow + extras (and it has a far better and cleaner set of extras -
> finally you can reproducibly install airflow with `all` extra if you
> ***REALLY*** want :).
> 
> Reproducibility (also for other PMC members doing the check): I found a
> small glitch in the "reproducible" part of verifying the packages. While
> wheel and sdist packages are exactly the same binary-wise, the
> source-tarball was binarry different for me. I decompressed it and compared
> the content - they are identical - but there is one difference which I
> overlooked - the group permissions for files in Ephraim's tarball are
> different from mine. I have totally forgotten about the fact that umask
> might set different group/other permissions when checking out the files
> from git. Fix will be coming shortly - in the meantime I recommend anyone
> who will be doing the comparison to uncompress both tarballs and compare
> the contents with `diff -r`.
> 
> 
> 
>> On Tue, Jan 16, 2024 at 11:30 AM Ephraim Anierobi <
>> ephraimanier...@apache.org> wrote:
>> 
>> Hey fellow Airflowers,
>> 
>> I have cut Airflow 2.8.1rc1. This email is calling a vote on the release,
>> which will last at least 72 hours, from Tuesday, January 16, 2024 at 10:30
>> am UTC
>> until Friday, January 19, 2024, at 10:30 am UTC
>> <
>> https://www.timeanddate.com/worldclock/fixedtime.html?msg=8=20240119T1030=1440
>>> ,
>> and until 3 binding +1 votes have been received.
>> 
>> Status of testing of the release is kept at
>> https://github.com/apache/airflow/issues/36808
>> 
>> Consider this my (binding) +1.
>> 
>> Airflow 2.8.1rc1 is available at:
>> https://dist.apache.org/repos/dist/dev/airflow/2.8.1rc1/
>> 
>> *apache-airflow-2.8.1-source.tar.gz* is a source release that comes with
>> INSTALL instructions.
>> *apache-airflow-2.8.1.tar.gz* is the binary Python "sdist" release.
>> *apache_airflow-2.8.1-py3-none-any.whl* is the binary Python wheel "binary"
>> release.
>> 
>> Public keys are available at:
>> https://dist.apache.org/repos/dist/release/airflow/KEYS
>> 
>> Please vote accordingly:
>> 
>> [ ] +1 approve
>> [ ] +0 no opinion
>> [ ] -1 disapprove with the reason
>> 
>> Only votes from PMC members are binding, but all members of the community
>> are encouraged to test the release and vote with "(non-binding)".
>> 
>> The test procedure for PMC members is described in:
>> 
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-pmc-members
>> 
>> The test procedure for and Contributors who would like to test this RC is
>> described in:
>> 
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-contributors
>> 
>> 
>> Please note that the version number excludes the `rcX` string, so it's now
>> simply 2.8.1. This will allow us to rename the artifact without modifying
>> the artifact checksums when we actually release.
>> 
>> Release Notes:
>> https://github.com/apache/airflow/blob/2.8.1rc1/RELEASE_NOTES.rst
>> 
>> Changes since 2.8.0:
>> 
>> *Significant Changes*
>> 
>> *Target version for core dependency ``pendulum`` package set to 3
>> (#36281).*
>> Support for pendulum 2.1.2 will be saved for a while, presumably until the
>> next feature version of Airflow.
>> It is advised to upgrade user code to use pendulum 3 as soon as possible.
>> 
>> *Airflow packaging specification follows modern Python packaging standards
>> (#36537).*
>> We standardized Airflow dependency configuration to follow latest
>> development in Python packaging by
>> using ``pyproject.toml``. Airflow is now compliant with those accepted
>> PEPs:
>> 
>> * `PEP-440 Version Identification and Dependency Specification <
>> https://www.python.org/dev/peps/pep-0440/>`__
>> * `PEP-517 A build-system independent format for source trees <
>> https://www.python.org/dev/peps/pep-0517/>`__
>> * `PEP-518 Specifying Minimum Build System Requirements for Python Projects
>> `__
>> * `PEP-561 Distributing and Packaging Type Information <
>> https://www.python.org/dev/peps/pep-0561/>`__
>> * `PEP-621 Storing project metadata in pyproject.toml <
>>