Re: [VOTE] Airflow Providers prepared on October 10, 2024
+1 non binding Tested my dag set, looks good! Thanks & Regards, Amogh Desai On Fri, Oct 11, 2024 at 9:03 AM Wei Lee wrote: > +1 (non-binding), tested my changes and a few example dags > > Best, > Wei > > > On Oct 11, 2024, at 5:23 AM, Jens Scheffler > wrote: > > > > +1 (binding) - Checked SVN, Check in Docker, Reproducible package build, > > Licenses, Signatures > > > > On 10.10.24 13:15, Freddy Demiane wrote: > >> +1 (non-binding) on google providers package > >> > >> On Thu, Oct 10, 2024 at 3:40 AM Elad Kalif wrote: > >> > >>> Hey all, > >>> > >>> I have just cut the new wave Airflow Providers packages. This email is > >>> calling a vote on the release, > >>> which will last for 72 hours - which means that it will end on October > 13, > >>> 2024 01:35 AM UTC and until 3 binding +1 votes have been received. > >>> > >>> Consider this my (binding) +1. > >>> > >>> Airflow Providers are available at: > >>> https://dist.apache.org/repos/dist/dev/airflow/providers/ > >>> > >>> *apache-airflow-providers--*.tar.gz* are the binary > >>> Python "sdist" release - they are also official "sources" for the > provider > >>> packages. > >>> > >>> *apache_airflow_providers_-*.whl are the binary > >>> Python "wheel" release. > >>> > >>> The test procedure for PMC members is described in > >>> > >>> > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members > >>> > >>> The test procedure for and Contributors who would like to test this RC > is > >>> described in: > >>> > >>> > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors > >>> > >>> > >>> Public keys are available at: > >>> https://dist.apache.org/repos/dist/release/airflow/KEYS > >>> > >>> Please vote accordingly: > >>> > >>> [ ] +1 approve > >>> [ ] +0 no opinion > >>> [ ] -1 disapprove with the reason > >>> > >>> Only votes from PMC members are binding, but members of the community > are > >>> encouraged to test the release and vote with "(non-binding)". > >>> > >>> Please note that the version number excludes the 'rcX' string. > >>> This will allow us to rename the artifact without modifying > >>> the artifact checksums when we actually release. > >>> > >>> The status of testing the providers by the community is kept here: > >>> https://github.com/apache/airflow/issues/42882 > >>> > >>> The issue is also the easiest way to see important PRs included in the > RC > >>> candidates. > >>> Detailed changelog for the providers will be published in the > documentation > >>> after the > >>> RC candidates are released. > >>> > >>> You can find the RC packages in PyPI following these links: > >>> > >>> https://pypi.org/project/apache-airflow-providers-amazon/9.0.0rc1/ > >>> > https://pypi.org/project/apache-airflow-providers-apache-beam/5.8.1rc1/ > >>> > https://pypi.org/project/apache-airflow-providers-apache-kafka/1.6.1rc1/ > >>> > https://pypi.org/project/apache-airflow-providers-apache-spark/4.11.1rc1/ > >>> https://pypi.org/project/apache-airflow-providers-celery/3.8.3rc1/ > >>> https://pypi.org/project/apache-airflow-providers-cloudant/4.0.1rc1/ > >>> > https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/9.0.0rc1/ > >>> > https://pypi.org/project/apache-airflow-providers-common-compat/1.2.1rc1/ > >>> https://pypi.org/project/apache-airflow-providers-common-io/1.4.2rc1/ > >>> > https://pypi.org/project/apache-airflow-providers-common-sql/1.18.0rc1/ > >>> > https://pypi.org/project/apache-airflow-providers-databricks/6.11.0rc1/ > >>> https://pypi.org/project/apache-airflow-providers-dbt-cloud/3.11.0rc1/ > >>> > https://pypi.org/project/apache-airflow-providers-elasticsearch/5.5.2rc1/ > >>> https://pypi.org/project/apache-airflow-providers-fab/1.4.1rc1/ > >>> https://pypi.org/project/apache-airflow-providers-google/10.24.0rc1/ > >>> https://pypi.org/project/apach
Re: [ANNOUNCE] Slack notification for canary build failures
I love it too. I just checked up on this today and my thoughts were exactly what Niko wrote. I commented on the channel on Slack, and realised that Jarek's reasoning makes sense, so I am going to correct myself! Thanks & Regards, Amogh Desai On Thu, Oct 3, 2024 at 4:16 AM Jarek Potiuk wrote: > false positives -> false negatives of course :) > > On Wed, Oct 2, 2024 at 3:45 PM Jarek Potiuk wrote: > > > I love it too. > > > > I actually like the idea of having it in the same channel where we have > > discussions. This has the nice feature of being slightly "annoying" for > > those who use that channel. And being "slightly annoying" - is all but > > guaranteed that eventually someone will take care of the failures. This > is > > the "social" aspect of such integrations in the open-source community > that > > I think is quite important. > > > > One of the things such "annoying" notifications will have is that we > > (collectively) will have to make sure that we take better care about > false > > positives. > > > > And I really love that - we can add some retries etc. to make some > > regular "environmental" issues workaround, and if we will be sufficiently > > "annoyed" by the false positives to finally fix or workaround them, > that's > > a "net-positive" effect. > > > > J. > > > > > > > > On Wed, Oct 2, 2024 at 2:56 PM Oliveira, Niko > > > wrote: > > > >> I love this! Thanks for doing that! > >> > >> Have we considered sending the messages to a dedicated alerting channel? > >> E.g. #internal-airflow-ci-cd-alerts > >> That way you could set notifications to all messages of that channel, to > >> get a notification when main is failing, without getting a notification > for > >> every unrelated message that comes up in the existing > >> #internal-airflow-ci-cd channel (which has other discussions and topics > >> being discussed at any one time). > >> > >> Cheers, > >> Niko > >> > >> > >> From: rom sharon > >> Sent: Wednesday, October 2, 2024 2:27:49 PM > >> To: dev@airflow.apache.org > >> Subject: [EXT] [ANNOUNCE] Slack notification for canary build failures > >> > >> CAUTION: This email originated from outside of the organization. Do not > >> click links or open attachments unless you can confirm the sender and > know > >> the content is safe. > >> > >> > >> > >> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > externe. > >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > pouvez > >> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain > que > >> le contenu ne présente aucun risque. > >> > >> > >> > >> I've set up Slack notifications for failed scheduled canary builds. (pr > >> <https://github.com/apache/airflow/pull/42394>) > >> These alerts will be sent to the #internal-ci-cd > >> <https://apache-airflow.slack.com/archives/C015SLQF059> channel. > >> > >> Feel free to join the channel and check for any failure notifications > that > >> appear. > >> > >> Thanks, Rom. > >> > > >
Re: [DISCUSSION] Proposal to Add Couchbase Provider to Apache Airflow
I second the thought of Jarek here. I also think it is better to go with third party providers in terms of long term commitment to the provider and the fact that it would provide better sustainability. Thanks & Regards, Amogh Desai On Tue, Oct 1, 2024 at 11:36 AM Jarek Potiuk wrote: > Why not release your own provider and expose it via > > https://airflow.apache.org/ecosystem/#third-party-airflow-plugins-and-providers > . > - there are plenty of 3rd-party providers written, maintained and released > by respective owners. We are reluctant in accepting new providers because - > to put it bluntly - this means that the community has to make sure it > works, test it and whenever couchbase changes or modifies their API, also > follow that up. It's much easier the other way round - when the respective > owner maintains their own provider (and many do as you can see). > > What makes you think that couchbase will do better as "community managed" ? > > J. > > On Thu, Sep 19, 2024 at 10:12 PM Shyam Rajamannar > wrote: > > > Hi, > > > > My name is Shyam, and I work at Couchbase. On behalf of Couchbase, I > would > > like to contribute to Apache Airflow by adding a Couchbase provider. This > > contribution is aimed at enabling a wider reach of Couchbase as a viable > > option for users of Apache Airflow, facilitating easier integration and > use > > in ETL and data pipeline processes. > > > > I have forked the repository and implemented the initial code for the > > Couchbase provider. You can find the draft PR for your review here: > > https://github.com/apache/airflow/pull/42271 > > > > Summary of Changes: We created a hook for Couchbase that exposes the > > Couchbase client, scope, and collection. Additionally, the code includes > > functionality to modify the connection field information. > > > > Please let me know if you have any feedback or suggestions. > > > > Thank you! > > > > Best regards, > > Shyam Venkat > > Couchbase > > >
Re: [VOTE] Airflow Providers prepared on September 27, 2024
+1 non binding Tested some example jobs. No regressions. Thanks & Regards, Amogh Desai On Sat, Sep 28, 2024 at 9:27 PM Vishnu Chilukoori wrote: > +1 (non-binding) > > > -- > Regards, > Vishnu C. > > On Sat, Sep 28, 2024, 17:23 Shahar Epstein wrote: > > > +1 (non-binding) > > > > On Fri, Sep 27, 2024 at 11:58 AM Elad Kalif wrote: > > > > > Hey all, > > > > > > I have just cut the adhoc wave Airflow Providers packages. This email > is > > > calling a vote on the release, which will last for 72 hours - which > means > > > that it will end on September 30, 2024 08:55 AM UTC and until 3 binding > > +1 > > > votes have been received. > > > > > > Consider this my (binding) +1. > > > > > > Airflow Providers are available at: > > > https://dist.apache.org/repos/dist/dev/airflow/providers/ > > > > > > *apache-airflow-providers--*.tar.gz* are the binary > > > Python "sdist" release - they are also official "sources" for the > > provider > > > packages. > > > > > > *apache_airflow_providers_-*.whl are the binary Python > "wheel" > > > release. > > > > > > The test procedure for PMC members is described in > > > > > > > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members > > > > > > The test procedure for and Contributors who would like to test this RC > is > > > described in: > > > > > > > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors > > > > > > > > > Public keys are available at: > > > https://dist.apache.org/repos/dist/release/airflow/KEYS > > > > > > Please vote accordingly: > > > > > > [ ] +1 approve > > > [ ] +0 no opinion > > > [ ] -1 disapprove with the reason > > > > > > Only votes from PMC members are binding, but members of the community > are > > > encouraged to test the release and vote with "(non-binding)". > > > > > > Please note that the version number excludes the 'rcX' string. > > > This will allow us to rename the artifact without modifying > > > the artifact checksums when we actually release. > > > > > > The status of testing the providers by the community is kept here: > > > https://github.com/apache/airflow/issues/42538 > > > > > > The issue is also the easiest way to see important PRs included in the > RC > > > candidates. > > > Detailed changelog for the providers will be published in the > > documentation > > > after the > > > RC candidates are released. > > > > > > You can find the RC packages in PyPI following these links: > > > > > > > https://pypi.org/project/apache-airflow-providers-common-sql/1.17.1rc1/ > > > > https://pypi.org/project/apache-airflow-providers-openlineage/1.12.1rc1/ > > > > > > Cheers, > > > Elad Kalif > > > > > >
Re: [VOTE] September 2024 PR of the Month
+1 for 42004. Good job! Thanks & Regards, Amogh Desai On Thu, Sep 26, 2024 at 12:31 PM Pavankumar Gopidesu < gopidesupa...@gmail.com> wrote: > +1 for 42004 > > Regards, > Pavan Kumar > > On Thu, Sep 26, 2024, 04:06 Wei Lee wrote: > > > +1 for 42004 > > > > Best, > > Wei > > > > Mehta, Shubham 於 2024年9月26日 週四 上午11:18寫道： > > > > > +1 for 42004 > > > > > > Shubham > > > > > > On 2024-09-25, 7:00 PM, "Vishnu Chilukoori" < > vish.chiluko...@gmail.com > > > <mailto:vish.chiluko...@gmail.com>> wrote: > > > > > > > > > CAUTION: This email originated from outside of the organization. Do not > > > click links or open attachments unless you can confirm the sender and > > know > > > the content is safe. > > > > > > > > > > > > > > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > externe. > > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > > pouvez > > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain > > que > > > le contenu ne présente aucun risque. > > > > > > > > > > > > > > > > > > > > > +1 for 42004 > > > > > > > > > > > > > > > -- > > > Regards, > > > Vishnu. Chilukoori > > > > > > > > > > > > > > > On Thu, Sep 26, 2024, 06:32 Kaxil Naik > > kaxiln...@gmail.com>> wrote: > > > > > > > > > > +1 for 42004 > > > > > > > > On Wed, 25 Sept 2024 at 21:07, Bishundeo, Rajeshwar > > > > mailto:rbish...@amazon.com.inva>lid> > wrote: > > > > > > > > > #42004 - simple auth manager -> big exciting changes for Airflow > > 3.0!!! > > > > > > > > > > -- Rajesh > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 2024-09-25, 3:36 PM, "Jens Scheffler" > > <mailto:jsche...@web.de.inva> > > > > jsche...@web.de.inva <mailto:jsche...@web.de.inva>>LID> wrote: > > > > > > > > > > > > > > > CAUTION: This email originated from outside of the organization. Do > > not > > > > > click links or open attachments unless you can confirm the sender > and > > > > know > > > > > the content is safe. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > > > externe. > > > > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous > ne > > > > pouvez > > > > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > > certain > > > > que > > > > > le contenu ne présente aucun risque. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > +1 for #42004 > > > > > > > > > > > > > > > On 25.09.24 20:52, Shahar Epstein wrote: > > > > > > #42004 - Simple is best :) > > > > > > > > > > > > On Mon, Sep 23, 2024 at 10:10 PM Briana Okyere > > > > > > > > briana.oky...@astronomer.io.inva> > > > > briana.oky...@astronomer.io.inva > > briana.oky...@astronomer.io.inva>>lid> wrote: > > > > > > > > > > > >> Hey All, > > > > > >> > > > > > >> It’s once again time to vote for the PR of the Month! > > > > > >> > > > > > >> With the help of the `get_important_pr_candidates` script in > > > > dev/stats, > > > > > >> we've identified the following candidates: > > > > > >> > > > > > >> PR #42004: Implement `SimpleAuthManager` < > > > > > >> https://github.com/apache/airflow/pull/42004> < > > > https://github.com/apache/airflow/pull/42004>;> < > > > > > https://github.com/apache/airflow/pull/42004>;> < > > > https://github.com/apache/airflow/pull/42004>
Re: [RESULT][VOTE] Airflow Providers - release of September 21, 2024
Elad, hussein got added to non binding as well. Thanks & Regards, Amogh Desai On Tue, 24 Sep 2024 at 7:07 PM, Elad Kalif wrote: > Hello, > > Apache Airflow Providers prepared on September 21, 2024 have been accepted. > > 4 "+1" binding votes received: > - Elad Kalif > - Ephraim Anierobi > - Jed Cunningham > - Hussein Awala > > 6 "+1" non binding votes received: > - Amogh Desai > - Vishnu Chilukoori > - Vincent Beck > - Rahul Vats > - Hussein Awala > - Freddy Demiane > > Vote thread: > https://lists.apache.org/thread/9j6p6c9fcc9cz2tpysr7m0rj0g9o96hv > > I'll continue with the release process, and the release announcement will > follow shortly. > > Cheers, > Elad Kalif >
Re: [VOTE] Airflow Providers prepared on September 21, 2024
+1 non binding Ran couple of dags for cncf and hive providers. Works as expected Thanks & Regards, Amogh Desai On Sat, 21 Sep 2024 at 2:13 PM, Elad Kalif wrote: > Hey all, > > I have just cut the new wave Airflow Providers packages. This email is > calling a vote on the release, > which will last for 72 hours - which means that it will end on September > 24, 2024 08:40 AM UTC and until 3 binding +1 votes have been received. > > Consider this my (binding) +1. > > Airflow Providers are available at: > https://dist.apache.org/repos/dist/dev/airflow/providers/ > > *apache-airflow-providers--*.tar.gz* are the binary > Python "sdist" release - they are also official "sources" for the provider > packages. > > *apache_airflow_providers_-*.whl are the binary > Python "wheel" release. > > The test procedure for PMC members is described in > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members > > The test procedure for and Contributors who would like to test this RC is > described in: > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors > > > Public keys are available at: > https://dist.apache.org/repos/dist/release/airflow/KEYS > > Please vote accordingly: > > [ ] +1 approve > [ ] +0 no opinion > [ ] -1 disapprove with the reason > > Only votes from PMC members are binding, but members of the community are > encouraged to test the release and vote with "(non-binding)". > > Please note that the version number excludes the 'rcX' string. > This will allow us to rename the artifact without modifying > the artifact checksums when we actually release. > > The status of testing the providers by the community is kept here: > https://github.com/apache/airflow/issues/42393 > > The issue is also the easiest way to see important PRs included in the RC > candidates. > Detailed changelog for the providers will be published in the documentation > after the > RC candidates are released. > > You can find the RC packages in PyPI following these links: > > https://pypi.org/project/apache-airflow-providers-airbyte/4.0.0rc1/ > https://pypi.org/project/apache-airflow-providers-alibaba/2.9.1rc1/ > https://pypi.org/project/apache-airflow-providers-amazon/8.29.0rc1/ > https://pypi.org/project/apache-airflow-providers-apache-flink/1.5.1rc1/ > https://pypi.org/project/apache-airflow-providers-apache-hdfs/4.5.1rc1/ > https://pypi.org/project/apache-airflow-providers-apache-impala/1.5.1rc1/ > https://pypi.org/project/apache-airflow-providers-apache-livy/3.9.1rc1/ > https://pypi.org/project/apache-airflow-providers-apache-spark/4.11.0rc1/ > https://pypi.org/project/apache-airflow-providers-celery/3.8.2rc1/ > https://pypi.org/project/apache-airflow-providers-cloudant/4.0.0rc1/ > https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/8.4.2rc1/ > https://pypi.org/project/apache-airflow-providers-common-io/1.4.1rc1/ > https://pypi.org/project/apache-airflow-providers-common-sql/1.17.0rc1/ > https://pypi.org/project/apache-airflow-providers-databricks/6.10.0rc1/ > https://pypi.org/project/apache-airflow-providers-datadog/3.7.1rc1/ > https://pypi.org/project/apache-airflow-providers-dbt-cloud/3.10.1rc1/ > https://pypi.org/project/apache-airflow-providers-docker/3.14.0rc1/ > https://pypi.org/project/apache-airflow-providers-elasticsearch/5.5.1rc1/ > https://pypi.org/project/apache-airflow-providers-fab/1.4.0rc1/ > https://pypi.org/project/apache-airflow-providers-ftp/3.11.1rc1/ > https://pypi.org/project/apache-airflow-providers-google/10.23.0rc1/ > https://pypi.org/project/apache-airflow-providers-http/4.13.1rc1/ > https://pypi.org/project/apache-airflow-providers-influxdb/2.7.1rc1/ > https://pypi.org/project/apache-airflow-providers-jdbc/4.5.1rc1/ > https://pypi.org/project/apache-airflow-providers-jenkins/3.7.1rc1/ > > https://pypi.org/project/apache-airflow-providers-microsoft-azure/10.5.0rc1/ > https://pypi.org/project/apache-airflow-providers-microsoft-mssql/3.9.1rc1/ > https://pypi.org/project/apache-airflow-providers-mongo/4.2.1rc1/ > https://pypi.org/project/apache-airflow-providers-mysql/5.7.1rc1/ > https://pypi.org/project/apache-airflow-providers-odbc/4.7.1rc1/ > https://pypi.org/project/apache-airflow-providers-openai/1.4.0rc1/ > https://pypi.org/project/apache-airflow-providers-openlineage/1.12.0rc1/ > https://pypi.org/project/apache-airflow-providers-papermill/3.8.1rc1/ > https://pypi.org/project/apache-airflow-providers-postgres/5.13.0rc1/ > https://pypi.org/project/apache-airflow-providers-sftp/4.11.1rc1/ > https://pypi.org/project/apache-airflow-providers-snowflake/5.7.1rc1/ > https://pypi.org/project/apache-airflow-providers-tableau/4.6.1rc1/ > > Cheers, > Elad Kalif >
Re: [VOTE] Release Airflow 2.10.1 from 2.10.1rc1
+1 non binding I managed to test my changes #41783 <https://github.com/apache/airflow/pull/41783>, #41680 <https://github.com/apache/airflow/pull/41680>, #41694 <https://github.com/apache/airflow/pull/41694>, #41890 <https://github.com/apache/airflow/pull/41890> and all of them work as expected. I also tested a few generic DAGs and they all seem to work well. One thing I wanted to bring up however was that, for example #41890 <https://github.com/apache/airflow/pull/41890> - I am the author for the original PR and was cherry picked by Jarek, but I wasn't tagged in the 2.10.1RC testing Github issue, so it is easy to overlook it. I propose that we add a note to include the author in the "description" field when merging the PR to the Airflow3 readme doc that Elad proposes here https://github.com/apache/airflow/pull/41457 to avoid such issues. Thanks & Regards, Amogh Desai On Tue, Sep 3, 2024 at 7:57 AM Utkarsh Sharma wrote: > Hey fellow Airflowers, > > I have cut Airflow 2.10.1rc1. This email is calling for a vote on the > release, which will last at least 72 hours, from Tuesday, September 3, > 2024, at 2:30 AM UTC until Friday, September 6, 2024, at 2:30 AM UTC > < > https://www.timeanddate.com/worldclock/fixedtime.html?msg=8&iso=20240906T0230&p1=1440 > >, > and until 3 binding +1 votes have been received. > > The status of testing of the release is kept at > https://github.com/apache/airflow/issues/41956 > > Consider this my (non-binding) +1. As I’m not a member of the PMC, Ephraim > signed the distribution. > > Airflow 2.10.1rc1 is available at: > https://dist.apache.org/repos/dist/dev/airflow/2.10.1rc1/ > > *apache-airflow-2.10.1-source.tar.gz* is a source release that comes with > INSTALL instructions. > *apache-airflow-2.10.1.tar.gz* is the binary Python "sdist" release. > *apache_airflow-2.10.1-py3-none-any.whl* is the binary Python wheel > "binary" release. > > Public keys are available at: > https://dist.apache.org/repos/dist/release/airflow/KEYS > > Please vote accordingly: > > [ ] +1 approve > [ ] +0 no opinion > [ ] -1 disapprove with the reason > > Only votes from PMC members are binding, but all members of the community > are encouraged to test the release and vote with "(non-binding)". > > The test procedure for PMC members is described in: > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-pmc-members > > The test procedure for contributors and members of the community who would > like to test this RC is described in: > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-contributors > > > Please note that the version number excludes the `rcX` string, so it's now > simply 2.10.1. This will allow us to rename the artifact without modifying > the artifact checksums when we actually release it. > > Release Notes: > https://github.com/apache/airflow/blob/2.10.1rc1/RELEASE_NOTES.rst > > For information on what goes into a release please see: > > https://github.com/apache/airflow/blob/main/dev/WHAT_GOES_INTO_THE_NEXT_RELEASE.md > > *Changes since 2.10.0:* > > *Bugs:* > - Handle Example dags case when checking for missing files (#41874) > - Fix logout link in "no roles" error page (#41845) > - Set end_date and duration for triggers completed with end_from_trigger as > True. (#41834) > - DAGs are not marked as stale if the dags folder change (#41829) > - Fix compatibility with FAB provider versions <1.3.0 (#41809) > - Don't Fail LocalTaskJob on heartbeat (#41810) > - Remove deprecation warning for cgitb in Plugins Manager (#41793) > - Fix log for notifier(instance) without __name__ (#41699) > - Splitting syspath preparation into stages (#41694) > - Adding url sanitization for extra links (#41680) > - Fix InletEventsAccessors type stub (#41607) > - Fix UI rendering when XCom is INT, FLOAT, BOOL or NULL (#41605) > - Fix try selector refresh (#41503) > - Incorrect try number subtraction producing invalid span id for OTEL > airflow (#41535) > - Add WebEncoder for trigger page rendering to avoid render failure > (#41485) > - Adding ``tojson`` filter to example_inlet_event_extra example dag > (#41890) > - Add backward compatibility check for executors that don't inherit > BaseExecutor (#41927) > > > *Miscellaneous:* > - Bump webpack from 5.76.0 to 5.94.0 in /airflow/www (#41879) > - Adding rel property to hyperlinks in logs (#41783) > - Field Deletion Warning when editing Connections (#41504) > - Make Scarf usage reporting in major+minor versions and counters in > buckets (#41900) > - Lower down universal-pathlib minimum to 0.2.2 (#41943) > - Protect against None components of universal pathlib xcom backend > (#41938) > > *Doc Only Change:* > - Remove Debian bullseye support (#41569) > - Add an example for auth with ``keycloak`` (#41791) > > Cheers, > Utkarsh Sharma >
Re: [VOTE] August 2024 PR of the Month
+1 for #41390 It takes courage to do something like this! Thanks & Regards, Amogh Desai On Thu, Aug 29, 2024 at 7:24 AM Vishnu Chilukoori wrote: > +1 for 41390 (Non-binding) > > > -- > Regards, > Vishnu. Chilukoori > > > On Wed, Aug 28, 2024, 18:51 Wei Lee wrote: > > > +1 for #41390 > > > > Best, > > Wei > > > > > On Aug 29, 2024, at 2:32 AM, Vishnu Chilukoori < > > vish.chiluko...@gmail.com> wrote: > > > > > > +1 for #41390 (Non-binding) > > > > > > > > > -- > > > Regards, > > > Vishnu C. > > > > > > On Wed, Aug 28, 2024 at 8:51 AM Buğra Öztürk > > > wrote: > > > > > >> +1 for #41390 > > >> > > >> On Wed, 28 Aug 2024, 15:10 Brent Bovenzi, > > > >> wrote: > > >> > > >>> +1 for #41390 > > >>> > > >>> On Wed, Aug 28, 2024 at 5:03 AM Pierre Jeambrun < > pierrejb...@gmail.com > > > > > >>> wrote: > > >>> > > >>>> +1 for PR #41390 > > >>>> > > >>>> On Tue, Aug 27, 2024 at 10:59 PM Scheffler Jens (XC-AS/EAE-ADA-T) > > >>>> wrote: > > >>>> > > >>>>> +1 binding for 41390 - subdags gone is a great step forward! > > >>>>> > > >>>>> Sent from Outlook for iOS<https://aka.ms/o0ukef> > > >>>>> > > >>>>> From: Briana Okyere > > >>>>> Sent: Tuesday, August 27, 2024 10:40:07 PM > > >>>>> To: dev@airflow.apache.org > > >>>>> Subject: [VOTE] August 2024 PR of the Month > > >>>>> > > >>>>> Hey All, > > >>>>> > > >>>>> It’s once again time to vote for the PR of the Month! > > >>>>> > > >>>>> With the help of the `get_important_pr_candidates` script in > > >> dev/stats, > > >>>>> we've identified the following candidates: > > >>>>> > > >>>>> PR #41039: Send context using in venv operator < > > >>>>> > > >>>>> > > >>>> > > >>> > > >> > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F41039&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7Cdeed61adb6a74d7625d708dcc6d88d80%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638603880598025351%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=CBk7sNhDIp4RjFrHMuC6QMp8MsQnVNO0zsXPc%2BCxa0g%3D&reserved=0 > > >>>>>> <https://github.com/apache/airflow/pull/41039> > > >>>>> > > >>>>> PR #41554: feat(providers/openai): support batch api in > > >>>>> hook/operator/trigger < > > >>>>> > > >>>> > > >>> > > >> > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F41554&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7Cdeed61adb6a74d7625d708dcc6d88d80%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638603880598036094%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=e0fR7Y8DBdCwIbQ7ywp%2F0LhDVE5fxg%2BPG1rsQELb04M%3D&reserved=0 > > >>>>> <https://github.com/apache/airflow/pull/41554>> > > >>>>> > > >>>>> PR #40008: Add `CloudRunServiceHook` and > > >>> `CloudRunCreateServiceOperator` > > >>>> < > > >>>>> > > >>>>> > > >>>> > > >>> > > >> > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F40008&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7Cdeed61adb6a74d7625d708dcc6d88d80%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638603880598043192%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=w1MeMWLjCxzqzqqpHTJ3ZmeafBV71t2l%2FuAg4xucH9w%3D&reserved=0 > > >>>>>> <https://github.com/apache/airflow/pull/40008> > > >>>>> > > >>>>> PR #41304: Add incremental export and cross account export > > >>> functionality > > >>>> in > &
Re: [RESULT][VOTE] Airflow Providers - release of August 19, 2024
Small typo for non binding votes. It mentions both as binding Thanks & Regards, Amogh Desai On Thu, 22 Aug 2024 at 3:44 PM, Elad Kalif wrote: > Hello, > > Apache Airflow Providers prepared on August 19, 2024 have been accepted. > Provider openlineage is excluded from this release. > > 4 "+1" binding votes received: > - Elad Kalif > - Jarek Potiuk > - Ash Berlin-Taylor > - Hussein Awala > > > 3 "+1" binding votes received: > - Vincent Beck > - Freddy Demiane > - Amogh Desai > - Rahul Vats > > > > Vote thread: > https://lists.apache.org/thread/jfxnssx45hjqnc12xyo54b190blo5zx7 > > I'll continue with the release process, and the release announcement will > follow shortly. > > Cheers, > Elad Kalif >
Re: [VOTE] New Provider for Core operators/sensor
1. Provider name +1 for standard, core +0.5 for essential/s - it gives me a sense that it is mandatory -1 for common, shared, builtin, primary 2. -1 for placing under common Thanks & Regards, Amogh Desai On Wed, Aug 21, 2024 at 2:10 AM Pankaj Singh wrote: > +1 core > The implementation of certain operators and sensors, such as > TriggerDagRunOperator, ExternalTaskSensor, relies on the core module in a > way, so it makes sense to have the apache-airflow-providers-core for them. > We may keep other operators such as python, bash here. > > +1 (common.time) > There are many -1 votes for this option, and the reasoning that "Common > should be used by at least two other providers" seems valid. However, I > believe it is reasonable to keep sensors and operators related to time in > the common.time module. This approach appears consistent with the common-io > module and other modules within common. > > + 0 (others) > > Thanks > Pankaj > > On Tue, Aug 20, 2024 at 5:24 PM Aritra Basu > wrote: > > > -1 on common as well, +1 to standard/essentials (non-binding) > > -- > > Regards, > > Aritra Basu > > > > On Tue, 20 Aug 2024, 5:03 pm Elad Kalif, wrote: > > > > > > IMHO this vote is about a code change > > > > > > Then we will consider PMCs -1 as veto which disqualifies the specific > > name. > > > Looks like standard is the leading option which got enough +1 and no > > > vetos but lets see how it proceeds till vote time is closed. > > > > > > On Tue, Aug 20, 2024 at 2:11 PM Hussein Awala > wrote: > > > > > > > > Hussein I believe the intent is that the provider comes as one unit > > > with > > > > Airflow (it will be part of the pre-installed providers like: sqlite, > > > HTTP, > > > > ...) so in that spirit is essential. > > > > > > > > What about installing Airflow 3 by default without any provider > > (minimal > > > > version), and adding the current default providers to a new extra > basic > > > or > > > > something else? > > > > > > > > Even if we decide to install it by default as we do with SQLite and > > HTTP > > > > providers, that doesn't make it "essential", where we can use Airflow > > and > > > > create dags without importing from that provider, except if we move > > some > > > of > > > > our base classes to this provider, which I challenged in my previous > > > email. > > > > > > > > > just to clarify PMC voting -1 is considered veto but the rule is > > > applied > > > > to code change, I am not sure what it means for naming > > > > > > > > IMHO this vote is about a code change > > > > > > > > On Tue, Aug 20, 2024 at 7:33 AM Elad Kalif > wrote: > > > > > > > > > +1 binding on essential/essentials, standard, builtin, primary, > > > > > - 0 binding on core, shared, base > > > > > > > > > > -1 binding on common path > > > > > > > > > > > > > > > > > > > > On Tue, Aug 20, 2024 at 12:45 AM Ephraim Anierobi < > > > > > ephraimanier...@gmail.com> > > > > > wrote: > > > > > > > > > > > +1 standard > > > > > > +0 essential/essentials (makes it seem required) > > > > > > > > > > > > If it were available, I would have chosen the "core-add-on" > option. > > > It > > > > > > gives the feeling that it's a provider that complements the > > > > > > core(apache-airflow-providers-core-add-on). > > > > > > > > > > > > - 1 under common > > > > > > > > > > > > On Mon, 19 Aug 2024 at 22:12, Elad Kalif > > wrote: > > > > > > > > > > > > > Hussein I believe the intent is that the provider comes as one > > unit > > > > > with > > > > > > > Airflow (it will be part of the pre-installed providers like: > > > sqlite, > > > > > > http, > > > > > > > ...) > > > > > > > so in that spirit is essential. > > > > > > > > > > > > > > just to clarify PMC voting -1 is considered veto but the rule > is > > > > > applied > > > > > > to > > > > > > > code change, I am not
Re: [DISCUSS] Airflow 2.11 as bridge release
+1 for this idea but I concur with Elad's thought here Thanks & Regards, Amogh Desai On Tue, Aug 20, 2024 at 1:56 PM Ephraim Anierobi wrote: > +1 > > On Tue, 20 Aug 2024 at 06:40, Utkarsh Sharma > wrote: > > > +1 non binding > > > > Thanks, > > Utkarsh Sharma > > > > On Tue, Aug 20, 2024 at 10:58 AM Elad Kalif wrote: > > > > > + 1 > > > but I think we should clarify the mental model of the 2.11 release. > > > > > > I wouldn't say it will have no features. If there are no features then > > why > > > 2.11? it can just be 2.10.x > > > I rather, we agree that it will have features which are relevant for > the > > > migration (for example: introduce new settings that we think can make > > > migration easier etc...) > > > We should leave room for exceptions, if a community member thinks that > a > > > specific feature is crucial for 2.x and can not wait for Airflow 3 I > > would > > > rather give them the option to make their case rather than saying no in > > > advance. I suggest that such cases will have a mailing list discussion > / > > > Airflow 3 call discussion and lazy consensus vote. With clarification > > that > > > the technical effort in getting the feature to 2.11 is on the community > > > member who advocated for it. > > > > > > > > > On Tue, Aug 20, 2024 at 6:56 AM Vishnu Chilukoori < > > > vish.chiluko...@gmail.com> > > > wrote: > > > > > > > +1 non binding. > > > > > > > > -- > > > > Regards, > > > > Vishnu Chilukoori > > > > > > > > On Mon, Aug 19, 2024 at 6:40 PM Vikram Koka > > > > > > > > > wrote: > > > > > > > > > +1 > > > > > > > > > > Vikram > > > > > > > > > > > > > > > On Mon, Aug 19, 2024 at 12:36 PM Mehta, Shubham > > > > > > > > > > > > > > wrote: > > > > > > > > > > > +1 > > > > > > > > > > > > Shubham > > > > > > > > > > > > On 2024-08-19, 11:22 AM, "Michał Modras" > > > > > > > > <mailto:michalmod...@google.com.inva>LID> wrote: > > > > > > > > > > > > > > > > > > CAUTION: This email originated from outside of the organization. > Do > > > not > > > > > > click links or open attachments unless you can confirm the sender > > and > > > > > know > > > > > > the content is safe. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > > > > externe. > > > > > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous > > ne > > > > > pouvez > > > > > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > > > certain > > > > > que > > > > > > le contenu ne présente aucun risque. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > +1 > > > > > > > > > > > > > > > > > > On Mon, Aug 19, 2024 at 4:21 PM Wei Lee > > > > > > > weilee...@gmail.com>> wrote: > > > > > > > > > > > > > > > > > > > +1 > > > > > > > > > > > > > > Best, > > > > > > > Wei > > > > > > > > > > > > > > > On Aug 19, 2024, at 9:59 PM, Pierre Jeambrun < > > > > pierrejb...@gmail.com > > > > > > <mailto:pierrejb...@gmail.com>> > > > > > > > wrote: > > > > > > > > > > > > > > > > +1 > > > > > > > > > > > > > > > > On Mon, Aug 19, 2024 at 3:29 PM Jed Cunningham < > > > > > > jedcunning...@apache.org <mailto:jedcunning...@apache.org> > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > >> +1 > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > > > > > > dev-unsubscr...@airflow.apache.org> > > > > > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > > dev-h...@airflow.apache.org> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > > > > > > > > > > > > > >
Re: [VOTE] Airflow Providers prepared on August 19, 2024
+1 non binding Ran some example dags for https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/8.4.0rc1/ and https://pypi.org/project/apache-airflow-providers-apache-hive/8.2.0rc1/. The results look fine. Thanks & Regards, Amogh Desai On Tue, Aug 20, 2024 at 1:34 PM Freddy Demiane wrote: > +1 (non binding) on Google Providers Package. > > On Tue, Aug 20, 2024 at 7:44 AM Elad Kalif wrote: > > > provider openlineage is excluded from this wave due to bugs discovered. > > please continue the vote excluding this provider > > > > On Mon, Aug 19, 2024 at 5:27 PM Vincent Beck > wrote: > > > > > +1 non binding. All AWS system tests are running successfully against > > > apache-airflow-providers-amazon==8.28.0rc1. You can see the results > here: > > > > > > https://aws-mwaa.github.io/#/open-source/system-tests/version/8def71479a8a4e943f8c39f07793c9fb2cf1fcc6_8.28.0rc1.html > > > > > > On 2024/08/19 11:06:07 Jarek Potiuk wrote: > > > > +1 (binding) - checked reproducibility, licences, checksums, > > signatures, > > > > checked all the changes of mine (mostly dependency upgrades). > > > > > > > > On Mon, Aug 19, 2024 at 10:18 AM Elad Kalif > > wrote: > > > > > > > > > Hey all, > > > > > > > > > > I have just cut the new wave Airflow Providers packages. This email > > is > > > > > calling a vote on the release, > > > > > which will last for 72 hours - which means that it will end on > August > > > 22, > > > > > 2024 08:15 AM UTC and until 3 binding +1 votes have been received. > > > > > > > > > > > > > > > Consider this my (binding) +1. > > > > > > > > > > Note: package tabular was built but not prepared as RC, please > ignore > > > it. > > > > > We will mark this package as to be removed for next release since > it > > > was > > > > > replaced by apache.iceberg provider > > > > > > > > > > Airflow Providers are available at: > > > > > https://dist.apache.org/repos/dist/dev/airflow/providers/ > > > > > > > > > > *apache-airflow-providers--*.tar.gz* are the binary > > > > > Python "sdist" release - they are also official "sources" for the > > > provider > > > > > packages. > > > > > > > > > > *apache_airflow_providers_-*.whl are the binary > > > > > Python "wheel" release. > > > > > > > > > > The test procedure for PMC members is described in > > > > > > > > > > > > > > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md\#verify-the-release-candidate-by-pmc-members > > > > > > > > > > The test procedure for and Contributors who would like to test this > > RC > > > is > > > > > described in: > > > > > > > > > > > > > > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md\#verify-the-release-candidate-by-contributors > > > > > > > > > > > > > > > Public keys are available at: > > > > > https://dist.apache.org/repos/dist/release/airflow/KEYS > > > > > > > > > > Please vote accordingly: > > > > > > > > > > [ ] +1 approve > > > > > [ ] +0 no opinion > > > > > [ ] -1 disapprove with the reason > > > > > > > > > > Only votes from PMC members are binding, but members of the > community > > > are > > > > > encouraged to test the release and vote with "(non-binding)". > > > > > > > > > > Please note that the version number excludes the 'rcX' string. > > > > > This will allow us to rename the artifact without modifying > > > > > the artifact checksums when we actually release. > > > > > > > > > > The status of testing the providers by the community is kept here: > > > > > https://github.com/apache/airflow/issues/41577 > > > > > > > > > > The issue is also the easiest way to see important PRs included in > > the > > > RC > > > > > candidates. > > > > > Detailed changelog for the providers will be published in the > > > documentation > > > > &
Re: [Discussion] Add a new TriggerRule: Never
I like the idea of having a trigger rule of `never`. I have also come across scenarios while migrating some example_dags to system tests that don't work right out of docs (intentional too, it should be good to run by setting up external systems) I guess if we can have the new trigger rule and by default we modify them to run with `never` and they can be configured (using some env or so) to run with actual trigger rules. Somewhat like this: *DEFAULT_TRIGGER_RULE = TriggerRule.NEVER (override this one based on maybe an env)* *with DAG(dag_id='example_dag') as dag:* *task_1 = DummyOperator(task_id='task_1')* *task_2 = DummyOperator(task_id='task_2')* *for task in [task_1, task_2]:* *task.trigger_rule = DEFAULT_TRIGGER_RULE* Thanks & Regards, Amogh Desai On Mon, Aug 19, 2024 at 11:09 PM Jarek Potiuk wrote: > > Example dags are one place we have historically put dags / tasks that go > in > docs but are not run as system tests. Is that not still the case? > > I think from the very beginning when I joined airflow - example dags were > meant to be "runnable" - so when you include core example dags in airflow - > you were supposed to run them in the UI when you enable them. This is for > "core" examples. > > Extending that concept, when the system tests were introduced ,"provider" > examples had the same assumption - that they are "runnable" examples (this > is basically what "system test" is - it's a runnable DAG that requires an > external system to be configured, but should "work" when used as a DAG in > general). > This has been improved via AIP-47 > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-47+New+design+of+Airflow+System+Tests > - where system tests were simplified and you could really have "runnable > examples" to become "system tests" - AIP-47 defined the way how you can get > the example dag and run it as "system test" and what conditions need to be > fulfilled. > Due to that providers do not have separe "examples" folder - they all place > examples in "tests/system". > > In case we have dashboards (google, amazon, LLM, Teradata) - they are > actually run and tested regularly (including for example today's voting for > providers - where Vincent confirmed they are passing with the new > providers. > In case we do not have dashboards, then they are "possibly runnable > manually" (and people use them when they are developing providers as a way > to test e2e if they work in a number of cases). > > So - similarly as core "example_dags" the function of provider > "example_dags" in providers is three-fold: > > * documentation snippets > * complete examples for people to look at > * system tests if you want to test the provider with it (either run > manually or via automated tests) > > The only difference is that the "provider" example dags require the system > tests and some configuration to connect them - but both "core" examples and > "provider" examples have those three functions: docs, complete examples, > and being runnable. > > Possibly the "manually run" example_dags are always runnable as system > tests and have some errors - but usually they are "close to be" - which > should decrease the burden of having to create them when you add or modify > a provider. And those that are regularly executed, they are quite likely > "working" - by just the fact of running them > > Here is a nice readme describing it in contributing guide: > > https://github.com/apache/airflow/blob/main/contributing-docs/testing/system_tests.rst > And more specific documentation on how to run them: > https://github.com/apache/airflow/tree/main/tests/system > > J. > > > > On Mon, Aug 19, 2024 at 6:17 PM Daniel Standish > wrote: > > > Example dags are one place we have historically put dags / tasks that go > in > > docs but are not run as system tests. Is that not still the case? > > > > On Mon, Aug 19, 2024 at 9:01 AM Ferruzzi, Dennis > > wrote: > > > > > Another part I had not considered is that since it's part of the > > hot-loop, > > > then a user-defined callable could very possibly bring the whole thing > > to a > > > grinding stop, either intentionally or not. So yeah, I guess without > > > massive work to somehow decouple that so the whole scheduler loop > didn't > > > freeze waiting for a callable trigger rule, that could be a very bad > > idea. > > > > > > > > > - ferruzzi > > > > > &
Re: System Test Dashboards - Phase Two??
Thanks for the reply, good to have your ongoing support. @Ferruzzi, Dennis Please help in getting them onboarded if needed. Thanks & Regards, Amogh Desai On Mon, Aug 19, 2024 at 11:34 AM Chinthanippu, Satish wrote: > https://lists.apache.org/thread/wnt3pxt3xxgw3fw4vdscd2wkfl13829z > > Hi Team, > > Replying to this email thread from Teradata Provider, > > > To provide some context on how the Teradata providers dashboard functions: > 1. Execute Teradata system tests using the command `pytest > --system=teradata --junitxml=report_test.xml > tests/system/providers/teradata` on the latest version of Airflow and its > providers. > 2. Extract test result metrics from the generated JUnit XML file - > `report_test.xml ` and save these results in CSV format, which will then be > uploaded as an artifact to GitHub. > 3. Create a dashboard based on the CSV data and publish it to a GitHub > page. > We are committed to adopting and following the required JSON > specifications. > > Regards, > Satish >
Re: [DISCUSS] Airflow 2/3 providers versioning support
That looks quite good to me because as per the below quote from you: > Assuming that we release Airflow 3.0.0 in March 2025, if we follow this rule, we will be able to drop Airflow 2 support in providers in March 2026 - so we give about a year to Airflow 2 users to migrate to Airflow 3 - if they will want new features and bug-fixes from new providers. Also it means that we have 1.5 year of supporting both Airflow 2 and Airflow 3 in providers (that for example includes the old Templated Field back-compatibility support assuming that we will want to go ahead with the change) It seems like there still will be 18 months for users to comfortably migrate. Thanks & Regards, Amogh Desai On Fri, Aug 16, 2024 at 7:36 PM Jarek Potiuk wrote: > Yes. Agree with Ash that when we have "sdk" that will be all-but-guaranteed > (or at least intended to be) backwards compatible and no DB access from > workers, or DAG processor, then yes we can change the rules for Airflow 3 > going forward, but Airflow 2 support question remains. > > Basically it boils down to "How long are we going to support Airflow 2 in > providers released while Airflow 3 is already out?". 12 months > after Airflow 3.0 is released is if we use the current rule. > > No strong opinions of that but that looks good. > > J > > > > On Thu, Aug 15, 2024 at 1:57 PM Ash Berlin-Taylor wrote: > > > Going forward with Airflow 3 I also think we should re-consider what we > > version/depend upon from providers. > > > > If we move all providers (and likley also utility functions!) out of > > airflow core then the thing that a given provider needs to depend upon is > > the shared library/provider versions and the "Task SDK” that we are > > introducing with AIP-72, and the version of Airflow core doesn’t matter > > anymore. > > > > This doesn’t take away the need to think about how long we continue to > > support Airflow 2 in our providers, but that going forward I think we > need > > to subtly shift how we think about this. The first “key use case” we > > mention in AIP 72 is this: > > > > > > 1. Ensure that this interface reduces the interaction between the code > > running within the Task and the rest of Airflow. This is to address > > unintended ripple effects from core Airflow changes which has caused > > numerous Airflow upgrade issues, because Task (i.e. DAG) code relied on > > Core Airflow abstractions. This has been a common problem pointed out by > > numerous Airflow users including early adopters. This proposal would > enable > > Airflow users to upgrade Airflow system components (Scheduler, et al), > > without impacting DAG user code. > > > > I don’t yet have a clear proposal of what it should be, but I do want to > > surface this. > > > > -ash > > > > > On 15 Aug 2024, at 12:05, Amogh Desai > wrote: > > > > > > Nice points to ponder upon, Jarek. Both topics are essential for > ensuring > > > that users have a clear path forwars. > > > > > >> We currently guarantee that the minimum Airflow version supported by a > > > provider is the release date of the next minor version, plus 12 months. > > > You’re asking if this should be adjusted for Airflow 3, correct? > > > When we say adjust, what are we looking at? Are we thinking of > extending > > > the 12 month period, to lets say 18 months > > > or something else? > > > > > > Few thoughts that come to my mind when I read this email: > > > - A potential adjustment would be to extend the window to 18 months > post > > > the first Airflow 3 release, > > > allowing users more time to transition without feeling > > > rushed. Communicating this timeline clearly > > > to the community will be key here. We should *emphasize* that while > they > > > extra time for the transition, > > > they will eventually need to migrate to Airflow 3 to access the newer > > > features and bug fixes. > > > - We can introduce a policy (and document it well, maybe in > > > > > > https://github.com/apache/airflow/blob/main/PROVIDERS.rst#upgrading-minimum-supported-version-of-airflow > > > ) > > > where providers are guaranteed to work with all Airflow versions > released > > > within 12 months (or 18) > > > of the provider release. After that, users *should* expect that older > > > providers may not be fully compatible > > > with newer Airflow versions, and any issues arising from this won't be > > > treated as bugs. > > > > > > We should also document t
Re: Airflow debugging story survey
That indeed is a good idea. I got a chance to skim through the initial draft, mostly looks good. I also left a comment suggesting that we finalise the draft and distribute it in person at the AF summit. Thanks & Regards, Amogh Desai On Thu, Aug 8, 2024 at 5:19 PM Kaxil Naik wrote: > One option might be to create a Google Form so that people can leave > responses easily and you can share the editor rights to PMC members so we > can look at the responses too. > > What do you think? > > On Thu, 8 Aug 2024 at 08:45, Dev iL wrote: > > > Hello everyone, > > > > In the context of improving the Airflow debugging experience improving > the > > Airflow debugging experience > > <https://github.com/apache/airflow/issues/40975>, a survey is being > > created > > to identify potential areas of improvement. Currently, this effort is > > taking place in this shared document > > < > > > https://docs.google.com/document/d/1QcQaejbBgpHzAjLHyx9KNG1d1JKdYkWjhrR6XzZ_0tE/edit?usp=sharing > > >. > > The document is open for commenting by anyone, and if you want editor > > access - please send me your google account in a PM on Slack (@Dev-iL). > > >
Re: [DISCUSS] New provider Common.time
+1 for this idea. I like the idea of moving the core operators entirely out to the providers. When it comes to naming, I do not have a strong preference but if we are looking at some alternatives to "common", my suggestions would be to try something like: - "shared": this will make some sense as we plan to share these across more than one provider(s) - "foundation": so that it can serve as a "base" or foundation for the other provider(s). My issue with "base" is that we do use "base" in some other places, for example, BaseOperator, could be confusing. Thanks & Regards, Amogh Desai On Thu, Aug 15, 2024 at 12:37 PM Wei Lee wrote: > +1 for moving core operators to providers. > > I also agree with Hussein's statement that common should only include > things that are used at least twice in other providers. For this specific > case, would `airflow.providers.time` probably work? > > Best, > Wei > > > On Aug 15, 2024, at 3:46 AM, Hussein Awala wrote: > > > > +1 for moving all the core operators, sensors, and triggers to > new/existing > > providers. > > > >> New provider Common.time > > > > I agree with others who comment on the name. IMHO common providers should > > have abstract (or generic) providers/sensors/triggers used by at least > two > > other providers, which is not the case here, but it's a good opportunity > to > > discuss the policy to create a new common provider. > > > > > > On Wed, Aug 14, 2024 at 8:53 PM Jarek Potiuk wrote: > > > >>> It can mean multiple things, but I’d like it if we used it to mean one > >> thing in Airflow :) > >> > >> It does not have to be common. But Ash, if you have a proposal there > that > >> does not conflict with any other meaning used in Airflow already - > don't be > >> shy and constructively propose it :) > >> > >> On Wed, Aug 14, 2024 at 7:42 PM Ferruzzi, Dennis > >> wrote: > >> > >>> I like it. > >>> > >>> - ferruzzi > >>> > >>> > >>> > >>> From: Ash Berlin-Taylor > >>> Sent: Wednesday, August 14, 2024 7:50 AM > >>> To: dev@airflow.apache.org > >>> Subject: RE: [EXT] [DISCUSS] New provider Common.time > >>> > >>> CAUTION: This email originated from outside of the organization. Do not > >>> click links or open attachments unless you can confirm the sender and > >> know > >>> the content is safe. > >>> > >>> > >>> > >>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > externe. > >>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > >> pouvez > >>> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain > >> que > >>> le contenu ne présente aucun risque. > >>> > >>> > >>> > >>>> On 14 Aug 2024, at 15:11, Vincent Beck wrote: > >>>> > >>>> "common" can refer to "common use cases" or "common usage" which makes > >>> sense (at least to me). > >>> > >>> It can mean multiple things, but I’d like it if we used it to mean one > >>> thing in Airflow :) > >>> > >> > > > - > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
Re: Need more people engaged in our CI / infrastructure
I was away for the last few days too, and couldn't pay much attention to my email and Github. I am looking forward to Hussein's ppt in the next dev call, so that I can help in some of these areas as well. I have a decent knowledge of our CI and tests. Thanks & Regards, Amogh Desai On Wed, Aug 14, 2024 at 3:43 AM Jarek Potiuk wrote: > Fantastic ! That might help us to speed up our builds a lot - and we need > it looking at the number of PRs we keep on receiving with Airflow 3. Also > Self-hosted runners from ASF are not helping - thay are pretty unstable and > ARM runners are on ubuntu 20.04 which does not have ARM Python, and the > issue is not solved for a long time already > https://issues.apache.org/jira/projects/INFRA/issues/INFRA-25990 > > On Wed, Aug 14, 2024 at 12:05 AM Hussein Awala wrote: > > > I can prepare a short presentation for the next dev call (22 Aug) to > > explain the architecture we tried to implement, why we chose it, and what > > the blockers are (mainly related to the infrastructure team). > > > > Also, I am still interested in completing this work. > > > > On Mon, Aug 12, 2024 at 3:58 AM Kaxil Naik wrote: > > > > > Added as agenda item for the next dev call (22 Aug) > > > > > > On Mon, 12 Aug 2024 at 00:25, Jarek Potiuk wrote: > > > > > > > I will let Hussein (if he has time) to share some more details :). > > > > > > > > Generally speaking we are using Github Actions as CI - so what we > > > > **really** need is auto-scaling k8S cluster where K8S Controller is > > > deployd > > > > and connected (via ASF infrastructure's Github APP) > > > > https://github.com/actions/actions-runner-controller. The last state > > we > > > > had > > > > - as far as I remember - Hussein already had a (Terraform?) > deployment > > > for > > > > it and it generally was depending on the ASF/ Infra authorisation / > > > setup. > > > > Then some fine-tuning / labels (small/medium/big instances) to > > > > define/findalize and extend it to be able to also run ARM instances. > > > > > > > > J. > > > > > > > > On Mon, Aug 12, 2024 at 1:10 AM Neil wrote: > > > > > > > > > I have solid AWS and EKS knowledge, I'd offer my help if my skills > > are > > > > > applicable. > > > > > Which Infrastructure as Code and CI/CD frameworks are being > utilized > > > for > > > > > the testing Terraform Cloudformation? > > > > > I've had good experiences with Pulumi python. > > > > > Have you considered using EFS to handle the disk space needs? > > > > > > > > > > On Sun, Aug 11, 2024 at 6:18 PM Jarek Potiuk > > wrote: > > > > > > > > > > > Hello here, > > > > > > > > > > > > It would be great to have someone (or better two people) to get > > > engaged > > > > > in > > > > > > our test infrastructure work - this will improve everyone's > > > experience. > > > > > I > > > > > > **REALLY** think we should have other people that have engaged so > > > far, > > > > so > > > > > > that we can decrease the bus factor we have for our > infrastructure. > > > > > > > > > > > > Just after I was away for 5 days and without too much > connectivity > > > our > > > > > main > > > > > > was broken (lack of disk space for constraints generation) and > some > > > > mypy > > > > > > checks were failing for the last few days. > > > > > > > > > > > > This is unsustainable and we need to find people who will know > and > > be > > > > > able > > > > > > to fix this infrastructure. > > > > > > > > > > > > *Early warning* - I am planning 3 weeks holidays after Airflow > > > Summit - > > > > > and > > > > > > I won't be looking at my email/github during those days, which > > means > > > > that > > > > > > whoever will be working on Airflow 3 might be severely impacted > by > > > some > > > > > of > > > > > > those failures. > > > > > > > > > > > > Just to remind - until we have the k8S controller set up on our > AWS > > > > > > account and connected to our repo - we won't be able to use the > > > > credits > > > > > > that we got recently. So this is a good start. > > > > > > > > > > > > I created a high-level issue for that > > > > > > https://github.com/apache/airflow/issues/41388 and it waits for > > some > > > > > > volunteers to pick it up. It's a very important thing to do - we > > can > > > > > speed > > > > > > up many parts of our builds (for example release preparation - > but > > > also > > > > > > likely most of our tests) up to 4 times, which means that a lot > of > > > time > > > > > can > > > > > > be saved for waiting. > > > > > > > > > > > > Kaxil - I propose we should add a point at the next devcall - and > > > keep > > > > it > > > > > > as an unresolved Airflow 3 issue until it is well, unresolved. > > > > > > > > > > > > J. > > > > > > > > > > > > > > > > > > > > >
Re: [DISCUSS] Airflow 2/3 providers versioning support
Nice points to ponder upon, Jarek. Both topics are essential for ensuring that users have a clear path forwars. > We currently guarantee that the minimum Airflow version supported by a provider is the release date of the next minor version, plus 12 months. You’re asking if this should be adjusted for Airflow 3, correct? When we say adjust, what are we looking at? Are we thinking of extending the 12 month period, to lets say 18 months or something else? Few thoughts that come to my mind when I read this email: - A potential adjustment would be to extend the window to 18 months post the first Airflow 3 release, allowing users more time to transition without feeling rushed. Communicating this timeline clearly to the community will be key here. We should *emphasize* that while they extra time for the transition, they will eventually need to migrate to Airflow 3 to access the newer features and bug fixes. - We can introduce a policy (and document it well, maybe in https://github.com/apache/airflow/blob/main/PROVIDERS.rst#upgrading-minimum-supported-version-of-airflow ) where providers are guaranteed to work with all Airflow versions released within 12 months (or 18) of the provider release. After that, users *should* expect that older providers may not be fully compatible with newer Airflow versions, and any issues arising from this won't be treated as bugs. We should also document these rules clearly to provide users with the context and rationale behind these decisions. Thanks & Regards, Amogh Desai On Wed, Aug 14, 2024 at 3:37 PM Jarek Potiuk wrote: > > We currently guarantee that the minimum Airflow version supported by a > provider is the release date of the next minor version, plus 12 months. > You’re asking if this should be adjusted for Airflow 3, correct? > > Yes > > > Are you asking for guarantees the other way around, so Airflow version -> > guaranteed provider package min version support? > > Yes. > > On Wed, Aug 14, 2024 at 12:04 PM Bas Harenslak > wrote: > > > Sorry I’m a bit lost in the long text. Is my understanding of the 2 > > questions correct? > > > > We currently guarantee that the minimum Airflow version supported by a > > provider is the release date of the next minor version, plus 12 months. > > You’re asking if this should be adjusted for Airflow 3, correct? > > Are you asking for guarantees the other way around, so Airflow version -> > > guaranteed provider package min version support? > > > > Bas > > > > > On 14 Aug 2024, at 11:22, Jarek Potiuk wrote: > > > > > > Hello everyone, > > > > > > I have two important topics to discuss regarding Provider <> Airflow > > > version support rules. > > > > > > *1) What is the "min Airflow version" support while we are > transitioning > > to > > > Airflow 3? * > > > > > > So far we had the [1] rule: > > > > > >> The default support timespan for the minimum version of Airflow (there > > > could be justified exceptions) is that we increase the minimum Airflow > > > version to the next MINOR release, when 12 months passed since the > first > > > release for the MINOR version of Airflow. > > > > > >> For example this means that by default we upgrade the minimum version > of > > > Airflow supported by providers to 2.8.0 in the first Provider's release > > > after 18th of December 2024. 18th of December 2023 is the date when the > > > first PATCHLEVEL of 2.8 (2.8.0) has been released. > > > > > > The next cutoff time (2.9.0 + 12 months) will be in April 2025, And the > > > next one - 2.10.0 - assuming we release 2.10.0 in August - for min > > airflow > > > version = 2.10.0 will be in August 2025. If we have "bridge" 2.11.0 > > release > > > (very likely) - say in December, then min_version = 2.11.0 will be in > > > December 2025. > > > > > > Assuming that we release Airflow 3.0.0 in March 2025, if we follow this > > > rule, we will be able to drop Airflow 2 support in providers in March > > 2026 > > > - so we give about a year to Airflow 2 users to migrate to Airflow 3 - > > if > > > they will want new features and bug-fixes from new providers. Also it > > means > > > that we have 1.5 year of supporting both Airflow 2 and Airflow 3 in > > > providers (that for example includes the old Templated Field > > > back-compatibility support assuming that we will want to go ahead with > > the > > > change) > > > > > > That provides (as was the original idea of this rolling min-version > > >
Re: [ANNOUNCE] New PMC member: Jens Schaffler
Many Congratulations, Jens! Very well deserved! Thanks & Regards, Amogh Desai On Tue, Aug 13, 2024 at 6:55 AM Neil wrote: > Congrats Jens!! > > On Mon, Aug 12, 2024 at 8:29 PM Pierre Jeambrun > wrote: > > > Congratulations Jens! > > > > Le lun. 12 août 2024 à 20:53, Ferruzzi, Dennis > > > > > a écrit : > > > > > Congratulations Jens, well deserved. > > > > > > > > > - ferruzzi > > > > > > > > > > > > From: Mehta, Shubham > > > Sent: Wednesday, August 7, 2024 1:39 PM > > > To: dev@airflow.apache.org > > > Subject: RE: [EXT] [ANNOUNCE] New PMC member: Jens Schaffler > > > > > > CAUTION: This email originated from outside of the organization. Do not > > > click links or open attachments unless you can confirm the sender and > > know > > > the content is safe. > > > > > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > externe. > > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > > pouvez > > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain > > que > > > le contenu ne présente aucun risque. > > > > > > > > > > > > Many congratulations, Jens!! Thankful for your continued efforts and > > > amazing work. > > > > > > Shubham > > > > > > On 2024-08-07, 11:42 AM, "Bishundeo, Rajeshwar" > > > > <mailto:rbish...@amazon.com.inva>LID> wrote: > > > > > > > > > CAUTION: This email originated from outside of the organization. Do not > > > click links or open attachments unless you can confirm the sender and > > know > > > the content is safe. > > > > > > > > > > > > > > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > externe. > > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > > pouvez > > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain > > que > > > le contenu ne présente aucun risque. > > > > > > > > > > > > > > > > > > > > > Congratulations Jens!! Very well deserved for your great work and > > > dedication to the continued success of Airflow!! > > > > > > > > > -- Rajesh > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 2024-08-07, 1:56 PM, "Shahar Epstein" > > sha...@apache.org> <mailto:sha...@apache.org <mailto:sha...@apache.org > > >>> > > > wrote: > > > > > > > > > > > > > > > CAUTION: This email originated from outside of the organization. Do not > > > click links or open attachments unless you can confirm the sender and > > know > > > the content is safe. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > externe. > > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > > pouvez > > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain > > que > > > le contenu ne présente aucun risque. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Congratulations Jens! You definitely deserve it :) > > > > > > > > > > > > > > > On 2024/08/06 07:50:44 Jarek Potiuk wrote: > > > > The Project Management Committee (PMC) for Apache [PROJECT] > > > > has invited Jens to become a PMC member and we are pleased > > > > to announce that they have accepted. > > > > > > > > Jens has contributed for a long time already in quite a few areas > > > > > > > > > > https://github.com/apache/airflow/pulls?q=is%3Apr+author%3Ajscheffl+is%3Aclosed > > > < > > > > > > https://github.com/apache/airflow/pulls?q=is%3Apr+author%3Ajscheffl+is%3Aclosed > > > > > > < > > > > > > https://github.com/a
Re: The Airflow main branch is now Airflow 3
Awesome, thanks for summarising nicely! Thanks & Regards, Amogh Desai On Mon, Aug 12, 2024 at 5:55 AM Jarek Potiuk wrote: > Indeed exciting!. I already cherry-picked a few changes, following the > process. > > J. > > > On Fri, Aug 9, 2024 at 3:49 PM Kaxil Naik wrote: > > > 🎉 Exciting! > > > > On Fri, 9 Aug 2024 at 14:17, Ephraim Anierobi < > ephraimanier...@apache.org> > > wrote: > > > > > Fellow Airflowers, as agreed in the Airflow 3 dev call, we have > branched > > > off for the Airflow 2.10.0 release, and now Airflow's main branch is > > > Airflow 3! > > > > > > Airflow 2.10.0rc1 will be created in a few hours from v2-10-stable > > branch. > > > If rc1 does not succeed and if we need to create rc2, the bugfixes > should > > > be created to v2-10-test & main branch both if there are merge > conflicts. > > > If there aren’t merge conflicts, feel free to target either branch & > the > > > release manager will take care of making sure that those are both in > main > > > and 2.10.0 – but please notify the release manager. > > > > > > After the Airflow 2.10.0 release, PRs targeting Airflow 2 and not > Airflow > > > 3 should be made against v2-10-test (e.g. code to integrate Fab). > > > > > > For Contributors: > > > All bug fixes after 2.10.0 release will target Airflow 3. We will make > > the > > > best effort to make them available in 2.10.x, but if somebody wants to > > > guarantee that a fix is included in 2.10.x, they need to raise the PR > > > explicitly to the v2-10-test branch too. > > > > > > For Committers: > > > When merging bugfix PRs to the main branch, the committers should also > > try > > > to cherry-pick it to v2-10-test branch. If there are merge conflicts, > the > > > committer should add a comment on the original PR, informing the author > > and > > > asking them to raise a separate PR against v2-10-test branch. If this > > > doesn't happen, there is no guarantee that the PR will be part of 2.10. > > > > > > We will do a POC separately to evaluate cherrypicker [1] to make this > > > process easier in the coming weeks and report back. > > > > > > All of the above has also been included in our Contributing Guide in > this > > > PR [2]. > > > > > > Start merging those Airflow 3 PRs now! :) > > > > > > [1]: https://pypi.org/project/cherry_picker/ > > > [2]: https://github.com/apache/airflow/pull/41354 > > > > > > - > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > >
Re: [RESULT][VOTE] AIP-67 Multi-team deployment of Airflow Components
Late to the game here, but my vote stays with a strong +1 Thanks & Regards, Amogh Desai On Sun, Aug 11, 2024 at 11:47 PM Jarek Potiuk wrote: > The vote for AIP-67 has passed. > > AIP-67 > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components > > Discussion thread: > https://lists.apache.org/thread/d7qdf6zbv9t98nkmqpzdlb76lsnc89vq > > Vote thread: > https://lists.apache.org/thread/kvc0stsbm25fngmld3npv2xcpxz3o2kt > > Binding +1 votes (9): > - Jarek Potiuk > - Jens Scheffler > - Kaxil Naik > - Shubham Mehta > - Vincent Beck > - Vikram Koka > - Shahar Epstein > - Oliveira, Niko > - Hussein Awala > > Non-binding +1 votes: > > - Buğra Öztürk > - Rajeshwar Bishundeo > - Guangyang Li > - Jorrick Sleijster > - Pavankumar Gopidesu - Vishu Chilukoori > > J. >
Re: Airflow 3 AIP Gold Rush! -- Great job everyone
Great job! Seeing a flood of emails (after coming back to the dev mail post a vacation) was scary at first, but it's awesome to see the kind of effort that was put in by everyone. Kudos to everyone! Glad to be part of such a happening community Thanks & Regards, Amogh Desai On Mon, Aug 5, 2024 at 7:16 PM Kaxil Naik wrote: > Just noticed I had an incomplete sentence :) > > I meant: There is mutual respect between the members since they know that > everyone is working for what's better for the Airflow Community. > > On Fri, 2 Aug 2024 at 22:16, Kaxil Naik wrote: > > > Hi team, > > > > Before we head for the weekend, I wanted to take a moment to express my > > gratitude and pride in being part of this incredible community > > > > I am so honoured, proud and happy! Over the past few months, and > > especially in the last two weeks, I’ve witnessed our brilliant community > > members come together to draft AIPs, review each other’s work, provide > > constructive feedback, and align on delivering what’s best for Airflow > 3.0 > > despite occasional disagreements. There is mutual respect between the > > members since they know > > > > This has to be the most activity I have seen on our dev mailing list, > even > > more than 2.0. Kudos to all of you!!. In the last 10 days, we have had 12 > > AIPs put to a VOTE, and in the last 24 hours alone, 5 AIPs are being > VOTED > > on. The context needed for all these amazing AIPs is immense, so a > > huge shoutout to all the devs who dedicated time to read, understand and > > provide feedback on these discussions. All of these while still working > on > > Airflow 2.10, providers, helm chart releases & planning for Airflow > Summit > > -- that's incredible. > > > > A huge thank you to all Stakeholders (managed Airflow service providers) > > who are dedicating engineers for Airflow 3. Special shoutout to the: > > - Amazon team for taking on Event-driven scheduling and getting the > Hybrid > > Executor along the line for 2.10 release. > > - Astronomer team with 12 AIPs !! (putting my OSS hat on). Even excluding > > the sub-AIPs, there are still 7 big AIPs. > > - Google team for starting the discussion for 3.1 with Extendable DAG > > parsing controls AIP > > > > And a special mention to some of our passionate individuals: Jarek, Jens, > > Buğra, and Elad for leading some of the big workstreams. Airflow remains > a > > leader in this space because of individuals like you. > > > > We have 15-16 AIPs targeted for Airflow 3 [1] !! If we are able to get > all > > of these, it will be a freaking awesome release with a huge impact on > > Airflow and the larger Data Engineering community. The real fun starts > now > > as we shift gears from planning to execution as VOTEs start getting > > concluded. > > > > The next few months will be game time and I am sure we will have a fun > and > > fulfilling ride. Please enjoy every part of it. > > > > Have a great weekend. > > > > Regards, > > Kaxil > > > > > > [1]: > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Workstreams > > > > > > >
Re: [RESULT][VOTE] July 2024 PR of the Month
Congratulations Niko! Thanks & Regards, Amogh Desai On Sat, Aug 3, 2024 at 12:01 AM Bishundeo, Rajeshwar wrote: > Awesome job Niko!! > > -- Rajesh > > > > > > > On 2024-08-02, 1:51 PM, "Buğra Öztürk" ozturkbugr...@gmail.com>> wrote: > > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que > le contenu ne présente aucun risque. > > > > > > > Congrats, Niko! 🥳 > > > On Fri, Aug 2, 2024 at 3:16 AM Jarek Potiuk ja...@potiuk.com>> wrote: > > > > Yep. Grats! > > > > On Fri, Aug 2, 2024 at 2:52 AM Pavankumar Gopidesu < > > gopidesupa...@gmail.com <mailto:gopidesupa...@gmail.com>> > > wrote: > > > > > Congratulations Niko 🎉🎉. > > > > > > Regards, > > > Pavan > > > > > > On Fri, Aug 2, 2024 at 1:45 AM Vikram Koka <mailto:vik...@astronomer.io.inva>lid > > > > > > wrote: > > > > > > > Very cool! Congratulations Niko! > > > > > > > > > > > > On Thu, Aug 1, 2024 at 5:42 PM Kaxil Naik <mailto:kaxiln...@gmail.com>> wrote: > > > > > > > > > Congrats Niko > > > > > > > > > > On Fri, 2 Aug 2024 at 00:00, Ferruzzi, Dennis > > > > mailto:ferru...@amazon.com.inva>lid > > > > > > > > > > > wrote: > > > > > > > > > > > Congrats Niko, and almost Vincent :P > > > > > > > > > > > > - ferruzzi > > > > > > > > > > > > > > > > > > > > > > > > From: Briana Okyere briana.oky...@astronomer.io.inva>LID> > > > > > > Sent: Wednesday, July 31, 2024 3:11 PM > > > > > > To: dev@airflow.apache.org <mailto:dev@airflow.apache.org> > > > > > > Subject: RE: [EXT] [RESULT][VOTE] July 2024 PR of the Month > > > > > > > > > > > > CAUTION: This email originated from outside of the organization. > Do > > > not > > > > > > click links or open attachments unless you can confirm the sender > > and > > > > > know > > > > > > the content is safe. > > > > > > > > > > > > > > > > > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > > > > externe. > > > > > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous > > ne > > > > > pouvez > > > > > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > > > certain > > > > > que > > > > > > le contenu ne présente aucun risque. > > > > > > > > > > > > > > > > > > > > > > > > Hey All, > > > > > > > > > > > > My apologies- I mixed up the contender! Niko Oliveira is the > winner > > > of > > > > PR > > > > > > of the Month for PR #40017 > > > > > > > > > > > > Congratulations Niko! > > > > > > > > > > > > On Wed, Jul 31, 2024 at 2:12 PM Briana Okyere < > > > > > briana.oky...@astronomer.io <mailto:briana.oky...@astronomer.io> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hey All, > > > > > > > > > > > > > > The results are in! Congratulations to Vincent Beck for winning > > PR > > > of > > > > > the > > > > > > > Month for PR #40017: AIP-61 - Scheduler job: main loop and > event > > > > > > > processing for multi executors < > > > > > > > https://github.com/apache/airflow/pull/40017> < > https://github.com/apache/airflow/pull/40017>;> > > > > > > > > > > > > > > PR #40017 will be featured in the July 2024 Newsletter, and > thank > > > you > > > > > to > > > > > > > all those who made contributions this month. > > > > > > > > > > > > > > -- > > > > > > > Briana Okyere > > > > > > > Community Manager > > > > > > > Astronomer > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > Bugra Ozturk > > > > > - > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org >
Re: [DISCUSS] Approaches to bugfixes for 2.10 after main becomes 3.0
Late to the discussion here but I agree that we should document the process (if we haven't already). I also think that the automatic cherry picker is worth exploring. I also came across one of the github action plugins for backporting and it looks pretty good: https://github.com/marketplace/actions/backport-action. This action allows us to do the same stuff as TP suggested, along with label-wise filtering. Thanks & Regards, Amogh Desai On Sun, Jul 28, 2024 at 6:18 PM Kaxil Naik wrote: > This is also what we did during Airflow 2 release and worked well for us > where some changes were in Airflow 1 branch only since it had Flask Admin > vs FAB, and their were no concept of Provider in v1, Python 2 support etc > > > On Sun, 28 Jul 2024 at 13:45, Kaxil Naik wrote: > > > @Jens: yup precisely, I do expect changes to only v2 branch. This is also > > what we did during v2 release and worked well for us. > > > > On Sun, 28 Jul 2024 at 11:59, Ephraim Anierobi < > ephraimanier...@gmail.com> > > wrote: > > > >> From how the cherrypicker works, it looks like something we need. We > could > >> have a bot that adds a "review comment" that must be resolved before > >> merging. The review comment can ask for a label to be added to the PR if > >> the PR should be backported to Airflow 2. Whoever is merging the PR can > >> determine if it should be backported or not and add the appropriate > label. > >> > >> > >> On Fri, 26 Jul 2024 at 20:14, Jarek Potiuk wrote: > >> > >> > Same comment as Jens. There will be things removed in Airflow-3 that > >> will > >> > need only-Airflow-2 fixes. > >> > > >> > I like the idea of automated cherry-picker, but it only make sense if > we > >> > pay attention to which PRs are going to be cherry-picked and do so at > >> the > >> > moment of merging - we are going to have many changes that will not be > >> > supposed to be cherry-picked, so we need to have "opt-in" approach > >> where we > >> > explicitly say "this PR should be cherry-picked to v2". > >> > > >> > I think it's absolutely worth doing it- cherry-picking in sequential > >> order > >> > as things are merged makes a lot of sense - but it needs some > >> deliberation > >> > and decision making at the moment of merging (should this one be > >> > cherry-picked?) - and assigning responsibility to who should be making > >> that > >> > decision. But 100% I think it makes sense to do it at "merge" time > >> rather > >> > than release time, because rather than having to make 100 decisions > by a > >> > single person at release time, 100 people might make separate > decisions > >> > (and follow-up when cherry-pick fails) for a long time. > >> > > >> > So technically it's fine, but it needs a well defined process and > (say) > >> > everyone who is merging PRs to pay attention to it. It's more of a > >> social > >> > than technical problem. Not sure if we will be able to make it work > with > >> > our distributed setup. > >> > > >> > My 3 cents. > >> > > >> > J. > >> > > >> > > >> > On Fri, Jul 26, 2024 at 8:45 PM Scheffler Jens (XC-AS/EAE-ADA-T) > >> > wrote: > >> > > >> > > I fully agree. We should document it. > >> > > > >> > > There is one exceptional case. In my view this should be planned and > >> > > documented as exception: if something needs to be fixed that is not > on > >> > main > >> > > anymore (e.g. code to integrate Fab) then a fix pr mist be made > >> against > >> > > v2-10-test branch as it can not be made against main - also assuming > >> that > >> > > such big is not relevant for 3.x line. > >> > > > >> > > Jens > >> > > > >> > > Sent from Outlook for iOS<https://aka.ms/o0ukef> > >> > > > >> > > From: Kaxil Naik > >> > > Sent: Friday, July 26, 2024 7:41 PM > >> > > To: dev@airflow.apache.org > >> > > Subject: Re: [DISCUSS] Approaches to bugfixes for 2.10 after main > >> becomes > >> > > 3.0 > >> > > > >> > > Nice, that is indeed promising > >> > > > >> > > On
Re: [VOTE] AIP-66: DAG Bundles and Parsing
+1 binding I really love the way the AIP has been written to target small pieces instead of doing everything related to versioning together. Thanks & Regards, Amogh Desai On Sun, Jul 21, 2024 at 1:03 AM Aritra Basu wrote: > +1 non-binding > > -- > Regards, > Aritra Basu > > On Sat, 20 Jul 2024, 10:40 pm Igor Kholopov, > > wrote: > > > +1 (non-binding) > > > > On Sat, Jul 20, 2024 at 4:31 PM Jarek Potiuk wrote: > > > > > +1 (binding) > > > > > > On Sat, Jul 20, 2024 at 12:39 PM Rahul Vats > > > wrote: > > > > > > > +1 (non-binding) > > > > Regards, > > > > Rahul Vats > > > > 9953794332 > > > > > > > > > > > > On Sat, 20 Jul 2024 at 02:14, Jed Cunningham < > jedcunning...@apache.org > > > > > > > wrote: > > > > > > > > > I’m calling for a vote on this AIP 66: > > > > > https://cwiki.apache.org/confluence/x/ZIqSEQ > > > > > > > > > > Discussion thread: > > > > > https://lists.apache.org/thread/l8ksl144xd43jfk1wk3kz77t1xgbbq7z > > > > > > > > > > The vote will run for 5 days and last till next Wednesday, > > 2024-07-24 > > > > > 21:00 UTC. > > > > > > > > > > Everyone is encouraged to vote, although only PMC members and > > > Committer's > > > > > votes are considered binding. > > > > > > > > > > This is my +1. > > > > > > > > > > > > > > >
Re: [VOTE] AIP-65: Improve DAG history in UI
+1 binding Glad to see this happening Thanks & Regards, Amogh Desai On Sat, Jul 20, 2024 at 9:59 PM Aritra Basu wrote: > +1 non-binding > -- > Regards, > Aritra Basu > > On Sat, Jul 20, 2024, 8:03 PM Jarek Potiuk wrote: > > > +1 (binding) > > > > J. > > > > > > On Sat, Jul 20, 2024 at 12:15 PM Wei Lee wrote: > > > > > +1 (binding) > > > > > > Best, > > > Wei > > > > > > > On Jul 20, 2024, at 5:18 AM, Vikram Koka > > > > > > wrote: > > > > > > > > Sorry, should have said +1 (binding) > > > > > > > > On Fri, Jul 19, 2024 at 2:18 PM Vikram Koka > > > wrote: > > > > > > > >> +1 > > > >> > > > >> This is possibly the highest voted feature from the Airflow survey. > > > >> > > > >> > > > >> On Fri, Jul 19, 2024 at 1:50 PM Jed Cunningham < > > > jedcunning...@apache.org> > > > >> wrote: > > > >> > > > >>> I’m calling for a vote on AIP 65: > > > >>> https://cwiki.apache.org/confluence/x/T4qSEQ > > > >>> > > > >>> Discussion thread: > > > >>> https://lists.apache.org/thread/vvm43tfchyo92hmf40fqvmq0f5845bjr > > > >>> > > > >>> The vote will run for 5 days and last till next Wednesday, > 2024-07-24 > > > >>> 21:00 > > > >>> UTC. > > > >>> > > > >>> Everyone is encouraged to vote, although only PMC members and > > > Committer's > > > >>> votes are considered binding. > > > >>> > > > >>> This is my +1. > > > >>> > > > >> > > > > > > > > > - > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > >
Re: [PROPOSE] Agree and document Ad-hoc release process for providers
Yeah, that does make sense. However, I was suggesting a screen recorded demo to allow folks get a hang of things rather than following the recording step by step. The rationale here would be to make people familiar with what needs to be done and "where". Thanks & Regards, Amogh Desai On Mon, Jul 22, 2024 at 12:48 PM Jarek Potiuk wrote: > Screen recording is problematic - it has the problem that you need to > re-record it when something changes and it limits flexibility - we had very > good experience so far with describing the steps in the docs and correcting > any mistakes and adding clarification every time someone run it. > > On Wed, Jul 17, 2024 at 10:56 AM Pankaj Koti > wrote: > > > +1 on clarifying the policy. > > > > +1 to Amogh's idea for checking the possibility of a screen > recording/video > > of the steps. If it it'd doable, I definitely would like to be one the > > John(s) > > Jarek is mentioning to test out the possibility of helping the RM. > > > > > > Best regards, > > > > *Pankaj Koti* > > Senior Software Engineer (Airflow OSS Engineering team) > > Location: Pune, Maharashtra, India > > Timezone: Indian Standard Time (IST) > > > > > > On Wed, Jul 17, 2024 at 12:42 PM Amogh Desai > > wrote: > > > > > Late to the party here but +1 on clarifying the policy. > > > > > > I am very much with Elad and Jed here regarding the chaos it could lead > > to > > > so > > > it definitely shouldn't be entirely driven by a non-RM (which is not > even > > > possible). > > > > > > One improvement I can think of is in the announcement we make when we > are > > > about > > > to cut the "RC". We should make it a point (along with instructions) to > > ask > > > stakeholders > > > test out the "early drop" of providers using breeze with clear > > indications > > > on how to and > > > with the "impact", if they don't. The impacts are what Jarek mentioned > > > earlier regarding, > > > but not limited to them waiting for the next wave of providers. > > > > > > Also, I think it will be really beneficial to have a "video" recording > > > (screen recording is good enough) > > > of a RM running the whole process so that interested folks can take up > > that > > > responsibility, if need be. > > > > > > > > > Thanks & Regards, > > > Amogh Desai > > > > > > > > > On Mon, Jul 15, 2024 at 3:18 PM Eugen Kosteev > wrote: > > > > > > > +1 to this > > > > > > > > On Mon, Jul 15, 2024 at 11:24 AM Jarek Potiuk > > wrote: > > > > > > > > > Ok. If there are no more comments - I will now do some more work on > > the > > > > > release process (I am going to finally look at the "Trusted > > Publishing" > > > > > change for PyPI, and i will review and update the processes and > along > > > the > > > > > way will mark and double check if the release process steps can be > > made > > > > by > > > > > non-PMC members/Release manager and find a way to nicely tag them > in > > > the > > > > > release process (I will do it also for other release processes - > just > > > tag > > > > > them). Then I will propose some updates to add the "ad-hoc" process > > > lead > > > > by > > > > > other people and once it's ready, we might have a test/dry run with > > it. > > > > > > > > > > J. > > > > > > > > > > On Thu, Jul 11, 2024 at 10:57 AM Jarek Potiuk > > > wrote: > > > > > > > > > > > Yes. I think we all agree this should be a truly exceptional case > > > with > > > > > > wide impact and one that has no easy workaround (an example of > such > > > > > > issues from the past were when we have not realized that an older > > > > > > version of ads-sdk has been disabled and none of the previous > > > > > > providers would work. Or when the current version of the provider > > > > > > released causes huge data loss (in which case we should yank it > as > > > > > > well) or similar kinds of issues. I think when someone wants to > do > > > it, > > > > > > they should be absolutely convinced that it
Re: [DISCUSS] AIP-67 Multi-team deployment of Airflow components (reloaded)
Nice, thanks for clarifying all this! Now that I read the new proposal, it is adding up to me why certain decisions were made. The decision to separate the "common" part from the "per team" part adds up now. It is a traditional paradigm of separating "control plane" from "compute". Thanks & Regards, Amogh Desai On Mon, Jul 15, 2024 at 8:53 PM Jarek Potiuk wrote: > I got the transcript and chat from the last call (thanks Kaxil!) and it > allowed me to answer a few questions that were asked during my presentation > about AIP-67. I updated the AIP document but here is summary: > > 1) What about Pools (asked by Elad and Jed, Jorrick): I thought about it > and I propose that pools could have (optional) team_id added. This will > allow users to keep common pools (no team_id assigned) and have > team-specific ones. DAG file processor specific for each team will fail DAG > if it tries using a pool that is not common and belonging to "other team". > Also each team will be able to have their own "default_pool" configured. > This will give enough flexibility on 'common vs. team exclusive" use of > pools. > > 2) Isolation for connections (John, Filip, Elad, Kaxil, Amogh, Ash): yes. > That is part of the design. The connections and variables can be accessed > per team - AIP-72 will only provide the tasks with connections that belong > to the team. Ash mentioned OPA (which might be used for that purpose). It's > not defined how exactly it will be implemented in AIP-72, it's not detailed > enough, but it can use the very mechanisms that AIP-72 - by only allowing > "global" connections and "my team" connections to be passed by AIP-72 API > to the task and DAG file processor. > > 3) Whether "Team=deployment" - Igor / Vikram ? -> depends on what you > understand by deployment. i'd say "sub-deployment" - each deployment in a > "multi-team" environment will consist of the "common" part and each team > will have their own part (where configuration and management of such team > deployment parts will be delegated to the team deployment manager). For > example such deployment managers will be able to build and publish the > environment (for example container images) used by team A to run Airflow. > Or change "team" specific configuration. > > 4) "This seems like quite a lot of work to share a scheduler and a web > server. What’s the net benefit of this complexity?" -> Ash, John, Amogh, > Maciej: Yes. I absolutely see it as a valuable option. It reflects > organizational structure and needs of many of our users, where they want to > manage part of the environment, monitoring of what's going in all of their > teams centrally (and manage things like upgrades of Airflow, security > centrally), while they want to delegate control of environments and > resources down to their teams. This is the need that I've heard from many > users who have a "data platform team" that makes Airflow available to their > several teams. I think the proposal I have is a nice middle ground that > follows Conway's law - that architecture of your system should reflect your > organizational structure - and what I separated out as "common" parts is > precisely what "data platform team" would like to manage, where "team > environment" is something that data platform should (and want to) delegate > to their teams. > > 5) "I am a little surprised by a shared dataset" - Vikram/Elad : The > datasets are defined by their URLs and as such - they don't have > "ownership". As I see it - It's really important who can trigger a DAG and > the controls I proposed allow the DAG author to specify "In this DAG it's > also ok when a different team (specified) triggered the dataset event". But > I left a note that it is AIP-73-dependent "Expanded Data Awareness" and > once we get that explained/clarified I am happy to coordinate with > Constance and see if we need to do more. Happy to hear more comments on > that one. > > I reflected the 2 points and 5) in the AIP. Looking forward to more > comments on the proposal - in the AIP or here. > > J. > > > > > On Tue, Jul 9, 2024 at 4:48 PM Jarek Potiuk wrote: > > > Hello Everyone, > > > > I would like to resume discussion on AIP-67. After going through a > > number of discussions and clarifications about the scope of Airflow 3, > > I rewrote the proposal for AIP-67 with the assumption that we will do > > it for Airflow 3 only - and that it will be based on the new proposed > > AIP-72 (Task Execution Interf
Re: More ruff style rules
+1 to what Ash has to say. +1 to enforcing E731 too. Just wondering what others think of D107 here? Thanks & Regards, Amogh Desai On Tue, Jul 16, 2024 at 6:40 AM Tzu-ping Chung wrote: > +1 to enforcing E731, using lambdas is objectively worse in every way > except you save two lines and a few characters at most. The lambda is only > “needed” in the sense that you can’t do (say) functool.partial due to > variable scoping. A nested function does exactly the same thing. > > -- > Sent from my iPhone > > > On 16 Jul 2024, at 00:14, Ferruzzi, Dennis > wrote: > > > > I forgot the formatting was going to get stripped on the email, sorry > about that. How do we feel about these two? > > > > > > "E731", Use a method instead of a lambda when declaring a variable, > currently 3 violations > > > > Seems easy enough, just replace a lambda with a getter function but > someone would need to look more into one spot [1] where there is a comment > which seems to say it needs a lambda, but I have not looked into it yet. > > > > "PT019", Use @pytest.mark.usefixtures instead of passing a fixture as a > parameter, currently 268 violations > > > > This one I really just don't know about. Seems like we gain some > clarification vs passing fixtures in as test parameters, which I kind of > like, but the more explicit declaration is pretty clunky. I'm inclined to > say it's more trouble than it is worth? > > > > > > > > To clean up the formatting mess from the original message and update it > with the discussion so far: > >To Discuss: > > > >"E731", Use a method instead of a lambda when declaring a > variable, currently 3 violations > > > >"PT019", Use @pytest.mark.usefixtures instead of passing a > fixture as a parameter, currently 268 violations > > > >To Do: > > > >"D214", Consistent multi-line docstring indentation, currently 0 > violations > > > >"D215", Enforce number of underscores if you use underlines in a > multi-line docstring, currently 0 violations > >"PT004", Fixture does not return anything, add leading > underscore, currently 311 violations > > > >"PT006", @pytest.mark.parametrize names should be a tuple, > currently 1215 violations > > > >"PT007", @pytest.mark.parametrize values should be a tuple, > currently 391 violations > >"PT011", pytest.raises() is too broad, set the match parameter, > currently 153 violations > >"TCH003", stdlib imports which are only used for typechecking go > in to TYPE_CHECKING block, currently 38 violations > > > >To Don't: > > > >"D100", # Docstring at the top of every file. > > > >"D102", # Missing docstring in public method, currently 2473 > violations > >"D103", # Missing docstring in public function, currently 318 > violations > >"D104", # Docstring at the top of every `__init__.py` file. > >"D105", # See > https://lists.apache.org/thread/8jbg1dd2lr2cfydtqbjxsd6pb6q2wkc3 > >"D107", # Docstring in every constructor is unnecessary if the > class has a docstring. > >"D203", # Conflicts with D211. Both can not be enabled. > >"D212", # Conflicts with D213. Both can not be enabled. > >"PT005", # Fixture returns a value, remove leading underscore, > currently 1 violation > > > > > > > > [1] > https://github.com/apache/airflow/blob/d029e77f2fd704bec4f4797b09d54c5c824a8536/dev/perf/scheduler_dag_execution_timing.py#L293 > > > > > > > > - ferruzzi > > > > > > > > From: Kaxil Naik > > Sent: Sunday, July 14, 2024 11:30 AM > > To: dev@airflow.apache.org > > Cc: Ferruzzi, Dennis > > Subject: RE: [EXT] More ruff style rules > > > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous > ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > certain que le contenu ne présente aucun risque. > > > > > > > > Yup agreed with both: going ahead wi
Re: [PROPOSE] Agree and document Ad-hoc release process for providers
Late to the party here but +1 on clarifying the policy. I am very much with Elad and Jed here regarding the chaos it could lead to so it definitely shouldn't be entirely driven by a non-RM (which is not even possible). One improvement I can think of is in the announcement we make when we are about to cut the "RC". We should make it a point (along with instructions) to ask stakeholders test out the "early drop" of providers using breeze with clear indications on how to and with the "impact", if they don't. The impacts are what Jarek mentioned earlier regarding, but not limited to them waiting for the next wave of providers. Also, I think it will be really beneficial to have a "video" recording (screen recording is good enough) of a RM running the whole process so that interested folks can take up that responsibility, if need be. Thanks & Regards, Amogh Desai On Mon, Jul 15, 2024 at 3:18 PM Eugen Kosteev wrote: > +1 to this > > On Mon, Jul 15, 2024 at 11:24 AM Jarek Potiuk wrote: > > > Ok. If there are no more comments - I will now do some more work on the > > release process (I am going to finally look at the "Trusted Publishing" > > change for PyPI, and i will review and update the processes and along the > > way will mark and double check if the release process steps can be made > by > > non-PMC members/Release manager and find a way to nicely tag them in the > > release process (I will do it also for other release processes - just tag > > them). Then I will propose some updates to add the "ad-hoc" process lead > by > > other people and once it's ready, we might have a test/dry run with it. > > > > J. > > > > On Thu, Jul 11, 2024 at 10:57 AM Jarek Potiuk wrote: > > > > > Yes. I think we all agree this should be a truly exceptional case with > > > wide impact and one that has no easy workaround (an example of such > > > issues from the past were when we have not realized that an older > > > version of ads-sdk has been disabled and none of the previous > > > providers would work. Or when the current version of the provider > > > released causes huge data loss (in which case we should yank it as > > > well) or similar kinds of issues. I think when someone wants to do it, > > > they should be absolutely convinced that it will be accepted, so if we > > > get in the scenario where most of such requests are rejected, it will > > > mean there is a misunderstanding that should be clarified. > > > > > > Re: Elad - yeah some of that for sure require committer to "merge" > > > things (but the PRs can be opened by anyone) - also some of the > > > announcements need apache email "from", but I think there are just a > > > few "touch points" with PMC member/release manager. We will find out > > > when we actually describe it (and try it once or twice). But I am > > > quite sure we can do it well - also to Kacper's comment, engaging more > > > people in the release process is generally a good idea. > > > > > > And I am very much with Vikram as well - that it should not introduce > > > chaos, so we should do it carefully. I don't think we are trading > > > speed for reliability here, if we all agree that this is just > > > "standard procedure for really exceptional situations". For me this is > > > mostly something like a fire-drill on a ship. You should be prepared > > > and have a process for it and do a "test run" from time to time so > > > that when it actually happens you can do it calmly, following all > > > trained and known processes and do it methodically and calmly. So I'd > > > say it's quite the reverse to introducing chaos - it's an attempt to > > > introduce an order to exceptional things that we know are going to > > > (rarely) happen and be prepared for it. > > > > > > J, > > > > > > On Thu, Jul 11, 2024 at 7:39 AM Vikram Koka > > > wrote: > > > > > > > > +1 on clarifying the policy for sure. > > > > > > > > However, I have concerns around the rest of this from a reliability > > > > standpoint. > > > > Unless this is to be used only for critical situations and for > > selective > > > > providers only, this seems like a speed vs. reliability tradeoff. > > > > > > > > I am uncomfortable with this. > > > > > > > > On Wed, Jul 10, 2024 at 12:22 PM Jed Cunningham < > > > jedcunning...@apache.org> > > > > wrote: > > > > > > > > > I'm kinda of the same opinion as Elad here. +1 for clarifying the > > > policy, > > > > > but I have some concerns on the rest of it. > > > > > > > > > > If we really do restrict these to critical situations, these should > > be > > > rare > > > > > enough that we aren't burning out RMs. Opening this door to only > say > > > "no, > > > > > not critical enough" more often than not (assuming folks are eager > to > > > do > > > > > frequent releases), seems a bit weird to me. > > > > > > > > > > > > > -- > Eugene >
Re: [DISCUSS] To AIP-44 or not to AIP-44
I'd prefer to move it completely to Airflow 3, because I don't really see spending cycles on developing something for the short term here. With Airflow 3, it would come out more complete and something that could be usable by users without having to put a halt to it and starting it again. I understand that it would affect the AIP 69 but as I whole i don't see doing AIP 44 for Airflow 2 as beneficial. Thanks & Regards, Amogh Desai On Fri, Jul 12, 2024 at 10:39 PM Ash Berlin-Taylor wrote: > -0.25 to me continuing - I don't see the value long term in putting more > effort into this, and any lessons from this have already been learnt, as > development on Airflow 3 has already started, so this would literally be > happening in parallel. > > Given we are planning, at most, 1 or two more releases of Airflow 2 this > feature will have to stay marked experimental so that it doesn't increase > our support burden. > > My TL;Dr: I see this feature as a dead end now, but if you want to work on > it by my guest, as long as it doesn't cause us more (support) work in the > long term. > > -ash > > On 12 July 2024 14:20:14 BST, "Michał Modras" > > wrote: > >I think we should finish the AIP within Airflow 2 - it will take time > until > >Airflow 3 is out, and I believe some learnings from finishing and running > >this AIP might be useful for Airflow 3. We plan to contribute to finishing > >this AIP. > > > >On Fri, Jul 12, 2024 at 7:52 AM Scheffler Jens (XC-AS/EAE-ADA-T) > > wrote: > > > >> Hi, > >> > >> I'd favor to make it usable - especially as we are at 80%. > >> > >> Main motivation is that with our environment we see stability problems > >> with the distributed setup and using Celery, which was the main > motivation > >> to spin the discussion about AIP-69. AIP-69 is depending on the feature. > >> Waiting another 12-18 months to be able to host a stable distributed > setup > >> based on Airflow 3 is something hard to argue. And I can confirm it is > >> working already in my AIP-69 PoC. > >> > >> In this light I could offer to move it to at least the level that it can > >> be used and is properly CI tested as using it for AIP-69 as first > consumer > >> (which could reduce the scope to task execution, DAG parsing and > triggered > >> could be taken out-of-scope for AIP-69 dependency for example). I could > >> offer supporting to close the gaps to completion. > >> > >> In regards of workload the completion should be a target before the > >> cut-off to Airflow 3, so I would assume only "keeping the lights on" > would > >> be a distraction while developing Airflow 3. > >> > >> Mit freundlichen Grüßen / Best regards > >> > >> Jens Scheffler > >> > >> Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T) > >> Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen | > >> GERMANY | www.bosch.com > >> Tel. +49 711 811-91508 <+49%20711%2081191508> | Mobil +49 160 90417410 > >> <+49%20160%2090417410> | jens.scheff...@de.bosch.com > >> > >> Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000; > >> Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer; > >> Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. Markus > >> Forschner, > >> Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert > >> > >> -Original Message- > >> From: Jed Cunningham > >> Sent: Thursday, July 11, 2024 10:41 PM > >> To: dev@airflow.apache.org > >> Subject: Re: [DISCUSS] To AIP-44 or not to AIP-44 > >> > >> It feels a little weird to add a new "forever" experimental feature in > >> Airflow 2 that we already know won't be there in Airflow 3. Not > something > >> I'd want to be really user facing at this point in time either. > >> > >> Given the short timeline for Airflow 3, I imagine we'd be better off > >> spending those cycles elsewhere. My 2c - not my cycles :) > >> > > - > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
Re: [VOTE] AIP-72: Task Execution Interface aka Task SDK
+1 binding This AIP will finally attempt to solve some of the tough scaling challenges and despite the fact that it will break a few things here and there, this is something that needs to be done sooner rather than later. Thanks & Regards, Amogh Desai On Fri, Jul 12, 2024 at 2:55 PM Michał Modras wrote: > +1 non-binding > > In general I think the AIP makes sense, and there are of course details to > clarify and iron out - but this can happen on the way. > > On Fri, Jul 12, 2024 at 11:20 AM Ephraim Anierobi < > ephraimanier...@gmail.com> > wrote: > > > +1 binding > > > > On Fri, 12 Jul 2024 at 06:41, Scheffler Jens (XC-AS/EAE-ADA-T) > > wrote: > > > > > +1 binding > > > > > > Mit freundlichen Grüßen / Best regards > > > > > > Jens Scheffler > > > > > > Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T) > > > Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen | > > > GERMANY | www.bosch.com > > > Tel. +49 711 811-91508 <+49%20711%2081191508> | Mobil +49 160 90417410 > > <+49%20160%2090417410> | > > > jens.scheff...@de.bosch.com > > > > > > Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000; > > > Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer; > > > Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. Markus > > > Forschner, > > > Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert > > > > > > -Original Message- > > > From: Oliveira, Niko > > > Sent: Thursday, July 11, 2024 6:26 PM > > > To: dev@airflow.apache.org > > > Subject: Re: [VOTE] AIP-72: Task Execution Interface aka Task SDK > > > > > > +1 binding > > > > > > As Jarek mentioned, still some discussions ongoing but I think that can > > be > > > hashed out later. Looks good overall. > > > > > > > > > From: Elad Kalif > > > Sent: Thursday, July 11, 2024 9:11:19 AM > > > To: dev@airflow.apache.org > > > Subject: RE: [EXT] [VOTE] AIP-72: Task Execution Interface aka Task SDK > > > > > > CAUTION: This email originated from outside of the organization. Do not > > > click links or open attachments unless you can confirm the sender and > > know > > > the content is safe. > > > > > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > externe. > > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > > pouvez > > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain > > que > > > le contenu ne présente aucun risque. > > > > > > > > > > > > +1 binding > > > > > > On Thu, Jul 11, 2024 at 5:50 PM Bishundeo, Rajeshwar > > > wrote: > > > > > > > +1 non-binding > > > > > > > > -- Rajesh > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 2024-07-11, 9:57 AM, "Vincent Beck" > > > vincb...@apache.org>> wrote: > > > > > > > > > > > > CAUTION: This email originated from outside of the organization. Do > > > > not click links or open attachments unless you can confirm the sender > > > > and know the content is safe. > > > > > > > > > > > > > > > > > > > > > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > > externe. > > > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > > > > pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > > > > certain que le contenu ne présente aucun risque. > > > > > > > > > > > > > > > > > > > > > > > > > > > > +1 binding > > > > > > > > > > > > On 2024/07/11 13:32:14 Igor Kholopov wrote: > > > > > +1, non-binding > > > > > > > > > > Some alignment with AIP-66 might be required, but the general > vision > > > > > implementation looks clear to me. > > > > > > > > > > Thanks for leading this effort! > > > > > > > > > > On Thu, Jul 11, 2024 at 3:21 PM Jed Cunningham > > > > > > > > <mailto:jedcunning...@apache.org>> > > > > > wrote: > > > > > > > > > > > +1 binding > > > > > > > > > > > > > > > > > > > > > > > - > > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > dev-unsubscr...@airflow.apache.org> > > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > dev-h...@airflow.apache.org> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: [ANNOUNCE] New committers: Rom Sharon & Shahar Epstein
This is awesome news! Congratulations Rom and Shahar 🎊! Well deserved! Thanks & Regards, Amogh Desai On Mon, Jul 15, 2024 at 5:33 PM Elad Kalif wrote: > The Project Management Committee (PMC) for Apache Airflow > has invited Rom Sharon & Shahar Epstein to become a committers and we are > pleased > to announce that they have accepted. > > Congratulations Rom & Shahar and welcome! >
Re: [VOTE] Release Airflow 2.9.3 from 2.9.3rc1
+1 non binding. Tested my changes, and ran a couple of tests as well. It looks good Thanks & Regards, Amogh Desai On Mon, Jul 15, 2024 at 2:22 PM Jarek Potiuk wrote: > +1 (binding): Checked reproducibility, signatures, checksums, licences, run > a few test cases, checked all the "tooling" upgrades that are properly > applied. All good. > > I found an issue with latest FAB provider that it had check for <= 2.9.2 > for CSRF protection on logout, but we have not cherry-picked the fix ( > https://github.com/apache/airflow/pull/40145) as it was marked as 2.10.0. > to 2.9.3, but we will fix it in the next version of FAB provider by > https://github.com/apache/airflow/pull/40784 (unless there will be reason > for rc2, in which case we might attempt to cherry-pick the #40784 to rc2). > Not critical however so I don't think it should block 2.9.3 rc1 on its own. > > On Fri, Jul 12, 2024 at 7:07 PM Utkarsh Sharma > wrote: > > > Hey fellow Airflowers, > > > > I have cut Airflow 2.9.3rc1. This email is calling for a vote on the > > release, > > which will last at least 72 hours, from Friday, July 12, 2024 at 5:10 pm > > UTC > > until Monday, July 15, 2024, at 5:10 pm UTC > > < > > > https://www.timeanddate.com/worldclock/fixedtime.html?msg=8&iso=20240715T1710&p1=1440 > > >, > > and until 3 binding +1 votes have been received. > > > > Status of testing of the release is kept in > > https://github.com/apache/airflow/issues/40756 > > > > Consider this my (non-binding) +1. As I’m not a member of the PMC, > Ephraim > > signed the distribution. > > > > Airflow 2.9.3rc1 is available at: > > https://dist.apache.org/repos/dist/dev/airflow/2.9.3rc1/ > > > > *apache-airflow-2.9.3-source.tar.gz* is a source release that comes with > > INSTALL instructions. > > *apache-airflow-2.9.3.tar.gz* is the binary Python "sdist" release. > > *apache_airflow-2.9.3-py3-none-any.whl* is the binary Python wheel > "binary" > > release. > > > > Public keys are available at: > > https://dist.apache.org/repos/dist/release/airflow/KEYS > > > > Please vote accordingly: > > > > [ ] +1 approve > > [ ] +0 no opinion > > [ ] -1 disapprove with the reason > > > > Only votes from PMC members are binding, but all members of the community > > are encouraged to test the release and vote with "(non-binding)". > > > > The test procedure for PMC members is described in: > > > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-pmc-members > > > > The test procedure for contributors and members of the community who > would > > like to test this RC is described in: > > > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-contributors > > > > > > Please note that the version number excludes the `rcX` string, so it's > now > > simply 2.9.3. This will allow us to rename the artifact without modifying > > the artifact checksums when we actually release. > > > > Release Notes: > > https://github.com/apache/airflow/blob/2.9.3rc1/RELEASE_NOTES.rst > > > > For information on what goes into a release please see: > > > > > https://github.com/apache/airflow/blob/main/dev/WHAT_GOES_INTO_THE_NEXT_RELEASE.md > > > > *Changes since 2.9.2:* > > > > *Significant Changes* > > > > *Time unit for ``scheduled_duration`` and ``queued_duration`` changed > > (#37936)* > > > > ``scheduled_duration`` and ``queued_duration`` metrics are now emitted in > > milliseconds instead of seconds. > > > > By convention all statsd metrics should be emitted in milliseconds, this > is > > later expected in e.g. ``prometheus`` statsd-exporter. > > > > > > *Support for OpenTelemetry Metrics is no longer "Experimental" (#40286)* > > > > Experimental support for OpenTelemetry was added in 2.7.0 since then > fixes > > and improvements were added and now we announce the feature as stable. > > > > > > *Bug Fixes* > > > > - Fix calendar view scroll (#40458) > > - Validating provider description for urls in provider list view (#40475) > > - Fix compatibility with old MySQL 8.0 (#40314) > > - Fix dag (un)pausing won't work on environment where dag files are > missing > > (#40345) > > - Extra being passed to SQLalchemy (#40391) > > - Handle unsupported operand int + str when value of tag is int (job_id
Re: [VOTE] Airflow Providers prepared on July 09, 2024
+1 non binding Ran example dags for cncf and docker, thing seem to work fine. Thanks & Regards, Amogh Desai On Wed, Jul 10, 2024 at 7:22 AM Wei Lee wrote: > +1 for providers other than weaviate. Ran our example DAGs and worked fine. > > Best, > Wei > > > > On Jul 10, 2024, at 6:42 AM, Vincent Beck wrote: > > > > +1 non binding. All AWS system tests are running successfully against > apache-airflow-providers-amazon==8.26.0rc1 > > > > On 2024/07/09 15:01:36 Elad Kalif wrote: > >> weaviate provider is excluded from rc1 > >> please continue testing and voting on rest of the providers > >> > >> On Tue, Jul 9, 2024 at 4:57 PM Wei Lee wrote: > >> > >>> -1 for weaviate due to https://github.com/apache/airflow/pull/40670 > >>> (Thanks, Tamara, for fixing the issue!) > >>> > >>> Best, > >>> Wei > >>> > >>> > >>>> On Jul 9, 2024, at 9:43 PM, Jarek Potiuk wrote: > >>>> > >>>> Yep: -1 on weaviate following 40670 > >>>> > >>>> On Tue, Jul 9, 2024 at 3:25 PM Hussein Awala > wrote: > >>>>> > >>>>> -1 (binding) for Weaviate provider - > >>>>> https://github.com/apache/airflow/pull/40670 > >>>>> > >>>>> +1 (binding) for the rest - checked licences, checksums, signatures, > and > >>>>> sources. All looks good. > >>>>> > >>>>> On Tue, Jul 9, 2024 at 10:10 AM Jarek Potiuk > wrote: > >>>>> > >>>>>> +1 (binding) - checked all my changes, checked reproducibility, > >>>>>> signatures, licences, checksums, all looks good. > >>>>>> > >>>>>> One small thing I found was that changelog for weaviate was slightly > >>>>>> broken (2.0.0 changelog swallowed already released 1.4.2) - I > created > >>>>>> a PR fixing it https://github.com/apache/airflow/pull/40663 but it > >>>>>> does not affect the packagesand can be fixed directly by > regenerating > >>>>>> the docs without regenerating the package, so it does not affect > >>>>>> voting status. > >>>>>> > >>>>>> On Tue, Jul 9, 2024 at 9:16 AM Elad Kalif > wrote: > >>>>>>> > >>>>>>> Hey all, > >>>>>>> > >>>>>>> I have just cut the new wave Airflow Providers packages. This > email is > >>>>>>> calling a vote on the release, > >>>>>>> which will last for 72 hours - which means that it will end on July > >>> 12, > >>>>>>> 2024 07:15 AM UTC and until 3 binding +1 votes have been received. > >>>>>>> > >>>>>>> Consider this my (binding) +1. > >>>>>>> > >>>>>>> Airflow Providers are available at: > >>>>>>> https://dist.apache.org/repos/dist/dev/airflow/providers/ > >>>>>>> > >>>>>>> *apache-airflow-providers--*.tar.gz* are the binary > >>>>>>> Python "sdist" release - they are also official "sources" for the > >>>>>> provider > >>>>>>> packages. > >>>>>>> > >>>>>>> *apache_airflow_providers_-*.whl are the binary > >>>>>>> Python "wheel" release. > >>>>>>> > >>>>>>> The test procedure for PMC members is described in > >>>>>>> > >>>>>> > >>> > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members > >>>>>>> > >>>>>>> The test procedure for and Contributors who would like to test this > >>> RC is > >>>>>>> described in: > >>>>>>> > >>>>>> > >>> > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors > >>>>>>> > >>>>>>> > >>>>>>> Public keys are available at: > >>>>>>> https://dist.apache.org/repos/dist/release/airflow/KEYS > >>>>>>> > >>>>>>> Please vote accordingly: > >>>
Re: ASF download link broken
Thanks Jarek. Yes this change was intentional. Thanks & Regards, Amogh Desai On Fri, 5 Jul 2024 at 7:25 PM, Jarek Potiuk wrote: > > Thanks. Looks better now. You might want to consider linking directly to > the airflow directory: > > I think it's better to keep it this way - it was quite deliberate (and > that's why we named the link in generic way to point to "all" ASF > downloads. > > Few lines down we have direct links to point to the actual packages to > download directly (and those save the users even more clicks - as they > directly download the packages, signatures, checksums) :) > > J. > > > > > On Fri, Jul 5, 2024 at 3:49 PM Mark Thomas wrote: > > > Thanks. Looks better now. You might want to consider linking directly to > > the airflow directory: > > > > https://dlcdn.apache.org/airflow/ > > > > Since Airflow is near the top of the list it only saves you users a > click. > > > > Mark > > > > > > On 05/07/2024 14:46, Jarek Potiuk wrote: > > > Note - you might need Shift-Refresh to see it :) > > > > > > On Fri, Jul 5, 2024 at 3:45 PM Jarek Potiuk > > <mailto:ja...@potiuk.com>> wrote: > > > > > > I just checked, the links are fixed after the docs have been > > > rebuilt. Thanks Amogh for fixing it that quickly! > > > > > > On Fri, Jul 5, 2024 at 12:00 PM Amogh Desai > > > mailto:amoghdesai@gmail.com>> > wrote: > > > > > > Alright, noted. > > > > > > Thanks & Regards, > > > Amogh Desai > > > > > > > > > On Fri, Jul 5, 2024 at 2:29 PM Jarek Potiuk > > <mailto:ja...@potiuk.com>> wrote: > > > > > > > One think also - we will have to manually post-process html > > > documents for > > > > airflow-site as well as the links are generated in html for > > > old versions of > > > > docs as well. But this should be a simple global > > > 'search-and-replace' > > > > > > > > pt., 5 lip 2024, 10:39 użytkownik Amogh Desai > > > mailto:amoghdesai@gmail.com>> > > > > napisał: > > > > > > > > > Should be a quick fix in that case. Thanks for letting me > > know > > > > > > > > > > Thanks & Regards, > > > > > Amogh Desai > > > > > > > > > > > > > > > On Fri, 5 Jul 2024 at 2:07 PM, Jarek Potiuk > > > mailto:ja...@potiuk.com>> wrote: > > > > > > > > > > > I think it should be https://dlcdn.apache.org/ > > > <https://dlcdn.apache.org/> - I think the link is > > > > > from > > > > > > the times when we did not have CDN but mirroring system > > > and it's > > > > changed > > > > > > since then. > > > > > > > > > > > > We will fix it shortly. > > > > > > > > > > > > J. > > > > > > > > > > > > On Fri, Jul 5, 2024 at 9:04 AM Amogh Desai > > > mailto:amoghdesai@gmail.com>> > > > > > > wrote: > > > > > > > > > > > > > Hi Mark, > > > > > > > > > > > > > > Thanks for your email. I can see that the link is > > > active and leads > > > > to a > > > > > > > webpage, but the CSS seems to be broken. > > > > > > > Is that your issue or something else? > > > > > > > > > > > > > > Thanks & Regards, > > > > > > > Amogh Desai > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 5, 2024 at 12:31 PM Mark Thomas > > > mailto:ma...@apache.org>> > > > > wrote: > > > > > > > > > > > > > > > Hi Airflow commun
Re: ASF download link broken
Alright, noted. Thanks & Regards, Amogh Desai On Fri, Jul 5, 2024 at 2:29 PM Jarek Potiuk wrote: > One think also - we will have to manually post-process html documents for > airflow-site as well as the links are generated in html for old versions of > docs as well. But this should be a simple global 'search-and-replace' > > pt., 5 lip 2024, 10:39 użytkownik Amogh Desai > napisał: > > > Should be a quick fix in that case. Thanks for letting me know > > > > Thanks & Regards, > > Amogh Desai > > > > > > On Fri, 5 Jul 2024 at 2:07 PM, Jarek Potiuk wrote: > > > > > I think it should be https://dlcdn.apache.org/ - I think the link is > > from > > > the times when we did not have CDN but mirroring system and it's > changed > > > since then. > > > > > > We will fix it shortly. > > > > > > J. > > > > > > On Fri, Jul 5, 2024 at 9:04 AM Amogh Desai > > > wrote: > > > > > > > Hi Mark, > > > > > > > > Thanks for your email. I can see that the link is active and leads > to a > > > > webpage, but the CSS seems to be broken. > > > > Is that your issue or something else? > > > > > > > > Thanks & Regards, > > > > Amogh Desai > > > > > > > > > > > > On Fri, Jul 5, 2024 at 12:31 PM Mark Thomas > wrote: > > > > > > > > > Hi Airflow community, > > > > > > > > > > While reviewing a moderation message for an Airflow release > > > announcement > > > > > I noticed that the link to "Official Apache Software Foundations > > > > > Downloads" on > > > > > > > > > > > > > > > > > > > > https://airflow.apache.org/docs/apache-airflow-providers-airbyte/stable/installing-providers-from-sources.html > > > > > is broken and needs to be fixed. > > > > > > > > > > Mark > > > > > > > > > > > - > > > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > > > > > > > > > > > > >
Re: ASF download link broken
Should be a quick fix in that case. Thanks for letting me know Thanks & Regards, Amogh Desai On Fri, 5 Jul 2024 at 2:07 PM, Jarek Potiuk wrote: > I think it should be https://dlcdn.apache.org/ - I think the link is from > the times when we did not have CDN but mirroring system and it's changed > since then. > > We will fix it shortly. > > J. > > On Fri, Jul 5, 2024 at 9:04 AM Amogh Desai > wrote: > > > Hi Mark, > > > > Thanks for your email. I can see that the link is active and leads to a > > webpage, but the CSS seems to be broken. > > Is that your issue or something else? > > > > Thanks & Regards, > > Amogh Desai > > > > > > On Fri, Jul 5, 2024 at 12:31 PM Mark Thomas wrote: > > > > > Hi Airflow community, > > > > > > While reviewing a moderation message for an Airflow release > announcement > > > I noticed that the link to "Official Apache Software Foundations > > > Downloads" on > > > > > > > > > https://airflow.apache.org/docs/apache-airflow-providers-airbyte/stable/installing-providers-from-sources.html > > > is broken and needs to be fixed. > > > > > > Mark > > > > > > - > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > >
Re: ASF download link broken
Hi Mark, Thanks for your email. I can see that the link is active and leads to a webpage, but the CSS seems to be broken. Is that your issue or something else? Thanks & Regards, Amogh Desai On Fri, Jul 5, 2024 at 12:31 PM Mark Thomas wrote: > Hi Airflow community, > > While reviewing a moderation message for an Airflow release announcement > I noticed that the link to "Official Apache Software Foundations > Downloads" on > > https://airflow.apache.org/docs/apache-airflow-providers-airbyte/stable/installing-providers-from-sources.html > is broken and needs to be fixed. > > Mark > > - > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
Re: [DISCUSS] @remove provide_bucket_name and @unify_bucket_name_and_key?
I understand that having multiple ways of doing the same thing is not needed and we need to slowly and eventually clean up these things to simplify the codebase and also make it maintainable. Aligning with one of the principles of Airflow3, we need to simplify things, nuke unnecessary stuff, remove deprecated code etc. This will have multiple benefits for the ecosystem: 1. Maintainability is not a headache. We will know for sure that users are using certain things in a certain way. 2. It will make it easier for us to make some decisions like this one without major discussions and edge case considerations because we know how users are using certain patterns (aligns with point 1) 3. Easier for new contributors to contribute to :) I am all up for cleaning up the codebase if we have something of this nature. My 2c. Thanks & Regards, Amogh Desai On Tue, Jul 2, 2024 at 4:29 AM Daniel Standish wrote: > Sometimes in software, we try to be helpful by doing a bunch of work for > the user, but it ends up causing a bunch of confusion and maintenance > headaches. I think these decorators are an example of that. > > What `unify_bucket_name_and_key` does is, it tries to make it so you could > pass the full S3 URI to `key`, or you can provide them separately as bucket > and key, and the decorator will "figure it out". > > Then this decorator can also be combined with another one, > @provide_bucket_name, which, if you don't provide the bucket name, it will > grab it from an airflow connection. > > Then, sometimes the params are again passed through another function > `get_s3_bucket_key`. > > This requires us to add tests to ensure that the decorators are applied in > the correct order, and now, to handle the various async cases (the > decorated fn may be either coroutine or async iterator). > > It's a bunch of code and complexity that, really does not need to exist. > I.e. what if bucket were simply bucket, and key were simply key, always? > > If it were up to me, I'd just remove all of it. Just force the user to > supply a bucket and key separately. > > We can of course do this since semver allows it. But some portion of users > will be depending on this change and will have to change their code. > > Curious what others think. >
Re: [ANNOUNCE] New committer: Ryan Hatter
Congratulations Ryan! Welcome onboard :) Thanks & Regards, Amogh Desai On Fri, Jun 28, 2024 at 10:32 PM Jarek Potiuk wrote: > Congrats! > > On Fri, Jun 28, 2024 at 6:59 PM Jed Cunningham > wrote: > > > The Project Management Committee (PMC) for Apache Airflow > > has invited Ryan Hatter to become a committer and we are pleased > > to announce that they have accepted. > > > > Ryan has been involved in the Airflow community for a few years now - > > contributing code, triaging and creating issues, answering questions on > > Slack, etc. > > He also contributed the custom names for mapped tasks feature in Airflow > > 2.9, > > which was a long time ask from the community. > > > > Congratulations Ryan, and welcome! > > >
Re: Using AI / Dosu to help us with triaging issues
> The current idea we have, how we could still benefit from having AI/Bot generated answers, is for maintainers and possibly triage teams to be able to review the proposed answers and "approve" them before they are posted. Such an answer might then be marked as "bot/automatically/AI generated" but it will also get a "Maintainer approved" or "Triage team approved" badge, This way anyone can see that the answer was reviewed by a maintainer/triage team member. This way we can get way more efficiency in handling issues (time for reviewing answers will be far less than writing it). Also maintainers/triage team members will be able to correct the answers if they are misleading / wrong. Who approved/corrected it will be visible for the triage/maintainer team in the DoSu interface, but not seen in the Github UI who actually approved/corrected it. Love this! One thing we need to be a little sure of is the "honesty" while answering these bot suggestions, at least initially, so that the bot settles well into our ecosystem. I am happy with the direction this is heading into Thanks & Regards, Amogh Desai On Thu, Jun 27, 2024 at 1:31 PM Wei Lee wrote: > +1 for this idea. I remember we had this discussion during PyCon US, haha. > > So, in the first phase, we'll need a maintainer's approval before the AI > response is sent. I'm not sure whether this will be a burden to > maintainers. I'm thinking of adding a label to that AI response answer; the > user can decide whether to accept that even before the maintainers' > approval. If maintainers check the response and it's ok, they can remove > that label. > > Best, > Wei > > > On Jun 27, 2024, at 3:07 PM, Jarek Potiuk wrote: > > > >> > >> > >> I guess it's too early to discuss what the workflow we're picturing for > it > >> is. But I would like it if a reporter is unable to see the comment until > >> it's approved (which I guess is exactly what you're also suggesting > Jarek). > >> > > > > Absolutely. Actually - we (few of maintainers) can already see the > answers > > generated in a DoSU interface as "drafts" - and once we enable labelling > we > > can on-board more maintainers and triage team and work with DoSu team on > > implementing a workflow that might be good to start with. > > > - > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
Re: [VOTE] June 2024 PR of the Month
My vote would definitely go to https://github.com/apache/airflow/pull/39355! Developers love dark mode, and I mean it is done fantastically! Thanks & Regards, Amogh Desai On Thu, Jun 27, 2024 at 3:57 AM Kaxil Naik wrote: > My vote would be biased to #39355 because it is DARK MODE : > https://github.com/apache/airflow/pull/39355 :) > > On Wed, 26 Jun 2024 at 21:28, Jarek Potiuk wrote: > > > IF (and only IF) the votes from Jens and Vincent go to > > https://github.com/apache/airflow/pull/39355 (#39*3*55) then it's my +1 > as > > well. Otherwise I am NOT voting for > > https://github.com/apache/airflow/pull/39555 but > > https://github.com/apache/airflow/pull/39355 has still my vote > > > > :P > > > > On Wed, Jun 26, 2024 at 10:22 PM Vincent Beck > wrote: > > > > > Same! +1 for #39555. Quite an impactful contribution for a first time > > > contributor!! > > > > > > On 2024/06/26 20:17:47 "Scheffler Jens (XC-AS/EAE-ADA-T)" wrote: > > > > Thanks for the proposals... > > > > > > > > There was a "late arrival" and month is not over... as it was a long > > > runner and we had a first-time contributor with a long breath making it > > to > > > the finish line "never gonna give you up" (--> > > > https://www.youtube.com/watch?v=dQw4w9WgXcQ)... > > > > > > > > I'd like to add and +1 vote for #39555 > > > > > > > > Mit freundlichen Grüßen / Best regards > > > > > > > > Jens Scheffler > > > > > > > > Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T) > > > > Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen | > > > GERMANY | http://www.bosch.com/ > > > > Tel. +49 711 811-91508 | Mobil +49 160 90417410 | > > > jens.scheff...@de.bosch.com > > > > > > > > Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000; > > > > Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer; > > > > Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. > Markus > > > Forschner, > > > > Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert > > > > > > > > -Original Message- > > > > From: Briana Okyere > > > > Sent: Tuesday, June 25, 2024 9:25 PM > > > > To: dev@airflow.apache.org > > > > Subject: [VOTE] June 2024 PR of the Month > > > > > > > > Hey All, > > > > > > > > It's once again time to vote for the PR of the Month! > > > > > > > > With the help of the `get_important_pr_candidates` script in > dev/stats, > > > we've identified the following candidates: > > > > > > > > PR #37948: [AIP-49] OpenTelemetry Traces for Apache Airflow < > > > https://github.com/apache/airflow/pull/37948> > > > > > > > > PR #39936: add git-sync-ssh secret template < > > > https://github.com/apache/airflow/pull/39936> > > > > > > > > PR #39585: Prevent start trigger initialization in scheduler < > > > https://github.com/apache/airflow/pull/39585> > > > > > > > > PR #38524: Chart: Allow AWS ECS Executor < > > > https://github.com/apache/airflow/pull/38524> > > > > > > > > PR #40034: AIP-61 - Add executor field to the task instance API < > > > https://github.com/apache/airflow/pull/40034> > > > > > > > > Please reply to this thread with your selection or offer your own > > > nominee(s). > > > > > > > > Voting will close on Friday June 28th at 10 AM PST. The winner(s) > will > > > be featured in the next issue of the Airflow newsletter. > > > > > > > > Also, if there's an article or event that you think should be > included > > > in this or a future issue of the newsletter, please drop me a line at < > > > briana.oky...@astronomer.io> > > > > > > > > -- > > > > Briana Okyere > > > > Community Manager > > > > Astronomer > > > > > > > > - > > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > > > > > > > - > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > >
Re: Using AI / Dosu to help us with triaging issues
Agreed with Aritra here. I personally would love to have the "auto" labelling portion, which would ensure that issues get immediate "right" labels, without relying too much on the triaging team but I personally am not too comfortable with a bot answering my questions for two reasons: 1. Maintainer/Contributor responses are tailored personally to a question and the level of the issue reporter 2. Bot responses sometimes are disregarded by readers, just because its a "bot" Thanks & Regards, Amogh Desai On Thu, Jun 27, 2024 at 8:53 AM Aritra Basu wrote: > I'm a bit averse to having it answer questions (the similar issues part) as > I believe that's a slippery slope to where it could start being used to > provide actual answers. I personally would be more comfortable with > answering still being restricted to maintainers/contributors and maybe > restricting the bot to just tagging similar issues while still requiring a > human to provide advice based on that similarity. > > Just my personal 2 cents. I'm not too strongly opposed to using it. > -- > Regards, > Aritra Basu > > On Thu, Jun 27, 2024, 2:41 AM Julian LaNeve > wrote: > > > Not sure if I get an official vote here but we've been working with Devin > > and the Dosu team for Cosmos ( > > https://github.com/astronomer/astronomer-cosmos ) and it's been working > > great. Excited to see this in Airflow itself! > > > > Plus they use Airflow to power their data platform behind the scenes so > > they have a vested interest in this one :) > > > > -- > > > > *Julian LaNeve* > > CTO > > > > Email: jul...@astronomer.io > > ( jul...@astronomer.io ) Mobile: 330 509 5792 > > > > On Wed, Jun 26, 2024 at 4:58 PM, Vikram Koka < > vik...@astronomer.io.invalid > > > wrote: > > > > > > > > > > > > > > +1 > > > > > > > > > > > > Love it! > > > > > > > > > > > > On Wed, Jun 26, 2024 at 1:23 PM Vincent Beck < vincbeck@ apache. org ( > > > vincb...@apache.org ) > wrote: > > > > > > > > >> > > >> > > >> Fantastic idea! > > >> > > >> > > >> > > >> On 2024/06/26 20:12:43 Jarek Potiuk wrote: > > >> > > >> > > >>> > > >>> > > >>> Hello everyone, > > >>> > > >>> > > >>> > > >>> Together with Elad, Kaxil, and the Dosu team [1], we’ve been looking > > into > > >>> employing AI / Natural Language processing to help us triage issues > for > > >>> Apache Airflow. We do not want to go “all-in” into getting a chatbot > to > > >>> respond to all our issues because we believe this is not how the > > >>> > > >>> > > >> > > >> > > >> > > >> community > > >> > > >> > > >>> > > >>> > > >>> is being built. We looked at various ways we can start exploring the > > >>> capabilities of the new ML/AI/Natural Language processing available. > > >>> > > >>> > > >>> > > >>> We worked with the Dosu team. They are approved by the Apache > Software > > >>> Foundation infrastructure as Github integration and few ASF projects > > >>> already use it (including our friends at Superset) - they have a > > >>> > > >>> > > >> > > >> > > >> > > >> fantastic > > >> > > >> > > >>> > > >>> > > >>> offer to provide free service for open-source projects like Airflow. > > >>> Together we evaluated what we can start with and initially we have a > > >>> proposal to use auto-labeling of issues created in the Airflow > > >>> > > >>> > > >> > > >> > > >> > > >> repository. > > >> > > >> > > >>> > > >>> > > >>> We have a number of rules that are established for the triage team > [2] > > >>> > > >>> > > >> > > >> > > >> > > >> but > > >> > > >> > > >>> > > >>> > > >>> those rules are mundane and difficult to follow, and generally a lot > of &
Re: [DISCUSS][AIP-38 Modern Web Application]
I like this idea too. The ability to extend filtering at this level would be fantastic. IIUC, the filtering is at a "UI" only level and is transient too. Moving this logic to the backend using a new API (or existing ones) and grouping the dags would be a nice feature to have, making Airflow UI much more intuitive. I am not so sure what you mean by "domain" though.. Thanks & Regards, Amogh Desai On Wed, Jun 26, 2024 at 8:03 PM Constance Martineau wrote: > I love it and 100% agree. Thinking "Dag Groups", where you can group dags > (static & dynamic) into a subfolder. Tags are great for filtering, but they > aren't a replacement for dirs especially at a large scale. We have some > deployments with 20k dags and as designed today, it's not navigable at that > scale. This could help > > On Wed, Jun 26, 2024 at 9:04 AM Blain David > wrote: > > > Beside the new interface and getting rid off FAB in Airflow 3.0, a cool > > and handy feature would be to be able to group multiple DAG's so you > could > > order them by like domain or whatever grouping you want to achieve. > > Okay, you can achieve the same with filtering, and maybe the we could use > > that feature to achieve the grouping but still it would make the UI more > > convenient to use, especially if you have to manage multiple dynamic > DAG's > > which are related to the same domain. It would be nice I you could like > > create a group which always apply the filtering in a stateful manner. Or > > we could opt to really implement a dedicated grouping mechanism so that > you > > could for example specify in your DAG to which group it belongs. What do > > you guys think? I would be willing to help and contribute of course. > > > > > > >
Re: [PROPOSAL] Use Trusted Publishing workflow for Airflow releases to PyPI
Excellent proposal! I see no down-side to the proposal Good investigation on the higher level implementation part as well. Thanks & Regards, Amogh Desai On Wed, Jun 26, 2024 at 10:28 AM Poorvi Rohidekar < poorvirohidekar@gmail.com> wrote: > Looks like a good proposal. > > Regards, > Poorvi Rohidekar > > On Wed, 26 Jun 2024 at 00:28, Aritra Basu > wrote: > > > Agreed, overall sounds like a positive change. Don't see any issues with > it > > -- > > Regards, > > Aritra Basu > > > > On Tue, Jun 25, 2024, 10:28 PM Ferruzzi, Dennis > > > > wrote: > > > > > Sounds good, I don't see a down side and "supply chain security" has > been > > > a big concern lately. > > > > > > > > > - ferruzzi > > > > > > > > > > > > From: Wei Lee > > > Sent: Tuesday, June 25, 2024 8:07 AM > > > To: dev@airflow.apache.org > > > Subject: RE: [EXT] [PROPOSAL] Use Trusted Publishing workflow for > Airflow > > > releases to PyPI > > > > > > CAUTION: This email originated from outside of the organization. Do not > > > click links or open attachments unless you can confirm the sender and > > know > > > the content is safe. > > > > > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > externe. > > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > > pouvez > > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain > > que > > > le contenu ne présente aucun risque. > > > > > > > > > > > > This proposal is great! PyPI security has been valued a lot these days. > > > Glad we're also joining. > > > > > > Best, > > > Wei > > > > > > > On Jun 25, 2024, at 8:01 PM, Jarek Potiuk wrote: > > > > > > > > Yes and no :) > > > > > > > > We publish alpha/betas - yes. No change there. But for RCs what we > > > publish > > > > in SVN currently are the packages that are built fro RC tag but > without > > > rc > > > > suffix - so that when they pass the voting we upload them to PyPI > > without > > > > regenerating them (RC becomes final). > > > > > > > > But we do not publish the PYPI RCs - since PYPI uploads are > immutable, > > we > > > > need to publish PYPI RCs with the rc suffixes. So far we just > generated > > > > them and published to PyPI for testing but we did not upload them to > > SVN. > > > > > > > > > > > > So if we want to pull RCs from SVN - we need to upload there both: > the > > RC > > > > version for PyPI (with RC suffix) and the no-suffix candidate that > > might > > > > become the final version once voted. > > > > > > > > J > > > > > > > > > - > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > >
Re: [VOTE] Airflow Providers prepared on June 22, 2024
+1 non binding Tested some example dags in cncf and hive providers. My changes work well too! Thanks & Regards, Amogh Desai On Mon, Jun 24, 2024 at 1:02 AM Marcus Eagan wrote: > Amazing work! > > On Sat, Jun 22, 2024 at 07:47 Jarek Potiuk wrote: > > > +1 (binding): tested signatures, checksums, licences, reproducibility, > > checked my changes. All looks good. > > > > On Sat, Jun 22, 2024 at 3:26 PM Elad Kalif wrote: > > > > > Hey all, > > > > > > I have just cut the new wave Airflow Providers packages. This email is > > > calling a vote on the release, > > > which will last for 72 hours - which means that it will end on June 25, > > > 2024 13:25 PM UTC and until 3 binding +1 votes have been received. > > > > > > Consider this my (binding) +1. > > > > > > Airflow Providers are available at: > > > https://dist.apache.org/repos/dist/dev/airflow/providers/ > > > > > > *apache-airflow-providers--*.tar.gz* are the binary > > > Python "sdist" release - they are also official "sources" for the > > provider > > > packages. > > > > > > *apache_airflow_providers_-*.whl are the binary > > > Python "wheel" release. > > > > > > The test procedure for PMC members is described in > > > > > > > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members > > > > > > The test procedure for and Contributors who would like to test this RC > is > > > described in: > > > > > > > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors > > > > > > > > > Public keys are available at: > > > https://dist.apache.org/repos/dist/release/airflow/KEYS > > > > > > Please vote accordingly: > > > > > > [ ] +1 approve > > > [ ] +0 no opinion > > > [ ] -1 disapprove with the reason > > > > > > Only votes from PMC members are binding, but members of the community > are > > > encouraged to test the release and vote with "(non-binding)". > > > > > > Please note that the version number excludes the 'rcX' string. > > > This will allow us to rename the artifact without modifying > > > the artifact checksums when we actually release. > > > > > > The status of testing the providers by the community is kept here: > > > https://github.com/apache/airflow/issues/40382 > > > > > > The issue is also the easiest way to see important PRs included in the > RC > > > candidates. > > > Detailed changelog for the providers will be published in the > > documentation > > > after the > > > RC candidates are released. > > > > > > You can find the RC packages in PyPI following these links: > > > https://pypi.org/project/apache-airflow-providers-amazon/8.25.0rc1/ > > > > https://pypi.org/project/apache-airflow-providers-apache-drill/2.7.2rc1/ > > > > https://pypi.org/project/apache-airflow-providers-apache-flink/1.4.2rc1/ > > > > https://pypi.org/project/apache-airflow-providers-apache-hdfs/4.4.2rc1/ > > > > https://pypi.org/project/apache-airflow-providers-apache-hive/8.1.2rc1/ > > > > https://pypi.org/project/apache-airflow-providers-apache-kafka/1.5.0rc1/ > > > > https://pypi.org/project/apache-airflow-providers-apache-kylin/3.6.2rc1/ > > > > https://pypi.org/project/apache-airflow-providers-apache-spark/4.8.2rc1/ > > > https://pypi.org/project/apache-airflow-providers-cloudant/3.5.2rc1/ > > > > > > https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/8.3.2rc1/ > > > > > > https://pypi.org/project/apache-airflow-providers-common-compat/1.0.0rc1/ > > > > https://pypi.org/project/apache-airflow-providers-common-sql/1.14.1rc1/ > > > https://pypi.org/project/apache-airflow-providers-databricks/6.6.0rc1/ > > > https://pypi.org/project/apache-airflow-providers-dbt-cloud/3.9.0rc1/ > > > https://pypi.org/project/apache-airflow-providers-docker/3.12.1rc1/ > > > https://pypi.org/project/apache-airflow-providers-exasol/4.5.2rc1/ > > > https://pypi.org/project/apache-airflow-providers-fab/1.2.0rc1/ > > > https://pypi.org/project/apache-airflow-providers-facebook/3.5.2rc1/ > > > https://pypi.org/project/apache-airflow-providers-ftp/3.10.0rc1/ > > > https://pypi.org/proj
Re: [LAZY CONSENSUS] Introduce common.compat provider
This is awesome. Waiting to see how it works in action! Thanks & Regards, Amogh Desai On Sat, Jun 22, 2024 at 2:04 PM Jarek Potiuk wrote: > Common compat provider is added. Now we can use it for any kind of > compatibility code with earlier Airflow versions. We might add later some > protection against adding breaking changes to the provider (it should be > always backwards-compatible by nature), but for now let's see how it works > ( > https://github.com/apache/airflow/pull/39530 is going to be the first one > to benefit from it by avoiding a lot of code duplication for compatibility > code). > > On Thu, Jun 20, 2024 at 2:54 PM Jarek Potiuk wrote: > > > Lazy consensus has been reached. We will proceed with common.compat > > implementation then :). > > > > On Mon, Jun 17, 2024 at 12:54 PM Maciej Obuchowski < > mobuchow...@apache.org> > > wrote: > > > >> I know it's a lazy consensus, but I want to explicitly +1 this :) > >> > >> pon., 17 cze 2024 o 11:15 Jarek Potiuk napisał(a): > >> > >> > Also the discussion was continued there > >> > https://lists.apache.org/thread/hjmrv1wc2rxhkvozpvccs4mhgwg5sw2f (for > >> > posterity) > >> > > >> > On Mon, Jun 17, 2024 at 11:06 AM Jarek Potiuk > wrote: > >> > > >> > > Hello everyone, > >> > > > >> > > Following the discussion > >> > > https://lists.apache.org/thread/r302p5b03dpyncswn11nkjn98cpb6r5z I > >> would > >> > > like to call for a lazy consensus on introdiucing "common.compat" > >> > provider. > >> > > > >> > > Summarizing what has been discussed there (and some of my further > >> > > thoughts): > >> > > > >> > > * common.compat provider will contain compatibility code for earlier > >> > > versions of Airflow, that providers might use in case they need to > >> handle > >> > > some back-compatibility code (example where it might be needed is > here > >> > > https://github.com/apache/airflow/pull/39530) > >> > > > >> > > * as a rule, we should only keep compatibility code there (including > >> > > indication of version of Airflow where compatibility code is not > >> needed > >> > any > >> > > more). New versions of providers should switch to the "airflow" code > >> > > gradually as soon as "min airflow version" is bumped to the version > >> where > >> > > compatibility code is not needed. > >> > > > >> > > * the "common.compat" provider should be strictly > >> backwards-compatible. > >> > We > >> > > will release version 1.0 and for a foreseeable future we will have > to > >> > keep > >> > > it compatible in the future, which means that we will get some old > >> > "compat" > >> > > code lying around for quite some time. We might change it if we add > >> some > >> > > retention policies for older versions of providers (We do not have > >> them > >> > > now). This code should generally be not touched, and does not have > to > >> > have > >> > > test coverage other than test coverage from providers using it) - so > >> cost > >> > > of maintenance of that code should be **low** to **none**. > >> > > > >> > > * this will also make it possible to get some compatibility code for > >> > > Airflow 2 / Airflow 3 likely. > >> > > > >> > > The lazy consensus will last for 72 HRS and it will be effective as > of > >> > > Thursday 20th of June ~ noon CEST. > >> > > > >> > > J. > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > >> > > >
Re: [Meeting Notes] Airflow 3.0 Dev call - 13 June 2024
You have covered the minutes of the meeting thoroughly, Kaxil. Finally got the calls added to my calendar too :) Thanks & Regards, Amogh Desai On Mon, Jun 17, 2024 at 6:55 AM Wei Lee wrote: > Thanks Kaxil for summarising! Also tried to update the invitations through > the link. It works perfectly. > > Best, > Wei > > > On Jun 17, 2024, at 3:18 AM, Kaxil Naik wrote: > > > > Cool, thanks > > > > On Sun, 16 Jun 2024 at 20:15, Jarek Potiuk wrote: > > > >> The summary looks good ! > >> > >> BTW. Maybe attaching an .ics from the Zoom meeting to the meeting > summary > >> page might help with re-adding calendar meetings without > re-registering. I > >> added it as attachment and updated the doc with link to it. > >> > >> On Sun, Jun 16, 2024 at 9:07 PM Kaxil Naik wrote: > >> > >>> I have also added a meeting for this week, so if you don't have it on > >> your > >>> calendar, just open the registration link again. Adding it again should > >>> update your calendar. In case you have any issues, drop me a note here > or > >>> on Slack. > >>> > >>> The Airflow 3 Home page on cWiki > >>> <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3.0> has > >> also > >>> been updated to simplify and reflect discussions from the last 2 calls. > >>> > >>> Regards, > >>> Kaxil > >>> > >>> > >>> On Sun, 16 Jun 2024 at 19:56, Kaxil Naik wrote: > >>> > >>>> Hey all, > >>>> > >>>> I have updated our meeting notes document to summarize the discussion > >>>> from our 13th June dev call for Airflow 3.0. > >>>> > >>>> Link: > >>>> > >>> > >> > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-Summary.1 > >>>> > >>>> To all those who attended, can you please double-check and add if I > >> have > >>> missed > >>>> anything? > >>>> > >>>> To all those who didn't join, if you disagree with anything in the > >>>> Summary, please voice your opinion. > >>>> > >>>> I will send a separate email for the agenda for the next meeting on > >> 20th > >>>> June. > >>>> > >>>> Regards, > >>>> Kaxil > >>>> > >>>> -- > >>>> > >>>> Including the Summary here too (might break formatting): > >>>> > >>>> *Drafting proposals & getting feedback* > >>>> > >>>> - The team discussed the challenges of Apache Confluence access and > >>>> considered using Google Docs for initial feedback before moving to > >>>> Confluence. > >>>> - The general consensus was that developers could utilize the tool > >> of > >>>> their choice to get initial feedback (on their high-level draft) as > >>> long as > >>>> the final AIP is published on Airflow's Confluence space for AIPs > >>>> < > >>> > >> > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals > >>>> > >>>> . > >>>> - If someone does not have an account on Apache Confluence to > >> comment > >>>> on existing proposals or create a new proposal, reach out on > >>>> #airflow-3-dev Slack channel > >>>> < > >>> > https://apache-airflow.slack.com/archives/C07813CNKA8/p1718292608810799> > >>>> . > >>>> > >>>> > >>>> *Personas for the Airflow User* > >>>> > >>>> - Elad prepared a document on Airflow Actors > >>>> <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Actors > >>> > >>> that > >>>> the group discussed to get initial feedback. Several good > >> suggestions > >>> were > >>>> made, so it was decided to take the discussion offline/async and > >>> continue > >>>> the discussion on that page. > >>>> > >>>> > >>>> *Task Context draft AIP* > >>>> > >>>> - Ash presented the initial draft of the Task Exec
Re: [VOTE] Release Apache Airflow Helm Chart 1.14.0 based on 1.14.0rc1
+1 non binding Installed the chart in a Kind cluster and ran a few simple example dags. Looks fine! Thanks & Regards, Amogh Desai On Sat, Jun 15, 2024 at 8:49 PM Jarek Potiuk wrote: > +1 (binding). Tested reproducibility, checksums, signatures, licences. All > good. Installed the chart locally and it nicely pulled all latest > dependencies and airflow was flawlessly installed (2.9.2). All looks good. > > On Fri, Jun 14, 2024 at 10:53 PM Jed Cunningham > wrote: > > > Hello Apache Airflow Community, > > > > This is a call for the vote to release Helm Chart version 1.14.0. > > > > The release candidate is available at: > > https://dist.apache.org/repos/dist/dev/airflow/helm-chart/1.14.0rc1/ > > > > airflow-chart-1.14.0-source.tar.gz - is the "main source release" that > > comes with INSTALL instructions. > > airflow-1.14.0.tgz - is the binary Helm Chart release. > > > > Public keys are available at: https://www.apache.org/dist/airflow/KEYS > > > > For convenience "index.yaml" has been uploaded (though excluded from > > voting), so you can also run the below commands. > > > > helm repo add apache-airflow-dev > > https://dist.apache.org/repos/dist/dev/airflow/helm-chart/1.14.0rc1/ > > helm repo update > > helm install airflow apache-airflow-dev/airflow > > > > airflow-1.14.0.tgz.prov - is also uploaded for verifying Chart Integrity, > > though not strictly required for releasing the artifact based on ASF > > Guidelines. > > > > $ helm gpg verify airflow-1.14.0.tgz > > gpg: Signature made Fri Jun 14 14:41:31 2024 MDT > > gpg:using RSA key > E1A1E984F55B8F280BD9CBA20BB7163892A2E48E > > gpg:issuer "jedcunning...@apache.org" > > gpg: Good signature from "Jed Cunningham " > > [ultimate] > > plugin: Chart SHA verified. > > sha256:206d7eae00697bfd2fbe896b58adf9d66f928f70ee75a07e5772be86c9ed6185 > > > > The vote will be open for at least 72 hours (2024-06-17 20:59 UTC) or > until > > the necessary number of votes is reached. > > > > > > > https://www.timeanddate.com/countdown/to?iso=20240617T2059&p0=136&font=cursive > > > > Please vote accordingly: > > > > [ ] +1 approve > > [ ] +0 no opinion > > [ ] -1 disapprove with the reason > > > > Only votes from PMC members are binding, but members of the community are > > encouraged to test the release and vote with "(non-binding)". > > > > Consider this my (binding) +1. > > > > For license checks, the .rat-excludes files is included, so you can run > the > > following to verify licenses (just update your path to rat): > > > > tar -xvf airflow-chart-1.14.0-source.tar.gz > > cd airflow-chart-1.14.0 > > java -jar apache-rat-0.13.jar chart -E .rat-excludes > > > > Please note that the version number excludes the `rcX` string, so it's > now > > simply 1.14.0. This will allow us to rename the artifact without > modifying > > the artifact checksums when we actually release it. > > > > The status of testing the Helm Chart by the community is kept here: > > https://github.com/apache/airflow/issues/40248 > > > > Thanks, > > Jed > > >
Re: Call with Nielsen team demoing their DAG debugging feature
I agree with you Jarek. Every developer has their own way of debugging things and sharing those with the community including the best practices. There is always scope for improvement to the documentation! Thanks & Regards, Amogh Desai On Fri, Jun 14, 2024 at 12:17 PM Jarek Potiuk wrote: > Also I would like to drag attention of those who were interested in the > subject on two related PRs: > > * https://github.com/apache/airflow/pull/40010 by jannisko (sorry if you > see it - I do not know your real name :) ) -> where you can run a dag test > while skipping (or actually mark as success) some tasks (for example > sensors) > * https://github.com/apache/airflow/pull/40205 by Vincent -> where you can > run dag test using executor rather than `_run_raw_task` > > I think - the debug feature that Nielsen showcased on the call falls in the > same pattern "make our airflow dags test` more powerful - and it would be > great if we could incorporate similar pattern - where you can recreate task > context from already executed dag_run - as part of the `airflow dags test` > CLI command and `dag.test()` method. > > This is also I think a good opportunity to enhance documentation and > explain all those patterns on how you can debug dag - in > > https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/debug.html#testing-dags-with-dag-test > - for now the documentation is rather bare-bone, but it would be great if > we explain some of the best practices and use cases how dag debugging might > be done using the `airflow dags test` command. > > I think maybe other people have their own patterns of testing DAGs that > they could contribute here - both as documentation update and maybe new > features of our existing "airlfow dags test" command. > > WDYT? > > J. > > > On Thu, Jun 13, 2024 at 10:29 PM Stefan Krawczyk > wrote: > > > +1 for the recording please. > > > > On Thu, Jun 13, 2024, 1:26 PM Jarek Potiuk wrote: > > > > > Just summarizing the call: > > > > > > * we had demo from Nielsen showing their debugging feature > > > * It's based on environments created in the research environment where > > > users can run DAGs and debug individual tasks - basically replaying and > > > debugging the tasks based on existing DAG runs but without saving any > > > changes to state of the dag in the DB > > > * pretty useful thing is the way how they can use existing DAG run to > > > recreate the context of execution based on existing dag run > > > * Nielsen team used it with Airflow 1.10 and they will look into how > the > > > new `dag.test()` feature from Airflow 2.5 can be plugged into it and > come > > > back to it > > > * nice thing is that they hooked it up with VSCode plugin where they > can > > > easily do all that within VSCode and can debug it out-of-the-box > > > * possibly they could generalise it either as a "what could be done by > > > others" description or maybe even having VSCode-from-airflow > > out-of-the-box > > > (the latter was my brainstorming idea). > > > > > > I have a recording - I do not want to publish it on the public devlist, > > but > > > If anyone is interested - let me know and I will share. > > > > > > J. > > > > > > > > > On Thu, Jun 13, 2024 at 6:49 AM Albert Okiri > > > wrote: > > > > > > > Hi Jarek, I'm interested in joining this call. > > > > > > > > Regards, > > > > Albert. > > > > > > > > On Thu, 13 Jun 2024, 07:43 Poorvi Rohidekar, < > > > > poorvirohidekar@gmail.com> > > > > wrote: > > > > > > > > > Hi Jarek, > > > > > > > > > > I'd be interested in joining this call. > > > > > > > > > > Regards, > > > > > Poorvi > > > > > > > > > > On Tue, 11 Jun 2024 at 21:42, Jarek Potiuk > wrote: > > > > > > > > > > > And yes. I will check with them about recording :) > > > > > > > > > > > > On Tue, Jun 11, 2024 at 4:41 PM Jarek Potiuk > > > wrote: > > > > > > > > > > > > > I think I have to warn Nielsen team that we are going to have a > > big > > > > > crowd > > > > > > > :) > > > > > > > > > > > > > > On Tue, Jun 11, 2024 at 4:40 PM Bishundeo, Rajeshwar > > > > > > > wrote: > > >
Re: [Meeting Notes] Airflow 3.0 Dev call - 4 June 2024
Amazing summary @Kaxil Naik ! I reviewed the discussion points and I am happy with them so far. I only have a few minor comments which I have posted on confluence. Looking forward to attending these calls in future. Thanks & Regards, Amogh Desai On Sat, Jun 8, 2024 at 12:15 PM Jarek Potiuk wrote: > Very good summary Kaxil. Reviewed it and it looks like all things we > discussed are captured well. Also I am really happy that we are starting to > formalize and converge on the set of basic principles - they will help us > to focus on particular "streams" while keeping those in mind when we are > going to more and more details in each of the "streams" independently. > > J. > > > On Sat, Jun 8, 2024 at 4:17 AM Kaxil Naik wrote: > > > Hey all, > > > > I have updated our meeting notes document to summarize the discussion > from > > our dev call for Airflow 3.0 on 4th June. > > > > Link: > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024 > > > > To all those who attended, can you please double-check and add if I have > > missed anything? > > > > To all those who didn't join, if you disagree with anything in the > Summary, > > please voice your opinion. > > > > I will send a separate email for the agenda for the next meeting on 13 > > June. > > > > Regards, > > Kaxil > > > > -- > > > > Including the Summary here too (might break formatting): > > > > The following *principles* were agreed upon to drive Airflow 3 > development: > > 1) For the features that require breaking changes, ship Airflow 3 with > the > > foundational code to allow for iterative development, optimizing for > speed > > and a quicker feedback cycle. > > - Additional features that do not require breaking changes can be > > included in minor releases such as 3.1, 3.2, and beyond since we follow > > SemVer. > > - Discussion points: > > - Examples: > > - Add the hook points with a few references (fetcher for GCS, S3, > > Git) for DAG Versioning’s Execution AIP (AIP-66). > > - Removing dependency on FAB in Core: The new plugin framework > might > > support only a few functionalities, which are then built upon later. > > - For the multi-language AIP, start with only Python + one more > > language in AF 3.0 and then 3.1 and later minor versions can have more > > support for languages like Typescript. > > - Identify users who would be willing to give feedback during dev and > > beta snapshots. The Astronomer, MWAA, and GCC teams can help identify > > these. > > > > 2) Ensure a smoother migration path between Airflow 2 and 3, particularly > > for DAG authors using the existing official Airflow providers. > > - Directionally, the time required to update DAGs should be measured in > > hours, not days or months. > > - Action Items: > > - Update the AIP template to ask for the level of effort (manual and > > automated) needed for the users to adapt to the breaking changes. This > > should include high-level details on what could go in the upgrade > > utilities. This will help the AIP authors consciously think through the > > migration efforts. (Kaxil Naik) > > - For AIPs, be explicit about what’s for AF 3.0 and what’s for the > next > > minor releases (3.1, 3.2, ...). > > - Action Items: > > - Complete the housekeeping of the AIPs (Kaxil Naik): > > - Move the AIPs that we don’t plan to work on in Abandoned > state. > > - Add labels if an AIP is for Airflow 2, 3.0 or >= 3.1. These > > labels will be used via Macros to auto-populate the tables in Airflow > > Improvement Proposals, making it a good page for Roadmap items. > > > > 3) Build features that solidify Airflow as the modern Orchestrator that > has > > state-of-the-art support for Data, AI & ML workloads. > > - This includes enhancing scalability, performance, and > enterprise-level > > security, adhering to the principle of least privilege. > > - Making Airflow aware of what’s happening in the task to provide > better > > auditability, lineage & observability. > > > > 4) Set up the codebase for the next 3-5 years. > > - Reducing the matrix of supported combinations for reducing complexity > > in testing & development. E.g., Remove MySQL support to reduce the test > > matrix. > > - Simplifying codebase & standardize architecture (e.g., consolidating > > se
Re: [DISCUSS] common.compat provider (WAS: Common.util provider?)
I like the idea of having a common.compat provider too. (common.util was
actually kinda confusing)
* when providers get >= airflow 2.10 - we change them to import from
> `airflow.openlineage` rather than from "airflow.providers.common.compat".
>
I am not so sure why this is the case. Can you elaborate?
Thanks & Regards,
Amogh Desai
On Mon, Jun 10, 2024 at 4:15 PM Jakub Dardziński wrote:
> As the author of https://github.com/apache/airflow/pull/39530 I love the
> idea.
>
> * when providers get >= airflow 2.10 - we change them to import from
> > `airflow.openlineage` rather than from "airflow.providers.common.compat".
> >
> What's the reasoning behind that? How would Airflow core release impact
> providers dependencies?
>
> pon., 10 cze 2024 o 11:08 Maciej Obuchowski
> napisał(a):
>
> > I think it's a good solution.
> > The only known problem with that idea is that the common code has to live
> > "forever" - as long as someone can use the older providers (or older
> > Airflow version).
> > The solution would be to introduce some explicit deprecation or
> versioning
> > for provider dependencies - but that's not really possible due to lack of
> > constraints
> > for optional dependencies.
> >
> > sob., 8 cze 2024 o 22:00 Jarek Potiuk napisał(a):
> >
> > > I have an idea about that one, and probably that one will fulfill the
> > > "polyfill" approach discussed earlier.
> > >
> > > I think we should not name the provider "common.util" but
> > "common.compat" -
> > > because all the code that we need to put there is really about keeping
> > > compatibility.
> > >
> > > For example look here https://github.com/apache/airflow/pull/39530
> > >
> > > We have a need to have a "compatibility" code somewhere that a number
> of
> > > providers could use in case we want to keep some backwards
> compatibility.
> > >
> > > So having a "common.compat" provider would likely nicely full-fill the
> > > polyfill approach - It should only contain the code that we aim to keep
> > > backwards compatibility
> > >
> > > Example for https://github.com/apache/airflow/pull/39530
> > >
> > > * we add the complex compatibility code (see
> > > https://github.com/apache/airflow/pull/39530#issuecomment-2145670785)
> in
> > > the "common.compat" provider - and to airflow.openlineage in this case
> > > * we import it from there in all providers that need it (this will
> > > automatically add dependency)
> > > * when providers get >= airflow 2.10 - we change them to import from
> > > `airflow.openlineage` rather than from
> "airflow.providers.common.compat".
> > >
> > > We could apply similar approach for other "compatibility" code
> > >
> > > J.
> > >
> > >
> > >
> > >
> > > On Thu, Apr 11, 2024 at 10:22 AM Jarek Potiuk
> wrote:
> > >
> > > > Any other ideas or suggestions here? Can someone explain how the
> > > > "polypill" approach would look like, maybe? How do we imagine this
> > > working?
> > > >
> > > > Just to continue this discussion - another example.
> > > >
> > > > Small thing that David wanted to add for changes in some sql
> providers:
> > > >
> > > > @contextmanager
> > > > def suppress_and_warn(*exceptions: type[BaseException]):
> > > > """Context manager that suppresses the given exceptions and logs
> a
> > > > warning message."""
> > > > try:
> > > > yield
> > > > except exceptions as e:
> > > > warnings.warn(f"Exception suppressed:
> > > > {e}\n{traceback.format_exc()}", category=UserWarning)
> > > >
> > > >
> > > >
> > > >
> > >
> >
> https://github.com/apache/airflow/pull/38707/files#diff-6e1b2f961cb951d05d66d2d814ef5f6d8f8bf8f43c40fb5d40e27a031fed8dd7R115
> > > >
> > > > This is a small thing - but adding it in `airflow` is problematic -
> > > > because it will only be released in 1.10, so we cannot use it in
> > > providers
> > > > if we do.
> > > > Currently - since it is used in sql providers, I suggested using
> > > > `common.sql` for that code (and add >= 1.12 for common-sql-providers
> >
Re: Call with Nielsen team demoing their DAG debugging feature
Hello Jarek, Please add me to the invite as well. Thanks & Regards, Amogh Desai On Mon, Jun 10, 2024 at 11:22 AM Abhishek Bhakat wrote: > Hi Jarek, > > I would also like to join as well, please. > > Thanks, > Avi > > On Sat, Jun 8, 2024 at 3:32 PM Buğra Öztürk > wrote: > > > Hello Jarek, > > > > Thanks for sharing! It sounds very interesting. I would like to join. > Could > > you please forward to me as well? > > > > Thanks! > > > > On Sat, 8 Jun 2024, 17:28 Jed Cunningham, > > wrote: > > > > > Interesting. Can you forward to me as well Jarek? Thanks! > > > > > >
Re: [PROPOSAL] Automated managemenet of lower-bounds of airflow dependencies
Great work! Thanks & Regards, Amogh Desai On Tue, Jun 4, 2024 at 10:42 AM Jarek Potiuk wrote: > And ... merged :). We should now have very nice automated upgrading of > lower-bound constraints for Airflow and Providers working for all PRs. If > there will be any problems with your PRs related to that (failing > "LowestDeps" job on CI) - ping me on Slack. I might not be very responsive > - I am at "Community over Code EU" conference and leading data engineering > track but I will keep an eye on those in case there are some teething > problems. > > J, > > On Mon, Jun 3, 2024 at 3:14 PM Jarek Potiuk wrote: > > > Thanks for that work - I'm sure it will find many bugs before they are > >> found by users :) > >> > > > > Yep. That's the main reason we want to do it :) > > > > BTW. I think I am about to complete iterating on something that Jens > > noticed - i.e. that we need to expand the concept to test it on all > Python > > versions and I hope we will be able to merge it really soon :). > > >
Re: [ANNOUNCE] Podcast Launch- The Data Flowcast: Mastering Airflow for Data Engineering & AI
This is very cool! Thanks & Regards, Amogh Desai On Sat, Jun 1, 2024 at 4:44 PM Jarek Potiuk wrote: > Cool > > On Thu, May 30, 2024 at 7:02 PM Briana Okyere > wrote: > > > Hey All, > > > > Very excited to announce the relaunch of the Airflow Podcast, now titled > > "the relaunch of our podcast, now titled "The Data Flowcast: Mastering > > Airflow for Data Engineering & AI." > > > > This podcast is specially designed for the Apache Airflow community and > > aims to share invaluable insights, useful tips, and engaging discussions > > about the current and future trends of Airflow. > > > > Our first episode features a discussion with Alexander Booth, Director of > > R&D at The Texas Rangers on how they powered a World Series victory with > > Airflow. < > > > > > https://open.spotify.com/episode/1wduswN7sZSebUVL6Y5hMH/?utm_source=slack&utm_medium=organicsocial&utm_campaign=podcastlaunch > > > > > > > Listen/Watch on Spotify, Apple Podcasts, and YouTube! > > > > Spotify: <https://open.spotify.com/show/7tOwbp0j47DEir7dP7WRJj> > > > > Apple: < > > > > > https://podcasts.apple.com/us/podcast/the-data-flowcast-mastering-airflow-for-data/id1337349579 > > > > > > > YouTube: < > > https://www.youtube.com/playlist?list=PLCi-q9vYo4x80aQhkskTqKmLjOPebum_B > > > > > > Webpage: <https://www.astronomer.io/podcast/> > > > > -- > > Briana Okyere > > Community Manager > > Astronomer > > >
Re: [REMINDER] Airflow 3 Dev call today
Hi Kaxil, I wont be able to make it today because I am travelling. Will we be recording these meetings as well? Thanks & Regards, Amogh Desai On Tue, 4 Jun 2024 at 3:37 PM, Kaxil Naik wrote: > Hello all, > > Just a reminder that we will have our dev call today. > > Agenda: > 1) Agreeing on the Principles to drive Airflow 3 development > 2) Agreeing on the Guidelines that help decide if a feature should be in > Airflow 3 or not > > The proposed principles & guidelines are at > > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes > > A summary of the call will be sent to the mailing list and also posted on > the wiki > < > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes > > > . > > Regards, > Kaxil >
Re: [VOTE] Airflow Providers prepared on May 26, 2024
+1 non binding Tested my changes for Hive Provider and things work fine. Thanks & Regards, Amogh Desai On Tue, May 28, 2024 at 8:40 AM Josh Fell wrote: > +1 (non-binding) > > On Mon, May 27, 2024 at 7:24 PM Wei Lee wrote: > > > -1 (non-binding) for apache-airflow-providers-cncf-kubernetes, as Rahul > > found an issue. > > The issue is fixed in <https://github.com/apache/airflow/pull/39874>, as > > mentioned. > > > > Best, > > Wei > > > > > > > On May 28, 2024, at 12:17 AM, Vincent Beck > wrote: > > > > > > +1 non binding. All AWS system tests are running successfully against > > apache-airflow-providers-amazon==8.23.0rc1. You can see the results here: > > > https://aws-mwaa.github.io/#/open-source/system-tests/version/98c5a3a2c6d1df722d56bb3748dfbc810d5952aa_8.23.0rc1.html > > > > > > On 2024/05/27 14:01:48 Rahul Vats wrote: > > >> -1 (non-binding) for cncf-kubernetes-provider. Our example DAGs are > > failing > > >> due to an unexpected argument 'pod' in the read_namespaced_pod_log > call. > > >> Thanks to Wei for raising the fix PR > > >> <https://github.com/apache/airflow/pull/39874>. > > >> > > >> Regards, > > >> Rahul Vats > > >> 9953794332 > > >> > > >> > > >> On Mon, 27 May 2024 at 01:31, Jarek Potiuk wrote: > > >> > > >>> +1 (binding) - checked reproducibility, checksums, licences. > > signatures. > > >>> Checked my changes. > > >>> > > >>> On Sun, May 26, 2024 at 11:03 AM Elad Kalif > > wrote: > > >>> > > >>>> Hey all, > > >>>> > > >>>> I have just cut the new wave Airflow Providers packages. This email > is > > >>>> calling a vote on the release, which will last for 72 hours - which > > means > > >>>> that it will end on May 29, 2024 09:00 AM UTC and until 3 binding +1 > > >>> votes > > >>>> have been received. > > >>>> > > >>>> Consider this my (binding) +1. > > >>>> > > >>>> Airflow Providers are available at: > > >>>> https://dist.apache.org/repos/dist/dev/airflow/providers/ > > >>>> > > >>>> *apache-airflow-providers--*.tar.gz* are the binary > > >>>> Python "sdist" release - they are also official "sources" for the > > >>> provider > > >>>> packages. > > >>>> > > >>>> *apache_airflow_providers_-*.whl are the binary > > >>>> Python "wheel" release. > > >>>> > > >>>> The test procedure for PMC members is described in > > >>>> > > >>>> > > >>> > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members > > >>>> > > >>>> The test procedure for and Contributors who would like to test this > > RC is > > >>>> described in: > > >>>> > > >>>> > > >>> > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors > > >>>> > > >>>> > > >>>> Public keys are available at: > > >>>> https://dist.apache.org/repos/dist/release/airflow/KEYS > > >>>> > > >>>> Please vote accordingly: > > >>>> > > >>>> [ ] +1 approve > > >>>> [ ] +0 no opinion > > >>>> [ ] -1 disapprove with the reason > > >>>> > > >>>> Only votes from PMC members are binding, but members of the > community > > are > > >>>> encouraged to test the release and vote with "(non-binding)". > > >>>> > > >>>> Please note that the version number excludes the 'rcX' string. > > >>>> This will allow us to rename the artifact without modifying > > >>>> the artifact checksums when we actually release. > > >>>> > > >>>> The status of testing the providers by the community is kept here: > > >>>> https://github.com/apache/airflow/issues/39842 > > >>>> > > >>>> The issue is also the easiest way to
Re: Airflow 3 Dev Calls: Registration + Schedule
Lovely! Looking forward to it. Btw, as Jarek mentioned, many folks would be travelling for Community Over Code, next week and we also have some public holidays in some countries. Could we revise based on that? Btw, @Kaxil Naik , the calendar link: https://astronomer.zoom.us/meeting/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG/calendar/google/add takes me to Astronomer SSO. Thanks & Regards, Amogh Desai On Tue, May 28, 2024 at 1:56 AM Jarek Potiuk wrote: > Just a note - the first meeting is on 30th of May which is holiday in quite > a number of countries (Corpus Cristi) > https://www.officeholidays.com/holidays/corpus-christi - and taking > into account it's always Thursday, usually people bridge it with weekend > :). I personally will not ne available. > > Maybe it's worth moving the first call to next Thursday instead (6th) ? > > Personally, that would also be after the Community Over Code conference > that happens Mon-Wed in Bratislava (I am leading data engineering track > there and talk about Airflow too). > > J > > > On Sat, May 25, 2024 at 1:36 AM Kaxil Naik wrote: > > > Hi all, > > > > If you would like to participate in the development of Airflow 3, please > > join the dev calls starting next week. The calls will be open to anyone > in > > the community. > > > > *Schedule*: Starting May 30, 2024, at 04:00 PM BST (3 PM GMT/UTC | 11 AM > > EST | 8 AM PST), it will be weekly until we agree on the principles and > > fortnightly after that. > > *One-time registration Link*: > > > > > https://astronomer.zoom.us/meeting/register/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG > > *Add to your calendar*: > > > > > https://astronomer.zoom.us/meeting/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG/calendar/google/add > > > > The meeting notes from the call will also be posted on the dev mailing > list > > and Confluence for archival purposes (for example > > <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes>). At > > the end of each call, I would solicit ideas for the agenda for the next > > call and propose it to the broader group on the mailing list. > > > > Some of the items that should be discussed in the upcoming calls IMO: > > > >- Agreeing on Principles > >- Agreeing on the Guidelines that help decide if a feature should be > in > >Airflow 3 or not. > >- Workstream & Stream Owners > >- Airflow 2 support policy > >- Separate discussions for each big workstream including one for items > >to remove & refactor (e.g dropping MySQL) > >- Discussion to streamline the development of Airflow 3 > >- Finalize Scope + Timelines > >- Migration Utilities > >- Progress check-ins > > > > I will send a separate email for the first dev call in the next few days. > > > > Regards, > > Kaxil > > >
Re: [DISCUSS] Proposal to enhance Backfills
Good proposal! I like the idea here but again talking in terms of timelines, do we make it in Airflow 2 if it's that critical or can it wait till Airflow 3? I think we should scope this out and add some technical data to back this up before making this an AIP. Thanks & Regards, Amogh Desai On Sun, May 26, 2024 at 4:25 PM Jarek Potiuk wrote: > Yes. Long time awaited - and indeed some implementation details would be > needed to get it to AIP. And I also think one important decision to > consider - should it be targeting Airflow 2? > > On Sun, May 26, 2024 at 12:26 PM Elad Kalif wrote: > > > > In order for this to become a reality, Backfills need to be handled by > > the > > Airflow Scheduler as a normal DAG execution > > > > I think it's a good idea. > > It should solve natively problems like > > https://github.com/apache/airflow/issues/11302 > > > > On Fri, May 24, 2024 at 10:58 PM Vikram Koka > > > > > wrote: > > > > > Fellow Airflowers, > > > > > > I am following up on some of the proposed changes in the Airflow 3 > > proposal > > > < > > > > > > https://docs.google.com/document/d/1MTr53101EISZaYidCUKcR6mRKshXGzW6DZFXGzetG3E/ > > > >, > > > where more information was requested by the community. > > > > > > One specific topic was "Running Backfills at scale". This is not yet a > > full > > > fledged AIP, but a starting point for the discussion leading towards an > > AIP > > > with fully defined technical details. > > > Backfills at scale > > > > > > Backfills in Airflow 2.x are treated as an exception and executed by an > > > incarnation of the BackfillJob, rather than the regular Airflow > Scheduler > > > itself. This results in unexpected interactions with the other DAGs > being > > > run by the main Airflow Scheduler at the same time including resource > > > contention and possibly unexpected delays because established > scalability > > > configuration settings such as Concurrency are not consistently > applied, > > > and also code-level complexity by having two somewhat-similar > > > implementations of scheduling logic. > > > > > > > > > However, with ML model training, backfills are a common operation and > > need > > > to be treated as a regular Airflow DAG / Task execution operation and > not > > > treated as an exception. It is also not possible to run a backfill > unless > > > you have direct access to the Airflow database/SSH access to the > Airflow > > > server , which is not possible for many/most data engineers. > > > > > > > > > In order for this to become a reality, Backfills need to be handled by > > the > > > Airflow Scheduler as a normal DAG execution, building on the Dynamic > Task > > > Mapping execution pattern, rather than an exception. Additionally, > > Backfill > > > tasks will now ONLY be executed by the Airflow Workers, for obvious > > reasons > > > including scalability. A less obvious, but important reason is > Security, > > > since it is ideal to have data connections to Enterprise data only > happen > > > through Airflow Workers, rather than any Airflow system components. > > > > > > > > > As part of making Backfill support cleaner in Airflow, Backfill DAG > > > execution will also be supported in the Airflow REST API. > > > > > > > > > This proposal is purposefully light on exact implementation details but > > > will include at least: > > > > > > > > > > > >- > > > > > >Making the Airflow Scheduler responsible for scheduling decisions on > > all > > >DagRuns (instead of the current where it purposefully ignores > backfill > > > runs) > > >- > > > > > >A new API endpoint to submit a "backfill request". > > > > > > > > > -- > > > > > > > > > Best regards, > > > Vikram Koka, Ash Berlin-Taylor, Kaxil Naik, and Constance Martineau > > > > > >
Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach
I agree with Andrey too on this.
Thanks & Regards,
Amogh Desai
On Fri, May 17, 2024 at 7:42 PM Kaxil Naik wrote:
> Agreed on your points @andrey.ans...@taragol.is
>
> On Fri, 17 May 2024 at 15:01, Andrey Anshin
> wrote:
>
> > IMHO, In case if we decide to keep only Postgres support we need to have
> > really powerful arguments to provide an interface which helps integrate
> > with other DBs.
> >
> > In this case, we must clearly understand what the community is
> responsible
> > for in this case and how it can be sure that nothing is broken
> >
> > Especially if we take in account the Airflow has very tight integrations
> > with specific Databases, requires a lot of effort to support additional
> > ones (MS SQL case), and the DB part is not a part of the Public Interface
> > of Airflow [1].
> >
> > So I would consider that this should be two separate decisions:
> > 1. Keep only Postgres (vanila, not forks) as supported/tested backend in
> > Production. SQLite remains as development DB.
> > 2. Provide public interface to DB integrations between Airflow and DB for
> > third parties
> >
> > [1]:
> >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/public-airflow-interface.html
> >
> >
> > On Tue, 14 May 2024 at 12:15, Kaxil Naik wrote:
> >
> > > Yeah, that works for me
> > >
> > > Can we have it possible to have two (or maybe three -
> > > > like a sub-committee) co-owners of topics?
> > >
> > >
> > > On Tue, 14 May 2024 at 06:15, Vikram Koka >
> > > wrote:
> > >
> > > > Definitely a fast moving thread on the mailing list. I haven’t been
> > able
> > > to
> > > > respond for a few days and feel very far behind already.
> > > >
> > > > A few comments on topics discussed the last few days:
> > > > - Jarek, in response to your comments around being more aggressive
> than
> > > in
> > > > Airflow 2 about deprecation and drops of functionality, I am very
> > > > supportive of that stance. I completely agree that we could have been
> > > more
> > > > aggressive as part of Airflow 2.
> > > > However, I would like to ask that as we go forward, we make sure that
> > we
> > > > have clean interfaces to be able to add support, even if we choose a
> > > single
> > > > implementation. For example, with respect to dropping MySQL support.
> I
> > > can
> > > > understand the perspective of the project that this should be
> > deprecated
> > > > from an Airflow OSS perspective. However, even if the only OSS
> > supported
> > > DB
> > > > is Postgres, I would like to ensure that a clean interface exists for
> > > > interaction with the DB, so that other databases such as MySQL or
> > others
> > > > CAN be supported by a third party or at a later date.
> > > > I realize that this may seem onerous, but I believe that it enables
> us
> > to
> > > > be more flexible in the long run, rather than locking us into a
> single
> > DB
> > > > implementation.
> > > >
> > > > - Bolke, Daniel Standish, Ash, et al on the task execution contract,
> > > > definitely looking forward to this.
> > > >
> > > > - To those that I proposed a couple of more detailed write ups, I
> still
> > > > plan to do that, at the latest by early next week.
> > > >
> > > > Vikram
> > > >
> > > >
> > > >
> > > > On Mon, May 13, 2024 at 9:30 PM Jarek Potiuk
> wrote:
> > > >
> > > > > Super-excited about that.
> > > > >
> > > > > Question/Proposal: Can we have it possible to have two (or maybe
> > three
> > > -
> > > > > like a sub-committee) co-owners of topics? I think it's a lot to
> put
> > on
> > > > > one's head to "own" a topic and given circumstances/ volunteer time
> > of
> > > > > people, interruptions (and life intervening), it might be a bit
> risky
> > > to
> > > > > put it on one's shoulders only.
> > > > >
> > > > > I know it's against the rule ("if it is owned by many, it's not
> owned
> > > by
> > > > > anyone") - but I think in our case there are at least some topics
> > that
> > > > > could benefit from havin
Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach
Thanks for initiating this discussion @Kaxil Naik ! Read through the document and the emails here and I like the direction in which we are proceeding. I also agree that removing code is almost as important as adding code. We need to have a clear checklist of what we want to remove and get a vote for the same from the community. I also see that we have multiple ways of doing a thing, which is not needed (serialising for example) and removing which can lead to simpler interfaces for users, and contributors too. I also find the *secret masker* overly complex personally, why can't it be simpler :) However, I have a small concern with the proportion of the users that would like to migrate to an early version of Airflow 3 if we do not support various deployment modes. If we dont get enough initial feedback, we do not know how the new major is doing, no? Thanks & Regards, Amogh Desai On Thu, May 9, 2024 at 1:03 PM Ash Berlin-Taylor wrote: > Strong agree - I think one of the things I regret most about Airflow 2 was > that we didn't remove enough. It probably eased the upgrade process for > users though. > > > but I was hoping one day when we remove providers > > from main development, I can do it personally and beat Ash to it. > > Grrr ;l > > The other thing I want to look at (but don't have concrete ideas of yet) > is some of the lesser used dag/task concurrency and dependency features - > if we can remove a few things and get a faster scheduler than it's worth > having a discussion over. > > On 9 May 2024 06:31:38 BST, Jarek Potiuk wrote: > >I hope more voices will be coming soon as well :). > > > >Constance - few words of explanation, for my "remove" things. I do not say > >we "have to" remove those things or even that we "should". > > > >My main motivation for mentioning the list is that we often forget that > >when we have a new major version of software - removing stuff is as > >important as adding. So I think when it comes to final decisions about > >Airflow 3 we should be well aware about both: what we add and what we > >remove. So my main point is - "removal" of things should be as important a > >part in our future discussions as "adding", and I'd say with 10 years of > >Airflow history, it's more important to remove things than to add them. > > > >Side comment: Ash for one mentioned at multiple occasions how proud he is > >that his contributions to Airflow are net-negative - he removed more code > >than added. Unfortunately it's not shown any more here > >https://github.com/apache/airflow/graphs/contributors - because we have > >more than 10.000 commits, but I was hoping one day when we remove > providers > >from main development, I can do it personally and beat Ash to it. > > > >Just wanted to stress a few things on why I think "discussing potential > >removals" is important: > > > >* I think it is super important to deliver Airflow 3 really fast. It took > > > >1.5 year to release Airflow 2 and it was far too long and keeping Airflow > 1 > >and 2 in parallel was a huge pain and drain and it slowed us down > >enormously. We should absolutely limit the time when we actively develop > >both Airflow 2 features and Airflow 3 features. > >* I think most of the current deprecations of the 100+ we have does not > >slow us down AT ALL. Removal of those is mostly cosmetic change, a little > >bit clutter to remove but they have little-to-no impact on actual speed of > >development, it's just an old code that keeps on being around > >* on the other hand - some of the things I mentioned ARE slowing us down A > >LOT and will continue to do so until we remove them. For example, I > believe > >"postgresql + mysql + sqlite complete versioning solution for all the > >possible variants of storage" will be quite a bit more complex and testing > >will take far more time than if we drop mysql and choose a single storage > >approach for Airflow 3.0. I have no hard data to back it up, but my gut > >feeling is that it can take at least twice as long to get it out in the > >hands of our users if we try to have Airflow 2 parity in Airflow 3.0 for > >all the options we have there. > >* the telemetry will be cool - but even if we add it for 2.10, the first > >time meaningful data will be available is maybe 2 years from now when we > do > >Airflow 4. For now we can completely forget about it because it's only > >going to show us some early adopters of Airflow 2.10. But we have surveys > + > >we can collect data from Astronomer, Google. Amazon for some us
Re: [VOTE] AIP-67 Multi-team deployment of Airflow components
I agree with whatever decision comes out of this conversation going forward. I personally think the below option is the best given the fact that we might be doing Airflow 3 with a certain theme in mind: *only implement multi-team as an Airflow 3 feature (which might be mucheasier to do - depending on the scope of Airflow 3 changes we will target -some of the changes proposed have a significant overlap with the multi-teamproposal and we should make sure to discuss it as part of our Airflow 3planning.* Thanks & Regards, Amogh Desai On Thu, May 9, 2024 at 1:35 PM Jarek Potiuk wrote: > Just to clarify the state for that one. > > I would like to put that one on-hold until we get clarity on Airflow 2 vs > Airflow 3 approach: > https://lists.apache.org/thread/3chvg9964zvh15mtrbl073f4oj3nlzp2 > > There is currently a veto from Ash, so until it is withdrawn or we change > the problematic "team" database schema modification approach. I think > the choice we made here depends a lot on the Airflow 3 discussions. > > We have those options here: > > * Treat Airflow 2 multi-team approach as a "tactical" solution and > implement it in a non-future compliant way (and make use of the Airflow 3 > feature to implement it better for Airflow 3). This one is simplest and has > very limited impact on UI/API/DB etc. (so basically ripple effect) > * Implement Multi-team as Future-proof in Airflow 2 with proper schema > changes and ripple effects it might have for the UI, API and all the other > components > * only implement multi-team as an Airflow 3 feature (which might be much > easier to do - depending on the scope of Airflow 3 changes we will target - > some of the changes proposed have a significant overlap with the multi-team > proposal and we should make sure to discuss it as part of our Airflow 3 > planning. > > I currently do not know which option is best - as a lot depends on Airflow > 3 discussions. So I think putting this on hold and deciding what to do > after we have more clarity is the best approach. > > J. > > > > > On Wed, Apr 24, 2024 at 9:06 PM Mehta, Shubham > wrote: > > > +1 (non-binding). Looking forward to this one. > > > > Shubham > > > > On 2024-04-22, 12:31 AM, "Amogh Desai" > amoghdesai@gmail.com>> wrote: > > > > > > CAUTION: This email originated from outside of the organization. Do not > > click links or open attachments unless you can confirm the sender and > know > > the content is safe. > > > > > > > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > pouvez > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain > que > > le contenu ne présente aucun risque. > > > > > > > > > > > > > > +1 binding. > > > > > > Excited to see this happen! > > > > > > Thanks & Regards, > > Amogh Desai > > > > > > > > > > On Sat, Apr 20, 2024 at 12:11 AM Igor Kholopov > <mailto:ikholo...@google.com.inva>lid> > > wrote: > > > > > > > +1 (non-binding) > > > > > > Great to see this happening, hope we will see more proposals towards > > making > > > Airflow more flexible! > > > > > > Regards, > > > Igor > > > > > > On Fri, Apr 19, 2024 at 8:10 PM Daniel Standish > > > > daniel.stand...@astronomer.io.inva>lid> wrote: > > > > > > > > > > > > > It doesn’t affect my vote on the API, but I am very strongly > against > > > this > > > > > one part of the AIP: > > > > > > … dag_id are namespaced with `:` prefix. > > > > > This specific part is getting an implementation/code veto from me. > We > > > > made > > > > > the mistake of overloading one column to store multiple things in > > > Airflow > > > > > before, and I’ve dealt with the fallout in other apps in the past. > > > Trust > > > > > me: do. not. do. this. > > > > > > > > I agree with Ash's sentiment. Is adding a tenant_id or something so > > > > unpalatable? > > > > > > > > > > > > > > > >
Re: [VOTE] Proposal for adding Telemetry via Scarf
+1 binding
Thanks & Regards,
Amogh Desai
On Thu, May 9, 2024 at 10:29 AM Jarek Potiuk wrote:
> Short reminder and correction :).
>
> Wei Lee - as a committer, your vote is binding for any votes except
> releases. Releases are special - they are a legal act of the
> Apache Software Foundation, so when you vote on releases, only PMC votes
> are binding.
>
> Generally "releasing software" is what the ASF Foundation does. The
> foundation does not create software, only releases it ("for the public
> good").
> The PMC member is an official role in the ASF bylaws
> https://www.apache.org/foundation/bylaws.html (Apache Airflow is a PMC).
> This is according to Delaware laws - that's where Foundation is registered
> as https://en.wikipedia.org/wiki/501(c)(3)_organization non-profit
> organisation,
>
> This allows us to release software without the fear that someone will sue
> us personally if they are harmed by it, because if - as PMC members - we
> follow ASF rules (minimum 3 PMC members, reproducibility check, signatures,
> etc.) ASF indemnifies us personally from any harm done to anyone using that
> released software.
>
> But all the other decisions in Airflow are voted by committers:
> https://github.com/apache/airflow?tab=readme-ov-file#voting-policy - so
> committer votes are binding (except releases).
>
> J.
>
> On Thu, May 9, 2024 at 6:46 AM Jarek Potiuk wrote:
>
> > +1 (binding)
> >
> > On Thu, May 9, 2024 at 4:47 AM Wei Lee wrote:
> >
> >> +1 non-binding
> >>
> >> Best,
> >> Wei
> >>
> >> > On May 9, 2024, at 10:39 AM, Phani Kumar .INVALID>
> >> wrote:
> >> >
> >> > +1 binding, looking forward to add Scarf
> >> >
> >> > On Thu, May 9, 2024 at 7:42 AM Kaxil Naik
> wrote:
> >> >
> >> >> Hi all,
> >> >>
> >> >> Discussion thread:
> >> >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m
> >> >>
> >> >> I would like to officially call for a vote to add Scarf as a
> Telemetry
> >> >> tool. Some other things:
> >> >>
> >> >> - Opt-in by default
> >> >> - Explicit documentation that we collect the telemetry data
> >> >> - Opt-out via airflow.cfg and env var
> >> >> - Works for air-gapped environments
> >> >> - Initial access to only PMC members
> >> >>
> >> >> I have created a free account on Scarf and added it to the shared
> >> 1password
> >> >> (only PMC members have access to it). For now, I am just playing
> around
> >> >> with how the information can be shown.
> >> >>
> >> >> I have a draft PR: https://github.com/apache/airflow/pull/39510 that
> >> >> collects some basic info, adds docs & tests.
> >> >>
> >> >> Looking forward to releasing this for Airflow 2.10.
> >> >>
> >> >> Consider this my +1 binding vote.
> >> >>
> >> >> The vote will last until 04:20 GMT/UTC on May 16, 2024, and until at
> >> >> least 3 binding votes have been cast.
> >> >>
> >> >> Please vote accordingly:
> >> >>
> >> >> [ ] + 1 approve
> >> >> [ ] + 0 no opinion
> >> >> [ ] - 1 disapprove with the reason
> >> >>
> >> >> Only votes from PMC members and committers are binding, but other
> >> members
> >> >> of the community are encouraged to check the AIP and vote with
> >> >> "(non-binding)".
> >> >>
> >> >> Regards,
> >> >> Kaxil
> >> >>
> >>
> >>
> >> -
> >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
> >> For additional commands, e-mail: dev-h...@airflow.apache.org
> >>
> >>
>
Re: [DISCUSS] simplifying try_number handling
Looks good to me. Personally I never ran into any issues with this so far but I agree with the issues it solves. Thanks & Regards, Amogh Desai On Fri, May 3, 2024 at 2:50 AM Vincent Beck wrote: > I am all +1 on this one. This thing gave me headaches when working on > AIP-44 and I could not understand the difference between the private > "_try_number" and the public "try_number". Thanks for simplifying it! > > This is obviously assuming it does not break anything I am not aware of :) > > On 2024/05/02 19:37:32 Daniel Standish wrote: > > TLDR > > * changing handling of try_number in > > https://github.com/apache/airflow/pull/39336 > > * no more private attr > > * no more getter that changes value based on state of task > > * no more decrementing > > * try number now only handled by scheduler > > * hope that sounds good to all of you > > > > For more detail read on... > > > > In https://github.com/apache/airflow/pull/39336 I am doing some work to > > resolve some longstanding pain and frustration caused by try_number. > > > > The way we handle try_number has for quite some time been messy and > > problematic. > > > > For example, if you access `ti.try_number` and then change the state to > or > > from RUNNING, you will get a different value if you access it again! > > > > And the responsibility for managing this number has been distributed > > throughout the codebase. For example the task itself always increments > > when it starts running. But then if it defers or reschedules itself, it > > decrements it back down so that when it runs again and naively > increments, > > then it will be right again. > > > > Recently more issues have become visible as I have worked with AIP-44 > > because for example pydantic does not like private attrs and it's just > > awkward to know *what value to use* when serializing it when the TI will > > give you a different answer depending on the state of the task! > > > > And there's yet another edge case being solved in this community PR > > <https://github.com/apache/airflow/pull/38984#issuecomment-2090944403>. > > And then when we start looking at try history and AIP-64, it also > forces a > > look at this. > > > > So it all sounds bad and indeed it is bad but I think I have a solution. > > > > What I do is, just have the scheduler increment try_number at the moment > > when it schedules the task. It alone will have the responsibility for > > incrementing try_number. And no where will it ever be decremented. It > > will not be incremented when resuming after deferral or reschedule. And > > that's about all there is to it. > > > > I've tested it out and it works. But I'm working through many test > > failures that need to be resolved (there's lots of asserts re > try_number). > > > > One small thing I just want to point out is that if a user were > previously > > to be doing `task.run()` sort of manually without the task having been > > scheduled by the scheduler, well now their try_number won't be > > automatically incremented. Same if they just do `airflow tasks run` -- > > because now the responsibility is going to be solely with the scheduler. > > But airflow was never designed to assume that tasks will be run without > > having been scheduled, so I do not think that counts as a breaking > change. > > So I don't think that's a blocker for this. > > > > Thanks for the consideration. Let me know if you have any concerns. > > > > - > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
Re: [VOTE] Release Airflow 2.9.1 from 2.9.1rc2
+1 non binding Performed the same tests as RC1 and found no issues. Installed the RC, ran a few DAGs from my tests, and random clicks on UI. Thanks & Regards, Amogh Desai On Fri, May 3, 2024 at 1:03 AM Ephraim Anierobi wrote: > Hey fellow Airflowers, > > I have cut Airflow 2.9.1rc2. This email is calling a vote on the release, > which will last at least 30 hours, from Thursday, May 2, 2024 at 07:0 pm > UTC > until Saturday, May 4, 2024, at 01:30 am UTC > < > https://www.timeanddate.com/worldclock/fixedtime.html?msg=8&iso=20240504T0130&p1=1440 > >, > and until 3 binding +1 votes have been received. > > The status of testing of the release is kept at > https://github.com/apache/airflow/issues/39326 > > Consider this my (binding) +1. > > Airflow 2.9.1rc2 is available at: > https://dist.apache.org/repos/dist/dev/airflow/2.9.1rc2/ > > *apache-airflow-2.9.1-source.tar.gz* is a source release that comes with > INSTALL instructions. > *apache-airflow-2.9.1.tar.gz* is the binary Python "sdist" release. > *apache_airflow-2.9.1-py3-none-any.whl* is the binary Python wheel "binary" > release. > > Public keys are available at: > https://dist.apache.org/repos/dist/release/airflow/KEYS > > Please vote accordingly: > > [ ] +1 approve > [ ] +0 no opinion > [ ] -1 disapprove with the reason > > Only votes from PMC members are binding, but all members of the community > are encouraged to test the release and vote with "(non-binding)". > > The test procedure for PMC members is described in: > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-pmc-members > > The test procedure for contributors and members of the community who would > like to test this RC is described in: > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-contributors > > > Please note that the version number excludes the `rcX` string, so it's now > simply 2.9.1. This will allow us to rename the artifact without modifying > the artifact checksums when we actually release. > > Release Notes: > https://github.com/apache/airflow/blob/2.9.1rc2/RELEASE_NOTES.rst > > For information on what goes into a release please see: > > https://github.com/apache/airflow/blob/main/dev/WHAT_GOES_INTO_THE_NEXT_RELEASE.md > > Changes since 2.9.0rc1: > > *Bugs*: > Update trigger kwargs migration to specify existing_nullable ( #39361, > #39374) > > Cheers, > Ephraim >
Re: [VOTE] Airflow Providers prepared on May 01, 2024
+1 non binding Installed the tarball and ran some example DAGs for hive and cncf provider, works as expected. Thanks & Regards, Amogh Desai On Wed, May 1, 2024 at 10:49 PM Vincent Beck wrote: > +1 non binding. All AWS system tests are working successfully against > apache-airflow-providers-amazon==8.21.0rc1. You can see the results here: > https://aws-mwaa.github.io/#/open-source/system-tests/version/fe4605a10e26f1b8a180979ba5765d1cb7fb0111_8.21.0rc1.html. > The only failure (example_bedrock) is a known issue in the test itself and > is currently being worked on. > > On 2024/05/01 13:03:07 Elad Kalif wrote: > > Hey all, > > > > I have just cut the new wave Airflow Providers packages. This email is > > calling a vote on the release, > > which will last for 72 hours - which means that it will end on May 04, > 2024 > > 13:00 PM UTC and until 3 binding +1 votes have been received. > > > > > > Consider this my (binding) +1. > > > > *Note some of the providers are rc2 and some rc1.* > > > > Airflow Providers are available at: > > https://dist.apache.org/repos/dist/dev/airflow/providers/ > > > > *apache-airflow-providers--*.tar.gz* are the binary > > Python "sdist" release - they are also official "sources" for the > provider > > packages. > > > > *apache_airflow_providers_-*.whl are the binary > > Python "wheel" release. > > > > The test procedure for PMC members is described in > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members > > > > The test procedure for and Contributors who would like to test this RC is > > described in: > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors > > > > > > Public keys are available at: > > https://dist.apache.org/repos/dist/release/airflow/KEYS > > > > Please vote accordingly: > > > > [ ] +1 approve > > [ ] +0 no opinion > > [ ] -1 disapprove with the reason > > > > Only votes from PMC members are binding, but members of the community are > > encouraged to test the release and vote with "(non-binding)". > > > > Please note that the version number excludes the 'rcX' string. > > This will allow us to rename the artifact without modifying > > the artifact checksums when we actually release. > > > > The status of testing the providers by the community is kept here: > > https://github.com/apache/airflow/issues/39346 > > > > The issue is also the easiest way to see important PRs included in the RC > > candidates. > > Detailed changelog for the providers will be published in the > documentation > > after the > > RC candidates are released. > > > > You can find the RC packages in PyPI following these links: > > > > https://pypi.org/project/apache-airflow-providers-airbyte/3.8.0rc1/ > > https://pypi.org/project/apache-airflow-providers-alibaba/2.8.0rc1/ > > https://pypi.org/project/apache-airflow-providers-amazon/8.21.0rc1/ > > https://pypi.org/project/apache-airflow-providers-apache-beam/5.7.0rc1/ > > > https://pypi.org/project/apache-airflow-providers-apache-cassandra/3.5.0rc1/ > > https://pypi.org/project/apache-airflow-providers-apache-drill/2.7.0rc1/ > > > https://pypi.org/project/apache-airflow-providers-apache-druid/3.10.0rc1/ > > https://pypi.org/project/apache-airflow-providers-apache-flink/1.4.0rc1/ > > https://pypi.org/project/apache-airflow-providers-apache-hdfs/4.4.0rc2/ > > https://pypi.org/project/apache-airflow-providers-apache-hive/8.1.0rc1/ > > > https://pypi.org/project/apache-airflow-providers-apache-impala/1.4.0rc1/ > > https://pypi.org/project/apache-airflow-providers-apache-kafka/1.4.0rc2/ > > https://pypi.org/project/apache-airflow-providers-apache-kylin/3.6.0rc2/ > > https://pypi.org/project/apache-airflow-providers-apache-livy/3.8.0rc1/ > > https://pypi.org/project/apache-airflow-providers-apache-pig/4.4.0rc2/ > > https://pypi.org/project/apache-airflow-providers-apache-pinot/4.4.0rc2/ > > https://pypi.org/project/apache-airflow-providers-apache-spark/4.8.0rc2/ > > https://pypi.org/project/apache-airflow-providers-apprise/1.3.0rc2/ > > https://pypi.org/project/apache-airflow-providers-arangodb/2.5.0rc1/ > > https://pypi.org/project/apache-airflow-providers-asana/2.5.0rc1/ > > > https://pypi.org/project/apache-airflow-providers-atlassian-jira/2.6.0rc2/ > > https://pypi.org/project/apache-airflow-providers-celery/3.7.0rc1/
Re: [VOTE] Release Airflow 2.9.1 from 2.9.1rc1
+1 non binding Did a general testing by installing the RC, ran a few DAGs, performed some random clicks on the UI, things seem to be working as expected. Thanks & Regards, Amogh Desai On Wed, May 1, 2024 at 10:32 PM Scheffler Jens (XC-AS/EAE-ADA-T) wrote: > +1 (non binding) - tested 2.9.1rc1 and works great, especially the UI > glitches are removed. > > Mit freundlichen Grüßen / Best regards > > Jens Scheffler > > Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T) > Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen | > GERMANY | http://www.bosch.com/ > Tel. +49 711 811-91508 | Mobil +49 160 90417410 | > jens.scheff...@de.bosch.com > > Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000; > Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer; > Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. Markus > Forschner, > Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert > > -Original Message- > From: Ephraim Anierobi > Sent: Tuesday, April 30, 2024 1:04 PM > To: dev@airflow.apache.org > Subject: [VOTE] Release Airflow 2.9.1 from 2.9.1rc1 > > Hey fellow Airflowers, > > I have cut Airflow 2.9.1rc1. This email is calling a vote on the release, > which will last at least 72 hours, from Tuesday, April 30, 2024 at 11:00 am > UTC until Friday, May 3, 2024, at 11:00 am UTC < > https://www.timeanddate.com/worldclock/fixedtime.html?msg=8&iso=20240503T1100&p1=1440 > >, > and until 3 binding +1 votes have been received. > > The status of testing of the release is kept at > https://github.com/apache/airflow/issues/39326 > > Consider this my (binding) +1. > > Airflow 2.9.1rc1 is available at: > https://dist.apache.org/repos/dist/dev/airflow/2.9.1rc1/ > > *apache-airflow-2.9.1-source.tar.gz* is a source release that comes with > INSTALL instructions. > *apache-airflow-2.9.1.tar.gz* is the binary Python "sdist" release. > *apache_airflow-2.9.1-py3-none-any.whl* is the binary Python wheel "binary" > release. > > Public keys are available at: > https://dist.apache.org/repos/dist/release/airflow/KEYS > > Please vote accordingly: > > [ ] +1 approve > [ ] +0 no opinion > [ ] -1 disapprove with the reason > > Only votes from PMC members are binding, but all members of the community > are encouraged to test the release and vote with "(non-binding)". > > The test procedure for PMC members is described in: > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md/#verify-the-release-candidate-by-pmc-members > > The test procedure for contributors and members of the community who would > like to test this RC is described in: > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md/#verify-the-release-candidate-by-contributors > > > Please note that the version number excludes the `rcX` string, so it's now > simply 2.9.1. This will allow us to rename the artifact without modifying > the artifact checksums when we actually release. > > Release Notes: > https://github.com/apache/airflow/blob/2.9.1rc1/RELEASE_NOTES.rst > > For information on what goes into a release please see: > > https://github.com/apache/airflow/blob/main/dev/WHAT_GOES_INTO_THE_NEXT_RELEASE.md > > Changes since 2.9.0: > > *Significant Changes* > > *Stackdriver logging bugfix requires Google provider ``10.17.0`` or later > (#38071)* > > If you use Stackdriver logging, you must use Google provider version > ``10.17.0`` or later. Airflow ``2.9.1`` now passes ``gcp_log_name`` to the > ``StackdriverTaskHandler`` instead of ``name``, and this will fail on > earlier provider versions. > > This fixes a bug where the log name configured in ``[logging] > remove_base_log_folder`` was overridden when Airflow configured logging, > resulting in task logs going to the wrong destination. > > *Bug Fixes* > - Make task log messages include run_id (#39280) > - Copy menu_item ``href`` for nav bar (#39282) > - Fix trigger kwarg encryption migration (#39246) > - Add workaround for datetime-local input in ``firefox`` (#39261) > - Add Grid button to Task Instance view (#39223) > - Get served logs when remote or executor logs not available for > non-running task try (#39177) > - Fixed side effect of menu filtering causing disappearing menus (#39229) > - Use grid view for Task Instance's ``log_url`` (#39183) > - Improve task filtering ``UX`` (#39119) > - Improve rendered_template ``ux`` in react dag page (#39122) > - Graph view improvements (#38940) > - Check that the dataset<>task exists before trying to render graph > (#39069) > - Hostname was "redacted", not &
Re: [VOTE] April 2024 PR of the Month
My vote goes to #38674. Awesome work Thanks & Regards, Amogh Desai On Wed, 1 May 2024 at 9:21 PM, Jed Cunningham wrote: > I'll also vote for 38674 - cool stuff! >
Re: [VOTE] Release Apache Airflow Python Client 2.9.0 from 2.9.0rc1
+1 non binding Installed the RC with breeze and was able to test some basic functionalities of the client like listing DAGs, getting tasks, creating DAGruns, retrieving configuration, mainly through test_python_client.py. Looks good to me. Thanks & Regards, Amogh Desai On Tue, Apr 23, 2024 at 6:03 PM Pankaj Koti wrote: > +1 (non-binding) > > Tested the RC client to do some basic functionalities like get DAGs, tasks > for a DAG, > trigger a DAG, get Airflow configuration. Everything went well! > > Best regards, > > *Pankaj Koti* > Senior Software Engineer (Airflow OSS Engineering team) > Location: Pune, Maharashtra, India > Timezone: Indian Standard Time (IST) > > > On Tue, Apr 23, 2024 at 5:39 PM Jarek Potiuk wrote: > > > +1 (binding): tested reproducibility, signatures, checksums, licences > (code > > is autogenerated), run some basic tests with Airflow 2.9.0 with the RC1 > > client > > > > On Tue, Apr 23, 2024 at 1:05 AM Jed Cunningham > > > wrote: > > > > > Hey fellow Airflowers, > > > > > > I have cut the first release candidate for the Apache Airflow Python > > Client > > > 2.9.0. > > > This email is calling for a vote on the release, > > > which will last for 72 hours. Consider this my (binding) +1. > > > > > > Airflow Client 2.9.0rc1 is available at: > > > > https://dist.apache.org/repos/dist/dev/airflow/clients/python/2.9.0rc1/ > > > > > > The apache_airflow_client-2.9.0.tar.gz is an sdist release that > contains > > > INSTALL instructions, and also is the official source release. > > > > > > The apache_airflow_client-2.9.0-py3-none-any.whl is a binary wheel > > release > > > that pip can install. > > > > > > Those packages do not contain .rc* version as, when approved, they will > > be > > > released as the final version. > > > > > > The rc packages are also available at PyPI (with rc suffix) and you can > > > install it with pip as usual: > > > https://pypi.org/project/apache-airflow-client/2.9.0rc1/ > > > > > > Public keys are available at: > > > https://dist.apache.org/repos/dist/release/airflow/KEYS > > > > > > Only votes from PMC members are binding, but all members of the > community > > > are encouraged to test the release and vote with "(non-binding)". > > > > > > The test procedure for PMC members is described in: > > > > > > > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PYTHON_CLIENT.md#verify-the-release-candidate-by-pmc-members > > > > > > The test procedure for contributors and members of the community who > > would > > > like to test this RC is described in: > > > > > > > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PYTHON_CLIENT.md#verify-the-release-candidate-by-contributors > > > > > > *Changelog:* > > > > > > *Major changes:* > > > > > > - Allow users to write dag_id and task_id in their national characters, > > > added display name for dag / task (v2) (#38446) > > > - Add dataset_expression to grid dag details (#38121) > > > - Adding run_id column to log table (#37731) > > > - Show custom instance names for a mapped task in UI (#36797) > > > - Add excluded/included events to get_event_logs api (#37641) > > > - Filter Datasets by associated dag_ids (GET /datasets) (#37512) > > > - Add data_interval_start and data_interval_end in dagrun create API > > > endpoint (#36630) > > > - Return the specified field when get dag/dagRun (#36641) > > > > > > *New API supported:* > > > > > > - Add post endpoint for dataset events (#37570) > > > - Add "queuedEvent" endpoint to get/delete DatasetDagRunQueue (#37176) > > > > > > Thanks, > > > Jed > > > > > >
Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach
Thanks for starting the discussion, Jarek! I too agree that with the new upcoming features and AIPs, it might just be the right time to discuss the possibility of having Airflow 3. I agree with most reasons pointed out by others and I would love to see it happen, and also be a part of it. Since this is a major step for the future of Airflow, we need to carefully consider the user experience for users coming from versions of Airflow 2 and would not want this migration to be a pain. Btw, I concur with Jens and I too am not very clear when we say that "Gen AI is going to be the new trigger for Airflow". Would be obliged if someone could explain that portion to me :) Thanks & Regards, Amogh Desai On Mon, Apr 22, 2024 at 1:52 PM Jarek Potiuk wrote: > Just one comment here - while maybe "shocking" for some cases - yes, this > one has been clearly coming. Actually, it took a lot of my brain cycles > recently to think "what's next". Too much, to the point that I started the > thread. > I thought it might be quite a valuable opening from someone who always said > "well, we have to have **really** good reason to do Airflow 3" and "maybe > there will not be Airflow 3". > > And I quite agree with Kaxil - that trying to organise our thoughts around > what to do and how our Approach for Airflow 3 based on just this thread is > a bit too early. > I do not think this one thread here will lead to us deciding what to do - > if we try to do it now in a discussion thread or even a confluence doc, we > might fail achieving the goal. > > My main point here was to really get the feel and open thoughts of those > who are actively involved in Airflow - on what we should do next. And to > see if this is the right time to start thinking in "two" modes: Airflow 2 > and Airflow 3 (even if we do not know yet what Airflow 3 will be). > > I'd rather let a free stream of thoughts of what people think should happen > here continue. Merely opening our minds to the possibility of Airflow 3. > And I would love to keep it flowing for others - without the goal of > organizing it or achieving consensus. > > And I think all that Kaxil writes about - starting a series of calls, > organizing our discussions, getting "product manager(s)" working on > organizing those discussions is the **right** thing to do. > How exactly to do that, how to make sure everyone is involved, while we are > not tied up in endless discussions and bike-shedding, should materialize > from our discussion. > > But I would propose (and encourage) others' thoughts here as well - just a > free stream of those - then it might provide valuable feedback to what's > next. > > J. > > > On Mon, Apr 22, 2024 at 4:14 AM Kaxil Naik wrote: > > > Hello all, > > > > I didn't anticipate reading an Airflow 3 email from a sunny beach in > Nice, > > France <https://en.wikipedia.org/wiki/Nice> -- I had a great time there > > over the weekend, highly recommended :D > > > > I say that because, as Vikram pointed out, some of us at Astronomer have > > been polishing up the doc to propose Airflow 3 to the community in the > > coming week. Such is the beauty of the open-source project that multiple > > people (in the form of developers, committers, PMC members and various > > Stakeholders) think the same. From the Astronomer front, Constance had > been > > championing a doc with Vikram & myself, with inputs from various other > > committers & users, to have a good blend of different perspectives -- > > Product, PMC member & Industry leaders that cover several areas from > > User-facing pain points, Industry trends in the Orchestration space, > > Innovation in AI & ML space & opportunities as well as maintainability of > > the current codebase. We would love to share it this week that goes into > > some of the details to share our perspective on Why Airflow 3.0 & Why now > > etc. > > > > I would like to reiterate my statement from last year's panel session at > > the Airflow Summit with Marc, Jarek & Pierre > > <https://airflowsummit.org/sessions/2023/panels/panel-faces-airflow/>: > > "We, > > as the Airflow project, have maintained a great balance of Innovation & > > Stability", and I truly believe in that and is clearly visible in the > > number of downloads and Airflow's popularity as the leader in the > Workflow > > Orchestration space. Our industry is rapidly evolving, especially in the > > Data, AI & ML space. The role and expectations of the Data Orchestrator > > (more specialized than a generic Workflow Orches
Re: [DISCUSS] DRAFT AIP-68 Extended Plugin Interface + AIP-69 Remote Executor
Good work on the AIPs @Scheffler Jens (XC-DX/ETV5) ! I spent some time understanding your thoughts and I like the ideas and the direction you are heading too. I also agree with Jarek, that we might need to start thinking in terms of Airflow 3, especially now that you brought up AIP 68. Thanks & Regards, Amogh Desai On Sat, Apr 20, 2024 at 3:55 PM Jarek Potiuk wrote: > Hey Jens, > > I looked at the AIPs when you created them and I very much like the > directions put there - but it also got me into a lot of thinking on the > future of Airflow and AIPs. See the thread I started > https://lists.apache.org/thread/3chvg9964zvh15mtrbl073f4oj3nlzp2 - about > Airflow 3. > > I think in both cases (especially about AIP-689 but also AIP-69) - it would > make a wealth of difference if we treat them in the context of Airflow 2, > or (maybe) in the context of Airflow 3 which might start from taking the > best of Airflow 2 and get rid of all the unnecessary baggage it has. In the > past many similar efforts like AIP-69 had stalled in general because they > were far too complex to implement taking into account backwards > compatibility expectations of Airflow 2. > > And I think it's the right time we should get to terms with the future of > Airflow - whether/to what extent we want Airflow 3 to come, what level of > compatibility it should have, which assumptions should be dropped. I > personally have a feeling that AIP-69 would have been way easier to bring > as one of Airflow 3 "foundational" AIP that could define the "new" remote > architecture of Airflow rather than "plugin" to existing one. Dropping > Celery & K8s Options, leaving the Remote + Local variant of it as the only > ones, without the direct DB communication channel we have now and replacing > it with smth else. > > That would be my first comment on it and a question - should we get a bit > more clarity on what "future" of Airflow is before we discuss details and > approach there. > > J. > > On Fri, Apr 19, 2024 at 10:06 PM Scheffler Jens (XC-AS/EAE-ADA-T) > wrote: > > > Dear Dev-Community, > > > > mainly triggered by the deadline for CFP for Summit I dropped two “brand > > new” AIP’s as ideas that are running in my head for a longer time. Note > > these are DRAFT versions as a first write-up as solution concept and are > > lagging technical design and implementation yet. > > > > I’d kindly ask for feedback and review in Confluence cWiki (via Comments) > > – and also am seeking for people who like to join forces. > > > > AIP-68: Extended Plugin Interface for Custom Grid View Panels > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-68+Extended+Plugin+Interface+for+Custom+Grid+View+Panels > > > > Main motivation is that the UI has developed a lot in the recent time and > > AIP-38 is near completion. But it is focusing on technical details and > logs > > – and for most business users it is hard to read, missing business > > perspective. I propose to extend the Plugin interface allowing > > customizations on various new levels such that customer specific business > > information can be embedded, > > > > AIP-69: Remote Executor > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-69+Remote+Executor > > > > Airflow can be deployed in cloud or on-prem and various options of > > deployment are possible. But it lags the option to easily form a secure > and > > lean distributed setup for cases where individual nodes are far far away > > from the core of deployment. This imposes problems in opening firewalls > and > > might raise risk of security. Therefore I propose to add a “Remote > > Executor” such that workload can easily distributed to remote locations – > > also with a chance that it is easier for cases where people want to > > distribute workload to Windows (yeah there are really people around who > > still have this). > > > > Looking forward for (constructive) feedback, discussion and opinions. > > > > Again, note: DRAFT means open to discussion, nothing fixed, nothing coded > > yet. Many implementation options possible – and in case of interest > please > > join forces with me 😃 > > Once the discussion is calming I’d call for a vote separately like > usually. > > > > Mit freundlichen Grüßen / Best regards > > > > Jens Scheffler > > > > Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T) > > Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen | > > GERMANY | www.bosch.com > > Tel. +49 711 811-91508 | Mobil +49 160 90417410 | > > jens.scheff...@de.bosch.com<mailto:jens.scheff...@de.bosch.com> > > > > Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000; > > Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer; > > Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. Markus > > Forschner, > > Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert > > > > >
Re: [VOTE] AIP-67 Multi-team deployment of Airflow components
+1 binding. Excited to see this happen! Thanks & Regards, Amogh Desai On Sat, Apr 20, 2024 at 12:11 AM Igor Kholopov wrote: > +1 (non-binding) > > Great to see this happening, hope we will see more proposals towards making > Airflow more flexible! > > Regards, > Igor > > On Fri, Apr 19, 2024 at 8:10 PM Daniel Standish > wrote: > > > > > > > It doesn’t affect my vote on the API, but I am very strongly against > this > > > one part of the AIP: > > > > … dag_id are namespaced with `:` prefix. > > > This specific part is getting an implementation/code veto from me. We > > made > > > the mistake of overloading one column to store multiple things in > Airflow > > > before, and I’ve dealt with the fallout in other apps in the past. > Trust > > > me: do. not. do. this. > > > > I agree with Ash's sentiment. Is adding a tenant_id or something so > > unpalatable? > > >
Re: [VOTE] Airflow Providers prepared on April 16, 2024
+1 non binding Tested few example DAGs with cncf. Works fine. Thanks & Regards, Amogh Desai On Wed, 17 Apr 2024 at 12:21 PM, Pankaj Koti wrote: > +1 (non-binding) Concurring with Wei. > > > Best regards, > > *Pankaj Koti* > Senior Software Engineer (Airflow OSS Engineering team) > Location: Pune, Maharashtra, India > Timezone: Indian Standard Time (IST) > > > On Wed, Apr 17, 2024 at 9:01 AM Wei Lee wrote: > > > +1 (non-binding) > > > > I tested my change and ran example DAGs with cncf, databircks providers > > RCs, and it worked fine. > > > > Best, > > Wei > > > > > On Apr 16, 2024, at 8:40 PM, Elad Kalif wrote: > > > > > > Hey all, > > > > > > I have just cut the new wave Airflow Providers packages. This email is > > > calling a vote on the release, > > > which will last for 72 hours - which means that it will end on April > 19, > > > 2024 12:40 PM UTC and until 3 binding +1 votes have been received. > > > > > > > > > Consider this my (binding) +1. > > > > > > Airflow Providers are available at: > > > https://dist.apache.org/repos/dist/dev/airflow/providers/ > > > > > > *apache-airflow-providers--*.tar.gz* are the binary > > > Python "sdist" release - they are also official "sources" for the > > provider > > > packages. > > > > > > *apache_airflow_providers_-*.whl are the binary > > > Python "wheel" release. > > > > > > The test procedure for PMC members is described in > > > > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members > > > > > > The test procedure for and Contributors who would like to test this RC > is > > > described in: > > > > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors > > > > > > > > > Public keys are available at: > > > https://dist.apache.org/repos/dist/release/airflow/KEYS > > > > > > Please vote accordingly: > > > > > > [ ] +1 approve > > > [ ] +0 no opinion > > > [ ] -1 disapprove with the reason > > > > > > Only votes from PMC members are binding, but members of the community > are > > > encouraged to test the release and vote with "(non-binding)". > > > > > > Please note that the version number excludes the 'rcX' string. > > > This will allow us to rename the artifact without modifying > > > the artifact checksums when we actually release. > > > > > > The status of testing the providers by the community is kept here: > > > https://github.com/apache/airflow/issues/39063 > > > > > > The issue is also the easiest way to see important PRs included in the > RC > > > candidates. > > > Detailed changelog for the providers will be published in the > > documentation > > > after the > > > RC candidates are released. > > > > > > You can find the RC packages in PyPI following these links: > > > > > > > > > https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/8.1.1rc1/ > > > https://pypi.org/project/apache-airflow-providers-databricks/6.3.0rc3/ > > > https://pypi.org/project/apache-airflow-providers-fab/1.0.4rc1/ > > > > > > Cheers, > > > Elad Kalif > > > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > >
Re: [PROPOSAL] Keep >= LATEST MINOR for all providers using common.sql provider
I agree with the points made by Andrey here. > End users use amazon provider and google provider, and do not use common.sql and both of them have mandatory common.sql as dependency. On the contrary, would it be better to remove common.sql as mandatory dependencies for providers that do not use it? That way we would avoid one of the bigger problems which is maintaining the providers that have common.sql as dependency but do not need it as and when we someday deprecate common.sql? Thanks & Regards, Amogh Desai On Sun, Apr 14, 2024 at 2:20 PM Wei Lee wrote: > I feel like this is the responsibility of the provider contributor 🤔 > Maybe we could check whether the PR contains common.sql usage and raise a > warning? > > Best, > Wei > > > On Apr 12, 2024, at 12:31 AM, Andrey Anshin > wrote: > > > > There are some drawbacks I saw here, it would force to upgrade other > > providers to the latest version. > > Some scenarios from the my head: > > > > End users use amazon provider and google provider, and do not use > > common.sql and both of them have mandatory common.sql as dependency. > > if everything works fine there is no problem, but it could be a situation > > that a new release of one of the providers is major or contains bugs and > > there is no possible way to downgrade one of them and keep a new version > of > > another without breaking dependencies. > > > > Other case if we need to exclude common.sql provider from provider > release > > wave than we also need to exclude all providers which depends on > common.sql > > even if it problem not affected directly, e.g. it is not a sql only > > provider, e.g. amazon, google, microsoft.azure > > > > If it is not a big deal I have no objections to adding this because I do > > not have a better solution which could be implemented "Here and Now". > > > > It could be resolved if we might run tests against released versions of > > common.sql, but after some brief investigation to run tests > > against installed versions and not a main version this might require > > changing quite a few things, which might take a time. > > > > > > > > > > On Thu, 11 Apr 2024 at 12:41, Jarek Potiuk ja...@potiuk.com>> wrote: > > > >> Hello here, > >> > >> I have a proposal to add a general policy that all our providers > depending > >> on common.sql provider always use >= LATEST_MNOR version of the > provider. > >> > >> For example, following the change here > >> https://github.com/apache/airflow/pull/38707 by David we will update > all > >> sql providers to have: airflow-providers-common-sql >= 1.12. We could of > >> course automate that with pre-commit so that we do not have to remember > >> about it. > >> > >> Why ? > >> > >> Because it's very easy by a provider to accidentally use a new feature > in > >> common.sql without bumping the version. Current situation is like in the > >> image attached (thanks David), but we cannot be certain that the > >> "min-versions" there are "good".. > >> > >> A bit more context: > >> > >> Generally speaking - common.sql **should** be backwards compatible - > like > >> - always. We should not make any changes in it's API (which is for-now > >> captured here: > >> > https://github.com/apache/airflow/blob/main/airflow/providers/common/sql/hooks/sql.pyi > >> ). And from time to time we add new things to common.sql that providers > >> might start using. > >> > >> Example: > >> > >> From the > https://lists.apache.org/thread/lzcpllwcgo7pc8g18l3b905kn8f9k4w8 > >> thread the new "suppress_and_warn" method is going to be added. > >> > >> Currently we have no good mechanism to verify if the min version in > >> provider dependencies is really "good". For example when we add > >> `suppress_and_warn` today and someone starts using `suppress_and_warn` > in > >> the "presto" (for example) provider tomorrow, without bumping > common-sql to > >>> = 1.12 - it will fail with 1.11 or earlier installed. > >> > >> On the other hand - all the tests we run in `main` use the "latest" > >> version of the `common.sql` and all the constraints we produce also use > the > >> latest version released, so we can be rather sure that the new providers > >> actually work well with the latest `common.sql` version. If there will > be a > >> bug about compatibility (happened in the past), then it should be fixed > by > >> fixing it in a new patchlevel of the `common.sql`. > >> > >> So in-general it should be "safe" to add >= LATEST MINOR of common.sql > in > >> all providers. > >> > >> Should we do (and automate) it ? Any other comments / proposals maybe ? > >> > >> Here is the current state of dependencies: > >> > >> > >> [image: image.png] > >
Re: [VOTE] Airflow Providers prepared on April 10, 2024
+1 non binding Installed and RC with breeze and tested my changes: - Setting use_beeline by default for hive cli connection (#38763) - Removing deprecated code in hive provider (#38859) - Adding support to hive hook for high availability Hive installations (#38651) - Fix TRY002 for apache hive provider (#38781) - Add ssl context for verification of certs in FTPS hook (#38266) All these changes are working as expected. Thanks & Regards, Amogh Desai On Thu, Apr 11, 2024 at 10:53 PM Elad Kalif wrote: > databricks and yandex providers are excluded from rc1. > Please continue testing/voting without considering these two providers. > > On Thu, Apr 11, 2024 at 3:25 PM Pankaj Koti > wrote: > > > +1 (non-binding) > > > > Verified my changes and our provider's example DAGs are running fine > > on the RCs. > > > > Best regards, > > > > *Pankaj Koti* > > Senior Software Engineer (Airflow OSS Engineering team) > > Location: Pune, Maharashtra, India > > Timezone: Indian Standard Time (IST) > > > > > > On Thu, Apr 11, 2024 at 3:01 PM Jarek Potiuk wrote: > > > > > +1 (binding). Checked all my changes, including the celery provider bug > > > that prevented 3.6.1 from running on Airflow 2.7.* (celery 3.6.2rc1 now > > > works just fine for Airflow 2.7). Checked reproducibility, licences, > > > signatures, checksums. All good. > > > > > > On Wed, Apr 10, 2024 at 8:11 PM Vincent Beck > > wrote: > > > > > > > +1 non binding. All AWS system tests are running successfully against > > > > apache-airflow-providers-amazon==8.20.0rc1 with the exception of > > > > example_bedrock that is failing due to a bug in the test itself (fix > > > here: > > > > https://github.com/apache/airflow/pull/38887). You can see the > results > > > > here: > > > > > > > > > > https://aws-mwaa.github.io/#/open-source/system-tests/version/3d804351aa7a875dfdba824c2b27300cc5ce9e92_8.20.0rc1.html > > > > > > > > On 2024/04/10 16:37:12 Elad Kalif wrote: > > > > > Hey all, > > > > > > > > > > I have just cut the new wave Airflow Providers packages. This email > > is > > > > > calling a vote on the release, > > > > > which will last for 72 hours - which means that it will end on > April > > > 13, > > > > > 2024 16:35 PM UTC and until 3 binding +1 votes have been received. > > > > > > > > > > > > > > > Consider this my (binding) +1. > > > > > > > > > > Airflow Providers are available at: > > > > > https://dist.apache.org/repos/dist/dev/airflow/providers/ > > > > > > > > > > *apache-airflow-providers--*.tar.gz* are the binary > > > > > Python "sdist" release - they are also official "sources" for the > > > > provider > > > > > packages. > > > > > > > > > > *apache_airflow_providers_-*.whl are the binary > > > > > Python "wheel" release. > > > > > > > > > > The test procedure for PMC members is described in > > > > > > > > > > > > > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members > > > > > > > > > > The test procedure for and Contributors who would like to test this > > RC > > > is > > > > > described in: > > > > > > > > > > > > > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors > > > > > > > > > > > > > > > Public keys are available at: > > > > > https://dist.apache.org/repos/dist/release/airflow/KEYS > > > > > > > > > > Please vote accordingly: > > > > > > > > > > [ ] +1 approve > > > > > [ ] +0 no opinion > > > > > [ ] -1 disapprove with the reason > > > > > > > > > > Only votes from PMC members are binding, but members of the > community > > > are > > > > > encouraged to test the release and vote with "(non-binding)". > > > > > > > > > > Please note that the version number excludes the 'rcX' string. > > > > > This will allow us to rename the artifact without modifying > > > > >
Re: [DISCUSS] Redis licencing changes impact
I too agree with what you are saying, Jarek. I do not think there is any direct dependency here from what I know so far. Looking forward to when someone adds Valkey support. Thanks & Regards, Amogh Desai On Thu, Apr 11, 2024 at 10:06 PM Vikram Koka wrote: > Agree with your assessment Jarek, and that we don't have to do anything > here. > > > On Thu, Apr 11, 2024 at 5:29 AM Jarek Potiuk wrote: > > > Hello here, > > > > I've raised the discussion on private@ and it seems that there are no > > private/controversies there, so I bring the discussion to devlist where > it > > belongs. > > > > You might want to be aware that Redis announced licensing changes that > make > > future Redis 7.4+ releases not good for "mandatory" dependency of > Airflow. > > [1]. This is a very similar move to one that MongoDB, Elasticsearch and > > Terraform did. > > > > The RSAL licence is explicitly mentioned in the "Category -X" [2] of > > 3rd-party licence policy by the ASF as not good as "mandatory" > dependency - > > but still allowed as optional [3]. > > > > Redis client is MIT licensed and there are no plans to change that - so > > technically speaking we do not have to limit it in any way. > > > > My - personal - assessment is that we are not really affected - even with > > our Celery Executor, rabbitmq is still a viable alternative for Redis > (not > > mentioning Local and Kubernetes executors), so redis is indeed an > optional > > feature. In the future, someone could contribute Apache Qpid [4] (which > is > > supported by Kombu), also an easier one to contribute is to add Valkey > [5] > > as an alternative. > > > > In the meantime I created https://github.com/apache/airflow/pull/38928 > to > > limit Redis image to 7-2-bookworm. > > > > Also I'd love personally to see someone to contribute direct Valkey > support > > (that would be great I think) - but other than that I think there is not > > much we have to do now. > > > > But maybe others have different opinions and proposals (and would love to > > follow up on it) ? > > > > J. > > > > > > [1] https://redis.io/blog/redis-adopts-dual-source-available-licensing/ > > [2] https://www.apache.org/legal/resolved.html#category-x > > [3] https://www.apache.org/legal/resolved.html#optional > > [4] https://qpid.apache.org > > [5] > > > > > https://www.linuxfoundation.org/press/linux-foundation-launches-open-source-valkey-community > > >
Re: Issues on the Airflow Slack Workspace
Thanks for your email, Kaxil. I have been facing similar issues as well. Thanks & Regards, Amogh Desai On Thu, 11 Apr 2024 at 9:45 PM, Kaxil Naik wrote: > Hi all, > > I wanted to let you all know that I am facing & aware that others like me > are facing issues with Airflow's Slack workspace. Things like: you don't > see member's name, messages aren't visible or don't load etc. > > I have raised a support ticket with Slack, sent them logs and they are > working on it. > > I'll keep this thread updated once I hear back from them. > > Regards, > Kaxil >
Re: [DISCUSS] Asynchronous SQLAlchemy
Thanks, Hussein and Ash, I am much clearer on the scope now and I am ok with the discussion going on in the thread so far. Thanks & Regards, Amogh Desai On Tue, Apr 9, 2024 at 2:25 AM Andrey Anshin wrote: > If I do not miss something, usage of DB is not covered by Airflow Public > Interface, in this case we could easily replace one-by-one sync methods by > async. > There is some places exists where it might be mixin, as mentioned before > Secrets Backend, but it could be done by wrapping it into the: > - sync_to_async > - asyncio.to_thread (Python 3.9+) > - One of the anyio capabilities to run sync code into the async (threads or > processes) > > > > On Tue, 9 Apr 2024 at 00:45, Daniel Imberman > wrote: > > > Yeah do we have concrete examples of places where asyncio would be a > > non-starter? Are there enough of these examples to kill this idea? I > really > > don't like the idea of needing to maintain both sync and async. > > > > On Mon, Apr 8, 2024 at 1:39 PM Hussein Awala wrote: > > > > > > we definitely need a way to opt-out for the ones who aren't > interested > > > > > > I disagree, what I propose is to infer the async connection from the > sync > > > configuration using a translation method, with the possibility of > > providing > > > the async connection configuration explicitly. This will help to > > completely > > > migrate the REST API and web server to the async version without > > > duplicating the code. > > > > > > > We should have a seamless fallback to sync if async doesn't work for > > > whatever reasons > > > > > > For the async version of connections/variables, we will use the sync > > method > > > wrapped by sync_to_async in the base class, in this case, the async > > methods > > > will work in the custom secrets backends without any issues and users > can > > > override the async methods for better implementation. > > > > > > > are we limiting the scope to lets say connections + variables and > > > expanding based on the results in the long term? > > > > > > This needs to be implemented step by step, the first step is to add > > > integration to the different providers and DB, then implement an async > > > version for the secrets backends, then migrate the REST API and web > > server, > > > and later migrate our official executors, which will need also > > integrating > > > other tools like kubernetes-asyncio, and async integration for celery. > > > > > > > I think this needs to be an all or nothing thing > > > > > > Here are some of the available drivers > > > https://github.com/apache/airflow/pull/36504#issuecomment-1872653755, > I > > > have already tested one for each database, so we will have async > support > > > for all supported databases. > > > > > > > having to maintain sync and async versions of functions/features is a > > > non-starter in my mind; > > > > > > During the migration, we will have both sync and async endpoints in the > > API > > > and the webserver (they will be migrated one by one and not at the same > > > time), but without any code duplication, in the worst case, instead of > > > duplicating a method, we can use sync_to_async, and optimize it later > > > after migrating all endpoints that use it. > > > But for Secrets Backends, we may have some duplicated code when it is > not > > > possible to export it to a common method shared between sync and async > > > versions. > > > > > > > how can we keep one codenase bit cooe with sqlite? > > > > > > For my PoC, I used https://github.com/omnilib/aiosqlite and it worked > > > without any issues. > > > > > > > > > > > > On Mon, Apr 8, 2024 at 10:08 PM Daniel Standish > > > wrote: > > > > > > > If nothing else, write an ugly adapter using sync_to_async? > > > > > > > > On Mon, Apr 8, 2024 at 1:06 PM Daniel Standish < > > > > daniel.stand...@astronomer.io> wrote: > > > > > > > > > https://github.com/omnilib/aiosqlite maybe? > > > > > > > > > > On Mon, Apr 8, 2024 at 1:03 PM Scheffler Jens (XC-AS/EAE-ADA-T) > > > > > wrote: > > > > > > > > > >> I understand the „all-in“ approach as we were dropping MSSQL… how > > can > > > we > > > > >> keep one codenase bit cooe w
Re: [ANNOUNCE] New committer: Wei Lee
Many congratulations, Wei! Welcome onboard! Thanks & Regards, Amogh Desai On Mon, 8 Apr 2024 at 2:26 PM, Jarek Potiuk wrote: > Congrats Wei! Indeed, well deserved! > > On Mon, Apr 8, 2024 at 10:54 AM Hussein Awala wrote: > > > Congrats Wei, well deserved! > > > > On Monday, April 8, 2024, Ash Berlin-Taylor wrote: > > > > > The Project Management Committee (PMC) for Apache Airflow has invited > Wei > > > Lee to become a committer and > > > we are pleased to announce that they have accepted. > > > > > > Wei has been contributing for a number of months he also participated a > > > lot in discussions and decisions > > > on many aspects of Airflow but also helps a lot our users and > > contributors > > > on Slack, Github, Discussions. > > > > > > I am looking forward to what the future holds with Wei becoming the > > > committer, > > > > > > Congratulations Jens, and welcome onboard! > > > > > > Being a committer enables easier contribution to the project since > there > > > is no need to go via the patch > > > submission process. This should enable better productivity. A PMC > member > > > helps manage and guide > > > the direction of the project. > > > - > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > >
Re: [DISCUSS] Asynchronous SQLAlchemy
I checked the content and the PR that you attached. The results do seem promising and I like the general idea of this approach. But as Jarek also mentioned on the PR: 1. Not everyone might be on the board to go all async due to certain limitations around access to the drivers, or corporate limitations. So, we definitely need a way to opt-out for the ones who aren't interested. 2. We should have a seamless fallback to sync if async doesn't work for whatever reasons. 3. Are we going all in or are we limiting the scope to lets say connections + variables and expanding based on the results in the long term? Looking forward to improvements async can bring in! Thanks & Regards, Amogh Desai On Sun, Apr 7, 2024 at 3:13 AM Hussein Awala wrote: > The Metadata Database is the brain of Airflow, where all scheduling > decisions, cross-communication, synchronization between components, and > management via the web server, are made using this database. > > One option to optimize the DB queries is to merge many into a single query > to reduce latency and overall time, but this is not always possible because > the queries are sometimes completely independent, and it is impossible/too > complicated to merge them. But in this case, we have another option which > is running them concurrently since they are independent. The only way to do > this currently is to use multithreading (the sync_to_async decorator > creates a thread and waits for it using an asyncio coroutine), which is > already a good start, but by using the asyncio extension for sqlalchemy we > will be able to create thousands of lightweight coroutines with the same > amount of resources as a few threads, which will also help to reduce > resources consumption. > > A few months ago I started a PoC to add support for this extension and > implement an asynchronous version of connections and variables to be able > to get/set them from triggers without blocking the event loop and affecting > the performance of the triggerer, and the result was impressive ( > https://github.com/apache/airflow/pull/36504). > > I see a good opportunity to improve the performance of our REST API and web > server (for example https://github.com/apache/airflow/issues/38776), > knowing that we can mix sync and async endpoints, which will help for a > smooth migration. > > I also think that it will be possible (and very useful) to migrate some of > our executors to a full asynchronous version to improve their performance > (kubernetes and celery) > > I use the sqlalchemy asyncio extension in many personal and company > projects, and I'm very happy with it, but I would like to hear from others > if they have any positive or negative feedback about it. > > I will create a new AIP for integrating the asyncio extension of > sqlaclhemy, and other following AIPs to migrate/support each component once > the first one is implemented, but first, I prefer to check what the > community and other committers think about this integration. >
Re: [VOTE] Release Airflow 2.9.0 from 2.9.0rc3
+1 (non binding) I was able to install the RC seamlessly and ran a few example DAGs along with a few random UI testing. Looks good! Thanks & Regards, Amogh Desai On Mon, Apr 8, 2024 at 9:38 AM Jed Cunningham wrote: > +1 (binding) > > Checked reproducibility, signatures, checksums, licences. Used it with the > helm chart with a few different configs. >
Re: [DISCUSS] Consider disabling self-hosted runners for commiter PRs
+1 I like the idea. Looking forward to seeing the difference. Thanks & Regards, Amogh Desai On Fri, Apr 5, 2024 at 3:54 AM Ferruzzi, Dennis wrote: > Interested in seeing the difference, +1 > > > - ferruzzi > > > > From: Oliveira, Niko > Sent: Thursday, April 4, 2024 2:00 PM > To: dev@airflow.apache.org > Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Consider disabling > self-hosted runners for commiter PRs > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que > le contenu ne présente aucun risque. > > > > +1I'd love to see this as well. > > In the past, stability and long queue times of PR builds have been very > frustrating. I'm not 100% sure this is due to using self hosted runners, > since 35 queue depth (to my mind) should be plenty. But something about > that setup has never seemed quite right to me with queuing. Switching to > public runners for a while to experiment would be great to see if it > improves. > > > From: Pankaj Koti > Sent: Thursday, April 4, 2024 12:41:02 PM > To: dev@airflow.apache.org > Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Consider disabling > self-hosted runners for commiter PRs > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que > le contenu ne présente aucun risque. > > > > +1 from me to this idea. > > Sounds very reasonable to me. > At times, my experience has been better with public runners instead of > self-hosted runners :) > > And like already mentioned in the discussion, I think having the ability of > a applying the label "use-self-hosted-runners" to be used for critical > times would be nice to have too. > > > On Fri, 5 Apr 2024, 00:50 Jarek Potiuk, wrote: > > > Hello everyone, > > > > TL;DR With some recent changes in GitHub Actions and the fact that ASF > has > > a lot of runners available donated for all the builds, I think we could > > experiment with disabling "self-hosted" runners for committer builds. > > > > The self-hosted runners of ours have been extremely helpful (and we > should > > again thank Amazon and Astronomer for donating credits / money for > those) - > > when the Github Public runners have been far less powerful - and we had > > less number of those available for ASF projects. This saved us a LOT of > > troubles where there was a contention between ASF projects. > > > > But as of recently both limitations have been largely removed: > > > > * ASF has 900 public runners donated by GitHub to all projects > > * Those public runners have (as of January) for open-source projects now > > have 4 CPUS and 16GB of memory - > > > > > https://github.blog/2024-01-17-github-hosted-runners-double-the-power-for-open-source/ > > > > > > While they are not as powerful as our self-hosted runners, the > parallelism > > we utilise for those brings those builds in not-that bad shape compared > to > > self-hosted runners. Typical differences between the public and > self-hosted > > runners now for the complete set of tests are ~ 20m for public runners > and > > ~14 m for self-hosted ones. > > > > But this is not the only factor - I think committers experience the "Job > > failed" for self-hosted runners generally much more often than > > non-committers (stability of our solution is not best, also we are using > > cheaper spot instances). Plus - we limit the total number of self-hosted > > runners (35) - so if several committers submit a few PRs and we have > canary > > build running, the jobs will wait until runners are available. > > > > And of course it costs the credits/money of sponsors which we could use > for > > other things. > > > > I have - as of recently - access to Github Actions metrics - and while > ASF > > is keeping an eye and stared limi
Re: [VOTE] Release Airflow 2.9.0 from 2.9.0rc2
+1 non binding Tested a few example DAGs and tested to see if my changes work as expected. It looks good to me. Thanks & Regards, Amogh Desai On Fri, Apr 5, 2024 at 4:04 AM Jarek Potiuk wrote: > +1 (binding) - checked reproducibility, signatures, checksums, licences - > > all good. Installed it, run a few dags, clicked through a number of > screens. All looks good. Also verified the final package and it looks good > with the right FAB >=1.0.2 dependency. All looks good. > > On Thu, Apr 4, 2024 at 11:25 PM Ephraim Anierobi < > ephraimanier...@apache.org> > wrote: > > > Hey fellow Airflowers, > > > > I have cut Airflow 2.9.0rc2. This email is calling a vote on the release, > > which will last at least 52 hours, from Thursday, April 4, 2024, at 9:00 > pm > > UTC > > until Sunday, April 7, 2024, at 1:00 am UTC > > < > > > https://www.timeanddate.com/worldclock/fixedtime.html?msg=8&iso=20240407T0100&p1=1440 > > >, > > and until 3 binding +1 votes have been received. > > > > Consider this my (binding) +1. > > > > Airflow 2.9.0rc2 is available at: > > https://dist.apache.org/repos/dist/dev/airflow/2.9.0rc2/ > > > > *apache-airflow-2.9.0-source.tar.gz* is a source release that comes with > > INSTALL instructions. > > *apache-airflow-2.9.0.tar.gz* is the binary Python "sdist" release. > > *apache_airflow-2.9.0-py3-none-any.whl* is the binary Python wheel > "binary" > > release. > > > > Public keys are available at: > > https://dist.apache.org/repos/dist/release/airflow/KEYS > > > > Please vote accordingly: > > > > [ ] +1 approve > > [ ] +0 no opinion > > [ ] -1 disapprove with the reason > > > > Only votes from PMC members are binding, but all members of the community > > are encouraged to test the release and vote with "(non-binding)". > > > > The test procedure for PMC members is described in: > > > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-pmc-members > > > > The test procedure for and Contributors who would like to test this RC is > > described in: > > > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-contributors > > > > > > Please note that the version number excludes the `rcX` string, so it's > now > > simply 2.9.0. This will allow us to rename the artifact without modifying > > the artifact checksums when we actually release. > > > > Release Notes: > > https://github.com/apache/airflow/blob/2.9.0rc2/RELEASE_NOTES.rst > > > > For information on what goes into a release please see: > > > > > https://github.com/apache/airflow/blob/main/dev/WHAT_GOES_INTO_THE_NEXT_RELEASE.md > > > > Changes since 2.9.0rc1: > > > > *Bug Fixes*: > > - Fix decryption of trigger kwargs when downgrading (#38743) > > - Fix grid header rendering (#38720) > > > > *Doc-only Change*: > > - Improve timetable documentation (#38505) > > - Reorder OpenAPI Spec tags alphabetically (#38717) > > > > Cheers, > > Ephraim > > >
Re: [VOTE] Release Airflow 2.9.0 from 2.9.0rc1
Btw, these links are wrong. Instead refer this: 1. For PMC: https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md#verify-the-release-candidate-by-pmc-members 2. For contributors: https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md#verify-the-release-candidate-by-contributors Thanks & Regards, Amogh Desai On Thu, Apr 4, 2024 at 10:01 AM Amogh Desai wrote: > +1 non binding > > Installed the RC with pip and ran it with breeze. > > Tested out a few example dags from my test suite and also checked if my > changes > work as expected. > > > > Thanks & Regards, > Amogh Desai > > > On Wed, Apr 3, 2024 at 1:46 PM Ephraim Anierobi < > ephraimanier...@apache.org> wrote: > >> Hey fellow Airflowers, >> >> I have cut Airflow 2.9.0rc1. This email is calling a vote on the release, >> which will last at least 72 hours, from Wednesday, April 3, 2024 at 8:15 >> am >> UTC >> until Saturday, April 6, 2024, at 8:15 am UTC >> < >> https://www.timeanddate.com/worldclock/fixedtime.html?msg=8&iso=20240406T0815&p1=1440 >> >, >> and until 3 binding +1 votes have been received. >> >> Consider this my (binding) +1. >> >> Airflow 2.9.0rc1 is available at: >> https://dist.apache.org/repos/dist/dev/airflow/2.9.0rc1/ >> >> *apache-airflow-2.9.0-source.tar.gz* is a source release that comes with >> INSTALL instructions. >> *apache-airflow-2.9.0.tar.gz* is the binary Python "sdist" release. >> *apache_airflow-2.9.0-py3-none-any.whl* is the binary Python wheel >> "binary" >> release. >> >> Public keys are available at: >> https://dist.apache.org/repos/dist/release/airflow/KEYS >> >> Please vote accordingly: >> >> [ ] +1 approve >> [ ] +0 no opinion >> [ ] -1 disapprove with the reason >> >> Only votes from PMC members are binding, but all members of the community >> are encouraged to test the release and vote with "(non-binding)". >> >> The test procedure for PMC members is described in: >> >> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-pmc-members >> >> The test procedure for and Contributors who would like to test this RC is >> described in: >> >> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-contributors >> >> >> Please note that the version number excludes the `rcX` string, so it's now >> simply 2.9.0. This will allow us to rename the artifact without modifying >> the artifact checksums when we actually release. >> >> Release Notes: >> https://github.com/apache/airflow/blob/2.9.0rc1/RELEASE_NOTES.rst >> >> For information on what goes into a release please see: >> >> https://github.com/apache/airflow/blob/main/dev/WHAT_GOES_INTO_THE_NEXT_RELEASE.md >> >> *Changes since 2.8.4:* >> >> *Significant Changes* >> >> *Following Listener API methods are considered stable and can be used for >> production system (were experimental feature in older Airflow versions) >> (#36376):* >> Lifecycle events: >> >> - ``on_starting`` >> - ``before_stopping`` >> >> DagRun State Change Events: >> >> - ``on_dag_run_running`` >> - ``on_dag_run_success`` >> - ``on_dag_run_failed`` >> >> TaskInstance State Change Events: >> >> - ``on_task_instance_running`` >> - ``on_task_instance_success`` >> - ``on_task_instance_failed`` >> >> *Support for Microsoft SQL-Server for Airflow Meta Database has been >> removed (#36514)* >> >> After `discussion < >> https://lists.apache.org/thread/r06j306hldg03g2my1pd4nyjxg78b3h4>`__ >> and a `voting process < >> https://lists.apache.org/thread/pgcgmhf6560k8jbsmz8nlyoxosvltph2>`__, >> the Airflow's PMC and Committers have reached a resolution to no longer >> maintain MsSQL as a supported Database Backend. >> >> As of Airflow 2.9.0 support of MsSQL has been removed for Airflow Database >> Backend. >> >> A migration script which can help migrating the database *before* >> upgrading >> to Airflow 2.9.0 is available in >> `airflow-mssql-migration repo on Github < >> https://github.com/apache/airflow-mssql-migration>`_. >> Note that the migration script is provided without support and warranty. >> >> This does not affect the existing provider packages (operators and hooks), >> DAGs can still access and process data from
Re: [VOTE] Release Airflow 2.9.0 from 2.9.0rc1
+1 non binding Installed the RC with pip and ran it with breeze. Tested out a few example dags from my test suite and also checked if my changes work as expected. Thanks & Regards, Amogh Desai On Wed, Apr 3, 2024 at 1:46 PM Ephraim Anierobi wrote: > Hey fellow Airflowers, > > I have cut Airflow 2.9.0rc1. This email is calling a vote on the release, > which will last at least 72 hours, from Wednesday, April 3, 2024 at 8:15 am > UTC > until Saturday, April 6, 2024, at 8:15 am UTC > < > https://www.timeanddate.com/worldclock/fixedtime.html?msg=8&iso=20240406T0815&p1=1440 > >, > and until 3 binding +1 votes have been received. > > Consider this my (binding) +1. > > Airflow 2.9.0rc1 is available at: > https://dist.apache.org/repos/dist/dev/airflow/2.9.0rc1/ > > *apache-airflow-2.9.0-source.tar.gz* is a source release that comes with > INSTALL instructions. > *apache-airflow-2.9.0.tar.gz* is the binary Python "sdist" release. > *apache_airflow-2.9.0-py3-none-any.whl* is the binary Python wheel "binary" > release. > > Public keys are available at: > https://dist.apache.org/repos/dist/release/airflow/KEYS > > Please vote accordingly: > > [ ] +1 approve > [ ] +0 no opinion > [ ] -1 disapprove with the reason > > Only votes from PMC members are binding, but all members of the community > are encouraged to test the release and vote with "(non-binding)". > > The test procedure for PMC members is described in: > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-pmc-members > > The test procedure for and Contributors who would like to test this RC is > described in: > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-contributors > > > Please note that the version number excludes the `rcX` string, so it's now > simply 2.9.0. This will allow us to rename the artifact without modifying > the artifact checksums when we actually release. > > Release Notes: > https://github.com/apache/airflow/blob/2.9.0rc1/RELEASE_NOTES.rst > > For information on what goes into a release please see: > > https://github.com/apache/airflow/blob/main/dev/WHAT_GOES_INTO_THE_NEXT_RELEASE.md > > *Changes since 2.8.4:* > > *Significant Changes* > > *Following Listener API methods are considered stable and can be used for > production system (were experimental feature in older Airflow versions) > (#36376):* > Lifecycle events: > > - ``on_starting`` > - ``before_stopping`` > > DagRun State Change Events: > > - ``on_dag_run_running`` > - ``on_dag_run_success`` > - ``on_dag_run_failed`` > > TaskInstance State Change Events: > > - ``on_task_instance_running`` > - ``on_task_instance_success`` > - ``on_task_instance_failed`` > > *Support for Microsoft SQL-Server for Airflow Meta Database has been > removed (#36514)* > > After `discussion < > https://lists.apache.org/thread/r06j306hldg03g2my1pd4nyjxg78b3h4>`__ > and a `voting process < > https://lists.apache.org/thread/pgcgmhf6560k8jbsmz8nlyoxosvltph2>`__, > the Airflow's PMC and Committers have reached a resolution to no longer > maintain MsSQL as a supported Database Backend. > > As of Airflow 2.9.0 support of MsSQL has been removed for Airflow Database > Backend. > > A migration script which can help migrating the database *before* upgrading > to Airflow 2.9.0 is available in > `airflow-mssql-migration repo on Github < > https://github.com/apache/airflow-mssql-migration>`_. > Note that the migration script is provided without support and warranty. > > This does not affect the existing provider packages (operators and hooks), > DAGs can still access and process data from MsSQL. > > *Dataset URIs are now validated on input (#37005)* > > Datasets must use a URI that conform to rules laid down in AIP-60, and the > value > will be automatically normalized when the DAG file is parsed. See > `documentation on Datasets < > > https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/datasets.html > >`_ > for > a more detailed description on the rules. > > You may need to change your Dataset identifiers if they look like a URI, > but are > used in a less mainstream way, such as relying on the URI's auth section, > or > have a case-sensitive protocol name. > > *The method ``get_permitted_menu_items`` in ``BaseAuthManager`` has been > renamed ``filter_permitted_menu_items`` (#37627)* > > *Add REST API actions to Audit Log events (#37734)* > > The Audit Log ``event`` name for REST API events will be prepended
Re: [DISCUSS] Proposal for adding Telemetry via Scarf
+1 looks like a good tool which could be super helpful. * We should have some transparency into the data that is collected or sent * We should have an option to optionally opt-out Thanks & Regards, Amogh Desai On Sun, Mar 31, 2024 at 7:53 AM Wei Lee wrote: > +1 to this. It would be really useful. As long as we can opt out, I think > we’re good. > > Best, > Wei > > > On Mar 31, 2024, at 12:47 AM, Kaxil Naik wrote: > > > > Grammar Correction: > > > > We should assume that those who deploy and upgrade Airflow - actually > read > >> and take into account what is written in the release notes - especially > if > >> they have security guys breathing their necks, similarly as we have to > >> assume they follow CVE announcements about security issues fixed. If we > >> are very straightforward and out-going about the change, inform very > >> clearly how to opt-out, I don't see a big problem with opt-out. > > > > > > I couldn't agree more; even though we shouldn't collect any data that > > hamper security (and we should aim to do the same), most security > concerned > > folks don't just upgrade, and we can rely on them regarding release notes > > or announcements and we can make it very clear in our announcements too; > > and in our installation guides. > > > > On Sat, 30 Mar 2024 at 16:47, Kaxil Naik wrote: > > > >> Grammar crrection: > >> > >> > >> On Sat, 30 Mar 2024 at 16:43, Kaxil Naik wrote: > >> > >>> Have this at the end of the email too: but if folks don't read until > the > >>> end and quoting Maxime from the use-case blog[1]: > >>> > >>> "I think people often ask ‘how do I contribute to open source?’, ‘I've > >>> got to get into the code’, or ‘ I’ve got to be an engineer.’ Actually, > the > >>> very simplest thing that you can do is just say, ‘my organization gets > real > >>> value from this piece of software.’ There are a bunch of ways to let > the > >>> people know about it – and now Scarf is there. If your organization is > >>> getting a lot of value from a piece of open source software, make sure > the > >>> devs know about it." > >>> > >>> What kind of edge cases are you thinking about? I don't think it makes > >>> sense to have "opt-in" at all. As the goal is to collect data for most > >>> Airflow installations except for those that don't want to give data, > then > >>> "opt-out" is the only way to maximize it. As long as we don't collect > any > >>> PII data, this is in-compliance as well. > >>> > >>> Imagine someone learning Airflow, if they have to opt-in via a config, > >>> they wouldn't even know or care about it, hence us losing most of the > data. > >>> I understand why some orgs & individuals may want to opt-out. > >>> > >>> Scarf Provides tracking pixels (essentially an HTML image tag) that you > >>> can place in your website or product to track visitors to that URL. If > >>> there were any concerns about Privacy, ASF wouldn't have approved it > at all. > >>> > >>> A few key details to note about the pixel: > >>> > >>> > >>> - No PII is tracked… Scarf does not capture/retain IP information… > >>> this information is discarded by the platform upon > processing/aggregating > >>> - Scarf pixels respect the Do Not Track (DNT) settings of browsers - > >>> these users will not be tracked whatsoever. > >>> > >>> > >>> All the ASF projects I had listed (whether they use Scarf gateway or > >>> Scarf pixel in product) are using opt-out. > >>> > >>> 1. Short opt-in period before opt-out. Test this feature with users who > >>>> trust and if it works great - make it public. I think it's wise to > handle > >>>> edge cases and configure collected data more accurately. > >>> > >>> > >>> > >>> It would be a pixel in the webserver, should affect nothing at all even > >>> in an air-gapped environment. > >>> > >>>> 2. It should not affect anything if access to the internet is > restricted > >>>> which is default for many companies. > >>> > >>> > >>> > >>> 100% agreed on the below: > >>> > >>>> I think we
Re: [VOTE] AIP-64: Keep TaskInstance try history
+1 binding I like the thought behind this and understand why this was repeatedly asked by the community. Good work on the proposal so far! Thanks & Regards, Amogh Desai On Tue, Mar 26, 2024 at 9:58 AM Rahul Vats wrote: > +1 (non-binding) > > Regards, > Rahul Vats > 9953794332 > > > On Mon, 25 Mar 2024 at 22:46, Jed Cunningham > wrote: > > > Hello Airflow Community, > > > > I would like to start a vote on AIP-64: Keep TaskInstance try history. > > > > You can find the AIP here: > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-64%3A+Keep+TaskInstance+try+history > > > > Discussion Thread: > > https://lists.apache.org/thread/vvm43tfchyo92hmf40fqvmq0f5845bjr > > > > This is the first step in the AIP-63 DAG Versioning journey, though this > > provides value in isolation as well: > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-63%3A+DAG+Versioning > > > > The vote will last until 2024-03-28 17:30 UTC and until at least 3 > binding > > votes have been cast. > > > > Consider this my binding +1. > > > > Please vote accordingly: > > > > [ ] + 1 approve > > [ ] + 0 no opinion > > [ ] - 1 disapprove with the reason > > > > Only votes from PMC members and committers are binding, but other members > > of the community are encouraged to check the AIP and vote with > > "(non-binding)". > > > > Thanks, > > Jed > > >
Re: [VOTE] March 2024 PR of the Month
My vote goes to #36755 Great teamwork, given that it wasn't so easy since so many providers were involved. Kudos! Thanks & Regards, Amogh Desai On Tue, Mar 26, 2024 at 5:46 AM Jarek Potiuk wrote: > +1 on Python 3.12 (#36755) - with the note that it's been a joint effort > of quite a few people - Andrey, Bolke, Gopal Dirisao who contributed to it > in Airflow, but also a number people who are maintainers of our > dependencies: Israel Fruchter and Bret McGuire from Cassandra, Andreas > Poehlmann from universal_pathlib , Jessie Whitehouse who figured out the > databricks driver (and general) coverage-induced slow-downs. > > I probably forgot someone :). > > J. > > On Mon, Mar 25, 2024 at 10:51 PM Scheffler Jens (XC-AS/EAE-ADA-T) > wrote: > > > My vote is +1 for #36755 - mainly because it was really a long - runner > > and important we made it! > > > > Sent from Outlook for iOS<https://aka.ms/o0ukef> > > > > From: Briana Okyere > > Sent: Monday, March 25, 2024 8:53:32 PM > > To: dev@airflow.apache.org > > Subject: [VOTE] March 2024 PR of the Month > > > > Hey All, > > > > It’s once again time to vote for the PR of the Month! > > > > With the help of the `get_important_pr_candidates` script in dev/stats, > > we've identified the following candidates: > > > > PR #37937: refactor: Refactored __new__ magic method of BaseOperatorMeta > > to avoid bad mixing classic and decorated operators < > > > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F37937&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C0a80d271194c4cb01fc508dc4d055a9e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638469932608262236%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=PgitHoOv1s7k1lqG9oaqog4Ajo2gm7K9bdPlkITx5yE%3D&reserved=0 > > ><https://github.com/apache/airflow/pull/37937> > > > > PR #37821: Add `ADLSCreateObjectOperator` < > > > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F37821&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C0a80d271194c4cb01fc508dc4d055a9e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638469932608272501%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=qV6tNfyonXsa0R7j%2FNJExEWIfQWdc2NWmpzokCY27jA%3D&reserved=0 > > ><https://github.com/apache/airflow/pull/37821> > > > > PR #37458: Add Yandex Query support from Yandex.Cloud < > > > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F37458&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C0a80d271194c4cb01fc508dc4d055a9e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638469932608279527%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=vodH9CqvR%2FQeDvkUt0YgkuS9WRJKD7BIDojjcrZETTU%3D&reserved=0 > > ><https://github.com/apache/airflow/pull/37458> > > > > PR #36935: Adding ability to automatically set DAG to off after X times > it > > failed sequentially > > < > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F36935&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C0a80d271194c4cb01fc508dc4d055a9e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638469932608284177%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=G7dIUFdwRlEoC8sI4m4gc%2BVmZgDVvJcw2ZYfu8MaD1c%3D&reserved=0 > > <https://github.com/apache/airflow/pull/36935>> > > > > PR #36755: Python3.12 < > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F36755&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C0a80d271194c4cb01fc508dc4d055a9e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638469932608288507%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=WfyCRalWyPhIADBxFhmcS0X7dWSdjNHPbbJPA0lMhcY%3D&reserved=0 > > <https://github.com/apache/airflow/pull/36755>> > > > > Please reply to this thread with your selection or offer your own > > nominee(s). > > > > Voting will close on Friday, March 29th at 10 AM PST. The winner(s) will > be > > featured in the next issue of the Airflow newsletter. > > > > Also, if there’s an article or event that you think should be included in > > this or a future issue of the newsletter, please drop me a line at < > > briana.oky...@astronomer.io> > > > > -- > > Briana Okyere > > Community Manager > > Astronomer > > >
Re: [VOTE] Airflow Providers prepared on March 25, 2024
+1 non binding Installed airflow from main with fab rc1 and I was able to create a few test users with various roles, check their rights on the elements, and run a few simple dags. Works fine from my end. Btw, @Elad Kalif there's a typo: *This release is only for FAB provider to be tested along with Airflow 2.9.0rc11* Thanks & Regards, Amogh Desai On Mon, Mar 25, 2024 at 10:34 PM Jarek Potiuk wrote: > +1 (binding): checked signatures, checksums, licences, reproducibility. > > I build latest main `airflow` (that will become b2) as `2.9.0rc1` package > and installed it together with apache-airflow-providers-fab `1.0.2rc1` and > they work nicely together. I was able to create users, authenticate, login > to webserver, run a few example dags. All looks good. > > On Mon, Mar 25, 2024 at 3:00 PM Elad Kalif wrote: > > > Hey all, > > > > I have just cut the new wave Airflow Providers packages. This email is > > calling a vote on the release, > > which will last for 72 hours - which means that it will end on March 28, > > 2024 13:55 PM UTC and until 3 binding +1 votes have been received. > > > > > > Consider this my (binding) +1. > > > > This release is only for FAB provider to be tested along with Airflow > 2.9.0 > > rc11 > > > > > > Airflow Providers are available at: > > https://dist.apache.org/repos/dist/dev/airflow/providers/ > > > > *apache-airflow-providers--*.tar.gz* are the binary > > Python "sdist" release - they are also official "sources" for the > provider > > packages. > > > > *apache_airflow_providers_-*.whl are the binary > > Python "wheel" release. > > > > The test procedure for PMC members is described in > > > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members > > > > The test procedure for and Contributors who would like to test this RC is > > described in: > > > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors > > > > > > Public keys are available at: > > https://dist.apache.org/repos/dist/release/airflow/KEYS > > > > Please vote accordingly: > > > > [ ] +1 approve > > [ ] +0 no opinion > > [ ] -1 disapprove with the reason > > > > Only votes from PMC members are binding, but members of the community are > > encouraged to test the release and vote with "(non-binding)". > > > > Please note that the version number excludes the 'rcX' string. > > This will allow us to rename the artifact without modifying > > the artifact checksums when we actually release. > > > > The status of testing the providers by the community is skipped because > we > > test the fab provider as a whole rather than specific commits. > > > > The issue is also the easiest way to see important PRs included in the RC > > candidates. > > Detailed changelog for the providers will be published in the > documentation > > after the > > RC candidates are released. > > > > You can find the RC packages in PyPI following these links: > > > > https://pypi.org/project/apache-airflow-providers-fab/1.0.2rc1/ > > > > Cheers, > > Elad Kalif > > >
Re: [VOTE] Release Airflow 2.8.4 from 2.8.4rc1
+1 non binding Verified my changes, and ran a few example DAGs. I see no regressions. Thanks & Regards, Amogh Desai On Sat, Mar 23, 2024 at 8:11 PM Andrey Anshin wrote: > +1 binding > > Checked files, licences, signatures and also my changes. > > One small nit that I've found is that the key which is used for sign > releases is expired, I'm not sure if it should be considered as a > showstopper. > > ❯ gpg --verify apache_airflow-2.8.4-py3-none-any.whl.asc > apache_airflow-2.8.4-py3-none-any.whl > gpg: Signature made Wed Mar 20 19:17:26 2024 +04 > gpg:using RSA key E1A1E984F55B8F280BD9CBA20BB7163892A2E48E > gpg:issuer "jedcunning...@apache.org" > gpg: Good signature from "Jed Cunningham " > [ultimate] > gpg: Note: This key has expired! > Primary key fingerprint: A020 DD36 34F1 A4D1 26A1 C554 7774 A4E5 90CB 0351 > Subkey fingerprint: E1A1 E984 F55B 8F28 0BD9 CBA2 0BB7 1638 92A2 E48E > > ❯ gpg --list-options show-unusable-subkeys --list-keys > A020DD3634F1A4D126A1C5547774A4E590CB0351 > pub rsa4096 2022-01-05 [C] > A020DD3634F1A4D126A1C5547774A4E590CB0351 > uid [ultimate] Jed Cunningham > sub rsa4096 2022-01-05 [A] [expired: 2024-01-05] > sub rsa4096 2022-01-05 [E] [expired: 2024-01-05] > sub rsa4096 2022-01-05 [S] [expired: 2024-01-05] > > > > > On Fri, 22 Mar 2024 at 15:24, Rahul Vats wrote: > > > +1 (non-binding) > > > > Verified running our regression DAGS and API tests. LGTM > > > > Regards, > > Rahul Vats > > 9953794332 > > > > > > On Fri, 22 Mar 2024 at 15:24, Hussein Awala wrote: > > > > > +1 (binding) checked licences, checksums, signatures, and source codes, > > and > > > ran some testing dags. All looks good. > > > > > > On Wed, Mar 20, 2024 at 7:54 PM Jarek Potiuk wrote: > > > > > > > +1 (binding) - tested / verified all changes I was involved (either > as > > > > fixer, bug introducer or both, particularly when both), verified > > > > reproducibility, licences, checksums, signatures, run a few DAGs - > all > > > > looks good. > > > > > > > > On Wed, Mar 20, 2024 at 4:56 PM Jed Cunningham < > > jedcunning...@apache.org > > > > > > > > wrote: > > > > > > > > > Hey fellow Airflowers, > > > > > > > > > > I have cut Airflow 2.8.4rc1. This email is calling a vote on the > > > release, > > > > > which will last at least 72 hours, from Wednesday, March 20, 2024 > at > > > 4:00 > > > > > pm UTC > > > > > until Saturday, March 23, 2024 at 4:00 pm UTC, and until 3 binding > +1 > > > > votes > > > > > have been received. > > > > > > > > > > > > > > > > > > > > > > > > > https://www.timeanddate.com/worldclock/fixedtime.html/?msg=8&iso=20240323T1600&p1=1440 > > > > > > > > > > Status of testing of the release is kept in > > > > > https://github.com/apache/airflow/issues/38334 > > > > > > > > > > Consider this my (binding) +1. > > > > > > > > > > Airflow 2.8.4rc1 is available at: > > > > > https://dist.apache.org/repos/dist/dev/airflow/2.8.4rc1/ > > > > > > > > > > *apache-airflow-2.8.4-source.tar.gz* is a source release that comes > > > with > > > > > INSTALL instructions. > > > > > *apache-airflow-2.8.4.tar.gz* is the binary Python "sdist" release. > > > > > *apache_airflow-2.8.4-py3-none-any.whl* is the binary Python wheel > > > > "binary" > > > > > release. > > > > > > > > > > Public keys are available at: > > > > > https://dist.apache.org/repos/dist/release/airflow/KEYS > > > > > > > > > > Please vote accordingly: > > > > > > > > > > [ ] +1 approve > > > > > [ ] +0 no opinion > > > > > [ ] -1 disapprove with the reason > > > > > > > > > > Only votes from PMC members are binding, but all members of the > > > community > > > > > are encouraged to test the release and vote with "(non-binding)". > > > > > > > > > > The test procedure for PMC members is described in: > > > > > > > > > > > > > > > > > > > > https://github.com
Re: [VOTE] Release Apache Airflow Helm Chart 1.13.1 based on 1.13.1rc1
+1 non binding Tested the chart bundle on a kind cluster, and things seem to work as expected. Thanks & Regards, Amogh Desai On Sat, Mar 23, 2024 at 3:03 AM Hussein Awala wrote: > +1 (binding) Checked signatures, licences, and checksums, and tested the > chart locally with kind; the duplicated labels issue seems to be fixed. > > On Fri, Mar 22, 2024 at 6:44 PM Jarek Potiuk wrote: > > > +1 binding. Verified reproducibility, signatures, licences, checksums. > > Installed it locally and got Airflow working. > > > > Though - one comment. While doing it now, the prometheus exporter image > > has not been pulled yet and the exporter is failing. The problem is that > ` > > quay.io` is going through an outage currently it seems. > > > > It's a RedHat service, so I presume it's a temporary blip, I am not sure > > how reliable it is, but it seems to be going for well over 30 minutes for > > me at least and does not show a sign of recovery, so maybe we > > should consider switching to dockerhub for all our images in the future - > > at least we will rely on a single service, not two of those. It seems > quay > > is far less reliable than dockerhub: > > > > Quay last 30 days:: > > > > [image: image.png] > > > > Last 24 hrs: > > [image: image.png] > > > > Docker: > > > > [image: image.png] > > > > > > J, > > > > > > On Thu, Mar 21, 2024 at 7:24 PM Jed Cunningham > > > wrote: > > > >> Hello Apache Airflow Community, > >> > >> This is a call for the vote to release Helm Chart version 1.13.1. > >> > >> The release candidate is available at: > >> https://dist.apache.org/repos/dist/dev/airflow/helm-chart/1.13.1rc1/ > >> > >> airflow-chart-1.13.1-source.tar.gz - is the "main source release" that > >> comes with INSTALL instructions. > >> airflow-1.13.1.tgz - is the binary Helm Chart release. > >> > >> Public keys are available at: https://www.apache.org/dist/airflow/KEYS > >> > >> For convenience "index.yaml" has been uploaded (though excluded from > >> voting), so you can also run the below commands. > >> > >> helm repo add apache-airflow-dev > >> https://dist.apache.org/repos/dist/dev/airflow/helm-chart/1.13.1rc1/ > >> helm repo update > >> helm install airflow apache-airflow-dev/airflow > >> > >> airflow-1.13.1.tgz.prov - is also uploaded for verifying Chart > Integrity, > >> though not strictly required for releasing the artifact based on ASF > >> Guidelines. > >> > >> $ helm gpg verify airflow-1.13.1.tgz > >> gpg: Signature made Thu Mar 21 12:13:02 2024 MDT > >> gpg:using RSA key > E1A1E984F55B8F280BD9CBA20BB7163892A2E48E > >> gpg:issuer "jedcunning...@apache.org" > >> gpg: Good signature from "Jed Cunningham " > >> [ultimate] > >> plugin: Chart SHA verified. > >> sha256:ed6b2dea0d8f99eb9bd9cd6bc418db95f88a7bb3b8d1afb7fdc266b1ea411a15 > >> > >> The vote will be open for at least 72 hours (2024-03-24 18:29 UTC) or > >> until > >> the necessary number of votes is reached. > >> > >> > >> > https://www.timeanddate.com/countdown/to?iso=20240324T1829&p0=136&font=cursive > >> > >> Please vote accordingly: > >> > >> [ ] +1 approve > >> [ ] +0 no opinion > >> [ ] -1 disapprove with the reason > >> > >> Only votes from PMC members are binding, but members of the community > are > >> encouraged to test the release and vote with "(non-binding)". > >> > >> Consider this my (binding) +1. > >> > >> For license checks, the .rat-excludes files is included, so you can run > >> the > >> following to verify licenses (just update your path to rat): > >> > >> tar -xvf airflow-chart-1.13.1-source.tar.gz > >> cd airflow-chart-1.13.1 > >> java -jar apache-rat-0.13.jar chart -E .rat-excludes > >> > >> Please note that the version number excludes the `rcX` string, so it's > now > >> simply 1.13.1. This will allow us to rename the artifact without > modifying > >> the artifact checksums when we actually release it. > >> > >> The status of testing the Helm Chart by the community is kept here: > >> https://github.com/apache/airflow/issues/38382 > >> > >> Thanks, > >> Jed > >> > > >
Re: [DISCUSS] Applying D105 rule for our codebase ("undocumented magic methods") ?
After reading the emails in this thread, I too think that this rule is not
generally very useful, but we should have
it for cases which are special (where we are overriding some magic method
to do something different or more
than its reserved meaning)
+1 for removal of the rule, with special cases being handled separately.
Thanks & Best Regards,
Amogh Desai
On Wed, Mar 20, 2024 at 11:46 PM Oliveira, Niko
wrote:
> I'm -1 to enabling D105
>
>
> I don't think it will lead to helpful documentation. I think for the rare
> cases it is required it can left up to the developer or caught in PR review.
>
> Cheers,
> Niko
>
>
> From: Vincent Beck
> Sent: Wednesday, March 20, 2024 5:51:43 AM
> To: dev@airflow.apache.org
> Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Applying D105 rule
> for our codebase ("undocumented magic methods") ?
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
> le contenu ne présente aucun risque.
>
>
>
> +1 for not enforcing as well. Let's leave to maintainers the flexibility
> to chose whether a given method should be documented.
>
> On 2024/03/20 08:28:51 Ash Berlin-Taylor wrote:
> > I'm for not enforcing this rule - as others have said its very unlikely
> to result in more useful docs for developers or end users.
> >
> > -asg
> >
> > On 20 March 2024 08:12:40 GMT, Andrey Anshin
> wrote:
> > >±0 from my side
> > >
> > >Maybe we have to review all current methods which do not follow this
> rule
> > >to find a really useful meaning, and do not enforce (disable it).
> > >So for avoid unnecessary changes we might close
> > >https://github.com/apache/airflow/issues/37523 and remove/mark
> completed
> > >into the https://github.com/apache/airflow/issues/10742
> > >
> > >
> > >
> > >
> > >
> > >
> > >On Wed, 20 Mar 2024 at 10:41, Pankaj Koti .invalid>
> > >wrote:
> > >
> > >> +1 to what Aritra is saying.
> > >>
> > >>
> > >> Best regards,
> > >>
> > >> *Pankaj Koti*
> > >> Senior Software Engineer (Airflow OSS Engineering team)
> > >> Location: Pune, Maharashtra, India
> > >> Timezone: Indian Standard Time (IST)
> > >> Phone: +91 9730079985
> > >>
> > >>
> > >> On Wed, Mar 20, 2024 at 12:05 PM Aritra Basu <
> aritrabasu1...@gmail.com>
> > >> wrote:
> > >>
> > >> > I'm in general not a huge fan of documenting for the sake of
> documenting,
> > >> > so I'd be in agreement of not enforcing it via code but rather be
> > >> enforced
> > >> > by the reviewers in cases they believe certain methods need
> documenting.
> > >> >
> > >> > --
> > >> > Regards,
> > >> > Aritra Basu
> > >> >
> > >> > On Wed, Mar 20, 2024, 9:39 AM Jarek Potiuk
> wrote:
> > >> >
> > >> > > Hey here,
> > >> > >
> > >> > > I wanted to quickly poll what people think about applying the
> > >> > > https://docs.astral.sh/ruff/rules/undocumented-magic-method/
> rule in
> > >> our
> > >> > > codebase. There are many uncontroversial rules - but that one is
> > >> somewhat
> > >> > > more controversial than others.
> > >> > >
> > >> > > See
> > >> https://github.com/apache/airflow/pull/37602#issuecomment-2001951402
> > >> > > and
> > >> > >
> > >> >
> > >>
> https://github.com/apache/airflow/pull/38277#pullrequestreview-1945745542
> > >> > > for example
> > >> > >
> > >> > > I think that even in the ruff example, but also in many cases
> requiring
> > >> > to
> > >> > > document the methods will lead to rather useless documentation:
> > >> > >
> > >> > > class Cat(Animal):
> > >> > > def __str__(self) -> str:
> > >> > > ""&
Re: [DISCUSS] DRAFT AIP-67 Multi-tenant deployment of Airflow components
Thanks Jarek and Shubham for the clarifications. Looking forward to this one! Thanks & Regards, Amogh Desai On Sat, Mar 16, 2024 at 10:10 AM Mehta, Shubham wrote: > Jarek - I totally agree. We had a similar conversation in Dec 2022, and > Filip K. from Google suggested [1] calling them "workspaces." But I think > most of our users and contributors are used to the word "tenant." Trying a > new term like "workspaces" just seems to make things more confusing. For > example, when I tried using it with a couple of developers at AWS, who were > somewhat familiar with Airflow, it immediately prompted questions about its > relation to "tenants." > > I also liked how you explained it in the AIP response. It's like when > Kubernetes talks about "multi-tenant" [2] and they mean it could be for > different customers or different teams. What we’re doing with Airflow is > for teams, not really different customers. But it's simpler to keep calling > it "multi-tenant," just like Kubernetes does, and make sure we explain that > we're talking about different teams. > > Reference: > 1. > https://docs.google.com/document/d/1n23h26p4_8F5-Cd0JGLPEnF3gumJ5hw3EpwUljz7HcE/edit?disco=lo3bv6Q > 2. https://kubernetes.io/docs/concepts/security/multi-tenancy/ > > Shubham > > On 2024-03-15, 2:50 AM, "Jarek Potiuk" ja...@potiuk.com>> wrote: > > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que > le contenu ne présente aucun risque. > > > > > > > Thaks Shubham - that's a really nice and succinct description of the vision > behind tht 'Multi-tenant deployment' AIP. The case you described is spot-on > where I see there is enough value for the organisation that would like to > create such multi-tenant deployment of Airflow. > > > And I think one comment fro TP in the AIP that summaries it well - this is > not really `Making Airflow multi-tenant'. It's merely enabling airflow > components to be put together in a single multi-tenant deployment - > consisting with a number of smaller sub-deployments that achieve a nice > IMHO way of being able to decompose monolithic Airflow installation Ina a > set of loosely cooperating 'sub-deployments' - achieving some (interesting) > centralisation property - while giving enough freedom for tenants to manage > their part separately > > > In a way it might be a response to the claims that airflow is this old > monolithic style application and it's not the new, cool micro-service > /serverless world. Which I think trades some problems with a set of other > problems mainly around managing all those micro-sevices to cooperate with > each other and introduces a lot of performance and service management > problems. > > > But with the architecture described here it is a bit connecting best things > if both worlds - splitting Airflow into 'midi-services' - where the > architecture of your deploynent nicely follows Conway's Law and > decomposition is done at the right boundaries - boundaries where Tenant > Deployment Managers might claim as 'theirs' while the 'Airflow Platform > Team' keeps the whole Airflow deployment under control because there are > clear boundaries what the Platform team is responsible for, and where each > Tenant has their own 'kingdom'. > > > I think once we get this one approved and some deployments out there in the > wild - we will be able to claim that the architecture is actually the next > gen after monolithic, micro-sevices and serverless - bit done better, > taking into account some real-life user scenarios and Making it relatively > easy to adopt such deployment because of Conway's Law limitations. > > > I thought a lot about Conway's Law playing a role in this whole design and > I think your Business Case, Shubham, is a perfect reflection of how well > the proposes architecture fits in the Business structures of organisation > that would like to deploy it. > > > J. > > > czw., 14 mar 2024, 20:21 użytkownik Mehta, Shubham > mailto:shu...@amazon.com.inva>lid> napisał: > > > > Hi folks, > > > > Firstly, thanks Jarek for putting together such a thorough and > > well-thought-out proposal. > > > > >
Re: [DISCUSS] DRAFT AIP-67 Multi-tenant deployment of Airflow components
@Mehta, Shubham Thanks a lot for the case study, it helps to visualise the AIP better. I would like to bring up a point about the User Interface changes that I could not catch in the AIP: 1. If we want to let the data engineering team view across tenants, we should have the --tenant flag run with multiple values (?) Not sure, if that is in scope or if it was considered 2. We need to have the user interface provide some sort of a filter to filter out on tenants and their resources (DAGs, DAG runs, connections, variables, etc.) Otherwise, seeing tons of these resources from every team would be overwhelming and impractical Thanks & Regards, Amogh Desai On Fri, Mar 15, 2024 at 5:52 AM Adam Dutko wrote: > Shubham, > > I typically lurk on this list. I find your example of [Rocket] and J's > request for more voices a good opportunity to speak up. > > As a platform/data engineer I see the utility of scoped access to multiple > tenants. Not only would this help with troubleshooting and optimization > like you mention but it would be useful to reference when discussing > Airflow with the CISO, CFO and CTO. > > There are "hacks" to fix things like Airflow dashboard scroll (TM) - a lot > of DAGs everyone" has access to, environment partitioning (put it in a new > tenant), scoped access to certain areas of the system and particular tenant > components etc. > > I admittedly don't know if other concerns may arise related to performance > and user interface design, only time will tell. > > Thank you for all you do for the Airflow community J. I hope the above > proves helpful in advancing this conversation. > > -Adam > > From: Mehta, Shubham > Sent: Thursday, March 14, 2024 3:21:12 PM > To: us...@airflow.apache.org ; > dev@airflow.apache.org > Subject: Re: [DISCUSS] DRAFT AIP-67 Multi-tenant deployment of Airflow > components > > Hi folks, > > Firstly, thanks Jarek for putting together such a thorough and > well-thought-out proposal. > > I am very much in support of the multi-tenancy proposal. Having discussed > this with over 30 customers (AWS and non-AWS), there's a clear desire to > shift focus from the complex management of multiple Airflow environments to > enhancing their capabilities, such as enabling data quality checks and > lineage. This proposal is a significant step towards achieving that goal. > > Acknowledging that not every Airflow user has enough time to thoroughly > review the AIP, I have drafted a user scenario that encapsulates what's > possible with the implementation of multi-tenancy support: > > Scenario: Multi-Tenancy in Apache Airflow at [Rocket] > [Rocket], a leading [mobile gaming platform], has adeptly structured its > cloud operations using Apache Airflow to provide an efficient and secure > multi-tenant environment for orchestrating their complex workflows. This > approach caters to the diverse needs of their three main user groups: the > Data Engineering team, the Data Science team, and the Data Analytics team. > > All teams share basic Airflow components like the Scheduler and Webserver, > providing centralized management with shared cost. Each team has its own > distinct tenant cluster, offering self-sufficiency, flexibility, and > isolation. The Data Engineering team builds ETL/ELT pipelines and produces > user profile, telemetry, and marketing data. The Analytics team works with > marketing data and user information to build comprehensive dashboards. The > Data Science team uses Kubernetes as their execution environment for > heavy-duty machine learning tasks, producing a churn prediction dataset. > > Members of each team can only see and work with their own workflows. > However, Data engineers are granted access to all tenants, enabling them to > assist with DAG troubleshooting and optimization across all teams. Upon > logging in, users are presented with a tenant-specific view, displaying > only the relevant DAGs and artifacts. For those with multi-tenant access, > seamless navigation between different tenant views is available without the > need for re-authentication. > > This setup lets each team work independently with their own tools and > data, while also getting help from data engineers when needed. It's secure, > efficient, and user-friendly. > > Image: > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fimgur.com%2Fgallery%2FuQNqiVc&data=05%7C02%7Cadam%40runbymany.com%7Ca5ad7fc69319431a5e7b08dc445bf26b%7C33632a8512c2443d9b064cfa3bf99965%7C0%7C0%7C638460408922495976%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=sLtUKZXCuHm1FKYE40Z6hx1ovbBjBfy14jhPO89BtYk%3D&reserved=0 >
Re: [VOTE] AIP-59 Performance testing framework
+1 binding Glad to see this happen. Go for it! Thanks & Regards, Amogh Desai On Thu, Mar 14, 2024 at 12:28 AM Rahul Vats wrote: > +1 (non- binding) > > On Tue, 12 Mar, 2024, 23:40 Bartosz Jankiewicz, > wrote: > > > Hi folks, > > > > The AIP for performance testing has been in review for quite some time > and > > I've included your feedback in the document. > > > > I'd like to call a vote, and if you agree I'd start a development of the > > framework. > > > > The AIP can be found below: > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-59+Performance+tests+framework > > > > Please vote accordingly: > > > > [ ] + 1 approve > > [ ] + 0 no opinion > > [ ] - 1 disapprove with the reason > > > > Thank you! > > >
Re: [VOTE] Release Airflow 2.8.3 from 2.8.3rc1
+1 non binding. Installed and ran some example DAGs. All looks good. Thanks & Regards, Amogh Desai On Fri, Mar 8, 2024 at 8:31 PM Vincent Beck wrote: > +1 non binding. I check my change and it works as expected. I also ran few > testing DAGs and they ran fine. > > On 2024/03/08 00:14:12 Jarek Potiuk wrote: > > +1 (binding): checked reproducibility, sources, licences, checksums, > > run a few dags, tested all my changes, all looks good. > > > > One caveat. I think we still have the change that makes the default > > image `airflow/2.8.3rc1` point to `airflow/2.8.3rc1-python3.11` > > instead of `airfllow/2.8.3rc1-python3.8`. This is an aftermath of > > moving the "default image" change to 2.9.* line after we found out the > > bug in 2.8.0 that prevented the change from going live in 2.8 (but > > then the fix crept-in the 2.8 branch) > > > > Similarly as for 2.8.2 last week - I fixed it manually by `docker > > login` followed by: > > > > regctl image copy --force-recursive --digest-tags > > apache/airflow:2.8.3rc1-python3.8 apache/airflow:2.8.3rc1 > > regctl image copy --force-recursive --digest-tags > > apache/airflow:slim-2.8.3rc1-python3.8 apache/airflow:slim-2.8.3rc1 > > > > I think it's not worth cancelling the release and fixing it in the > > code, we can "fix" the final images in the same way after they are > > pushed. It takes literally a few seconds, we just have to remember to > > do it before the announcement. > > > > J. > > > > On Thu, Mar 7, 2024 at 11:13 PM Ephraim Anierobi > > wrote: > > > > > > Hey fellow Airflowers, > > > > > > I have cut Airflow 2.8.3rc1. This email is calling a vote on the > release, > > > which will last at least 72 hours, from Thursday, March 7, 2024 at > 10:10 pm > > > UTC > > > until Sunday, March 10, 2024, at 10:10 pm UTC > > > < > https://www.timeanddate.com/worldclock/fixedtime.html?msg=8&iso=20240310T2210&p1=1440 > >, > > > and until 3 binding +1 votes have been received. > > > > > > > > > The status of testing of the release is kept at > > > https://github.com/apache/airflow/issues/37982 > > > > > > Consider this my (binding) +1. > > > > > > Airflow 2.8.3rc1 is available at: > > > https://dist.apache.org/repos/dist/dev/airflow/2.8.3rc1/ > > > > > > *apache-airflow-2.8.3-source.tar.gz* is a source release that comes > with > > > INSTALL instructions. > > > *apache-airflow-2.8.3.tar.gz* is the binary Python "sdist" release. > > > *apache_airflow-2.8.3-py3-none-any.whl* is the binary Python wheel > "binary" > > > release. > > > > > > Public keys are available at: > > > https://dist.apache.org/repos/dist/release/airflow/KEYS > > > > > > Please vote accordingly: > > > > > > [ ] +1 approve > > > [ ] +0 no opinion > > > [ ] -1 disapprove with the reason > > > > > > Only votes from PMC members are binding, but all members of the > community > > > are encouraged to test the release and vote with "(non-binding)". > > > > > > The test procedure for PMC members is described in: > > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-pmc-members > > > > > > The test procedure for and Contributors who would like to test this RC > is > > > described in: > > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-contributors > > > > > > > > > Please note that the version number excludes the `rcX` string, so it's > now > > > simply 2.8.3. This will allow us to rename the artifact without > modifying > > > the artifact checksums when we actually release. > > > > > > Release Notes: > > > https://github.com/apache/airflow/blob/2.8.3rc1/RELEASE_NOTES.rst > > > > > > For information on what goes into a release please see: > > > > https://github.com/apache/airflow/blob/main/dev/WHAT_GOES_INTO_THE_NEXT_RELEASE.md > > > > > > Changes since 2.8.2: > > > > > > *Significant Changes* > > > > > > The smtp provider is now pre-installed when you install Airflow. > (#37713) > > > > > > *Bug Fixes* > > > - Add "MENU" permission in auth manager (#37881) > > > - Fix external_executor_id being overwritten (#37784)
Re: [VOTE] Airflow Providers prepared on March 04, 2024
+1 non binding Verified some dags, mostly around cncf providers. Looks good 👍 On Wed, 6 Mar 2024 at 11:13 AM, Rahul Vats wrote: > +1 (non-binding). > > Verified running example DAG for below provides with no issues > > >- https://pypi.org/project/apache-airflow-providers-amazon/8.19.0rc1 >- > https://pypi.org/project/apache-airflow-providers-apache-hive/7.0.1rc1 >- > https://pypi.org/project/apache-airflow-providers-apache-livy/3.7.3rc1 >- > > https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/8.0.1rc1 >- https://pypi.org/project/apache-airflow-providers-weaviate/1.3.3rc1 >- > > https://pypi.org/project/apache-airflow-providers-microsoft-azure/9.0.1rc1 >- https://pypi.org/project/apache-airflow-providers-google/10.16.0rc1 >- https://pypi.org/project/apache-airflow-providers-http/4.10.0rc1 > > > > Regards, > Rahul Vats > 9953794332 > > > On Wed, 6 Mar 2024 at 10:24, Jarek Potiuk wrote: > > > +1 (binding): tested /checked my changes. Checked reproducibility, > > licences, signatures, checksums - all looks good. > > > > On Tue, Mar 5, 2024 at 5:51 PM Vincent Beck wrote: > > > > > > +1 non binding. All AWS system tests are running successfully against > > apache-airflow-providers-amazon==8.19.0rc1. You can see the results here: > > > https://aws-mwaa.github.io/#/open-source/system-tests/version/2852976ea6321b152ebc631d30d5526703bc6590_8.19.0rc1.html > > > > > > On 2024/03/04 21:34:04 Elad Kalif wrote: > > > > Hey all, > > > > > > > > I have just cut the new wave Airflow Providers packages. This email > is > > > > calling a vote on the release, > > > > which will last for 72 hours - which means that it will end on March > > 07, > > > > 2024 21:30 PM UTC and until 3 binding +1 votes have been received. > > > > > > > > Consider this my (binding) +1. > > > > > > > > Airflow Providers are available at: > > > > https://dist.apache.org/repos/dist/dev/airflow/providers/ > > > > > > > > *apache-airflow-providers--*.tar.gz* are the binary > > > > Python "sdist" release - they are also official "sources" for the > > provider > > > > packages. > > > > > > > > *apache_airflow_providers_-*.whl are the binary > > > > Python "wheel" release. > > > > > > > > The test procedure for PMC members is described in > > > > > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members > > > > > > > > The test procedure for and Contributors who would like to test this > RC > > is > > > > described in: > > > > > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors > > > > > > > > > > > > Public keys are available at: > > > > https://dist.apache.org/repos/dist/release/airflow/KEYS > > > > > > > > Please vote accordingly: > > > > > > > > [ ] +1 approve > > > > [ ] +0 no opinion > > > > [ ] -1 disapprove with the reason > > > > > > > > Only votes from PMC members are binding, but members of the community > > are > > > > encouraged to test the release and vote with "(non-binding)". > > > > > > > > Please note that the version number excludes the 'rcX' string. > > > > This will allow us to rename the artifact without modifying > > > > the artifact checksums when we actually release. > > > > > > > > The status of testing the providers by the community is kept here: > > > > https://github.com/apache/airflow/issues/37890 > > > > > > > > The issue is also the easiest way to see important PRs included in > the > > RC > > > > candidates. > > > > Detailed changelog for the providers will be published in the > > documentation > > > > after the > > > > RC candidates are released. > > > > > > > > You can find the RC packages in PyPI following these links: > > > > > > > > https://pypi.org/project/apache-airflow-providers-amazon/8.19.0rc1/ > > > > > > https://pypi.org/project/apache-airflow-providers-apache-beam/5.6.2rc1/ > > > > > > https://pypi.org/project/apache-airflow-providers-apache-druid/3.9.0rc1/ > > > > > > https://pypi.org/project/apache-airflow-providers-apache-hdfs/4.3.3rc1/ > > > > > > https://pypi.org/project/apache-airflow-providers-apache-hive/7.0.1rc1/ > > > > > > https://pypi.org/project/apache-airflow-providers-apache-livy/3.7.3rc1/ > > > > > > https://pypi.org/project/apache-airflow-providers-apache-pinot/4.3.1rc1/ > > > > https://pypi.org/project/apache-airflow-providers-celery/3.6.1rc1/ > > > > > > > https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/8.0.1rc1/ > > > > > > https://pypi.org/project/apache-airflow-providers-common-sql/1.11.1rc1/ > > > > > https://pypi.org/project/apache-airflow-providers-dbt-cloud/3.7.0rc1/ > > > > https://pypi.org/project/apache-airflow-providers-docker/3.9.2rc1/ > > > > https://pypi.org/project/apache-airflow-providers-exasol/4.4.3rc1/ > > > > https://pypi.org/project/apache-airflow-providers-google/10.16.0rc1/ > > > > > https://pypi.org/project/apache-airflow-providers-hashicorp
