[ANNOUNCE] Apache Airlfow 3.0.2 reference images rebuilt

2025-06-23 Thread Jarek Potiuk
We have just updated constraints and rebuilt reference images for Airflow 3.0.2 following release of common-messaging provider 1.0.3 and fab provider 2.2.1. If you already installed 3.0.2, please pull the images or install airflow locally, with the newly updated constraints to make sure you get th

[ANNOUNCE] Apache Airflow Providers prepared on June 20, 2025 are released

2025-06-23 Thread Elad Kalif
Dear Airflow community, I'm happy to announce that new versions of Airflow Providers packages prepared on June 20, 2025 were just released. Full list of PyPI packages released is added at the end of the message. The source release, as well as the binary releases, are available here: https://airf

Re: [Discussion] AIP-90: Should It Be Implemented in the Standard Provider or as a standalone provider?

2025-06-23 Thread Wei Lee
Hi everyone, Thank you for joining this discussion! At this point, it seems we have reached some consensus. For naming, we now agree to use human centric term. To embed the whole idea into operators, I will use HITL (apologies for the previous typo) and mention "Human in the Loop" in the docum

Re: [DISCUSS] What should we cherry-pick ?

2025-06-23 Thread Wei Lee
I’m +1 for 1-3 (assuming the doc changes relate to the backported version). +0.5 for 4. I hope that changes not related to new features will be backported when feasible; however, we can skip them if the required effort is substantial. This is because failing to backport these items could potentia

Re: [DISCUSS] What should we cherry-pick ?

2025-06-23 Thread Jens Scheffler
I am +1 for (1) to (3) [also assuming that (2) is mostly like a doc bugfix!] For (4) I am hesitant and would rather be conservative. Every cherry-pick has a risk to break something in old codebase. As branches change over time and backport PRs are clearly less cautious reviewed it might lead t

Re: [Discussion] AIP-90: Should It Be Implemented in the Standard Provider or as a standalone provider?

2025-06-23 Thread Jens Scheffler
Hi, Thanks Wei for taking the lead in starting to implement! Hope I can review the next days. as I was writing the AIP together with Vikram I was and still am for (=+1) to keep it "human" centric. Also adding an API such that somebody else is able to roll their whatever UI and not being lock

[RESULT][VOTE] Release Apache Airflow Helm Chart 1.17.0 based on 1.17.0rc2

2025-06-23 Thread Jed Cunningham
Hello all, The vote to release Apache Airflow Helm Chart version 1.17.0 based on 1.17.0rc2 is now closed. The vote PASSED with 4 binding "+1", 3 non-binding "+1" and 0 "-1" votes: "+1" Binding votes: - Jed Cunningham - Jarek Potiuk - Jens Scheffler - Kaxil Naik "+1" Non-Binding votes:

Re: [DISCUSS] What should we cherry-pick ?

2025-06-23 Thread Pavankumar Gopidesu
Thanks Jarek, for starting this discussion, I agree with all the points. The real intention behind to backport https://github.com/apache/airflow/pull/51992 is , this area has a lot of ongoing development going and I felt it's worth porting to v3-0-test. I myself faced situations where I tried to

Re: [DISCUSS] What should we cherry-pick ?

2025-06-23 Thread Jarek Potiuk
> I think We are approaching it from the same point of view, just that we have different conclusions. I agree. I think that simply there is one special case that we should "allow". Details below. > For 4, I do not have strong opinions on either front, but defining what to > do and what not should

Re: [DISCUSS] What should we cherry-pick ?

2025-06-23 Thread Amogh Desai
Agree with point 1 - 3 definitely. For 4, I do not have strong opinions on either front, but defining what to do and what not should probably be that if that change makes it easy to backport something else, maybe have it? For ex: *PR 1 changes file1.py and file2.py* *PR 2 changes some lines in fi

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

2025-06-23 Thread Jarek Potiuk
Just to clarify the relation - I updated the AIP now to refer to AIP-82 and to explain relation between the "cross-team" and "cross-airflow" asset triggering - this is what I added: Note that there is a relation between AIP-82 ("External Driven Scheduling") and this part of the functionality. When

Re: Python 3.13 is coming

2025-06-23 Thread Amogh Desai
Thanks for taking this on Jarek. These are not easy tasks to work on and I cannot think of a better person to do it than you! I will review it shortly this week. I think it is a good thing to introduce in Airflow 3.1 and I would not be too much in favour to cherry pick it to the 3.x series, just

Re: [VOTE] Release Apache Airflow Helm Chart 1.17.0 based on 1.17.0rc2

2025-06-23 Thread Amogh Desai
+1 non binding. Installed the helm chart in a local minikube cluster and was able to run airflow 3.0.2 without any issues. Ran a few example dags and played around a bit on the UI with port forwarding, seems ok. Thanks & Regards, Amogh Desai On Mon, Jun 23, 2025 at 9:47 AM Vishnu Chilukoori wr

Re: [DISCUSS] Dropping Python 3.9 support

2025-06-23 Thread Avi
+1, with this it will make it easier to adopt sqlalchemy 2.x On Mon, Jun 23, 2025 at 1:03 PM Wei Lee wrote: > +1, I even thought we had already done it 🤔 > > Best, > Wei > > > On Jun 23, 2025, at 1:50 PM, Amogh Desai > wrote: > > > > +1 to this. > > > > Since the tentative date for PY 3.9 depre

Re: [HELP NEEDED] Cleanup pytest.mark.db_tests

2025-06-23 Thread Amogh Desai
Thanks for the email, Jarek. A quick summary of the change: while working on moving BaseHook to the task SDK in #51873, I noticed that many providers rely on `db.merge_conn()` to set up test connections & there are thousands of occurrences across the codebase. Doing this comes with few drawbacks:

[Discussion] AIP-90: Should It Be Implemented in the Standard Provider or as a standalone provider?

2025-06-23 Thread Wei Lee
Hi fellow Airflower, I am currently working on a PoC for AIP-90. I've incorporated some suggestions based on comments in the voting thread and Jira page. Since they have not yet been included in the AIP, I want to confirm with everyone to ensure I'm on the right track. 1. Many have expressed c

Re: [Discussion] AIP-90: Should It Be Implemented in the Standard Provider or as a standalone provider?

2025-06-23 Thread Amogh Desai
I was not strongly against using "human" -- it just felt a little odd and confusing to me at first. Jarek's email has convinced me that having HITH is contextual in the AI space and it is kinda what we are doing with this AIP - 90. In fact, using "interactive" now seems odd that it is not descript

Re: [DISCUSS] What should we cherry-pick ?

2025-06-23 Thread Ash Berlin-Taylor
I think We are approaching it from the same point of view, just that we have different conclusions. Points 1-3 I agree with. We do already have this written up https://github.com/apache/airflow/blob/130e9600443e06c08acc1b28c69a62c858d6e6a2/dev/README_RELEASE_AIRFLOW.md?plain=1#L116-L129 On 4 I

Re: [VOTE] Release Apache Airflow Helm Chart 1.17.0 based on 1.17.0rc2

2025-06-23 Thread Kaxil Naik
+1 binding, ran a few dags On Mon, 23 Jun 2025 at 13:00, Amogh Desai wrote: > +1 non binding. > > Installed the helm chart in a local minikube cluster and was able to run > airflow 3.0.2 without any issues. Ran a few example dags and played around > a bit on the UI with port forwarding, seems ok

Re: [VOTE] Airflow Providers prepared on June 20, 2025

2025-06-23 Thread Amogh Desai
+1 non binding. I didn't have any changes this time, but I had raised an issue which seems to have been fixed. Also installed the bits with AF 3.0.2 and verified user creation in FAB, no issues. Thanks & Regards, Amogh Desai On Sat, Jun 21, 2025 at 12:02 PM Shahar Epstein wrote: > +1 non-bin

Re: [VOTE] Example dags

2025-06-23 Thread Jens Scheffler
Okay, then sorry for the noise... cancelling the VOTE and switching to a LAZY CONSENSUS in a fresh email thread. On 22.06.25 23:00, Jarek Potiuk wrote: I think it would be better to call for LAZY CONSENSUS with `[LAZY CONSENSUS]` title - calling a vote is indeed not needed, but having a vote in

Re: [DISCUSS] What should we cherry-pick ?

2025-06-23 Thread Jarek Potiuk
BTW. I'd be happy to capture result of this discussion if we can get to a consensus or vote eventually in the "cherry-picking" guidelines. On Mon, Jun 23, 2025 at 12:42 PM Jarek Potiuk wrote: > I wanted to start a discussion on "things that we cherry-pick" (to vX_Z > branch). > > I think there a

[DISCUSS] What should we cherry-pick ?

2025-06-23 Thread Jarek Potiuk
I wanted to start a discussion on "things that we cherry-pick" (to vX_Z branch). I think there are different opinions on what kind of changes should be cherry-picked and it might be a good idea to agree on a common approach. I think (following the comment of Ash here) https://github.com/apache/ai

Re: [DISCUSS] Dropping Python 3.9 support

2025-06-23 Thread Jarek Potiuk
Cool :) On Mon, Jun 23, 2025 at 10:33 AM Elad Kalif wrote: > I started a lazy consensus > https://lists.apache.org/thread/5b3w2ofb2hjqcszkd21sg815rc0yxovt > > On Mon, Jun 23, 2025 at 10:36 AM Avi wrote: > > > +1, with this it will make it easier to adopt sqlalchemy 2.x > > > > On Mon, Jun 23, 2

Re: [Discussion] AIP-90: Should It Be Implemented in the Standard Provider or as a standalone provider?

2025-06-23 Thread Wei Lee
Got it! Yes, it makes sense to keep the phrase widely used. Thanks a lot! As a compromise, I will try something like `HITHOperator`, which may address some of the concerns. We can always rename it to whatever we decide before the release. I will also send a follow-up email to this thread once it

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

2025-06-23 Thread Jarek Potiuk
My counter-points: > 1. Managing a multi team deployment is not materially different from > managing a deployment per team > It's a bit easier - especially when it comes to upgrades (especially in the case we are targetting when we are not targetting multi-tenant, but several relatively closely

Re: [LAZY CONSENSUS] Dropping Python 3.9 support

2025-06-23 Thread Elad Kalif
PR: https://github.com/apache/airflow/pull/52072 On Mon, Jun 23, 2025 at 11:30 AM Elad Kalif wrote: > Hi, > Following discussion thread: > https://lists.apache.org/thread/0th63mgfc51jy7bch032hskbgoyzyhs2 > I'm calling for a lazy consensus on Dropping Python 3.9 support. > > The lazy consensus pe

Re: [HELP NEEDED] Cleanup pytest.mark.db_tests

2025-06-23 Thread Jarek Potiuk
yep - there are a few providers that require more "thorough" changes - ssh for example :) ... We noticed when doing cleanup. But we already can clean-up and remove a lot of the pytest.mark.db_tests that were only there due to connections :) I am thrilled at the prospect of having all our "provide

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

2025-06-23 Thread Ash Berlin-Taylor
Sorry, I was not on emails last week, I’ll catch up, read the updated proposal and come back to this today or tomorrow. A few notes from skimming the thread. Yes, the original trigger for this was the discussion about the DB changes (was it in the list or in the dev call? Whichever it was) and

Re: [DISCUSS] Dropping Python 3.9 support

2025-06-23 Thread Elad Kalif
I started a lazy consensus https://lists.apache.org/thread/5b3w2ofb2hjqcszkd21sg815rc0yxovt On Mon, Jun 23, 2025 at 10:36 AM Avi wrote: > +1, with this it will make it easier to adopt sqlalchemy 2.x > > On Mon, Jun 23, 2025 at 1:03 PM Wei Lee wrote: > > > +1, I even thought we had already done

[LAZY CONSENSUS] Dropping Python 3.9 support

2025-06-23 Thread Elad Kalif
Hi, Following discussion thread: https://lists.apache.org/thread/0th63mgfc51jy7bch032hskbgoyzyhs2 I'm calling for a lazy consensus on Dropping Python 3.9 support. The lazy consensus period lasts for 72 hours until Thursday, June 26, 2025 08:30 AM UTC. Thanks, Elad

Re: [DISCUSS] Dropping Python 3.9 support

2025-06-23 Thread Wei Lee
+1, I even thought we had already done it 🤔 Best, Wei > On Jun 23, 2025, at 1:50 PM, Amogh Desai wrote: > > +1 to this. > > Since the tentative date for PY 3.9 deprecation is nearing (Oct 2025), > since we support > 3.10 (relatively easier migration), I do not see why we should delay the > sup

Re: [DISCUSS] Dropping Python 3.9 support

2025-06-23 Thread Amogh Desai
+1 to this. Since the tentative date for PY 3.9 deprecation is nearing (Oct 2025), since we support 3.10 (relatively easier migration), I do not see why we should delay the support here. Maintaining older versions is a nightmare in itself and becomes even more so with the combinations we have in