Re: [VOTE] AIP-82 - External event driven scheduling in Airflow

+1 binding Great work! On Sat, Aug 3, 2024 at 7:15 AM Jed Cunningham wrote: > +1 binding >

Re: [VOTE] AIP-82 - External event driven scheduling in Airflow

+1 binding

Re: [VOTE] AIP-82 - External event driven scheduling in Airflow

+1 (non-binding) It would be a great improvement if we can have external event trigger support! Guangyang On Fri, Aug 2, 2024 at 5:55 PM Vikram Koka wrote: > +1 binding. > > Thank you for the quick turnaround on this and being very open in the > discussion as well as incorporating all the div

Re: [VOTE] AIP-67 - Multi-team deplyment of Airflow Components

+1 binding Thank you Jarek for being very patient and working through all the twists and turns on this AIP. I do believe that the changes are all positive and I really like the evolution towards a more focused multi-team target. This has become an uber-AIP, which is now building on top of AIPs su

Re: Airflow 3 AIP Gold Rush! -- Great job everyone

Oh yes. All that. And all that while not forgetting that we still have Airflow 2 to develop and support. I think there is a delicate balance that we have to keep: * enthusiasm for the new things being developed and Airflow3-to-be * and making sure that we can maintain new development and respond

Re: [VOTE] AIP-82 - External event driven scheduling in Airflow

+1 binding. Thank you for the quick turnaround on this and being very open in the discussion as well as incorporating all the diverse feedback. Vikram On Fri, Aug 2, 2024 at 1:03 PM Bishundeo, Rajeshwar wrote: > +1 non-binding > > Very quick turnaround on this AIP, good job Vincent. > > -- Ra

Re: [DISCUSS] External event driven scheduling in Airflow

Thanks for the update Vincent! I also referenced a comment earlier about leveraging patterns for the 80-20 use cases in the poll-model. However, after seeing how you have defined the syntax for "Asset watchers", I believe that's already covered in your proposal. So, all good from my perspective.

Re: Airflow 3 AIP Gold Rush! -- Great job everyone

+100 to Kaxil's lovely email summary. I could not be more proud of the Airflow Community and I am honored to be part of it! Jarek sent out the email starting the discussion about Airflow towards the end of April, and in the three months since we have discussed, debated, and finally aligned on a s

Airflow 3 AIP Gold Rush! -- Great job everyone

Hi team, Before we head for the weekend, I wanted to take a moment to express my gratitude and pride in being part of this incredible community I am so honoured, proud and happy! Over the past few months, and especially in the last two weeks, I’ve witnessed our brilliant community members come to

Re: [VOTE] AIP-67 - Multi-team deplyment of Airflow Components

+1 binding. Very good demonstration of perseverance on this one :) On 2024/08/02 20:02:00 "Bishundeo, Rajeshwar" wrote: > +1 non-binding. > > Great job to all the folks getting this feature 1-step closer to reality! > > -- Rajesh > > > > > > > On 2024-08-02, 3:57 PM, "Mehta, Shubham"

Re: [VOTE] AIP-83 Remove Execution Date Unique Constraint from DAG Run

+1 non-binding Great cleanup initiative! -- Rajesh On 2024-08-02, 1:50 PM, "Buğra Öztürk" mailto:ozturkbugr...@gmail.com>> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the cont

Re: [VOTE] AIP-79 & AIP-84 Remove Flask AppBuilder as a Core Dependency & UI REST API

+1 non-binding -- Rajesh On 2024-08-02, 1:51 PM, "Buğra Öztürk" mailto:ozturkbugr...@gmail.com>> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. AVERTISS

Re: [VOTE] AIP-82 - External event driven scheduling in Airflow

+1 non-binding Very quick turnaround on this AIP, good job Vincent. -- Rajesh On 2024-08-02, 3:57 PM, "Mehta, Shubham" mailto:shu...@amazon.com.inva>LID> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confir

Re: [VOTE] AIP-67 - Multi-team deplyment of Airflow Components

+1 non-binding. Great job to all the folks getting this feature 1-step closer to reality! -- Rajesh On 2024-08-02, 3:57 PM, "Mehta, Shubham" mailto:shu...@amazon.com.inva>LID> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments

Re: [VOTE] AIP-82 - External event driven scheduling in Airflow

+1 (non-binding) Great work, Vincent! On 2024-08-02, 12:50 PM, "Pavankumar Gopidesu" mailto:gopidesupa...@gmail.com>> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

Re: [VOTE] AIP-67 - Multi-team deplyment of Airflow Components

+1 non-binding. Thanks, Jarek, for continuing to evolve this piece of work based on the hundreds of comments you received from everyone. While there are dependencies on other AIPs, I'd love to see this implemented in Airflow 3.0. I hope other AIP authors also consider this as a reality and plan

Re: [VOTE] AIP-82 - External event driven scheduling in Airflow

+1 (non-binding). Regards, Pavan On Fri, Aug 2, 2024 at 8:46 PM Kaxil Naik wrote: > +1 binding. > > Thanks for incorporating the feedback Vincent. > > On Fri, 2 Aug 2024 at 20:43, Scheffler Jens (XC-AS/EAE-ADA-T) > wrote: > > > +1 binding. - thanks for the discussion! > > > > Sent from Outlook

Re: [VOTE] AIP-82 - External event driven scheduling in Airflow

+1 binding. Thanks for incorporating the feedback Vincent. On Fri, 2 Aug 2024 at 20:43, Scheffler Jens (XC-AS/EAE-ADA-T) wrote: > +1 binding. - thanks for the discussion! > > Sent from Outlook for iOS > > From: Beck, Vincent > Sent: Frida

Re: [VOTE] AIP-82 - External event driven scheduling in Airflow

+1 (binding) On Fri, Aug 2, 2024 at 9:42 PM Scheffler Jens (XC-AS/EAE-ADA-T) wrote: > +1 binding. - thanks for the discussion! > > Sent from Outlook for iOS > > From: Beck, Vincent > Sent: Friday, August 2, 2024 9:41:16 PM > To: dev@airflo

Re: [VOTE] AIP-67 - Multi-team deplyment of Airflow Components

+1 binding Thank you for persisting with the questions & changes, great job. Realistically, though, I think this would happen in 3.1 due to dependencies, as you mentioned in the email. Looking forward. Regards, Kaxil On Fri, 2 Aug 2024 at 20:30, Scheffler Jens (XC-AS/EAE-ADA-T) wrote: > +1 b

Re: [VOTE] AIP-76: Asset Partitions

Yup, I am fine removing that language to make it explicit but leave it up to TP. On Fri, 2 Aug 2024 at 19:56, Daniel Standish wrote: > My concern with the AIP is the talk of support for incremental data > pipelines. In an incremental data pipeline, you don't think of a delta > load (let's say a

Re: [VOTE] AIP-82 - External event driven scheduling in Airflow

+1 binding. - thanks for the discussion! Sent from Outlook for iOS From: Beck, Vincent Sent: Friday, August 2, 2024 9:41:16 PM To: dev@airflow.apache.org Subject: [VOTE] AIP-82 - External event driven scheduling in Airflow Hi everyone, I

[VOTE] AIP-82 - External event driven scheduling in Airflow

Hi everyone, I would like to call for a vote on AIP-82 External event driven scheduling in Airflow. AIP link: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow Discussion thread: https://lists.apache.org/thread/b7kl9sng7zn81dchn5nqom336ql5p

Re: [VOTE] AIP-67 - Multi-team deplyment of Airflow Components

+1 binding. Many thougths and discussions went into this AIP. Also User survery showed a strong demand for this - same we have also in our house. A multi team setup is eagerly needed. Nevertheless the complexity and missing turn-key delivery will be a challenge. I assume it needs to evolve. Bes

Re: [DISCUSS] External event driven scheduling in Airflow

Hello everyone, After multiple conversations with different actors, I decided to move the push based event scheduling out of scope of this AIP. I could not find a solution for the push based approach that was satisfactory and I think more investigation and data from Airflow users is needed to m

Re: [VOTE] AIP-76: Asset Partitions

My concern with the AIP is the talk of support for incremental data pipelines. In an incremental data pipeline, you don't think of a delta load (let's say a collection of updated rows) as a partition. A partition in data is defined by a partition key, which should be an immutable field or fields

Re: [RESULT][VOTE] July 2024 PR of the Month

Awesome job Niko!! -- Rajesh On 2024-08-02, 1:51 PM, "Buğra Öztürk" mailto:ozturkbugr...@gmail.com>> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. AVER

Re: [RESULT][VOTE] July 2024 PR of the Month

Congratulatio Niko 🥳🥳🥳 On Fri, 2 Aug 2024, 23:21 Buğra Öztürk, wrote: > Congrats, Niko! 🥳 > > On Fri, Aug 2, 2024 at 3:16 AM Jarek Potiuk wrote: > > > Yep. Grats! > > > > On Fri, Aug 2, 2024 at 2:52 AM Pavankumar Gopidesu < > > gopidesupa...@gmail.com> > > wrote: > > > > > Congratulations Niko

Re: [VOTE] AIP-76: Asset Partitions

This has been one of the harder AIPs to review and I have asked a lot of questions on this, but I am happy about where it is now. +1 binding On Thu, Aug 1, 2024 at 7:39 PM Wei Lee wrote: > +1 (binding) > > Best, > Wei > > > On Aug 2, 2024, at 9:09 AM, Kaxil Naik wrote: > > > > We should resolv

Re: [VOTE] AIP-67 - Multi-team deplyment of Airflow Components

+1 non-binding I’m looking forward to seeing this AIP as it will pave the way for follow-up features that enable fully manageable multi-tenancy. I’m also excited about the opportunity to work on this. 🙌 On Thu, Aug 1, 2024 at 10:40 PM Jarek Potiuk wrote: > Voting ends a week from now - i.e Thurs

Re: [RESULT][VOTE] July 2024 PR of the Month

Congrats, Niko! 🥳 On Fri, Aug 2, 2024 at 3:16 AM Jarek Potiuk wrote: > Yep. Grats! > > On Fri, Aug 2, 2024 at 2:52 AM Pavankumar Gopidesu < > gopidesupa...@gmail.com> > wrote: > > > Congratulations Niko 🎉🎉. > > > > Regards, > > Pavan > > > > On Fri, Aug 2, 2024 at 1:45 AM Vikram Koka > > > wrot

Re: [VOTE] AIP-79 & AIP-84 Remove Flask AppBuilder as a Core Dependency & UI REST API

+1 non-binding On Fri, Aug 2, 2024 at 6:20 PM Elad Kalif wrote: > +1 binding > > I am not sure I agree with previous statements regarding the LDAP issue > (specifically about blocker or not blocker for Airflow 3) but let's not > discuss this here as it's out of scope for the AIPs we are voting h

Re: [VOTE] AIP-83 Remove Execution Date Unique Constraint from DAG Run

+1 non-binding On Fri, Aug 2, 2024 at 6:20 PM Elad Kalif wrote: > +1 binding > > On Fri, Aug 2, 2024 at 8:12 AM Phani Kumar > wrote: > > > +1 binding > > > > On Fri, Aug 2, 2024 at 8:30 AM Wei Lee wrote: > > > > > +1 binding. > > > > > > Best, > > > Wei > > > > > > > On Aug 2, 2024, at 5:06 AM

Re: [DISCUSS] AIP-81: Enhanced Security in CLI via Integration of API

Hi everyone, I have finalized the categorization, mapping, and authentication sections. Could you please review the AIP when you have a moment? Your feedback would be greatly appreciated and valuable. After this round of reviews, I think we can start voting for the AIP. Thanks! Kind regards, Bugr

Re: [DISCUSS] AIP-81: Enhanced Security in CLI via Integration of API

It's fun for me to make Airflow better. I have responded to them and adjusted the document accordingly. Thanks a lot, Kaxil! On Fri, Aug 2, 2024 at 3:06 AM Kaxil Naik wrote: > Thanks, Bugra, great turnaround on such a short notice. I have added my > comments too. > > Regards, > Kaxil > > On Sat,

Re: [VOTE] AIP-83 Remove Execution Date Unique Constraint from DAG Run

+1 binding On Fri, Aug 2, 2024 at 8:12 AM Phani Kumar wrote: > +1 binding > > On Fri, Aug 2, 2024 at 8:30 AM Wei Lee wrote: > > > +1 binding. > > > > Best, > > Wei > > > > > On Aug 2, 2024, at 5:06 AM, Vishnu Chilukoori < > vish.chiluko...@gmail.com> > > wrote: > > > > > > +1 non-binding > > >

Re: [VOTE] AIP-79 & AIP-84 Remove Flask AppBuilder as a Core Dependency & UI REST API

+1 binding I am not sure I agree with previous statements regarding the LDAP issue (specifically about blocker or not blocker for Airflow 3) but let's not discuss this here as it's out of scope for the AIPs we are voting here. On Fri, Aug 2, 2024 at 7:15 PM Vincent Beck wrote: > Big +1 to this

Re: [VOTE] AIP-79 & AIP-84 Remove Flask AppBuilder as a Core Dependency & UI REST API

Big +1 to this one. I think this is the major blocker on that workstream: deciding which tool we want to leverage to build the default auth manager for Airflow 3. Once that decision taken I'll be happy to help to implement the auth manager based upon that tool. On 2024/08/02 16:06:00 Jarek Pot

Re: [VOTE] AIP-79 & AIP-84 Remove Flask AppBuilder as a Core Dependency & UI REST API

Yeah. KeyCloak was just a repeating theme in a number of questions/issues some users raised (How do I integrate with KeyCloak) and it's well recognized in enterprise. But this is about as much as I know about it :) So yeah, if we have anyone who can dig deeper in the options we have or ideally som

Re: [VOTE] AIP-79 & AIP-84 Remove Flask AppBuilder as a Core Dependency & UI REST API

Agreed Jarek on the parallel workstream for auth and also that should not be a blocker for 3.0. I don't know if the right answer is actually Keycloak. There was some research done by my colleagues within Astronomer using Casbin for the same, but I don't know the differences between those and other

Re: AWS Open Source Credits Program

Great news..! On Thu, Aug 1, 2024 at 8:49 PM Wei Lee wrote: > Awesome! > > > On Jul 29, 2024, at 3:08 AM, Utkarsh Sharma < > utkarsh.sha...@astronomer.io.INVALID> wrote: > > > > Sounds great! :) > > > > On Sun, Jul 28, 2024 at 4:17 PM Pierre Jeambrun > > wrote: > > > >> Great news! > >> > >> On

Re: AWS Open Source Credits Program

Also that becomes quite a bit more important as currently we do not have ARM builds available in our CI - and (info from last 15 minutes) https://issues.apache.org/jira/browse/INFRA-25990 - we do not have suitable ARM runners available from the ASF and there is no timeline where we will have the ri

Re: [VOTE] AIP-79 & AIP-84 Remove Flask AppBuilder as a Core Dependency & UI REST API

Yeah. And (a little tangential) - I really feel that we should have a separate parallel workstream `Implement "proper" Auth Manager` (for example authorizing users via Keycloak) - which should be creating a new provider. Note that this provider should NOT have a way to manage users and roles - it s

Re: [VOTE] AIP-79 & AIP-84 Remove Flask AppBuilder as a Core Dependency & UI REST API

> Just to verify, users will still be able connect FAB to LDAP by installing > FAB provider explicitly? Yes. That and configuring the FAB auth manager as the auth manager, as it won't be the default most likely. Being able to maintain that is a primary goal of this AIP. > But I want to make sur

Re: [VOTE] AIP-79 & AIP-84 Remove Flask AppBuilder as a Core Dependency & UI REST API

generally +1 binding Just to verify, users will still be able connect FAB to LDAP by installing FAB provider explicitly? ‫בתאריך יום ו׳, 2 באוג׳ 2024 ב-2:09 מאת ‪Jarek Potiuk‬‏ <‪ja...@potiuk.com ‬‏>:‬ > +1 - with the same concerns as Kaxil. > > On Fri, Aug 2, 2024 at 1:02 AM Kaxil Naik wrote:

Re: AWS Open Source Credits Program

Now. Our previous infrastructure with self-hosted runners is broken just now, when we have the money. And we need to fix it to start using them. This will allow us to speed up our builds a lot - but we need someone who might complete the dev-ops work on setting up Kubernetes Controller as a self-h