Re: [DISCUSS] Time to say goodbye to the old UI?

+10 :) -- I can't wait to see all the old UI code being deleted. On Tue, 18 Feb 2025 at 21:25, Vikram Koka wrote: > Thanks for starting this thread, Jed. > > Around September last year, I was pessimistic about when this would happen. > I have been very impressed by our rate of progress on th

Re: [Help wanted] Check whether the significant newsfragment are correctly updated

This is awesome Wei. I have reviewed mine and added a comment. On Wed, 15 Jan 2025 at 17:28, Wei Lee wrote: > ## Why > Since https://github.com/apache/airflow/pull/44378, we have introduced a > newsfragments template to track our breaking changes. However, this > template was implemented after m

Re: [DISCUSSION] Replace Poke & Reschedule mode from Sensors for Airflow 3 in favor of Deferrable

Thanks everyone for the discussion. The concerns raised about deferrable are valid and given the time and scope we already have for 3.0, let's defer this to post 3.0. Let's defer the defer conversation to 3.x :) tldr: Things will be kept as-is! Regards, Kaxil On Tue, 26 Nov 2024 at 07:05, Andre

Re: [ANNOUNCE] Upgraded Sphinx to 7

Kudos Shahar and all who helped 🎉 On Mon, 13 Jan 2025 at 22:29, Jarek Potiuk wrote: > Also the remaining tasks are now nicely described as the cool new > sub-issues in > > https://github.com/orgs/apache/projects/403/views/2?pane=issue&itemId=88615825&issue=apache%7Cairflow%7C44373 > > > On Mon,

Re: [DISCUSS] Drop support for the DAG processor embedded in the scheduler

e some > kind of cgroups docker-like process separation, such forked DAG processor > (and this also means DAG author) can access those credentials, and access > database. This means pretty much that embedded DAG processor simply breaks > the "no DB access by DAG author" assu

Re: [DISCUSS] Drop support for the DAG processor embedded in the scheduler

+1 On Fri, 10 Jan 2025 at 07:43, Mehta, Shubham wrote: > + 1 on this as well. From what I have seen, standalone DAG processing > results in a minor performance advantage and, importantly, makes the > Scheduler loop more resilient to DAG processor crashes. > > Shubham > > On 2025-01-09, 4:02 PM,

Re: [ANNOUNCEMENT] Moving Providers to separate sub-projects soon-ish

Since each provider will self contained, it will be easier for isolation and moving providers one-by-one to Airflow 3. On Wed, 8 Jan 2025 at 00:04, Jarek Potiuk wrote: > I think it will be better to keep it. > > The reason we have varying levels were to group things together - mainly > Apache re

Re: [RESULT][VOTE] Release Airflow 2.10.4 from 2.10.4rc1

🎉 On Mon, 16 Dec 2024 at 15:21, Utkarsh Sharma wrote: > Hello, > > Apache Airflow 2.10.4 (based on RC1) has been accepted. > > 3 "+1" binding votes received: > - Jarek Potiuk > - Jens Scheffler > - Ephraim Anierobi > > 6 "+1" non-binding votes received: > - Utkarsh Sharma > - Shahar Epstein > -

Re: [LAZY CONSENSUS] Remove `conf` from Task Context for AF 3.0

Regardless of any option, there isn't a strong case to keep it in the Context dict -- which is why I want to separate the discussion of configs to worker/parser etc to https://github.com/apache/airflow/issues/44352 On Wed, 11 Dec 2024 at 00:04, Kaxil Naik wrote: > imo it is relat

Re: [LAZY CONSENSUS] Remove `conf` from Task Context for AF 3.0

ACE")` will not work because configuration > will not be present there and in this case what would make more sense is to > leave "conf" as part of the context but only pass the "needed" > configuration there. > If we go for option 1) then likely yes we could remov

Re: [LAZY CONSENSUS] Remove `conf` from Task Context for AF 3.0

> > KubernetesPodOperator( > namespace=namespace, > image="", > cmds=[""], > arguments=[""], > labels={"": ""}, > name="", > task_id="", > get_logs=True, >

Re: [Meeting Notes] Airflow 3.0 dev call - 5 Dec 2024

Thanks for the update Vikram, I missed the last call -- great progress 🎉 On Tue, 10 Dec 2024 at 06:33, Vikram Koka wrote: > Hey everyone, > > I updated our meeting notes document in the Airflow wiki to capture the > notes from our dev call on Thursday, the 5th of December. The link for > those n

[LAZY CONSENSUS] Remove `conf` from Task Context for AF 3.0

Hey team, I have a PR (https://github.com/apache/airflow/pull/44820) to remove conf object from the Task context dictionary. This was initially added (in 2015) in response to https://github.com/apache/airflow/issues/168. However, we now have `ti.log_url` that is used for that; example usages: ht

Re: AIP-84 Public API Migration

Awesome stuff, kudos to Pierre and everyone involved 🎉 On Thu, 28 Nov 2024 at 19:08, Pavankumar Gopidesu wrote: > Woohoo 🙌 thank you everyone :) > > On Thu, Nov 28, 2024, 18:13 Vincent Beck wrote: > > > HUGE! > > > > On 2024/11/28 17:25:45 Jarek Potiuk wrote: > > > WOOWOOWOW! > > > > > > O

Re: [DISCUSS] Avoid Yoda conditions in our code

Same as slack, prefer it on right. Right is right 😆 On Thu, 28 Nov 2024 at 23:48, Hussein Awala wrote: > +1 > > On Fri, Nov 29, 2024 at 12:28 AM Jarek Potiuk wrote: > > > FYI: This is how it will look like when SIM300 is applied: > > https://github.com/apache/airflow/pull/44466 > > > > On Thu,

Re: Adding a significant newsfragment template to ease breaking change tracking

Nice, a good step for categorization for breaking changes. On Wed, 27 Nov 2024 at 13:38, Amogh Desai wrote: > Nice! > > It was really hard to do back and forth with the precommit while adding a > significant > one. This solves that! > > Thanks & Regards, > Amogh Desai > > > On Wed, Nov 27, 2024

Re: [VOTE] November 2024 PR of the Month

+1 for 41916 given that it was their first time contribution and I would like to encourage them to continue contributing. On Wed, 27 Nov 2024 at 13:52, Phani Kumar wrote: > +1 for https://github.com/apache/airflow/pull/43902 > > I'd like to nominate PR #43502 as it was a monumental effort *(329

Re: [ANNOUNCE] Cherry Picker Automation Now in GitHub Actions!

Awesome stuff, Pavan 🎉 On Tue, 19 Nov 2024 at 10:27, Jarek Potiuk wrote: > I tried it last night and it's SO MUCH BETTER for regular cases where > cherry picking *just works* without conflicts. A lot of saved mechanical, > repetitive work. > > Thanks for TP for suggesting it and Pavan for implem

Re: [DISCUSSION] Defferrable and mutiple triggerers WAS: Replace Poke & Reschedule mode from Sensors for Airflow 3 in favor of Deferrable

; stories for Airflow 3. > > > > > > > > > > I have no data to back it up, (maybe some people here could have > > > > > it) > > > > > - > > > > but > > > > > my intuition tells me that: > > > > &

Re: [DISCUSSION] Replace Poke & Reschedule mode from Sensors for Airflow 3 in favor of Deferrable

g to clarify the options. > > If we can rule out any of the options I presented we'll have made some > progress. > > Do you have an opinion on any of those? > > I would probably lean towards keep reschedule interface, keep it on base > sensor, remove it from providers. > &

Re: [DISCUSSION] Replace Poke & Reschedule mode from Sensors for Airflow 3 in favor of Deferrable

> I think we need to separate what to do about the sensors we have in > providers from the core interface, essentially. I think they are interconnected, especially if one of the goals is to have the "most performant" option (deferral) as the default or the only option for the users, regardless of

Re: [DISCUSSION] Replace Poke & Reschedule mode from Sensors for Airflow 3 in favor of Deferrable

There is 4th option to keep things as-is too :) On Wed, 13 Nov 2024 at 12:19, Kaxil Naik wrote: > Hi all, > > Following up on the Dev call discussions last Thursday, I am opening this > up for discussion. > > Reschedule mode was introduced to improve efficiency over poke

[DISCUSSION] Replace Poke & Reschedule mode from Sensors for Airflow 3 in favor of Deferrable

Hi all, Following up on the Dev call discussions last Thursday, I am opening this up for discussion. Reschedule mode was introduced to improve efficiency over poke mode by allowing tasks to wait without holding a worker slot. Since the introduction of deferrable operators in Airflow 2.2, however,

[LAZY CONSENSUS] Drop XCom pickling for Airflow 3

Hello all, I am calling for a LAZY CONSENSUS to drop support for XCom pickling. In Airflow 2.0.0, we had already changed the default for the ``[core] enable_xcom_pickling`` config to False. PR with details are here: https://github.com/apache/airflow/pull/43905 If no one objects, this vote will p

Re: [VOTE] AIP-86 Deadline Alerts (Formerly SLAs)

+1 binding On Tue, 5 Nov 2024 at 21:25, Jens Scheffler wrote: > Was a longer discussion thread and filled my inbox - but I very much > assume it is worth it! > > Thanks Denis and the others for leading this forward! > > +1 binding! > > On 05.11.24 19:45, Buğra Öztürk wrote: > > +1 non-binding. T

Re: [VOTE] Release Airflow 2.10.3 from 2.10.3rc1

-1 binding too -- thanks Jens for finding this! On Thu, 31 Oct 2024 at 08:45, Jarek Potiuk wrote: > -1 binding - there is an issue with source packages prepared for 2.10.3rc1 > > The source tar.gz contains `2.10.0` version instead of `2.10.3`. This > might indicate that there might be some iss

Re: New committer: Pavan Kuman Gopidesu (gopidesupavan)

Congrats Pavan On Tue, 29 Oct 2024 at 21:55, Jens Scheffler wrote: > Welcome to the team! > > On 29.10.24 19:41, Vikram Koka wrote: > > Congratulations Pavan, well deserved! > > > > Vikram > > > > > > On Tue, Oct 29, 2024 at 12:53 PM Oliveira, Niko > > > wrote: > > > >> Congrats! Great to have

Re: Airflow should deprecate the term "DAG" for end users

Same agreed with Brent & Daniel -- maybe we re-kindle this discussion for Airflow 4 :) -- but right now it will cause too much disruption On Tue, 22 Oct 2024 at 21:27, Constance Martineau wrote: > In my experience, when you ask those with Airflow experience what a dag is, > they'll start talking

Re: [ANNOUNCEMENT] THE AIRFLOW SURVEY IS LIVE!

Nice, let's do it On Mon, 21 Oct 2024 at 19:19, Jarek Potiuk wrote: > Looking forward to see this year's results :) > > On Mon, Oct 21, 2024 at 6:40 PM Briana Okyere > wrote: > > > Hey All, > > > > Exciting news- the Airflow Survey is officially LIVE! Everyone on this > list > > unders

[ANNOUNCE] New PMC member: Vikram Koka

Dear Airflow Community, The Project Management Committee (PMC) for Apache Airflow has invited *Vikram Koka *to become a PMC member and we are excited to announce that he has accepted our invitation. Vikram has been an active member of the Airflow community for more than 5 years. He has been an Ai

Re: [DISCUSS] Airflow debugging survey report & action items

Great work here Omkar, Iliya & Amogh for all your hard work on this 👏🏻 On Fri, 18 Oct 2024 at 23:25, Omkar P wrote: > Hey all, > > Hope you are doing great! > > So recently the Airflow debugging survey made the rounds and we've > received a total of 69 responses. > > We've analyzed those respons

Re: [VOTE] Airflow Providers prepared on October 10, 2024

+1 binding on sigs, license & checksums On Fri, 11 Oct 2024 at 20:46, Jarek Potiuk wrote: > +1 (binding) - checked reproducibility, checksums, signatures, licences. > That also includes sdist packages tests. > > I also used the opportunity to improve our release manager and PMC steps > for repro

Re: [REMINDER] Airflow 3 Dev call is not today but 10th Oct (next Thursday)

Ref for [2]: https://lists.apache.org/thread/06cbld63g4ccj9ds5pc90xslc7zbb146 On Thu, 3 Oct 2024 at 21:33, Kaxil Naik wrote: > Hello all, > > Just a reminder that the next dev call is next Thursday, not today, at the > same time as discussed in the last dev call [1][2]. > >

[REMINDER] Airflow 3 Dev call is not today but 10th Oct (next Thursday)

Hello all, Just a reminder that the next dev call is next Thursday, not today, at the same time as discussed in the last dev call [1][2]. Regards, Kaxil [1] https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-Summary.9 [2] https://

[RESULT][VOTE] Airflow 2.11 as bridge release

The vote for Airflow 2.11 as bridge release has passed. Discussion thread: https://lists.apache.org/thread/7jf12p2mk0nr5495f26r67gnpm3jq8oj Vote thread: https://lists.apache.org/thread/0d3dlly0mbps8n58hlxmmpvdcv9kx68s Binding +1 votes (9): - Kaxil Naik - Jens Scheffler - Jarek Potiuk - Ephraim

Re: [RESULT][VOTE] Airflow 2.11 as bridge release

As part of this I have removed all the feature-related Issues/PR from 2.11 GitHub milestone. On Thu, 26 Sept 2024 at 04:08, Kaxil Naik wrote: > The vote for Airflow 2.11 as bridge release has passed. > > Discussion thread: > https://lists.apache.org/thread/7jf12p2mk0nr5495f26r

Re: [VOTE] Airflow 2.11 as bridge release

.x? > > > > On Wed, Sep 4, 2024 at 9:07 PM Kaxil Naik wrote: > > > > > Hi all, > > > > > > As discussed in > > > https://lists.apache.org/thread/7jf12p2mk0nr5495f26r67gnpm3jq8oj I am > > > calling for a lazy consensus on using marking 2.

[Meeting Notes] Airflow 3.0 Dev call - 19 Sep 2024

ache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-(Proposed)Agenda> . Regards, Kaxil -- Including the Summary here too (might break formatting): *Catch-up on action items from last call* - Kaxil Naik showed the GitHub project boards <http

Re: [LATE REMINDER] Airflow 3 Dev call on 19 September (today) in 5 mins with a Proposed Agenda

ty members, could we please pick some other > date? Thanks! > > Best, > Michal > > On Thu, Sep 19, 2024 at 4:56 PM Kaxil Naik wrote: > > > Hello all, > > > > Just a reminder that we will have our dev call today (Thursday) in 5 mins > > :), 19 Sep, at 4 PM BS

Re: [VOTE] September 2024 PR of the Month

+1 for 42004 On Wed, 25 Sept 2024 at 21:07, Bishundeo, Rajeshwar wrote: > #42004 - simple auth manager -> big exciting changes for Airflow 3.0!!! > > -- Rajesh > > > > > > > On 2024-09-25, 3:36 PM, "Jens Scheffler" jsche...@web.de.inva>LID> wrote: > > > CAUTION: This email originated from outs

[LATE REMINDER] Airflow 3 Dev call on 19 September (today) in 5 mins with a Proposed Agenda

Hello all, Just a reminder that we will have our dev call today (Thursday) in 5 mins :), 19 Sep, at 4 PM BST (3 PM GMT/UTC | 11 AM EST | 8 AM PST). *Proposed Agenda*: 1) Check in on the action items from the last call 2) Defining intermediate milestones to track progress & GitHub project boards (

Re: [Meeting Notes] Airflow 3.0 Dev call - 5 Sep 2024

The recording of this call was added to Airflow's YouTube channel as an unlisted video as there were a lot of discussions & useful presentations on this call: https://youtu.be/7UdvZNmVPHc On Wed, 18 Sept 2024 at 03:13, Kaxil Naik wrote: > Hey all, > > I have updated our meet

[Meeting Notes] Airflow 3.0 Dev call - 5 Sep 2024

Hey all, I have updated our meeting notes document to summarize the discussion from our dev call on 22nd Aug. Link: https://cwiki.apache.org/confluence/x/8ApeEg#Airflow3Devcall:MeetingNotes-5September2024 To all those who attended, can you please double-check and add if I have missed anything?

[REMINDER] Airflow 3 Dev call on 5 September (today) with a Proposed Agenda

Hello all, Just a reminder that we will have our dev call today (Thursday), 5 Sep, at 4 PM BST (3 PM GMT/UTC | 11 AM EST | 8 AM PST). *Proposed Agenda*: 1) Check in on the action items from the last call 2) Presentation: "Airflow Beach Cleaning" project – Security work (Jarek) 3) Presentation: UI

[VOTE] Airflow 2.11 as bridge release

Hi all, As discussed in https://lists.apache.org/thread/7jf12p2mk0nr5495f26r67gnpm3jq8oj I am calling for a lazy consensus on using marking 2.11 as a bridge release with the following ideas: 1. It would only have bug fixes & deprecation warnings as a BRIDGE release -- NO features. The exce

Re: [Meeting Notes] Airflow 3.0 Dev call - 22 Aug 2024

Here is the recording of @Hussein Awala 's presentation on Airflow CI infrastructure: https://youtu.be/Y949SQi7axI On Fri, 30 Aug 2024 at 15:11, Kaxil Naik wrote: > Hey all, > > I have updated our meeting notes document to summarize the discussion from our > dev call on 2

[Meeting Notes] Airflow 3.0 Dev call - 22 Aug 2024

Hey all, I have updated our meeting notes document to summarize the discussion from our dev call on 22nd Aug. Link: https://cwiki.apache.org/confluence/x/8ApeEg#Airflow3Devcall:MeetingNotes-22August2024 To all those who attended, can you please double-check and add if I have missed anything? To

[REMINDER] Airflow 3 Dev call on 22 August (Thursday) with a Proposed Agenda

Hello all, Just a reminder that we will have our dev call today (Thursday), 22 August, at 4 PM BST (3 PM GMT/UTC | 11 AM EST | 8 AM PST). *Proposed Agenda*: 1) Check in on the action items from the last call - AIP-79: Removing FAB as core dep. POC Update (Jed) - Update on development pr

Re: [VOTE] New Provider for Core operators/sensor

+1 standard & core -- can't pick one! -1 under common On Mon, 19 Aug 2024 at 15:29, Vishnu Chilukoori wrote: > +1 essential or essentials > -1 under common > (non-binding) > > > -- > Regards, > Vishnu > > On Mon, Aug 19, 2024 at 7:12 AM Wei Lee wrote: > > > Same here. > > > > 1. +1 essential/es

Re: [DISCUSS] Airflow 2.11 as bridge release

Once we have an explicit agreement (with a VOTE after this discussion), it would be added as part of the docs similar to Elad's PR: https://github.com/apache/airflow/pull/41457 On Mon, 19 Aug 2024 at 13:22, Kaxil Naik wrote: > Hi team, > > After a discussion on the dev call and wi

Re: [VOTE] AIP-80: Explicit Template Fields in Operator Arguments

Thanks TP & everyone for the discussion here: +1 binding On Mon, 19 Aug 2024 at 13:07, Jarek Potiuk wrote: > +1 (binding). Thanks for responding to the concerns of compatibility, I > think personally this is crucial to have good Airflow 3 adoption. > > On Mon, Aug 19, 2024 at 1:34 PM Tzu-ping Ch

[DISCUSS] Airflow 2.11 as bridge release

Hi team, After a discussion on the dev call and with some of you on various forums, I wanted to propose the following: 1. It would only have bug fixes & deprecation warnings as a BRIDGE release -- NO features 2. We would release it after December so Airflow 3.0 features & removals are

Re: [DISCUSS] New provider Common.time

Yeah “standard” or “builtin” are other options. But tbh I feel a “core provider” is different than “Airflow core” as it will be a Provider I feel. Don’t have a strong opinion on it though — naming is hard On Fri, 16 Aug 2024 at 16:22, Tzu-ping Chung wrote: > Random idea, how about standard? Lik

[Meeting Notes] Airflow 3.0 Dev call - 8 Aug 2024

Hey all, I have updated our meeting notes document to summarize the discussion from our dev call on 8th Aug. Link: https://cwiki.apache.org/confluence/x/8ApeEg#Airflow3Devcall:MeetingNotes-8August2024 To all those who attended, can you please double-check and add if I have missed anything? To a

Re: [DISCUSS] New provider Common.time

Yeah I would favour a single "core provider": `apache-airflow-providers-core-modules` or just `apache-airflow-providers-core` sounds more apt. Example: from airflow.providers.core.sensors.datetime import DateTimeSensor from airflow.providers.core.operators.python import PythonOperator from airflo

Re: [VOTE] Release Airflow 2.10.0 from 2.10.0rc1

+1 binding On Thu, 15 Aug 2024 at 16:42, Jed Cunningham wrote: > +1 (binding) Checked reproducibility, signatures, checksums, licences. Used > it with the helm chart with a few different configs. >

Re: [VOTE] Release Airflow 2.10.0 from 2.10.0rc1

/airflow:slim-2.10.0rc1-python3.12 On Mon, 12 Aug 2024 at 23:24, Kaxil Naik wrote: > non-slim images has been pushed and are now available. Slim images are in > progress. > > > https://hub.docker.com/repository/docker/apache/airflow/tags?page=&page_size=&ordering=&name=2.10.0

Re: [VOTE] Release Airflow 2.10.0 from 2.10.0rc1

non-slim images has been pushed and are now available. Slim images are in progress. https://hub.docker.com/repository/docker/apache/airflow/tags?page=&page_size=&ordering=&name=2.10.0rc1 On Mon, 12 Aug 2024 at 20:07, Utkarsh Sharma wrote: > Dear Airflow Community, > > We are currently experienc

Re: [DISCUSS] Sensor Improvements With Tirggers

Thanks for putting this together, I will take a look this week. On Fri, 9 Aug 2024 at 13:12, Pavankumar Gopidesu wrote: > Hi All, > > I have created a draft document for Sensor Improvements using triggers. > > Details: > > https://docs.google.com/document/d/1Kb_wL-T1DHkOpmR_QNa3O5p_2hMTLzM-sb_Hz

Re: [VOTE] AIP-81 Enhanced Security in CLI via Integration of API

+1 binding On Sun, 11 Aug 2024 at 19:30, Jarek Potiuk wrote: > +1 (binding) - there are likely some small details to work out during the > implementation, but I think they don't impact the overall AIP. > > On Fri, Aug 9, 2024 at 11:42 PM Scheffler Jens (XC-AS/EAE-ADA-T) > wrote: > > > Added a f

Re: Need more people engaged in our CI / infrastructure

Added as agenda item for the next dev call (22 Aug) On Mon, 12 Aug 2024 at 00:25, Jarek Potiuk wrote: > I will let Hussein (if he has time) to share some more details :). > > Generally speaking we are using Github Actions as CI - so what we > **really** need is auto-scaling k8S cluster where K8S

Re: The Airflow main branch is now Airflow 3

🎉 Exciting! On Fri, 9 Aug 2024 at 14:17, Ephraim Anierobi wrote: > Fellow Airflowers, as agreed in the Airflow 3 dev call, we have branched > off for the Airflow 2.10.0 release, and now Airflow's main branch is > Airflow 3! > > Airflow 2.10.0rc1 will be created in a few hours from v2-10-stable b

Re: Airflow debugging story survey

One option might be to create a Google Form so that people can leave responses easily and you can share the editor rights to PMC members so we can look at the responses too. What do you think? On Thu, 8 Aug 2024 at 08:45, Dev iL wrote: > Hello everyone, > > In the context of improving the Airfl

Re: [VOTE] AIP-80: Explicit Template Fields in Operator Arguments

@Emanuel You can send an email to dev-unsubscr...@airflow.apache.org On Thu, 8 Aug 2024 at 10:59, Emanuel Oliveira wrote: > How can i remove myself from these emails? i want to follow airflow project > technically but not interested on ongoing people-project management > thingies. > Thanks 🙏😊 >

[REMINDER] Airflow 3 Dev call on 8 August (Thursday) with a Proposed Agenda

Hello all, Just a reminder that we will have our dev call today (Thursday), 8 August, at 4 PM BST (3 PM GMT/UTC | 11 AM EST | 8 AM PST). *Proposed Agenda*: 1) Check in on the action items from the last call - Check-in on AIPs targeting 3.0 - AIP-79: Removing FAB as core dep. POC Update

Re: [ANNOUNCE] New PMC member: Jens Schaffler

Congratulations Jens, very well deserved! On Tue, 6 Aug 2024 at 09:24, Pierre Jeambrun wrote: > Congratulations Jens! > > On Tue 6 Aug 2024 at 10:08, Buğra Öztürk wrote: > > > Congratulations Jens 🎊🎉 > > > > On Tue, 6 Aug 2024, 10:04 Pankaj Koti, .invalid> > > wrote: > > > > > Many congratulat

Re: Airflow 3 AIP Gold Rush! -- Great job everyone

Just noticed I had an incomplete sentence :) I meant: There is mutual respect between the members since they know that everyone is working for what's better for the Airflow Community. On Fri, 2 Aug 2024 at 22:16, Kaxil Naik wrote: > Hi team, > > Before we head for the weekend, I

Airflow 3 AIP Gold Rush! -- Great job everyone

Hi team, Before we head for the weekend, I wanted to take a moment to express my gratitude and pride in being part of this incredible community I am so honoured, proud and happy! Over the past few months, and especially in the last two weeks, I’ve witnessed our brilliant community members come to

Re: [VOTE] AIP-82 - External event driven scheduling in Airflow

+1 binding. Thanks for incorporating the feedback Vincent. On Fri, 2 Aug 2024 at 20:43, Scheffler Jens (XC-AS/EAE-ADA-T) wrote: > +1 binding. - thanks for the discussion! > > Sent from Outlook for iOS > > From: Beck, Vincent > Sent: Frida

Re: [VOTE] AIP-67 - Multi-team deplyment of Airflow Components

+1 binding Thank you for persisting with the questions & changes, great job. Realistically, though, I think this would happen in 3.1 due to dependencies, as you mentioned in the email. Looking forward. Regards, Kaxil On Fri, 2 Aug 2024 at 20:30, Scheffler Jens (XC-AS/EAE-ADA-T) wrote: > +1 b

Re: [VOTE] AIP-76: Asset Partitions

Yup, I am fine removing that language to make it explicit but leave it up to TP. On Fri, 2 Aug 2024 at 19:56, Daniel Standish wrote: > My concern with the AIP is the talk of support for incremental data > pipelines. In an incremental data pipeline, you don't think of a delta > load (let's say a

Re: [VOTE] AIP-76: Asset Partitions

We should resolve the open comments on the AIP before we conclude the vote though On Fri, 2 Aug 2024 at 02:09, Kaxil Naik wrote: > +1 binding > > On Thu, 1 Aug 2024 at 18:07, Buğra Öztürk wrote: > >> +1 non-binding >> >> On Thu, 1 Aug 2024, 16:07 Brent Bovenzi,

Re: [VOTE] AIP-76: Asset Partitions

+1 binding On Thu, 1 Aug 2024 at 18:07, Buğra Öztürk wrote: > +1 non-binding > > On Thu, 1 Aug 2024, 16:07 Brent Bovenzi, > wrote: > > > +1 (binding) > > > > On Thu, Aug 1, 2024 at 3:10 AM Tzu-ping Chung > > wrote: > > > > > Hi all, > > > > > > Kicking start the vote on the final data awarenes

Re: [DISCUSS] AIP-81: Enhanced Security in CLI via Integration of API

Thanks, Bugra, great turnaround on such a short notice. I have added my comments too. Regards, Kaxil On Sat, 27 Jul 2024 at 17:08, Buğra Öztürk wrote: > Thanks, Vikram! Answered them in the proposal. > --- > Thanks, Jarek! It's a pleasure to be part of and work on this one! :) > > Thanks for th

Re: [RESULT][VOTE] July 2024 PR of the Month

Congrats Niko On Fri, 2 Aug 2024 at 00:00, Ferruzzi, Dennis wrote: > Congrats Niko, and almost Vincent :P > > - ferruzzi > > > > From: Briana Okyere > Sent: Wednesday, July 31, 2024 3:11 PM > To: dev@airflow.apache.org > Subject: RE: [EXT] [RESULT][VOTE] July 20

Re: [DISCUSS] External event driven scheduling in Airflow

I needs to be authorized - that part is not > changed). (unless of course you want to use the same user for all kinds of > external interfaces, which for security point of view is a very bad idea - > each external system should have their own "service account" - that 's the >

Re: [DISCUSS] External event driven scheduling in Airflow

PI. > > While I - myself - find it it a bit unusual, I think it might do the job, > But I wonder if there is any alternative solution to the problem of "Having > producer of an event generating even in their standard way, make it easy > for airflow to consume it as a dataset event with

Re: [VOTE] AIP-79 & AIP-84 Remove Flask AppBuilder as a Core Dependency & UI REST API

+1 binding on both AIPs. But I want to make sure that we add Connection form decoupling to AIP-79 (or other AIP) unless we rely on FAB for backwards compatibility. On Thu, 1 Aug 2024 at 23:49, Scheffler Jens (XC-AS/EAE-ADA-T) wrote: > +1 binding for both (lost overview about all the votes - did

Re: [DISCUSS] External event driven scheduling in Airflow

ready available with the Dataset Event Create API; the one user-to-one function mapping is an odd user experience. I'm curious to hear what others think. On Thu, 1 Aug 2024 at 17:39, Kaxil Naik wrote: > I agree with both of you that it is indeed a good idea and that it can be > adde

Re: [VOTE] AIP-83 Remove Execution Date Unique Constraint from DAG Run

+1 binding On Thu, 1 Aug 2024 at 15:34, Pavankumar Gopidesu wrote: > +1 (non-binding) > > On Thu, Aug 1, 2024 at 3:22 PM Vincent Beck wrote: > > > +1 binding > > > > On 2024/08/01 14:05:35 Brent Bovenzi wrote: > > > +1 (binding) > > > > > > On Thu, Aug 1, 2024 at 2:12 AM Tzu-ping Chung > > > >

Re: [DISCUSS] External event driven scheduling in Airflow

n > the > > > poll-based approach -- left a comment on the push-based: I am not sure > of > > > why we need a function since create asset event API endpoint should > have > > > all info needed for what the Asset was. > > > > > > On Thu, 1 Aug 2024

Re: [DISCUSS] External event driven scheduling in Airflow

over 80% of usage here is the question in my mind. Or, are you thinking of >> those as being covered in the "push" model? >> > >> > Could you share more details about what is this "20% of scenarios which >> cover well over 80% of usage" please? >> >

Re: [DISCUSS] External event driven scheduling in Airflow

tely makes sense for the long > tail here, but can we do better for the 20% of scenarios which cover well > over 80% of usage here is the question in my mind. Or, are you thinking of > those as being covered in the "push" model? > > > > Could you share more deta

Re: [RESULT][VOTE] AIP-68 Extended Plugin Interface for React Views

Whoops, late on this but +1 binding :) On Wed, 31 Jul 2024 at 20:09, Scheffler Jens (XC-AS/EAE-ADA-T) wrote: > Hi, > > > > The vote for AIP-68 has passed. > > > > Binding +1 votes: > > * Jens Scheffler > * Brent Bovenzi > * Vincent Beck > * Jarek Potiuk > * Wei Lee > * Sh

Re: [VOTE] Airflow Providers prepared on July 28, 2024

+1 binding for FAB. Could just add a 1-liner or a link to the comment of why we had to exclude Amazon & Google On Mon, 29 Jul 2024 at 16:21, Elad Kalif wrote: > google and amazon providers are excluded from this wave. > please continue vote for fab provider only > > On Sun, Jul 28, 2024 at 2:30

Re: [DISCUSS] Extendable parsing controls (a.k.a. Pluggable Parsers)

Hi Igor, I have added permissions to your cWiki username: ikholopov I am interested in this topic but my feedback will be a bit delayed since I am focussing on reviewing 3.0 items first. Regards, Kaxil On Tue, 30 Jul 2024 at 15:09, Igor Kholopov wrote: > Hi all, > > I've prepared a proposal o

Re: [RESULT][VOTE] AIP-78 Scheduler-managed backfills

Votes: +1 (binding): 7 +1 (non-binding): 2 No -1 votes On Wed, 31 Jul 2024 at 00:04, Kaxil Naik wrote: > The vote for AIP-78 has passed. > > Binding +1 votes: > - Kaxil Naik > - Jens Scheffler > - Jarek Potiuk > - Vincent Beck > - Vikram Koka > - Daniel Standish &

[RESULT][VOTE] AIP-78 Scheduler-managed backfills

The vote for AIP-78 has passed. Binding +1 votes: - Kaxil Naik - Jens Scheffler - Jarek Potiuk - Vincent Beck - Vikram Koka - Daniel Standish - Jed Cunningham Non-binding votes: - Eugen Kosteev - Igor Kholopov AIP-78 link: https://cwiki.apache.org/confluence/x/0Q_TEg Discussion thread https

Re: [VOTE] AIP-78 Scheduler-managed backfills

This vote has concluded, will summarize it in the separate thread, thanks for voting. On Tue, 30 Jul 2024 at 21:27, Jed Cunningham wrote: > +1 binding >

Re: [EXTERNAL] Re: [DISCUSS] Add the Microsoft Fabric Provider in Apache Airflow

; <https://outlook.office.com/bookwithme/user/1e04a093bb8944209a5749e9b82a9...@microsoft.com?anonymous&ep=signature> > > Book time to meet with me > <https://outlook.office.com/bookwithme/user/1e04a093bb8944209a5749e9b82a9...@microsoft.com?anonymous&ep=signature> > &

Re: [DISCUSS] AIP-76 Asset Partitions

hosepartitions, incremental updates can be tracked and > > processed.* > > > I'm not sure what you mean by this, particularly the bit I emphasized. Can > you try to clarify? > > > On Mon, Jul 29, 2024 at 11:01 AM Kaxil Naik wrote: > > > Yeah, TP and I discu

Re: [DISCUSS] AIP-76 Asset Partitions

Yeah, TP and I discussed that we aren't solving the incremental load problem; folks can use it to achieve it similar to how you achieved it by storing the Watermark in Variables and we can natively support it with a revised AIP-30 in one of the minor releases for Airflow 3. We should clarify in th

Re: [DISCUSS] External event driven scheduling in Airflow

Thanks Vincent for driving these, I have added my comments to the AIP too. Regards, Kaxil On Fri, 26 Jul 2024 at 20:16, Scheffler Jens (XC-AS/EAE-ADA-T) wrote: > +1 on the comments of Vikram and Jarek, added main points on confluence > > Sent from Outlook for iOS > __

Re: [VOTE] AIP-80: Explicit Template Fields in Operator Arguments

+1 binding. The concerns are 100% valid. However, I think we have a path ahead with all 3 options: 1. Airflow 2.11 with new Airflow 3-style templating 2. Compat package to include "Airflow3BaseOperator" similar to Python "six" package [1]. 3. Upgrade utilities by using PyBowler [2] or

Re: [EXTERNAL] Re: [DISCUSS] Add the Microsoft Fabric Provider in Apache Airflow

Airflow Summit Sponsorship is indeed a great and simple way to showcase "contributing back to the community.", maybe Infrabel is also interested in Sponsoring. On the System tests and Dashboard front, I would like to see it cover all the operators available in the existing MS Providers—there are fo

Re: [VOTE] Release Apache Airflow Python Client 2.9.1 from 2.9.1rc1

+1 binding On Mon, 29 Jul 2024 at 07:53, Ephraim Anierobi wrote: > +1 (binding) checked reproducibility & checksums > > On 2024/07/25 08:38:50 Utkarsh Sharma wrote: > > Hey fellow Airflowers, > > > > I have cut the first release candidate for the Apache Airflow Python > Client > > 2.9.1. > > Thi

Re: [DISCUSS] Approaches to bugfixes for 2.10 after main becomes 3.0

This is also what we did during Airflow 2 release and worked well for us where some changes were in Airflow 1 branch only since it had Flask Admin vs FAB, and their were no concept of Provider in v1, Python 2 support etc On Sun, 28 Jul 2024 at 13:45, Kaxil Naik wrote: > @Jens: yup precisely

Re: [DISCUSS] Approaches to bugfixes for 2.10 after main becomes 3.0

ly agree. We should document it. > > > > > > There is one exceptional case. In my view this should be planned and > > > documented as exception: if something needs to be fixed that is not on > > main > > > anymore (e.g. code to integrate Fab) then a fix pr

[Meeting Notes] Airflow 3.0 Dev call - 25 July 2024

non-unique for a DAG: Tzu-ping Chung - In the next couple of days, a discussion thread with a draft AIP will be created by Tzu-ping Chung , followed by a vote. - Splitting providers from the mono-repo (Kaxil Naik) - Kaxil is not going to propose this anymore as he agrees t

Re: AWS Open Source Credits Program

Awesome, thanks for the good news. On Fri, 26 Jul 2024 at 20:34, Oliveira, Niko wrote: > Hello folks, > > This is a short note to announce that the AWS Open Source Credits Program > has approved us for another round of credits this year. $31,000 of credits > have been deposited in our Apache Air

  1   2   3   4   5   6   7   8   9   10   >