Re: [VOTE] AIP-80: Explicit Template Fields in Operator Arguments

I understand the concern, but would also wonder that a code base, be it user DAGs or third-party operators, lacks support in such a way, they would be correct to not encourage users to upgrade anyway. Airflow 3 is already going to have various breakages, some of them significantly less obvious t

Re: More ruff style rules

+1 to not include D102, D103. +1 to enforcing E731. (I just introduced this in one of my latest PRs 🤦‍♂️ I will fix it) +1 to not-enforcing D107. I actually think we could enforce TCH003. This kind of pattern seems to be already applied to the 3rd party library. It could apply to the standard l

Re: [DISCUSS] AIP-81: Enhanced Security in CLI via Integration of API

Thanks for writing this up! I left a quick question as a comment in the proposal. Best regards, Vikram On Wed, Jul 24, 2024 at 2:15 PM Buğra Öztürk wrote: > Hey all, > I have created a proposal for an Airflow 3.0 workstream: to utilize API for > CLI > > Details in https://cwiki.apache.org/con

Re: [DISCUSSION] Proposal to Add OTEL Provider to Apache Airflow

Howard, I am intrigued by this, but unclear on what this would actually look like and what benefits it would add. Specifically, I believe that AIP-49 adds support for OTEL emission of metrics and traces, but NOT task logs from Airflow. I am probably being dense here, but I don't quite understand

Re: [VOTE] AIP-80: Explicit Template Fields in Operator Arguments

-1 (non-binding) While the cleaner approach to templates is appealing, the blast radius of this change in its current shape is enormous. I am worried that it would strongly impede migration of users from Airflow 2 to Airflow 3, especially that not all Airflow users are proficient in Airflow, and f

[DISCUSS] External event driven scheduling in Airflow

Hello everyone, I created a draft AIP regarding "External event driven scheduling in Airflow". This proposal is about adding capability in Airflow to schedule DAGs based on external events. Here are some examples of such external events: - A user signs up to one of the user pool defined in my cl

Re: [VOTE] AIP-80: Explicit Template Fields in Operator Arguments

I am personally perfectly fine with that approach. Looks like a good design for the approach when we decide that "breaking most DAGs is acceptable" in general. J.

Apache Airflow 2.10.0b1 available for testing

Hey, fellow Airflowers, I have cut Airflow 2.10.0b1 now that all the main features have been included. This "snapshot" is intended for members of the Airflow developer community to test the build and allow early testing of 2.10.0. Please test this beta and create GitHub issues wherever possible i

Re: [VOTE] July 2024 PR of the Month

+1 to #40017. I like the extra effort that was put in to benchmark the changes and ensure that there is no regression, and I think it should be the way with any change to scheduler logic going forward. Shubham On 2024-07-25, 10:11 AM, "Kaxil Naik" mailto:kaxiln...@gmail.com>> wrote: CAUTION

Re: [VOTE] July 2024 PR of the Month

I would vote for your PR Niko #40017 :) On Thu, 25 Jul 2024 at 18:01, Oliveira, Niko wrote: > I have to throw a shamless self-vote in for #40017 which represents the > main chunk (the scheduler changes) for AIP-61 Hybrid Executors (now > Multiple Executor Configuration) 🎉 > > ___

Re: [VOTE] July 2024 PR of the Month

I have to throw a shamless self-vote in for #40017 which represents the main chunk (the scheduler changes) for AIP-61 Hybrid Executors (now Multiple Executor Configuration) 🎉 From: Jarek Potiuk Sent: Thursday, July 25, 2024 5:12:41 AM To: dev@airflow.apache.org

Re: [DISCUSS] Add the Microsoft Fabric Provider in Apache Airflow

Hi! Welcome to the community.In addition to the comments below, I am curious why create a new provider instead of adopting and expanding the existing Microsoft provider package? [1] [1] https://github.com/apache/airflow/tree/main/airflow/providers/microsoft - ferruzzi _

Re: [VOTE] AIP-78 Scheduler-managed backfills

+1 binding On Thu, Jul 25, 2024 at 7:36 AM Eugen Kosteev wrote: > +1 (non-binding) > > On Thu, Jul 25, 2024 at 4:17 PM Vincent Beck wrote: > > > I am all in for simplification! +1 binding > > > > On 2024/07/25 12:16:19 Igor Kholopov wrote: > > > +1 (non-binding) > > > > > > On Thu, Jul 25, 2024

Re: [VOTE] AIP-78 Scheduler-managed backfills

+1 (non-binding) On Thu, Jul 25, 2024 at 4:17 PM Vincent Beck wrote: > I am all in for simplification! +1 binding > > On 2024/07/25 12:16:19 Igor Kholopov wrote: > > +1 (non-binding) > > > > On Thu, Jul 25, 2024 at 2:05 PM Jarek Potiuk wrote: > > > > > +1 (binding) - while this one has some imp

[ANNOUNCE] Apache Airflow Providers prepared on July 21, 2024 are released

Dear Airflow community, I'm happy to announce that new versions of Airflow Providers packages prepared on July 21, 2024 were just released. Full list of PyPI packages released is added at the end of the message. The source release, as well as the binary releases, are available here: https://airf

Re: [VOTE] AIP-78 Scheduler-managed backfills

I am all in for simplification! +1 binding On 2024/07/25 12:16:19 Igor Kholopov wrote: > +1 (non-binding) > > On Thu, Jul 25, 2024 at 2:05 PM Jarek Potiuk wrote: > > > +1 (binding) - while this one has some implications on scheduler- which I > > think are important enough to hash out and design

[RESULT][VOTE] Airflow Providers - release of July 21, 2024

Hello, Apache Airflow Providers prepared on July 21, 2024 have been accepted. provider apache.beam is excluded from this release. 4 "+1" binding votes received: - Elad Kalif - Jarek Potiuk - Kaxil Naik - Hussein Awala Vote thread: https://lists.apache.org/thread/95rj2846vgkncq37knlsncvhv68j1sl9

Re: [VOTE] Airflow Providers prepared on July 21, 2024

+1 binding On Thursday, July 25, 2024, Kaxil Naik wrote: > +1 binding for Spark > > On Wed, 24 Jul 2024 at 06:46, Elad Kalif wrote: > > > apache beam is removed from this wave due to bug found > > > > On Wed, Jul 24, 2024 at 1:14 AM Jarek Potiuk wrote: > > > > > +1 (binding) - checked reproduc

Re: [VOTE] Airflow Providers prepared on July 21, 2024

+1 binding for Spark On Wed, 24 Jul 2024 at 06:46, Elad Kalif wrote: > apache beam is removed from this wave due to bug found > > On Wed, Jul 24, 2024 at 1:14 AM Jarek Potiuk wrote: > > > +1 (binding) - checked reproducibility, signatures, checksums, licences - > > all looks good. > > > > On Su

Re: [VOTE] AIP-80: Explicit Template Fields in Operator Arguments

My idea toward managing migration is mostly drawn from how Python (eventually) managed to pull most people over, so let me use that as a parallel to describe what I want to do. Initially Python tried using an automated script (2to3) to magically convert code into Python 3 compatible, which peop

Re: [VOTE] AIP-78 Scheduler-managed backfills

+1 (non-binding) On Thu, Jul 25, 2024 at 2:05 PM Jarek Potiuk wrote: > +1 (binding) - while this one has some implications on scheduler- which I > think are important enough to hash out and design, I think it's good as is > - this can be done while we implement other things. > > On Thu, Jul 25,

Re: [VOTE] July 2024 PR of the Month

#40468 for me as well On Thu, Jul 25, 2024 at 7:18 AM Scheffler Jens (XC-AS/EAE-ADA-T) wrote: > My Vote this month is for 40468 - as I hope a couple of errors can be > better found by users in UI with this. > > Sent from Outlook for iOS > >

Re: [DISCUSS] AIP-67 Multi-team deployment of Airflow components (reloaded)

One more thing to add - if there will be no more comments, I will start a vote soon - but maybe some clarifications might help - I spoke to few other people about it: * this one is heavily based on other AIPs that are also part of Airlfow 3. And while some parts are independent, AIP-72 (Task Isola

Re: [VOTE] AIP-78 Scheduler-managed backfills

+1 (binding) - while this one has some implications on scheduler- which I think are important enough to hash out and design, I think it's good as is - this can be done while we implement other things. On Thu, Jul 25, 2024 at 8:25 AM Scheffler Jens (XC-AS/EAE-ADA-T) wrote: > +1 binding > (I am no

Re: [VOTE] AIP-80: Explicit Template Fields in Operator Arguments

Valid concerns about migrations -- I share the same concern On Thu, 25 Jul 2024 at 12:17, Jarek Potiuk wrote: > +0.5 (binding). (See https://www.apache.org/foundation/voting - for > fractional votes). > > A bit more comment here: > > See my last comment in the AIP. I am not sure if this is the r

Re: [VOTE] AIP-80: Explicit Template Fields in Operator Arguments

+0.5 (binding). (See https://www.apache.org/foundation/voting - for fractional votes). A bit more comment here: See my last comment in the AIP. I am not sure if this is the right way but it's conditional +1. I love the idea, and proposal. Mostly because it will make DAG authoring more "modern" lo

[VOTE] Release Apache Airflow Python Client 2.9.1 from 2.9.1rc1

Hey fellow Airflowers, I have cut the first release candidate for the Apache Airflow Python Client 2.9.1. This email is calling for a vote on the release, which will last for 72 hours. Consider this my (binding) +1. As I’m not a member of the PMC, Ephraim signed the distribution. Airflow Client 2

Re: [VOTE] Release Apache Airflow Python Client 2.9.3 from 2.9.3rc1

Hey fellow Airflowers, I’m canceling this vote due to the incorrect increment of the Python client version and will create a new release with the correct version 2.9.0rc1. Thanks, Utkarsh Sharma On Wed, Jul 24, 2024 at 10:00 PM Jarek Potiuk wrote: > Sure :) > > On Wed, Jul 24, 2024 at 6:09 PM

Re: [VOTE] AIP-80: Explicit Template Fields in Operator Arguments

+1 (non-binding) -- Regards, Aritra Basu On Thu, 25 Jul 2024, 1:14 pm Tzu-ping Chung, wrote: > Hi all, > > I’m calling for a vote on AIP-80: Explicit Template Fields in Operator > Arguments. > https://cwiki.apache.org/confluence/x/2grOEg > > This proposal aims to improve how Airflow defines temp

[VOTE] AIP-80: Explicit Template Fields in Operator Arguments

Hi all, I’m calling for a vote on AIP-80: Explicit Template Fields in Operator Arguments. https://cwiki.apache.org/confluence/x/2grOEg This proposal aims to improve how Airflow defines template fields, and help users avoid annoying pitfalls currently exist. Discussion thread: https://lists.apa