Re: [VOTE] May 2024 PR of the Month

Great contributions from everyone. Thanks Daniel for clean up with #39336, and thank you Kaxil for leading the scarf discussion, I'm looking forward to more data. My vote goes to #39650 as I sit in at least a few calls every month where users are looking for this information to address their

Re: Airflow 3 Dev Call #1 with Proposed Agenda

Kaxil - thank you for creating the wiki, setting up call invites, and starting this thread. The first draft of the principles looks great. I have commented on the wiki with some feedback and personal thoughts. I'm not adding the comments on this thread to keep the wiki as the single place for

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

Sorry for joining the discussion late - I was down under the weather for the past two weeks. First off, great discussion so far and appreciate the effort and thought everyone has put into exploring Airflow 3. The list from Kaxil, et al looks solid and I'm really looking forward to more

Re: [VOTE] AIP-67 Multi-team deployment of Airflow components

+1 (non-binding). Looking forward to this one. Shubham On 2024-04-22, 12:31 AM, "Amogh Desai" mailto:amoghdesai@gmail.com>> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the

Re: [DISCUSS] DRAFT AIP-67 Multi-tenant deployment of Airflow components

ds features > that would be useful for people who are happy with the current "shared" > infrastructure approach (i.e. happy with multiple teams in one Airflow, > using existing DAG level rbac etc) but would like the ability to add > Connection restriction without forcing them to (th

Re: [VOTE] AIP-64: Keep TaskInstance try history

+1 (non-binding). Shubham On 2024-03-25, 5:10 PM, "Wei Lee" mailto:weilee...@gmail.com>> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. AVERTISSEMENT: Ce

Re: [VOTE] Remove experimental API

Amazon MWAA should not act as a blocker to the removal of the experimental API. We have already surpassed the end of support date for v1.10 on MWAA, which was February 21st, 2024. While customers may continue to use their existing v1.10 environments, we do not intend to introduce new features.

Re: [DISCUSS] DRAFT AIP-67 Multi-tenant deployment of Airflow components

ection of how well the proposes architecture fits in the Business structures of organisation that would like to deploy it. J. czw., 14 mar 2024, 20:21 użytkownik Mehta, Shubham mailto:shu...@amazon.com.inva>lid> napisał: > Hi folks, > > Firstly, thanks Jarek for putting together such a thoro

Re: [DISCUSS] DRAFT AIP-67 Multi-tenant deployment of Airflow components

Cc: "dev@airflow.apache.org" , "Mehta, Shubham" Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] DRAFT AIP-67 Multi-tenant deployment of Airflow components CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless

Re: [DISCUSS] DRAFT AIP-67 Multi-tenant deployment of Airflow components

Hi folks, Firstly, thanks Jarek for putting together such a thorough and well-thought-out proposal. I am very much in support of the multi-tenancy proposal. Having discussed this with over 30 customers (AWS and non-AWS), there's a clear desire to shift focus from the complex management of

Re: [VOTE] AIP-59 Performance testing framework

+1 (non-binding) Great work, Bartosz! On 2024-03-14, 11:00 AM, "Xiaodong (XD) DENG" mailto:xd.d...@apple.com.inva>LID> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is

Re: [VOTE] AIP 61 - Hybrid Executors

+1 non-binding. Looking forward to trying this out with Celery and ECS Executor in version 2.9. No pressure __ Shubham On 2024-02-01, 7:27 PM, "Ryan Hatter" mailto:ryan.hat...@astronomer.io.inva>LID> wrote: CAUTION: This email originated from outside of the organization. Do not click links

Re: [VOTE] January 2024 PR of the Month

+1 for #36537. Shubham On 2024-01-23, 3:03 PM, "Scheffler Jens (XC-AS/EAE-ADA-T)" mailto:jens.scheff...@de.bosch.com.inva>LID> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the

Re: [ANNOUNCE] New PMC member: Andrey Anshin (taragolis)

Congratulations, Andrey!! On 2024-01-16, 6:38 AM, "Josh Fell" mailto:josh.d.f...@astronomer.io.inva>LID> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

Re: [VOTE] December PR of the Month

+1 for #35926 -- this was amazing contribution, backed by significant preparatory work. I love all the UI contributions, but I've noticed a bias towards recognizing UI work, perhaps due to immediate visibility. This is my small effort to help rebalance the scale. Shubham On 2024-01-03, 7:38

Re: [Discussion] Requesting Feedback on Airflow Survey

Hi Briana, Thank you for putting this together. It's looking pretty good for the first draft. I have added quite a few comments and left some suggestions for changes. A few important things to call out here: 1. We should work backwards from what our goals are with this survey and how we want

Re: [VOTE] September 2023 PR of the Month

+ 1 to #28900. That required a lot of perseverance Shubham On 2023-09-29, 2:09 PM, "Briana Okyere" mailto:briana.oky...@astronomer.io.inva>LID> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the

Re: Summary of the In-person informal workshops we had in Toronto

raim Anierobi , "i...@sayapin.biz" , John Jackson , Jed Cunningham , "kna...@google.com" , Nikolas Oliveira , Pierre Jeambrun , Rafał Biegacz , "Bishundeo, Rajeshwar" , Rich Bowen , "Mehta, Shubham" , Sam Talasila , Sung Yun , "Balog, Tarus&

Re: [DISCUSS] move from semver to a more "rolling" release cycle for core

I couldn’t agree more with Jarek and Niko's perspective on the importance of maintaining SemVer for Apache Airflow. I've had conversations with dozens of customers, and it was a lot easier to convince them to upgrade for a more stable and secure Airflow environment. The key selling point was

Re: [VOTE] August 2023 PR of the Month

I'm with Hussein on this, +1 to #30259. Can't wait to see the efficiency gains and savings it will bring for the users. Shubham On 2023-08-28, 3:54 PM, "Hussein Awala" mailto:huss...@awala.fr>> wrote: CAUTION: This email originated from outside of the organization. Do not click links or

Re: [ANNOUNCE] New PMC member: Hussein Awala

Many congratulations, Hussein!! On 2023-07-29, 10:34 AM, "Jarek Potiuk" mailto:ja...@potiuk.com>> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Hello Airflow

Re: [VOTE] PR of the Month for the Airflow Newsletter

+1 to #30727. Without going into details, 50-60% optimization in any aspect of Airflow is amazing. Shubham On 2023-07-26, 11:06 AM, "Michael Robinson" mailto:michael.robin...@astronomer.io.inva>LID> wrote: CAUTION: This email originated from outside of the organization. Do not click links

Re: [DISCUSS] Should we pre-install celery/k8s providers?

I second Jed's proposal and believe that the implications and benefits, as laid out, make a lot of sense. > we wouldn't have to set a minimum version for these 2 providers to ensure we > have the version with the executors, allowing an "old" k8s provider (KPO > itself is pretty common after

Re: [ANNOUNCE] New committers: Vincent Beck, Phani Kumar, Maciej Obuchowski

Congratulations everyone! Very well deserved!! On 2023-06-28, 4:22 PM, "Vikram Koka" mailto:vik...@astronomer.io.inva>LID> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content

Re: [VOTE] AIP-56 Extensible user management

+1 (non-binding) -- Looking forward to all the custom auth managers this will enable. Shubham On 2023-06-19, 9:48 AM, "Oliveira, Niko" mailto:oniko...@amazon.com.inva>LID> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless

Re: [VOTE] Airflow Providers prepared on April 25, 2023

pendency botocore. > # TODO: We can remove it once boto3 and aiobotocore both have compatible > botocore version or > # boto3 have native async support and we move away from aio aiobotocore > - name: aiobotocore > dependencies: > - aiobotocore>=2.1.1 > > > The first

Re: [VOTE] April 2023 PR of the Month

That's an amazing set of PRs. I vote for #29940. Kudos to Maciej for the effort he put into it and laying out the foundation for OL work. Shubham On 2023-04-25, 6:08 PM, "Michael Robinson" mailto:michael.robin...@astronomer.io.inva>LID> wrote: CAUTION: This email originated from outside of

Re: [VOTE] Airflow Providers prepared on April 25, 2023

Hey Pankaj, I have a few clarifying questions: 1. Can you specify the Airflow version used during testing? Based on the botocore version constraint, it appears to be v2.5.3. 2. Did you use a modified constraint file to install the Amazon provider v8.0.0rc2 with Airflow v2.5.3? If so, what were

Re: [ANNOUNCE] New committer Hussein Awala

Congrats, Hussein!! On 2023-04-11, 2:51 PM, "Jarek Potiuk" mailto:ja...@potiuk.com>> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Hello Airflow Community,

Re: AIP-56 Extensible user management

Bumping this up for feedback! @Vikram, @Kaxil, et al. - I understand that you're quite busy, but we would greatly appreciate your feedback here. Your inputs could help us improve the proposal and move forward with multi-tenancy work. Cheers, Shubham On 2023-03-28, 2:05 PM, "Beck, Vincent"

Re: [DISCUSS] AIP-55 Rule-based timetable with logical composition

Hi Malthe I really like this feature suggestion, especially since it's crucial for customers who are transitioning from Control-M or Autosys [1] to Airflow. Implementing this feature has the potential to bring Airflow's scheduling capabilities on par with these tools. > I actually added some

Re: [DISCUSS] a cache for Airflow Variables

Considering the current framing of the problem/solution - as a means to address poorly written DAGs - I agree with Elad's perspective. Nevertheless, I believe that caching for the secrets backend has the potential to enhance Airflow performance even for DAGs adhering to best practices,

Re: Request for feedback on proposal for new OpenLineage provider in Airflow

Please include me, I will try my best to join (shubhammehta...@gmail.com) Best, Shubham On 2023-03-22, 2:24 PM, "Jarek Potiuk" mailto:ja...@potiuk.com>> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the

Re: Seeking Feedback for Airflow Multi-Tenant Model Proposal

es will have no choice but to use the > provider installed by the Service of theirs > > I think that all gives the user the choice - if they want to go role > management and multi-tenant capabilities, fine but they will have to > mange the users outside of Airflo

Re: [VOTE] Move K8S / Celery (and related) executors to respective providers

+ 1 non-binding From: Elad Kalif Reply-To: "dev@airflow.apache.org" Date: Tuesday, February 21, 2023 at 10:44 AM To: "dev@airflow.apache.org" Subject: RE: [EXTERNAL][VOTE] Move K8S / Celery (and related) executors to respective providers CAUTION: This email originated from outside of the

Re: Seeking Feedback for Airflow Multi-Tenant Model Proposal

/apache-airflow/stable/configurations-ref.html#auth-backends > [2]: https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/security/access-control.html > > On Tue, 14 Feb 2023 at 02:06, Mehta, Shubham wrote: >> >> H

Re: Seeking Feedback for Airflow Multi-Tenant Model Proposal

Hi Vikram, Thank you for taking the time to review the proposal. I appreciate your insights — I will make sure to reach out to you directly in the future for feedback as that would've undoubtedly saved us some time and effort. In regards to the separation of user management, I understand your

Re: [VOTE] AIP-53 OpenLineage in Airflow

+1 non-binding. I'll be on the lookout for initial PRs to learn more about the implementation details of how System Tests will be extended to cover these changes, as well as the ongoing maintenance required from providers. The proposed changes should definitely make it easier for Airflow

Re: [DISCUSSION] Move K8S and Celery Executors (and related) to respective providers?

Thank you for the detailed reply! It takes me at least 2 cups of coffee to absorb all the information you provide in your emails :) > Release managers are merely executing the mechanics of the release > preparation and ca make decisions in case someone finds issues that are > potentially

Re: [DISCUSSION] Move K8S and Celery Executors (and related) to respective providers?

Big +1 to this proposal. Thanks for initiating it, Jarek. This will accelerate innovation for K8s and celery executor and encourage other providers, like Google and Amazon, to include cloud native executors in their packages. Couple of questions/concerns: 1. How does this impact work of

Seeking Feedback for Airflow Multi-Tenant Model Proposal

Hi folks, Over the past few weeks, @Vincent Beck and I have been working on a proposal for a multi-tenant model for Apache Airflow. Building on

Re: [ANNOUNCE] New committer Niko Oliveira (o-nikolas)

Congratulations, Niko!! From: Jarek Potiuk Reply-To: "dev@airflow.apache.org" Date: Monday, December 19, 2022 at 1:25 PM To: "dev@airflow.apache.org" Subject: RE: [EXTERNAL][ANNOUNCE] New committer Niko Oliveira (o-nikolas) CAUTION: This email originated from outside of the organization. Do

Re: [ANNOUNCE] New committer Andrey Anshin (Taragolis)

Congratulations Andrey!! On 2022-12-01, 11:31 PM, "Jarek Potiuk" wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Hello Airflow Community, I am happy

Re: [Discussion] Airflow Newsletter name and branding

+1 (non-binding) to the “Atmosphere: The Airflow Monthly” Other suggestion - “Symphony: The Airflow Monthly” Context – In one of the talks Jarek compared Airflow to orchestra who guides tasks for execution and that analogy stuck with me. I started looking into best orchestra, and “Symphony” is

Re: [DISCUSS] Airflow Scheduling Delay Metric Definition

Ping, I’m very interested in this as well. A good metric can help us benchmark and identify potential improvements in the scheduler performance. In order to understand the proposal better, can you please share where and how do you intend to use “Scheduling delay”? Is it meant for benchmarking

Re: Re: [DISCUSS] Approach for new providers of the community

Hi all, I’m Shubham, Sr. Product Manager at AWS, working closely with John and the MWAA team. Glad to see the Airflow community openly discussing this topic which will likely shape Airflow’s growth in the future. Firstly, I am with Elad and Dennis that we shouldn’t be gatekeeping the new