Re: Naming things: What should the imports in dag files for DAG etc. be?

I'm partial to everything that we expect users to use to be importable from `airflow`, but would love to hear other people's thoughts. On Fri, Aug 30, 2024 at 5:48 AM Ash Berlin-Taylor wrote: > Hi everyone, > > It’s time to have a another discussion about everyone's favourite > discussion - nami

Re: [DISCUSS] AIP-78 scheduler-managed backfill

Seems valid for default behaviour, but if I backfill for a year and realize there was something wrong with the code, I don't want to manually fail each dag run that is running. How about a force kill option? On Wed, Jul 10, 2024 at 9:28 AM Daniel Standish wrote: > Yup that's true @Tzu-ping Chung

Re: [DISCUSS][AIP-38 Modern Web Application]

I love it and 100% agree. Thinking "Dag Groups", where you can group dags (static & dynamic) into a subfolder. Tags are great for filtering, but they aren't a replacement for dirs especially at a large scale. We have some deployments with 20k dags and as designed today, it's not navigable at that s

Re: [DISCUSS] Number of queries to Airflow database in "DAG File Processing Stats"

I love the idea. If we were to store it in the DB, would we keep a history, or only the latest stats from the most recent dag parsing loop? DAG parsing by default is every 30s right? On Fri, Jun 14, 2024 at 6:53 AM Jarek Potiuk wrote: > > I think we still need to enable the ability for DAGs at p

Proposal for Enhanced Data Awareness in Airflow

nt. We will follow up with a set of formal AIPs. Constance -- Constance Martineau Senior Product Manager Email: consta...@astronomer.io Time zone: US Eastern (EST UTC-5 / EDT UTC-4)

Re: Call with Nielsen team demoing their DAG debugging feature

Hello again, Given all the enthusiasm - assuming Nielsen is ok with this - what if someone recorded the meeting so that it could be shared with those that are interested? Constance On Mon, Jun 10, 2024 at 1:48 PM Constance Martineau wrote: > Hi Jarek, > > Same :) > > Than

Re: Call with Nielsen team demoing their DAG debugging feature

Hi Jarek, Same :) Thanks, Constance On Mon, Jun 10, 2024 at 9:57 AM Amogh Desai wrote: > Hello Jarek, > > Please add me to the invite as well. > > Thanks & Regards, > Amogh Desai > > > On Mon, Jun 10, 2024 at 11:22 AM Abhishek Bhakat > wrote: > > >

Re: [VOTE] May 2024 PR of the Month

While I know that #39336 was a lot of work and big from a dev perspective (thanks @Daniel Standish !), my vote goes to #39650 as task-level CPU and memory metrics are a long-standing feature request. On Tue, May 28, 2024 at 1:42 PM Jarek Potiuk wrote: > #39336 hands down > > On Tue, May 28, 2024

Re: [DISCUSS] AIP-63, AIP-64, and AIP-65: DAG Versioning

Agreed. When Jed and team wrote the AIP, we intentionally limited the scope to DAGs since the AIPs were already really large, but the intention is to extend the concept to datasets. Funny that you bring up point #2. A few of us met last week to talk about DAG Versioning, and that use-case came up

Re: [VOTE] Proposal for adding Telemetry via Scarf

+1 non-binding On Thu, May 9, 2024 at 7:17 AM Tomasz Urbaszek wrote: > +1 binding > > On Thu, 9 May 2024 at 12:40, Andrey Anshin > wrote: > > > +1 binding > > > > > > > > > > > > On Thu, 9 May 2024 at 13:25, Wei Lee wrote: > > > > > Got it. Thanks Jarek for pointing out! > > > > > > Best, > >

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

ns on AIP-67 and others change > if we consider Airflow 3 is "starting". > > The price for back-compat is speed of development and quality. More > combinations to test, more unexpected issues uncovered, necessity to keep > parallel paths (old/new) while adding new features

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

at it continues to remain relevant for the next ten years. Constance On Mon, May 6, 2024 at 2:10 PM Ash Berlin-Taylor wrote: > There's a lot of technical debt hiding in Airflow, especially the > scheduler that makes it harder and harder to efficiently add new features. > > At some po

Re: [ANNOUNCE] New committer: Wei Lee

Congrats Wei! > On Apr 8, 2024, at 5:31 AM, Pankaj Singh wrote: > > Congrats Wei, very well deserved! > >> On Mon, Apr 8, 2024 at 2:33 PM Rahul Vats wrote: >> >> Congrats Wei, very well deserved! >> >> Regards, >> Rahul Vats >> >> >>> On Mon, 8 Apr 2024 at 14:27, Amogh Desai wrote: >>>

Re: [DISCUSS] Rename channels on slack

> Maybe we restrict who can post in development for a period of time with a message directing folks to the right places? As long as we don't make it committer only. If you're contributing something and want some help/feedback, it's not welcoming to find out that you're to be restricted from the de

Re: [DISCUSS] Rename channels on slack

+1 for #contributing and leaving #troubleshooting. Shorter names in slack are nice where possible. No strong opinion on the actual names. Agree that #development needs to be renamed to something more obvious though. On Thu, Feb 8, 2024 at 9:30 AM Vincent Beck wrote: > I am +1 in renaming these

Re: Idea for Discussion: custom TI dependencies

licy/task_instance_mutation_hook/pod_mutation_hook. > > So I do not think this violates our current preference in terms of > security. > > > Hi Constance, > > I thought the trigger is mainly for deferrable operator cases? It’s quite > different scenario from what I’m trying to

Re: Idea for Discussion: custom TI dependencies

Naive question: Instead of running the code on the scheduler - could the condition check be delegated to the triggerer? On Fri, Feb 2, 2024 at 2:33 PM Pierre Jeambrun wrote: > But maybe it’s time to reconsider that :), curious to see what others > think. > > On Fri 2 Feb 2024 at 20:30, Pierre Je

Re: [DISCUSSION] Enhanced Multi-Tenant Dataset Management in Airflow: Potential First Steps

ed to think carefully about how API-created > datasets would coexist with those defined in DAG files. However, in my > opinion, this is a natural transition that will likely become necessary as > Airflow is used in increasingly diverse environments and organizations, a > shift that seems

Re: [PROPOSE] Add A Code of Conduct for Slack and Meetups

t; we can ensure folks can anonymously submit a post as "breaking the > guidelines" if they cannot DM an individual. > > On Fri, Jan 26, 2024 at 10:16 AM Constance Martineau > wrote: > > > Wow Briana! This is fantastic, what a great idea! I added a few comments. > &

Re: [PROPOSE] Add A Code of Conduct for Slack and Meetups

Wow Briana! This is fantastic, what a great idea! I added a few comments. I also had a similar question as Jarek that I think merits a discussion: Should we have a committee or group to handle reported guideline violations? If we single out one person to report violations to, we'll have to continu

Re: [DISCUSSION] Enhanced Multi-Tenant Dataset Management in Airflow: Potential First Steps

ating Airflow) my actual > experience with Datasets is limited, I've been mainly observing what was > going on, so I would love to hear what those who created (and continue to > think about future of) the datasets :). > > J, > > On Wed, Jan 24, 2024 at 7:27 PM Constance Ma

Re: [DISCUSSION] Enhanced Multi-Tenant Dataset Management in Airflow: Potential First Steps

it to be a task output as part of a dag. The only valid reason to now allow it IMHO is because they were designed to be defined within a dag file, similar to a dag, and we don't want to deal with the impediment I laid out. On Wed, Jan 24, 2024 at 12:45 PM Jarek Potiuk wrote: > On Wed, Jan

Re: [DISCUSSION] Enhanced Multi-Tenant Dataset Management in Airflow: Potential First Steps

t; >> > > >> these tenants, we currently have a complex setup: > >> > > >> > >> > > >>1. Containers run on a schedule to export metadata to CosmosDB > >> > (these > >> > > >>will be replaced by the listener). > >> > > >>2. Additional scheduled containers pull data from CosmosDB and > >> > write > >> > > >>it to a shared file system, enabling generated DAGS to read it > >> and > >> > > mirror a > >> > > >>dataset across tenants. > >> > > >> > >> > > >> > >> > > >> Proposed Workflow > >> > > >> Here's a breakdown of our proposed workflow: > >> > > >> > >> > > >>1. Cross-Tenant Dataset Interaction: We have Dags in Tenant 1 > >> > > >>producing Dataset A. We need a mechanism to trigger all Dags > >> > > consuming > >> > > >>Dataset A in Tenant 2. This interaction is crucial for our > data > >> > > pipeline's > >> > > >>efficiency and synchronicity. > >> > > >>2. Dataset Listener Implementation: Our approach involves > >> > > >>implementing a Dataset listener that programmatically creates > >> > > Dataset A in > >> > > >>all tenants where it's not present (like Tenant 2) and export > >> > Dataset > >> > > >>updates when they happen. This would trigger an update on all > >> Dags > >> > > >>consuming from that Dataset. > >> > > >>3. Standardized Dataset Names: We plan to use standardized > >> dataset > >> > > >>names, which makes sense since a URI is its identifier and > >> > > uniqueness is a > >> > > >>logical requirement. > >> > > >> > >> > > >> [image: image.png] > >> > > >> > >> > > >> Why This Matters: > >> > > >> > >> > > >>- It offers a streamlined, automated way to manage datasets > >> across > >> > > >>different Airflow instances. > >> > > >>- It aligns with a need for efficient, interconnected > workflows > >> in > >> > a > >> > > >>multi-tenant environment. > >> > > >> > >> > > >> > >> > > >> I invite the community to discuss: > >> > > >> > >> > > >>- Are there alternative methods within Airflow's current > >> framework > >> > > >>that could achieve similar goals? > >> > > >>- Any insights or experiences that could inform our approach? > >> > > >> > >> > > >> Your feedback and suggestions are invaluable, and I look forward > >> to a > >> > > >> collaborative discussion. > >> > > >> > >> > > >> Best Regards, > >> > > >> Eduardo Nicastro > >> > > >> > >> > > > > >> > > > >> > > >> > > > -- Constance Martineau Senior Product Manager Email: consta...@astronomer.io Time zone: US Eastern (EST UTC-5 / EDT UTC-4) <https://www.astronomer.io/>

Re: [VOTE] January 2024 PR of the Month

or a future issue of the newsletter, please drop me a line at < > briana.oky...@astronomer.io>. > > -- > Briana Okyere > Community Manager > *Astronomer* > -- Constance Martineau Senior Product Manager Email: consta...@astronomer.io Time zone: US Eastern (EST UTC-5 / EDT UTC-4) <https://www.astronomer.io/>

Re: [VOTE] New Airflow Community Provider: Teradata

tests/providers/teradata > > > > > > > > System Tests: > > > > > > > > https://github.com/Teradata/airflow/tree/td_develop/tests/system/providers/teradata > > > > System Tests Dashboard: https://teradata.github.io/airflow/ > > > > > > > > > > > - > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > > -- Constance Martineau Senior Product Manager Email: consta...@astronomer.io Time zone: US Eastern (EST UTC-5 / EDT UTC-4) <https://www.astronomer.io/>

Re: [VOTE] December PR of the Month

+1 for 35719 > On Jan 2, 2024, at 11:14 PM, Amogh Desai wrote: > > My vote goes to #35719. Having a nice UI fix come in, that too from a first > time contributor > makes it a winner for me! > > Thanks & Regards, > Amogh Desai > >> On Wed, Jan 3, 2024 at 7:51 AM Wei Lee wrote: >> >> I want t

Re: [VOTE] October 2023 PR of the Month

Tue, Oct 31, 2023 at 6:00 PM Jed Cunningham > wrote: > > > The new OpenSearch provider gets my vote - 34705. > > > -- Constance Martineau Senior Product Manager Email: consta...@astronomer.io Time zone: US Eastern (EST UTC-5 / EDT UTC-4) <https://www.astronomer.io/>

Re: [VOTE] October 2023 PR of the Month

Hi Briana, My vote is for 34784. I love that we are resolving some inconsistencies within the AWS provider package Constance On Mon, Oct 30, 2023 at 1:19 PM Briana Okyere wrote: > Hey All, > > It’s once again time to vote for the PR of the Month. > > Please note that if th

Re: Lazy Consensus - Removing the Experimental tag for Pluggy

Oh! Yes I agree > and has been since it’s release in 2.6 Misunderstood. Thought it was only released a few months ago. Time flies! +1 (non-binding) > On Sep 8, 2023, at 3:44 PM, Hussein Awala wrote: > >  >> >> What’s the motivation to remove this now? > > The feature was introduced in Ai

Re: Lazy Consensus - Removing the Experimental tag for Pluggy

What’s the motivation to remove this now? > On Sep 8, 2023, at 2:55 AM, Cody Rich wrote: > > Hi Everyone, > > I'm calling a lazy consensus for removing the experimental tag for the > pluggy interface (pr #34174 ). > It's currently denoted as experi

Re: [RESULT][VOTE] PR of the Month for August Airflow Newsletter

+1 for special mention in newsletter. Fantastic idea Briana! > On Aug 30, 2023, at 3:59 PM, Briana Okyere > wrote: > > Hey All, > > Thanks for your engagement with our PR of the Month Vote for the > August Newsletter. > Lots of great PRs this month! > > The results are in at: > > + 7 to #30

Re: [VOTE] August 2023 PR of the Month

@eumiro deserves his own shoutout for the refactor work! That said, my vote is also #30259. > On Aug 28, 2023, at 6:53 PM, Hussein Awala wrote: > > I agree with Jed and Jarek on the great work by @eumiro, but my vote is for > #30259. > >> On Mon, Aug 28, 2023 at 10:49 PM Jarek Potiuk wrote:

Re: [VOTE] (extended) on AIP-50 (part 2) to finalize it

Hello, I think Option B is reasonable. +1 for B, non-binding. Constance On Mon, May 22, 2023 at 4:54 PM Scheffler Jens (XC-DX/ETV5) wrote: > Hi Airflow-Developers, > > It is not democracy if nobody makes a vote. I don't want to be a > "dictator" but still pr

Re: [VOTE] February PR of the Month

+1 for 27758On Feb 25, 2023, at 7:53 PM, Collin McNulty wrote:I’ll add my voice to the 27758 chorus. +1On Fri, Feb 24, 2023 at 6:09 PM Hussein Awala wrote: +1 to #27758 too! From: Vikram Koka Sent: Saturday, February 25, 2023 12:21:40 AM To: dev@airflow.apache.org Subj

Re: [VOTE] AIP-50 Trigger DAG UI Extension with Flexible User Form Concept

gistergericht: Amtsgericht Stuttgart, HRB 14000; > Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer; > Geschäftsführung: Dr. Stefan Hartung, > Dr. Christian Fischer, Filiz Albrecht, Dr. Markus Forschner, Dr. Markus > Heyn, Rolf Najork > ​ > -- Constance Martineau Product Manager Email: consta...@astronomer.io Time zone: US Eastern (EST UTC-5 / EDT UTC-4) <https://www.astronomer.io/>

Re: [VOTE] November 2022 PR of the Month

ttps://github.com/apache/airflow/pull/27540 >> >> >> >> [27597] Add max_wait for exponential_backoff in BaseSensor >> >> https://github.com/apache/airflow/pull/27597 >> >> >> >> [27506] Fix mini scheduler expansion of mapped task >> >> https://github.com/apache/airflow/pull/27506 >> >> >> >> [27526] Clean backcompat code kpo >> >> https://github.com/apache/airflow/pull/27526 >> >> >> >> Regards, >> >> John >> > >> > >> > -- Constance Martineau Product Manager Email: consta...@astronomer.io Time zone: US Eastern (EST UTC-5 / EDT UTC-4) <https://www.astronomer.io/>

Re: [VOTE] September 2022 PR of the Month

, Daniel Standish < > daniel.stand...@astronomer.io.INVALID> wrote: > > One vote for https://github.com/apache/airflow/pull/26400 (improved test > command) > > On Tue, Sep 27, 2022 at 10:50 AM Jed Cunningham > wrote: > >> My write-in is ExternalPythonOperator:

Re: [VOTE] August 2022 PR of the Month

r the local Quick Start in docs >>>> https://github.com/apache/airflow/pull/25888 >>>> >>>> [25788] Properly check the existence of missing mapped TIs >>>> https://github.com/apache/airflow/pull/25788 >>>> >>>> [25610] Grid logs for mapped instances >>>> https://github.com/apache/airflow/pull/25610 >>>> >>>> [25857] Add `RedshiftCreateClusterSnapshotOperator` >>>> https://github.com/apache/airflow/pull/25857 >>>> >>>> -- > > Collin McNulty > Lead Airflow Engineer > > Email: col...@astronomer.io > Time zone: US Central (CST UTC-6 / CDT UTC-5) > > > <https://www.astronomer.io/> > -- Constance Martineau Product Manager Email: consta...@astronomer.io Time zone: US Eastern (EST UTC-5 / EDT UTC-4) <https://www.astronomer.io/>

Re: [Proposal] Creating DAG through the REST api

n modules where strong authentication such as >>> Kerberos can be used. >>> 2. Execute DAG code as the API identity, i.e. A DAG created through the >>> API service will have run_as_user set to be the API identity. >>> 3. To enforce data access control on DAGs, the API identity should also >>> be used to access the data warehouse. >>> >>> We shared a demo based on a prototype implementation in the summit and >>> some details are described in this ppt >>> <https://drive.google.com/file/d/1luDGvWRA-hwn2NjPoobis2SL4_UNYfcM/view>, >>> and would love to get feedback and comments from the community about this >>> initiative. >>> >>> thanks >>> Mocheng >>> >> -- Constance Martineau Product Manager Email: consta...@astronomer.io Time zone: US Eastern (EST UTC-5 / EDT UTC-4) <https://www.astronomer.io/>

Re: [VOTE] AIP-48 Data Driven Scheduling

+1 (non-binding) > On Jun 4, 2022, at 7:45 PM, Igor Kholopov > wrote: > >  > +1 (non-binding) > Left some comments with my thoughts on the AIP wiki page. > >> On Fri, Jun 3, 2022 at 3:11 PM Ankit Chaurasia wrote: >> +1 (non-binding) >> >> Ankit Chaurasia >> HomePage | LinkedIn | +91-99873

Re: Implicit DAG registration

Am intrigued. Curious about dynamic dag pattern, where you create the DAG object in a a create_dag function and adding the DAG to globals. Would this new way prevent someone from modifying the dag object within the function, or returning it? > On Apr 27, 2022, at 11:20 AM, Ferruzzi, Dennis >

Re: Make first dag run optional when catchup is False

the next day. The oddness is amplified when you consider a monthly >> dag, where if you deploy now, start date is now, first schedulable run is >> next month, therefore first run _more_ than a month away. To fix this I >> think we need to add support in our timetables for running

Re: Make first dag run optional when catchup is False

2022 at 3:00 PM Philippe Lanoe >>>> wrote: >>>> >>>> >>>> >>>> Hello Daniel, >>>> >>>> >>>> >>>> Thank you for your answer. In your example, as I experienced, the >>>> first run would n

Re: K8s version/support policy: as long as K8s project, or as long as cloud providers?

+1 > On Mar 7, 2022, at 7:08 PM, Vikram Koka wrote: > >  > +1 > >> On Mon, Mar 7, 2022 at 3:01 PM Kaxil Naik wrote: >> +1 >> >>> On Mon, 7 Mar 2022 at 21:17, Jarek Potiuk wrote: >>> +1 >>> On Mon, Mar 7, 2022 at 10:12 PM Ash Berlin-Taylor wrote: Hey everyone, So Kube

Re: SSIS

but it is worth it. Your mileage may vary, but after a data scientist moved all his workloads to Airflow, he said that the time he spent debugging his pipelines went down 50%. Hope this helps, Constance On Tue, Jun 29, 2021 at 5:04 AM Neeku Endhuku Nenu cheppanu < idk.050...@gmail.com> wro

Re: [VOTE] AIP-38: Modern Web Application

+1 (non-binding) On Wed, Mar 3, 2021 at 10:31 AM Ryan Hamilton wrote: > Team, > > This email calls for a vote on the project proposed in AIP-38: > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-38+Modern+Web+Application > > Discussion thread: > > > https://lists.apache.org/thread.ht

RE: [DISCUSS] [AIP-37] Virtualenv management inside Airflow

his work if one were using the KubernetesExecutor? Cheers, Constance / CONSTANCE MARTINEAU | Développeuse Principale, Platformes et Exploitation | Tél: 514-847-7992 | comartin...@cdpq.com -Original Message- From: Tomasz Urbaszek Sent: Friday, January 8, 2021 11:48 AM To: dev@airflow.ap