RE: [DISCUSS] AIP-69 Remote Executor

Hi Airflow Dev's, After todays Airflow 3 planning and the "sneak preview" from Ash' AIP-72 I took some time today to update the AIP-69 Remote Executor with more technical details as I tried a PoC implementation. The AIP document is now updated and I call a second round for feedback /

Re: [DISCUSS] Restore the SQL server backend

I haven’t looked that much into the Airflow code yet, but the backend code must already be pluggable since it supports multiple databases. What if we were to maintain a separate repo and artifact for just an extension for the SQL Server backend? This seems like it’d be low-impact on the Airflow

Re: Call with Nielsen team demoing their DAG debugging feature

+1 for the recording please. On Thu, Jun 13, 2024, 1:26 PM Jarek Potiuk wrote: > Just summarizing the call: > > * we had demo from Nielsen showing their debugging feature > * It's based on environments created in the research environment where > users can run DAGs and debug individual tasks -

Re: Call with Nielsen team demoing their DAG debugging feature

Just summarizing the call: * we had demo from Nielsen showing their debugging feature * It's based on environments created in the research environment where users can run DAGs and debug individual tasks - basically replaying and debugging the tasks based on existing DAG runs but without saving

Re: [DISCUSS] Restore the SQL server backend

Hi Jarek, Thanks for your response. I would prefer to make this work part of the main Airflow repository. My previous experience with maintaining forks of Apache projects has always been that it fragments the community unnecessarily. Users of a company-specific fork might ask for features

Re: [DISCUSS] Restore the SQL server backend

Yep what Kaxil said and a separate repo does not change anything. Anything that we (Apache Airflow PMC) releases is something that we (Apache Airflow PMC maintain, contribute to and only committers can approve and merge the code) - so no matter if it is a separate repo, the cost is the same .

Re: [DISCUSS] Restore the SQL server backend

The cost of maintaining it in Airflow repo (with CI/CI, GH issues etc) is, unfortunately, just too much higher. On Thu, 13 Jun 2024 at 17:11, James Duong wrote: > Hi Jarek, > > Thanks for your response. > > I would prefer to make this work part of the main Airflow repository. > > My previous

Re: [DISCUSS] common.compat provider (WAS: Common.util provider?)

> The only known problem with that idea is that the common code has to live "forever" - as long as someone can use the older providers (or older Airflow version). Here Jakub is right - it's only 'older providers" not "older airlfow" - the provider will have "apache-airflow>=2.7.0" so older

Proposal for Enhanced Data Awareness in Airflow

Hi Airflow Dev Community! I am excited to share a new proposal written by TP and I titled "Enhanced Data Awareness in Airflow " that I believe will significantly advance our capabilities

Re: [Airflow 3] The Airflow actors

Awesome, thanks Elad On Thu, 13 Jun 2024 at 07:50, Jarek Potiuk wrote: > Nice one. > > On Thu, Jun 13, 2024 at 8:36 AM Elad Kalif wrote: > > > Hello everyone, > > > > As discussed in the last Airflow meeting I published a doc about the > > Airflow actors > >

[REMINDER] Airflow 3 Dev call today

Hello all, Just a reminder that we will have our dev call today at 4 PM BST (3 PM GMT/UTC | 11 AM EST | 8 AM PST). *Proposed Agenda*: 1) Check in on the action items from the last call 2) Discussion: How do we collaborate and discuss efficiently (e.g. Email, cwiki, Google docs)? 3) Discussion:

Re: [Airflow 3] The Airflow actors

Nice one. On Thu, Jun 13, 2024 at 8:36 AM Elad Kalif wrote: > Hello everyone, > > As discussed in the last Airflow meeting I published a doc about the > Airflow actors > https://cwiki.apache.org/confluence/x/1gmTEg > > > Thanks, > Elad >

[Airflow 3] The Airflow actors

Hello everyone, As discussed in the last Airflow meeting I published a doc about the Airflow actors https://cwiki.apache.org/confluence/x/1gmTEg Thanks, Elad