Re: [DISCUSS] Shaping the future of executors for Airflow (slowly phasing out Celery ?)

> We could even likely think about adding more options of similar kind for GCP/AWS/Azure - using native capabilities of those platforms rather than using generic "Kubernetes" as remote execution. I can imagine using Fargate (AWS team could contribute it ), Cloud Run (Google team), Azure Container

Re: [DISCUSS] Shaping the future of executors for Airflow (slowly phasing out Celery ?)

Just to throw I my 2 cents here, one huge benefit of the celery executor, and a large reason I use Airflow, is that it allows you to make use of multiple queues. We are in the process of transitioning workflows from on prem to cloud and the celery executor allows me to easily send tasks that

Re: [DISCUSS] AIP-1 and Airflow multi-tenancy

Just a reminder -> multi-tenancy meeting tomorrow. Few people worked on what will be presented tomorrow, and I am super excited we will be able to kick that one off - it has been a long time on my waiting list :) J. On Sat, Nov 20, 2021 at 10:14 AM Jarek Potiuk wrote: > > The meeting is set for

Re: [DISCUSS] Connection extra field widgets: long vs short name convention

> I don’t think what what being proposed by Daniel was to include both short > and long names in the UI fields, but just in the get_conn method, right? Hmm. That **might** work indeed. It will not be too helpful in having both forms possible and making long form "default" for UI, "short" default

Re: Lazy consensus: simplify XCom methods and remove broken "future" dates idea.

Draft PR here (I haven't finished writing/adding all the tests needed for that) On Thu, Nov 25 2021 at 14:11:38 +0100, Jarek Potiuk wrote: I can even actively agree. not even lazy consent to that one Ash :). Sounds like a super-obscure feature

Re: Lazy consensus: simplify XCom methods and remove broken "future" dates idea.

I can even actively agree. not even lazy consent to that one Ash :). Sounds like a super-obscure feature that likely no-one knows exists (and it does not work anyway). On Thu, Nov 25, 2021 at 11:36 AM Ash Berlin-Taylor wrote: > > Hi All, > > As part of finishing off AIP-39 (Timetables) where we

Re: [DISCUSS] Shaping the future of executors for Airflow (slowly phasing out Celery ?)

Very Good comments Ash ! Food for thought indeed - indeed LocalExecutor for multi-tenant is no-go (thought about it too :). I agree there are different cases and I agree totally that Celery will stay there for a looong time (maybe forever). Maybe the "phasing out" is too strong of a statement (I

Re: [DISCUSS] Shaping the future of executors for Airflow (slowly phasing out Celery ?)

Hi Jarek, Trigger does support multiple instances already. Deferrable tasks still need a normal task slot on a worker to start off and then defer to a trigger right now as well. While I have no love for Celery (or how we mis-use it in Airflow more accurately), and I agree that we aren't

[DISCUSS] Shaping the future of executors for Airflow (slowly phasing out Celery ?)

Hello Everyone, I recently had some discussions and thought about some new features implemented already and planned and in-progress work, and I had a thought - that maybe worth discussing here. It's very likely many of the people involved had similar discussion and thoughts, but maybe it's worth

Lazy consensus: simplify XCom methods and remove broken "future" dates idea.

Hi All, As part of finishing off AIP-39 (Timetables) where we re-keyed TaskInstance from (dag_id, task_id, exeuction_date) to (dag_id, task_id, run_id) I am looking at doing the same for XCom and ran into a few snags. Currently the docs for ti.xcom_push[1] say this: *execution_date*

Lazy consensus: simplify XCom methods and remove broken "future" dates idea

Hi All, As part of finishing off AIP-39 (Timetables) where we re-keyed TaskInstance from (dag_id, task_id, exeuction_date) to (dag_id, task_id, run_id) I am looking at doing the same for XCom and ran into a few snags. Currently the docs for ti.xcom_push[1] say this: *execution_date*