Re: [DISCUSSION] Move K8S and Celery Executors (and related) to respective providers?

I would love to see this. I think it would legitimize the interface a bit more and also help to encourage folks to not abuse/leak it in the future. AIP-51 is close to completion, I'd say 80%. We've boiled off most of the easier items and what's left is a few tricky decouplings (I have a PR for

Re: Thoughts on Adding Weaviate Provider?

J, Not discouraging at all. I reached out in search of clarity that I now have. Thanks for the detailed response. I think all of this makes a ton of sense. Upon first sight of how the provider ecosystem had grown, I was shocked. I will talk with the Weaviate team and community about Airflow's

[VOTE] Release Airflow Go Client 2.5.0 from 2.5.0rc1

Hey fellow Airflowers, I have cut the first release candidate for the Airflow Go Client 2.5.0rc1. The client consists of APIs corresponding to REST APIs available in *Apache Airflow 2.5.0*. This email is calling for a vote on the release, which will last for 72 hours. Consider this my

Re: Thoughts on Adding Weaviate Provider?

TL;DR; Generally we are very cautious about adding new "external service" providers to the community and it's highly unlikely we would want Weaviate in, but you are absolutely free (and encouraged) to release the provider on your own. Even if there is an open-source engine (like yours), when there

Thoughts on Adding Weaviate Provider?

Hi Devs, In keeping with the open source ethos and the need for DAG workflows in Neural Search pipelines, I welcome feedback on the idea of adding a Weaviate provider to Airflow. It's the best open source neural search engine. I see the need for a test

[VOTE] Release Airflow Python Client 2.5.1 from 2.5.1rc1

Hey fellow Airflowers, I have cut the first release candidate for the Airflow Python Client 2.5.1rc1. The client consists of APIs corresponding to REST APIs available in *Apache Airflow 2.5.0*. This email is calling for a vote on the release, which will last for 72 hours. Consider this my

Re: [Discussion] Deprecate auto cleanup RenderedTaskInstanceFields and decouple k8s_pod_yaml

Yep. Agree this is not an efficient query and dynamic task mapping makes the effect much worse. Generally speaking, selecting "what should be left" and then deleting stuff where the key is "not in" is never an efficient way of running an sql query. And the query not using index at all makes it

[DISCUSSION] Move K8S and Celery Executors (and related) to respective providers?

Hello Everyone, As a follow-up to AIP-51 - when it is completed (with few more quirks like the one described by Andrey in the "Rendered Task Instance Fields" discussion) - it should now be entirely possible to move Kubernetes Executor, Celery Executor and related "joined" executors to respective