I second Jed's proposal and believe that the implications and benefits, as laid out, make a lot of sense.
> we wouldn't have to set a minimum version for these 2 providers to ensure we > have the version with the executors, allowing an "old" k8s provider (KPO > itself is pretty common after all) to be used if you aren't using > KubernetesExecutor. This is great point. There are indeed many users who fall into this category, and not compelling them to upgrade their k8s provider is a more user-friendly approach. To mitigate issue of users not having a new enough provider, one that has their executor of choice, I agree that error messaging would be helpful. In addition to this, we should consider a UI status that clearly outlines the need to upgrade the provider to use the configured executor. Overall, the proposal helps the Airflow community adhere to the previous policies without significantly degrading the user experience. Shubham On 2023-07-21, 9:19 AM, "Jed Cunningham" <[email protected] <mailto:[email protected]>> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Hello everyone, After thinking through whether the dask provider should be pre-installed or not, it got me thinking about whether we should even be pre-installing the celery and k8s providers. Core Airflow so far hasn't had the dependencies required to make those executors functional either - users either had to use the extra or install the provider directly. So that doesn't really change if we choose not to preinstall the providers. Why would it be beneficial to not pre-install them? The most obvious benefit is you don't get a handful of new required deps you aren't using (like celery/flower/billiard/etc if you don't use CeleryExecutor). However another benefit is we wouldn't have to set a minimum version for these 2 providers to ensure we have the version with the executors, allowing an "old" k8s provider (KPO itself is pretty common after all) to be used if you aren't using KubernetesExecutor. The one downside here is how do we make sure users have a new enough provider, one that has their executor of choice in it. After chatting with Jarek, he brought up that we could also make the error messaging more helpful in this situation. This messaging would basically solve the problem without needing minimum version pinning in core. All that said, I think we should strongly consider not pre-installing the celery and k8s providers as well. Thoughts? Jed --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
