+1 to this! I also have a docs section half written on the executor interface and how to extend it. But I've been very busy with a few other items that are completing soon.
Cheers, Niko ________________________________ From: Pankaj Koti <pankaj.k...@astronomer.io.INVALID> Sent: Friday, September 8, 2023 12:03:35 PM To: dev@airflow.apache.org Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Executors docs should be published in Airflow core or providers? CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque. +1 to the proposal. I think core Airflow docs can contain details about the default executor that gets shipped with standalone Airflow installation and a short note about possibilities of using other (providers) executors in production and saying to look for detailed docs in the corresponding provider. Regards, Pankaj Koti *Senior Software Engineer, *OSS Engineering Team. Location: Pune, India Timezone: Indian Standard Time (IST) Email: pankaj.k...@astronomer.io Mobile: +91 9730079985 On Fri, Sep 8, 2023 at 11:28 PM Hussein Awala <huss...@awala.fr> wrote: > Since we moved the executors to the providers packages and made the > executor interface pluggable and extensible, we should move the docs to > their corresponding providers. However, we need to keep a doc in Airflow > core that explains how to use/configure a provider executor (as we have for > the secret managers and the task log handlers) and maybe how to create a > new custom one. > > On Fri, Sep 8, 2023 at 6:15 PM Elad Kalif <elad...@apache.org> wrote: > > > Hello everyone, > > > > This thread is opened due to open issue Migrate Celery/Dask/Kubernetes > > Executor docs to providers < > https://github.com/apache/airflow/issues/33916 > > > > > > > *Background:* > > We had a discussion about extracting Celery, Kubernetes, Dask executors > > from core to providers (discussion thread > > <https://lists.apache.org/thread/kwwhz62lddygodpgr3fk4b9tthtld9do>, vote > > thread <https://lists.apache.org/thread/7gyw7ty9vm0pokjxq7y3b1zw6mrlxfm8 > >) > > > > One of the things we voted on was: > > > > Also, resulting from the discussion we will keep documentation for > > > available executors in Airflow (so they will still be considered as THE > > > executors available and will be discoverable in the same way as today). > > > > > > *The problem:* > > Airflow Core and Providers do not share the same release cycle nor > cadance. > > This means that if we add new capabilities to executors or fix an issue > > which requires both code and doc update - the code will be delivered > > before/ahead of documentation. Both cases are not good. > > > > > > *My proposal:*Now that Celery, Kubernetes, Dask executors are in > providers. > > The section of core-concept > > < > > > https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/index.html > > > > > should > > contain only HIGH level information about the executors. It should not > > contain information about executors internals or how to address common > > problems. This information should be in the provider docs. The high level > > info should be short and on a level that is relevant for all executors > > which means it's not likely to have many changes over time. The > > core-concept should have links/refers for deep dive read to the provider > > docs. This is very similar to what we do with Notifiers > > < > > > https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/notifications.html > > > > > core > > contains high level information and a list of notifers that are linked to > > provider docs. > > > > WDYT? > > > > Also, resulting from the discussion we will keep documentation for > > available executors in Airflow (so they will still be considered as THE > > executors available and will be discoverable in the same way as > today).Moe > > K8S and Celery Executors (and related) to respective providers? > > >
