raphaelauv commented on code in PR #32310: URL: https://github.com/apache/airflow/pull/32310#discussion_r1251739035
########## docs/apache-airflow/core-concepts/executor/local.rst: ########## @@ -47,3 +47,12 @@ Arguably, :class:`~airflow.executors.sequential_executor.SequentialExecutor` cou parallelism of just 1 worker, i.e. ``self.parallelism = 1``. This option could lead to the unification of the executor implementations, running locally, into just one :class:`~airflow.executors.local_executor.LocalExecutor` with multiple modes. + +.. note:: + + When multiple Schedulers are configured with ``executor = LocalExecutor`` in the ``[core]`` section of your ``airflow.cfg``, each scheduler will run a LocalExecutor. This means tasks would be processed in a distributed fashion across the machines running the Schedulers. + + Some considerations should be taken into account: + + - Restarting a Scheduler: If a Scheduler is restarted, it may take some time for other Schedulers to recognize the orphaned tasks and restart or fail them. + - Sharing of instances: While a single LocalExecutor allows tasks to share the same Python process and exchange information via variables, this is not possible when using multiple Schedulers, as each Scheduler will run in its own separate Python process. Review Comment: what do you mean by `Sharing of instances` ? also I don't understand what you mean with `exchange information via variables` because only xcom is an airflow way to exchange data between tasks , everything else ( file system , mmap , external service ) are not manage by airflow -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
