Thanks Jarek. Yes, I agree there should be no doubt of the necessity. Another purpose I started this thread is to see if any folk has used this *multi_namespace_mode *option in real-world environments; if yes, how the issues I shared earlier were worked around, etc.
The PR should be ready very soon. We can continue the discussion either here or in the PR later. Inputs/suggestions/questions are always appreciated 😉 Regards, XD On Sun, Nov 27, 2022 at 5:20 AM Jarek Potiuk <[email protected]> wrote: > Looks solid. I don't even think it needs more discussion/consensus/voting > - that seems like a useful option to have in KubernetesExecutor and just > discussing details in PR / committer approval would be enough. > > On Mon, Nov 21, 2022 at 8:20 PM Ferruzzi, Dennis > <[email protected]> wrote: > >> On the surface this sounds like a solid improvement to me. I look >> forward to seeing the PR. >> >> >> ------------------------------ >> *From:* Xiaodong Deng <[email protected]> >> *Sent:* Friday, November 18, 2022 11:46 AM >> *To:* [email protected] >> *Subject:* [EXTERNAL] Make KubernetesExecutor's multi_namespace_mode >> more flexible & enterprise-ready >> >> >> *CAUTION*: This email originated from outside of the organization. Do >> not click links or open attachments unless you can confirm the sender and >> know the content is safe. >> >> Hi folks, >> >> I'm writing to share some thoughts and changes we would like to propose, >> in order to make *KubernetesExecutor*'s *multi_namespace_mode* more >> flexible & truly enterprise-ready. >> >> *# Background & Issue* >> If you check *KubernetesExecutor*'s *multi_namespace_mode *configuration >> <https://github.com/apache/airflow/blob/2.4.3/airflow/config_templates/default_airflow.cfg#L1186-L1187>, >> you will notice the statement "*Will require creating a cluster-role for >> the scheduler*". That's because under the hood KubernetesExecutor is >> using *list_pod_for_all_namespaces() >> <https://github.com/apache/airflow/blob/2.4.3/airflow/executors/kubernetes_executor.py#L140-L142>* >> when it is *multi_namespace_mode*. This is a cluster-wise query and >> would surely need a cluster role. >> >> This would become a problem in some of the enterprise environments. For >> example: >> - The Kubernetes clusters may be managed by a central team. Each user >> only gets full access to namespace(s), and is never allowed to have a >> cluster-role. >> - A Kuberntes cluster may be shared by multiple teams, and it's never a >> good idea to grant anyone cluster-role. >> >> This does block some users from using Airflow's *KubernetesExecutor* in >> the vanilla shape, and these users have to make certain customisations >> (like what our team has to do). >> >> *# Proposal* >> We would like to propose refactoring KubernetesExecutor so that users >> can provide a list of namespaces in the configuration, and then >> Scheduler/Executor only needs to have a namespace-role to these namespaces. >> >> We already have a draft ready to be shared really soon. We would like to >> hear feedback & thoughts from the community about this idea, before we >> proceed to create a PR. >> >> Looking forward to your thoughts, and wish everyone a good weekend (and >> for some folks a nice Thanksgiving break) ahead! >> >> >> Regards, >> XD >> >>
