I have a bit of a different point of view than Kamil - I think once we are authenticated, the Kubernetes API is pretty standard - so having hooks for different services that provide the same interface for the operators might make sense. Such hooks could authenticate in a "cloud-specific way" using the connection provided, but then after authentication it should be "generic".
I like the idea. J. On Sat, Jan 25, 2020 at 9:22 PM Kamil Breguła <[email protected]> wrote: > Hello, > > The issue of connection configuration for individual providers has not > been standardized in any way by the Kubernetes community. I'm afraid > we won't create a new solution. Some groups are trying to create their > solution[1][2], but for now it is a mess.[3] Google employees use > gcloud to obtain cluster credentials.[2] > > I am not sure if we want to create generic operators because they can > be very difficult to use in practice. The operator who will be > designed for a specific purpose will be easier to use and will work > better because we can improve it. Very generic solutions are very > difficult to improve and refactor. > > Best regards, > Kamil > > [1] > https://github.com/kubernetes/community/blob/master/contributors/design-proposals/auth/kubectl-exec-plugins.md > [2] > https://github.com/kubernetes-client/python-base/blob/a2d1024524de78b62e5f9aa72d34cb9ea9de2b97/config/exec_provider.py > [3] > https://github.com/kubernetes-client/python-base/blob/a2d1024524de78b62e5f9aa72d34cb9ea9de2b97/config/kube_config.py#L219-L224 > [4] > https://github.com/apache/airflow/pull/3532/files#diff-3cfc2b387652665d77ae50581081560eR266-R270 > > On Sat, Jan 25, 2020 at 7:51 PM Roi Teveth <[email protected]> wrote: > > > > hi all, > > I'm working at my company on: spark on Kubernetes POC during the work > I've > > built an operator for spark on Kubernetes and trying to contribute it to > > airflow(https://github.com/apache/airflow/pull/7163) in the process I > > started thinking about: > > 1. building hooks for managed Kubernetes engines on Amazon, GCP and > > others(EKS, GKE...) using their own connection, for example, using AWS > > connection to get kubeconfig then Kubernetes API client. > > 2. building more generalized Kubernetes operators: alongside > > sparkK8sOperator that send SparkApplication CRD to Kubernetes cluster, I > > can build kubernetesCrdOperator that will create any kind of CRD. and > > sensor that will be given which field in the CRD to check and the fail or > > success keywords. with the same principle, I can build > > kubernetesJobOperator that will create and sense Kubernetes Job(although > > it's very close to what Kubernetes Pod operator does) > > > > can you share your thoughts about it? and if it will be useful for > Airflow > > Thank you > > Roi Teveth > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
