Barni, Thank you so much for sharing this! I'm admittedly far from a Kubernetes guru, but I'm just trying to wrap my head around the reasons why we'd need a custom Kubernetes controller to manage Airflow's components, as opposed to the setup here https://github.com/mumoshu/kube-airflow wherein we simply deploy Airflow to Kubernetes using standard Kubernetes Services and Deployments. I assume you're doing this because, as you mention in the "Design" section of your repo, "In case of some stateful applications, the declarative models provided by kubernetes are not sufficient to handle fault remediation, scaling with data integrity and availability". Can you be more specific about what kinds of faults / scaling issues your Kubernetes Controller might handle for Airflow that otherwise would not be handled by built-in Kubernetes controllers (Services/Deployments)?
Thanks, Rob On Mon, Aug 6, 2018 at 1:55 AM Bolke de Bruin <bdbr...@gmail.com> wrote: > Really awesome stuff. We are in progress to move over to k8s for Airflow > (on prem though) and this is really helpful. > > B. > > Verstuurd vanaf mijn iPad > > > Op 3 aug. 2018 om 23:35 heeft Barni Seetharaman <ba...@google.com.INVALID> > het volgende geschreven: > > > > Hi > > > > We at Google just open-sourced a Kubernetes custom controller (also > called > > operator) to make deploying and managing Airflow on kubernetes simple. > > The operator pattern is a power abstraction in kubernetes. > > Please watch this repo (in the process of adding docs) for further > updates. > > > > https://github.com/GoogleCloudPlatform/airflow-operator > > > > Do reach out if you have any questions. > > > > Also created a channel in kubernetes slack (#airflow-operator) > > <https://kubernetes.slack.com/messages/CC1UAMYSV/details/> for any > > discussions specific to Airflow on Kubernetes (including Daniel's > > Kubernetes Executor, Kuberenetes operator and this custom controller also > > called kuberntes airflow operator). > > > > regs > > Barni >