Re: Airflow-GCP for Google Container Engine

Hey Alex, with resource management I meant the Pod CPU and Memory limits which can be set in k8s deployments. By default the pods will use all available resources on the node it's on. In theory this could mean a pod ends up by itself on 1 node when it

Re: Airflow-GCP for Google Container Engine

Hey Koen, I tried running it with 2 but one is for my use-case good enough. I don't exactly know what you "in-this-context" mean by managing resources, but on GCloud you want to delegate as much as possible to the services (DataProc/DataFlow/BigQuery) and soon also other docker containers in the

Re: Airflow-GCP for Google Container Engine

Nice one, thanks! I tried this a while ago, and got stuck on resource issues - probably because I capped the pods max resources to hard. I see in the worker yaml it creates 1 worker replica, are you running it with multiple workers as well? Are you managing any resources on the pods, or you let

Re: Airflow-GCP for Google Container Engine

Nice, thanks! :) On Fri, Dec 2, 2016 at 4:55 AM, Alex Van Boxel wrote: > Hi all, > > I think I pulled it off to have kind of "one-execute" install of Airflow on > Google Container Engine. Personally I find this my preferred setup (because > I can have a production environment