Hey Koen, I tried running it with 2 but one is for my use-case good enough. I don't exactly know what you "in-this-context" mean by managing resources, but on GCloud you want to delegate as much as possible to the services (DataProc/DataFlow/BigQuery) and soon also other docker containers in the same cluster, so I like to keep the workers as light as possible. But good point about the max (I should make it configurable).
My cluster for now is 2x4CPU for my Kubernetes cluster, but it has other stuff running aside from Airflow. On Mon, Dec 5, 2016 at 12:42 PM Koen Mevissen <kmevis...@travix.com> wrote: > Nice one, thanks! I tried this a while ago, and got stuck on resource > issues - probably because I capped the pods max resources to hard. > > I see in the worker yaml it creates 1 worker replica, are you running it > with multiple workers as well? Are you managing any resources on the pods, > or you let the workers consume whatever's available on the nodes? > > > On Fri, Dec 2, 2016 at 10:14 PM, Chris Riccomini <criccom...@apache.org> > wrote: > > > Nice, thanks! :) > > > > On Fri, Dec 2, 2016 at 4:55 AM, Alex Van Boxel <a...@vanboxel.be> wrote: > > > > > Hi all, > > > > > > I think I pulled it off to have kind of "one-execute" install of > Airflow > > on > > > Google Container Engine. Personally I find this my preferred setup > > (because > > > I can have a production environment and staging environment). It's > what I > > > use for our production/staging setup. I think someone else could pull > it > > > off with everything in this repo: > > > > > > https://github.com/alexvanboxel/airflow-gcp-k8s > > > > > > It has a README that should be enough to get it setup. > > > > > > Now I'll work on my sync'er so my dags will refresh as soon as I push > to > > > git :-) > > > > > > > > > -- > Kind regards, > Met vriendelijke groet, > > *Koen Mevissen* > Principal BI Developer > > > *Travix Nederland B.V.* > Piet Heinkade 55 > 1019 GM Amsterdam > The Netherlands > > T. +31 (0)20 203 3241 <+31%2020%20203%203241> > E: kmevis...@travix.com > www.travix.com > > *Brands: * CheapTickets | Vliegwinkel | Vayama | BudgetAir | > Flugladen >
