Note that on master (but not in the latest release), you can already specify resource constraints for tasks. They are not consumed anywhere in airflow itself yet, but you can use them in an operator if it fits your use case: https://github.com/apache/incubator-airflow/pull/1669
On Fri, Aug 5, 2016 at 4:07 PM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > We're adding cgroups semantics to Airflow's BaseOperator, along with > integration to activate those settings when firing up the task instance as > a subprocess. Paul Yang (@Airbnb) is working on a prototype / design doc > and should be able to share more shortly. > > It lines up nicely with the recent work around unix impersonation as they > are all new optional BaseOperator params that wraps the subprocess. There > features will require Airflow to run with elevated privileges, which > Airflow will "downstep" from as it runs the tasks instances as > subprocesses. > > Max > > On Fri, Aug 5, 2016 at 11:03 AM, Lance Norskog <lance.nors...@gmail.com> > wrote: > > > Linux cgroups gives Docker the ability to control memory use inside a > > container. > > > > https://www.cloudsigma.com/manage-docker-resources-with-cgroups/ > > > > Here is an example that allocates the CPU: > > http://blog.viktorpetersson.com/post/115562026784/using- > > cgroups-with-docker-on-ubuntu-1404 > > > > (I have not worked with cgroups.) > > > > On Fri, Aug 5, 2016 at 7:57 AM, wood stock <ws1...@gmail.com> wrote: > > > > > I think you can define a new operator which take an existing operator > > base > > > class and keep monitoring the memory usage (maybe base on airflow > > > heartbeat?). > > > > > > On Thu, Aug 4, 2016 at 8:23 PM, Adinata <mail.die...@gmail.com> wrote: > > > > > > > Is there any way for task instances to limit the memory task > execution? > > > > > > > > I run the worker inside docker container, hence when it uses high > > memory, > > > > the container was killed. > > > > Airflow then detect it as zombie (well the email notification failure > > > told > > > > me that). It would be great if > > > > I know it was killed because of oom, so I know what to fix after > > instead > > > of > > > > wondering first why it was > > > > becoming a zombie. > > > > > > > > Thanks > > > > -- > > > > *Adinata* > > > > Engineer - UrbanIndo.com > > > > > > > > > > > > > > > -- > > Lance Norskog > > lance.nors...@gmail.com > > Redwood City, CA > > >
