Linux cgroups gives Docker the ability to control memory use inside a container.
https://www.cloudsigma.com/manage-docker-resources-with-cgroups/ Here is an example that allocates the CPU: http://blog.viktorpetersson.com/post/115562026784/using-cgroups-with-docker-on-ubuntu-1404 (I have not worked with cgroups.) On Fri, Aug 5, 2016 at 7:57 AM, wood stock <[email protected]> wrote: > I think you can define a new operator which take an existing operator base > class and keep monitoring the memory usage (maybe base on airflow > heartbeat?). > > On Thu, Aug 4, 2016 at 8:23 PM, Adinata <[email protected]> wrote: > > > Is there any way for task instances to limit the memory task execution? > > > > I run the worker inside docker container, hence when it uses high memory, > > the container was killed. > > Airflow then detect it as zombie (well the email notification failure > told > > me that). It would be great if > > I know it was killed because of oom, so I know what to fix after instead > of > > wondering first why it was > > becoming a zombie. > > > > Thanks > > -- > > *Adinata* > > Engineer - UrbanIndo.com > > > -- Lance Norskog [email protected] Redwood City, CA
