We're adding cgroups semantics to Airflow's BaseOperator, along with integration to activate those settings when firing up the task instance as a subprocess. Paul Yang (@Airbnb) is working on a prototype / design doc and should be able to share more shortly.
It lines up nicely with the recent work around unix impersonation as they are all new optional BaseOperator params that wraps the subprocess. There features will require Airflow to run with elevated privileges, which Airflow will "downstep" from as it runs the tasks instances as subprocesses. Max On Fri, Aug 5, 2016 at 11:03 AM, Lance Norskog <[email protected]> wrote: > Linux cgroups gives Docker the ability to control memory use inside a > container. > > https://www.cloudsigma.com/manage-docker-resources-with-cgroups/ > > Here is an example that allocates the CPU: > http://blog.viktorpetersson.com/post/115562026784/using- > cgroups-with-docker-on-ubuntu-1404 > > (I have not worked with cgroups.) > > On Fri, Aug 5, 2016 at 7:57 AM, wood stock <[email protected]> wrote: > > > I think you can define a new operator which take an existing operator > base > > class and keep monitoring the memory usage (maybe base on airflow > > heartbeat?). > > > > On Thu, Aug 4, 2016 at 8:23 PM, Adinata <[email protected]> wrote: > > > > > Is there any way for task instances to limit the memory task execution? > > > > > > I run the worker inside docker container, hence when it uses high > memory, > > > the container was killed. > > > Airflow then detect it as zombie (well the email notification failure > > told > > > me that). It would be great if > > > I know it was killed because of oom, so I know what to fix after > instead > > of > > > wondering first why it was > > > becoming a zombie. > > > > > > Thanks > > > -- > > > *Adinata* > > > Engineer - UrbanIndo.com > > > > > > > > > -- > Lance Norskog > [email protected] > Redwood City, CA >
