I think you can define a new operator which take an existing operator base class and keep monitoring the memory usage (maybe base on airflow heartbeat?).
This is the easiest, I think On Thu, Aug 4, 2016 at 8:23 PM, Adinata <mail.die...@gmail.com> wrote: > Is there any way for task instances to limit the memory task execution? > > I run the worker inside docker container, hence when it uses high memory, > the container was killed. > Airflow then detect it as zombie (well the email notification failure told > me that). It would be great if > I know it was killed because of oom, so I know what to fix after instead of > wondering first why it was > becoming a zombie. > > Thanks > -- > *Adinata* > Engineer - UrbanIndo.com >
