The *current* correct behavior is to return STOP downstream. That is because the ownership of batches created by that operator's allocator may have been transferred downstream, and if we try to cleanup right away, it will result in apparent memory leaks.
Like Jacques mentioned, I am working on a patch that will change this behavior, and the reason we will be able to clean up right away is because there is work in progress to move to a model where ownership of buffers is transferred as well. On Mon, May 4, 2015 at 10:11 AM, Jacques Nadeau <jacq...@apache.org> wrote: > In general, an operator should release its resources as soon as it can. I > believe Steven is working on a patch to make all resource intensive > operators do just this. > > On Mon, May 4, 2015 at 9:47 AM, Abdel Hakim Deneche <adene...@maprtech.com > > > wrote: > > > What is the correct behavior for an operator when it receives an > > IterOutcome.STOP from upstream: > > 1. should it just stop right away and return STOP downstream ?, or > > 2. should it cleanup it's resources before returning STOP downstream ? > > > > Some operators follow 1 and others (like OrderedPartitionRecordBatch) > > follow 2. > > > > -- > > > > Abdelhakim Deneche > > > > Software Engineer > > > > <http://www.mapr.com/> > > > > > > Now Available - Free Hadoop On-Demand Training > > < > > > http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available > > > > > > -- Steven Phillips Software Engineer mapr.com
