Why limit operator keeps calling next() even after the limit is reached ?

[Abdel Hakim Deneche]

Hey all,

As part of DRILL-991 when LimitRecordBatch receives enough records it calls
kill() on it's upstream to inform the remaining operators and fragments
that they can stop sending batches.

But, limit operator will keep calling next() until it gets a NONE. Is there
a specific reason for this behavior ?

Thanks

-- 

Abdelhakim Deneche

Software Engineer

  


Now Available - Free Hadoop On-Demand Training

Re: Why limit operator keeps calling next() even after the limit is reached ?

[Steven Phillips]

I believe kill() will only stop the upstream fragments from sending
batches, but it does nothing about the batches that have already been sent.
When kill() is called on the RawBatchBuffer, this will release all of the
batches in the queue. But I believe it is still necessary to wait for all
remaining batches to arrive, so they can be cleared. It's possible that
it's not necessary to do this in LimitRecordBatch, and that we are handling
this in the RawBatchBuffer. You would have to examine the code to confirm.

On Thu, Oct 29, 2015 at 11:38 AM, Abdel Hakim Deneche  wrote:

> Hey all,
>
> As part of DRILL-991 when LimitRecordBatch receives enough records it calls
> kill() on it's upstream to inform the remaining operators and fragments
> that they can stop sending batches.
>
> But, limit operator will keep calling next() until it gets a NONE. Is there
> a specific reason for this behavior ?
>
> Thanks
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
>   
>
>
> Now Available - Free Hadoop On-Demand Training
> <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
>

Re: Why limit operator keeps calling next() even after the limit is reached ?

[Abdel Hakim Deneche]

Thanks Steven, I will look into RawBatchBuffer.

On Thu, Oct 29, 2015 at 11:52 AM, Steven Phillips  wrote:

> I believe kill() will only stop the upstream fragments from sending
> batches, but it does nothing about the batches that have already been sent.
> When kill() is called on the RawBatchBuffer, this will release all of the
> batches in the queue. But I believe it is still necessary to wait for all
> remaining batches to arrive, so they can be cleared. It's possible that
> it's not necessary to do this in LimitRecordBatch, and that we are handling
> this in the RawBatchBuffer. You would have to examine the code to confirm.
>
> On Thu, Oct 29, 2015 at 11:38 AM, Abdel Hakim Deneche <
> adene...@maprtech.com
> > wrote:
>
> > Hey all,
> >
> > As part of DRILL-991 when LimitRecordBatch receives enough records it
> calls
> > kill() on it's upstream to inform the remaining operators and fragments
> > that they can stop sending batches.
> >
> > But, limit operator will keep calling next() until it gets a NONE. Is
> there
> > a specific reason for this behavior ?
> >
> > Thanks
> >
> > --
> >
> > Abdelhakim Deneche
> >
> > Software Engineer
> >
> >   
> >
> >
> > Now Available - Free Hadoop On-Demand Training
> > <
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > >
> >
>



-- 

Abdelhakim Deneche

Software Engineer

  


Now Available - Free Hadoop On-Demand Training