Re: Out Of Memory Error (Possible Regression)

2015-12-30 Thread Abdel Hakim Deneche 
That must be it.

On Wed, Dec 30, 2015 at 1:22 PM, Steven Phillips  wrote:

> No, we are running on 4 4-core machines.
>
> On Wed, Dec 30, 2015 at 2:10 PM, Abdel Hakim Deneche <
> adene...@maprtech.com>
> wrote:
>
> > are you running the tests on 32 core machines ? a different number of
> cores
> > affects how much memory is available for the sort
> >
> > On Wed, Dec 30, 2015 at 1:02 PM, Abdel Hakim Deneche <
> > adene...@maprtech.com>
> > wrote:
> >
> > > The following tests are failing:
> > >
> > >
> > >>
> >
> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q163_DRILL-2046.q
> > >>
> > >>
> >
> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q177_DRILL-2046.q
> > >> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q174.q
> > >> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/
> > >> /Functional/window_functions/multiple_partitions/q35.sql
> > >>
> > >>
> >
> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q160_DRILL-1985.q
> > >>
> > >>
> >
> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q162_DRILL-1985.q
> > >> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q165.q
> > >> /Functional/window_functions/multiple_partitions/q37.sql
> > >> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q171.q
> > >>
> > >>
> >
> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q168_DRILL-2046.q
> > >> /Functional/window_functions/multiple_partitions/q36.sql
> > >>
> > >>
> >
> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q159_DRILL-2046.q
> > >> /Functional/window_functions/multiple_partitions/q30.sql
> > >>
> > >>
> >
> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/large/q157_DRILL-1985.q
> > >> /Functional/window_functions/multiple_partitions/q22.sql
> > >
> > >
> > > With one of the following errors:
> > >
> > > java.sql.SQLException: RESOURCE ERROR: One or more nodes ran out of
> > memory
> > >> while executing the query.
> > >> Caused by: org.apache.drill.exec.exception.OutOfMemoryException:
> > >> org.apache.drill.exec.exception.OutOfMemoryException: Unable to
> allocate
> > >> sv2, and not enough batchGroups to spill
> > >> at
> > >>
> >
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:356)
> > >> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> > >
> > >
> > > or
> > >
> > > java.sql.SQLException: SYSTEM ERROR: DrillRuntimeException: Failed to
> > >> pre-allocate memory for SV. Existing recordCount*4 = 0, incoming batch
> > >> recordCount*4 = 3340
> > >> Caused by: org.apache.drill.common.exceptions.DrillRuntimeException:
> > >> Failed to pre-allocate memory for SV. Existing recordCount*4 = 0,
> > incoming
> > >> batch recordCount*4 = 3340
> > >> at
> > >>
> >
> org.apache.drill.exec.physical.impl.sort.SortRecordBatchBuilder.add(SortRecordBatchBuilder.java:116)
> > >> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> > >> at
> > >>
> >
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:451)
> > >> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> > >
> > >
> > >
> > >
> > > On Wed, Dec 30, 2015 at 12:42 PM, Jacques Nadeau 
> > > wrote:
> > >
> > >> I'll let Steven answer your question directly.
> > >>
> > >> FYI, we are running a regression suite that was forked from the MapR
> > repo
> > >> a
> > >> month or so ago because we had to fix a bunch of things to make it
> work
> > >> with Apache Hadoop. (There was a thread about this back then and we
> > >> haven't
> > >> yet figured out how to merge both suites.) It is possible that he had
> a
> > >> successful run but the failures are happening on items that you've
> > >> recently
> > >> added to your suite.
> > >>
> > >> It is also possible (likely?) that the configuration settings for our
> > >> regression clusters are not the same.
> > >>
> > >> --
> > >> Jacques Nadeau
> > >> CTO and Co-Founder, Dremio
> > >>
> > >> On Wed, Dec 30, 2015 at 12:37 PM, Abdel Hakim Deneche <
> > >> adene...@maprtech.com
> > >> > wrote:
> > >>
> > >> > Steven,
> > >> >
> > >> > were you able to successfully run the regression tests on the
> transfer
> > >> > patch ? I just tried and saw several queries running out of memory !
> > >> >
> > >> > On Wed, Dec 30, 2015 at 11:46 AM, Abdel Hakim Deneche <
> > >> > adene...@maprtech.com
> > >> > > wrote:
> > >> >
> > >> > > Created DRILL-4236 <
> > https://issues.apache.org/jira/browse/DRILL-4236>
> > >> to
> > >> > > keep track of this improvement.
> > >> > >
> > >> > > On Wed, Dec 30, 2015 at 11:01 AM, Jacques Nadeau <
> > jacq...@dremio.com>
> > >> > > wrote:
> > >> > >
> > >> > >> Since the accounting changed (more accurate), the termination
> > >> condition
> > >> > >> for
> > >> > >> the sort operator will be different than before. In fact, this
> > likely
> > >> > will
> > >> > >> be sooner since our accounting is much larger than previously
> > (since
> > >> we
> > >> > >> correctly

Re: Out Of Memory Error (Possible Regression)

2015-12-30 Thread Steven Phillips 
No, we are running on 4 4-core machines.

On Wed, Dec 30, 2015 at 2:10 PM, Abdel Hakim Deneche 
wrote:

> are you running the tests on 32 core machines ? a different number of cores
> affects how much memory is available for the sort
>
> On Wed, Dec 30, 2015 at 1:02 PM, Abdel Hakim Deneche <
> adene...@maprtech.com>
> wrote:
>
> > The following tests are failing:
> >
> >
> >>
> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q163_DRILL-2046.q
> >>
> >>
> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q177_DRILL-2046.q
> >> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q174.q
> >> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/
> >> /Functional/window_functions/multiple_partitions/q35.sql
> >>
> >>
> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q160_DRILL-1985.q
> >>
> >>
> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q162_DRILL-1985.q
> >> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q165.q
> >> /Functional/window_functions/multiple_partitions/q37.sql
> >> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q171.q
> >>
> >>
> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q168_DRILL-2046.q
> >> /Functional/window_functions/multiple_partitions/q36.sql
> >>
> >>
> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q159_DRILL-2046.q
> >> /Functional/window_functions/multiple_partitions/q30.sql
> >>
> >>
> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/large/q157_DRILL-1985.q
> >> /Functional/window_functions/multiple_partitions/q22.sql
> >
> >
> > With one of the following errors:
> >
> > java.sql.SQLException: RESOURCE ERROR: One or more nodes ran out of
> memory
> >> while executing the query.
> >> Caused by: org.apache.drill.exec.exception.OutOfMemoryException:
> >> org.apache.drill.exec.exception.OutOfMemoryException: Unable to allocate
> >> sv2, and not enough batchGroups to spill
> >> at
> >>
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:356)
> >> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >
> >
> > or
> >
> > java.sql.SQLException: SYSTEM ERROR: DrillRuntimeException: Failed to
> >> pre-allocate memory for SV. Existing recordCount*4 = 0, incoming batch
> >> recordCount*4 = 3340
> >> Caused by: org.apache.drill.common.exceptions.DrillRuntimeException:
> >> Failed to pre-allocate memory for SV. Existing recordCount*4 = 0,
> incoming
> >> batch recordCount*4 = 3340
> >> at
> >>
> org.apache.drill.exec.physical.impl.sort.SortRecordBatchBuilder.add(SortRecordBatchBuilder.java:116)
> >> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >> at
> >>
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:451)
> >> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >
> >
> >
> >
> > On Wed, Dec 30, 2015 at 12:42 PM, Jacques Nadeau 
> > wrote:
> >
> >> I'll let Steven answer your question directly.
> >>
> >> FYI, we are running a regression suite that was forked from the MapR
> repo
> >> a
> >> month or so ago because we had to fix a bunch of things to make it work
> >> with Apache Hadoop. (There was a thread about this back then and we
> >> haven't
> >> yet figured out how to merge both suites.) It is possible that he had a
> >> successful run but the failures are happening on items that you've
> >> recently
> >> added to your suite.
> >>
> >> It is also possible (likely?) that the configuration settings for our
> >> regression clusters are not the same.
> >>
> >> --
> >> Jacques Nadeau
> >> CTO and Co-Founder, Dremio
> >>
> >> On Wed, Dec 30, 2015 at 12:37 PM, Abdel Hakim Deneche <
> >> adene...@maprtech.com
> >> > wrote:
> >>
> >> > Steven,
> >> >
> >> > were you able to successfully run the regression tests on the transfer
> >> > patch ? I just tried and saw several queries running out of memory !
> >> >
> >> > On Wed, Dec 30, 2015 at 11:46 AM, Abdel Hakim Deneche <
> >> > adene...@maprtech.com
> >> > > wrote:
> >> >
> >> > > Created DRILL-4236 <
> https://issues.apache.org/jira/browse/DRILL-4236>
> >> to
> >> > > keep track of this improvement.
> >> > >
> >> > > On Wed, Dec 30, 2015 at 11:01 AM, Jacques Nadeau <
> jacq...@dremio.com>
> >> > > wrote:
> >> > >
> >> > >> Since the accounting changed (more accurate), the termination
> >> condition
> >> > >> for
> >> > >> the sort operator will be different than before. In fact, this
> likely
> >> > will
> >> > >> be sooner since our accounting is much larger than previously
> (since
> >> we
> >> > >> correctly consider the entire allocation rather than simply the
> used
> >> > >> allocation).
> >> > >>
> >> > >> Hakim,
> >> > >> Steven and I were discussing the need to update the ExternalSort
> >> > operator
> >> > >> to use the new allocator functionality to better manage its memory
> >> > >> envelope. Would you be interested in working on this since you seem
> >> to
> >> > be
> >> > >> working with that

Re: Out Of Memory Error (Possible Regression)

2015-12-30 Thread Abdel Hakim Deneche 
are you running the tests on 32 core machines ? a different number of cores
affects how much memory is available for the sort

On Wed, Dec 30, 2015 at 1:02 PM, Abdel Hakim Deneche 
wrote:

> The following tests are failing:
>
>
>> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q163_DRILL-2046.q
>>
>> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q177_DRILL-2046.q
>> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q174.q
>> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/
>> /Functional/window_functions/multiple_partitions/q35.sql
>>
>> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q160_DRILL-1985.q
>>
>> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q162_DRILL-1985.q
>> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q165.q
>> /Functional/window_functions/multiple_partitions/q37.sql
>> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q171.q
>>
>> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q168_DRILL-2046.q
>> /Functional/window_functions/multiple_partitions/q36.sql
>>
>> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q159_DRILL-2046.q
>> /Functional/window_functions/multiple_partitions/q30.sql
>>
>> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/large/q157_DRILL-1985.q
>> /Functional/window_functions/multiple_partitions/q22.sql
>
>
> With one of the following errors:
>
> java.sql.SQLException: RESOURCE ERROR: One or more nodes ran out of memory
>> while executing the query.
>> Caused by: org.apache.drill.exec.exception.OutOfMemoryException:
>> org.apache.drill.exec.exception.OutOfMemoryException: Unable to allocate
>> sv2, and not enough batchGroups to spill
>> at
>> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:356)
>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>
>
> or
>
> java.sql.SQLException: SYSTEM ERROR: DrillRuntimeException: Failed to
>> pre-allocate memory for SV. Existing recordCount*4 = 0, incoming batch
>> recordCount*4 = 3340
>> Caused by: org.apache.drill.common.exceptions.DrillRuntimeException:
>> Failed to pre-allocate memory for SV. Existing recordCount*4 = 0, incoming
>> batch recordCount*4 = 3340
>> at
>> org.apache.drill.exec.physical.impl.sort.SortRecordBatchBuilder.add(SortRecordBatchBuilder.java:116)
>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>> at
>> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:451)
>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>
>
>
>
> On Wed, Dec 30, 2015 at 12:42 PM, Jacques Nadeau 
> wrote:
>
>> I'll let Steven answer your question directly.
>>
>> FYI, we are running a regression suite that was forked from the MapR repo
>> a
>> month or so ago because we had to fix a bunch of things to make it work
>> with Apache Hadoop. (There was a thread about this back then and we
>> haven't
>> yet figured out how to merge both suites.) It is possible that he had a
>> successful run but the failures are happening on items that you've
>> recently
>> added to your suite.
>>
>> It is also possible (likely?) that the configuration settings for our
>> regression clusters are not the same.
>>
>> --
>> Jacques Nadeau
>> CTO and Co-Founder, Dremio
>>
>> On Wed, Dec 30, 2015 at 12:37 PM, Abdel Hakim Deneche <
>> adene...@maprtech.com
>> > wrote:
>>
>> > Steven,
>> >
>> > were you able to successfully run the regression tests on the transfer
>> > patch ? I just tried and saw several queries running out of memory !
>> >
>> > On Wed, Dec 30, 2015 at 11:46 AM, Abdel Hakim Deneche <
>> > adene...@maprtech.com
>> > > wrote:
>> >
>> > > Created DRILL-4236 
>> to
>> > > keep track of this improvement.
>> > >
>> > > On Wed, Dec 30, 2015 at 11:01 AM, Jacques Nadeau 
>> > > wrote:
>> > >
>> > >> Since the accounting changed (more accurate), the termination
>> condition
>> > >> for
>> > >> the sort operator will be different than before. In fact, this likely
>> > will
>> > >> be sooner since our accounting is much larger than previously (since
>> we
>> > >> correctly consider the entire allocation rather than simply the used
>> > >> allocation).
>> > >>
>> > >> Hakim,
>> > >> Steven and I were discussing the need to update the ExternalSort
>> > operator
>> > >> to use the new allocator functionality to better manage its memory
>> > >> envelope. Would you be interested in working on this since you seem
>> to
>> > be
>> > >> working with that code the most? Basically, it used to be that there
>> was
>> > >> no
>> > >> way the sort operator would be able to correctly detect a memory
>> > condition
>> > >> and so it jumped through a bunch of hoops to try to figure out the
>> > >> termination condition.With the transfer accounting in place, this
>> code
>> > can
>> > >> be greatly simplified to just use the current operator memory
>> > allocation.
>> > >>
>> > >> --
>>

Re: Out Of Memory Error (Possible Regression)

2015-12-30 Thread Abdel Hakim Deneche 
The following tests are failing:

/Functional/data-shapes/wide-columns/5000/1000rows/parquet/q163_DRILL-2046.q
>
> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q177_DRILL-2046.q
> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q174.q
> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/
> /Functional/window_functions/multiple_partitions/q35.sql
>
> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q160_DRILL-1985.q
>
> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q162_DRILL-1985.q
> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q165.q
> /Functional/window_functions/multiple_partitions/q37.sql
> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q171.q
>
> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q168_DRILL-2046.q
> /Functional/window_functions/multiple_partitions/q36.sql
>
> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/q159_DRILL-2046.q
> /Functional/window_functions/multiple_partitions/q30.sql
>
> /Functional/data-shapes/wide-columns/5000/1000rows/parquet/large/q157_DRILL-1985.q
> /Functional/window_functions/multiple_partitions/q22.sql


With one of the following errors:

java.sql.SQLException: RESOURCE ERROR: One or more nodes ran out of memory
> while executing the query.
> Caused by: org.apache.drill.exec.exception.OutOfMemoryException:
> org.apache.drill.exec.exception.OutOfMemoryException: Unable to allocate
> sv2, and not enough batchGroups to spill
> at
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:356)
> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]


or

java.sql.SQLException: SYSTEM ERROR: DrillRuntimeException: Failed to
> pre-allocate memory for SV. Existing recordCount*4 = 0, incoming batch
> recordCount*4 = 3340
> Caused by: org.apache.drill.common.exceptions.DrillRuntimeException:
> Failed to pre-allocate memory for SV. Existing recordCount*4 = 0, incoming
> batch recordCount*4 = 3340
> at
> org.apache.drill.exec.physical.impl.sort.SortRecordBatchBuilder.add(SortRecordBatchBuilder.java:116)
> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:451)
> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]




On Wed, Dec 30, 2015 at 12:42 PM, Jacques Nadeau  wrote:

> I'll let Steven answer your question directly.
>
> FYI, we are running a regression suite that was forked from the MapR repo a
> month or so ago because we had to fix a bunch of things to make it work
> with Apache Hadoop. (There was a thread about this back then and we haven't
> yet figured out how to merge both suites.) It is possible that he had a
> successful run but the failures are happening on items that you've recently
> added to your suite.
>
> It is also possible (likely?) that the configuration settings for our
> regression clusters are not the same.
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Wed, Dec 30, 2015 at 12:37 PM, Abdel Hakim Deneche <
> adene...@maprtech.com
> > wrote:
>
> > Steven,
> >
> > were you able to successfully run the regression tests on the transfer
> > patch ? I just tried and saw several queries running out of memory !
> >
> > On Wed, Dec 30, 2015 at 11:46 AM, Abdel Hakim Deneche <
> > adene...@maprtech.com
> > > wrote:
> >
> > > Created DRILL-4236 
> to
> > > keep track of this improvement.
> > >
> > > On Wed, Dec 30, 2015 at 11:01 AM, Jacques Nadeau 
> > > wrote:
> > >
> > >> Since the accounting changed (more accurate), the termination
> condition
> > >> for
> > >> the sort operator will be different than before. In fact, this likely
> > will
> > >> be sooner since our accounting is much larger than previously (since
> we
> > >> correctly consider the entire allocation rather than simply the used
> > >> allocation).
> > >>
> > >> Hakim,
> > >> Steven and I were discussing the need to update the ExternalSort
> > operator
> > >> to use the new allocator functionality to better manage its memory
> > >> envelope. Would you be interested in working on this since you seem to
> > be
> > >> working with that code the most? Basically, it used to be that there
> was
> > >> no
> > >> way the sort operator would be able to correctly detect a memory
> > condition
> > >> and so it jumped through a bunch of hoops to try to figure out the
> > >> termination condition.With the transfer accounting in place, this code
> > can
> > >> be greatly simplified to just use the current operator memory
> > allocation.
> > >>
> > >> --
> > >> Jacques Nadeau
> > >> CTO and Co-Founder, Dremio
> > >>
> > >> On Wed, Dec 30, 2015 at 10:48 AM, rahul challapalli <
> > >> challapallira...@gmail.com> wrote:
> > >>
> > >> > I installed the latest master and ran this query. So
> > >> > planner.memory.max_query_memory_per_node should have been the
> default
> > >> > value. I switch

Re: Out Of Memory Error (Possible Regression)

2015-12-30 Thread rahul challapalli 
I am guessing the configuration is different since some of these failing
suites were added a while ago ( > 1 month)

On Wed, Dec 30, 2015 at 12:42 PM, Jacques Nadeau  wrote:

> I'll let Steven answer your question directly.
>
> FYI, we are running a regression suite that was forked from the MapR repo a
> month or so ago because we had to fix a bunch of things to make it work
> with Apache Hadoop. (There was a thread about this back then and we haven't
> yet figured out how to merge both suites.) It is possible that he had a
> successful run but the failures are happening on items that you've recently
> added to your suite.
>
> It is also possible (likely?) that the configuration settings for our
> regression clusters are not the same.
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Wed, Dec 30, 2015 at 12:37 PM, Abdel Hakim Deneche <
> adene...@maprtech.com
> > wrote:
>
> > Steven,
> >
> > were you able to successfully run the regression tests on the transfer
> > patch ? I just tried and saw several queries running out of memory !
> >
> > On Wed, Dec 30, 2015 at 11:46 AM, Abdel Hakim Deneche <
> > adene...@maprtech.com
> > > wrote:
> >
> > > Created DRILL-4236 
> to
> > > keep track of this improvement.
> > >
> > > On Wed, Dec 30, 2015 at 11:01 AM, Jacques Nadeau 
> > > wrote:
> > >
> > >> Since the accounting changed (more accurate), the termination
> condition
> > >> for
> > >> the sort operator will be different than before. In fact, this likely
> > will
> > >> be sooner since our accounting is much larger than previously (since
> we
> > >> correctly consider the entire allocation rather than simply the used
> > >> allocation).
> > >>
> > >> Hakim,
> > >> Steven and I were discussing the need to update the ExternalSort
> > operator
> > >> to use the new allocator functionality to better manage its memory
> > >> envelope. Would you be interested in working on this since you seem to
> > be
> > >> working with that code the most? Basically, it used to be that there
> was
> > >> no
> > >> way the sort operator would be able to correctly detect a memory
> > condition
> > >> and so it jumped through a bunch of hoops to try to figure out the
> > >> termination condition.With the transfer accounting in place, this code
> > can
> > >> be greatly simplified to just use the current operator memory
> > allocation.
> > >>
> > >> --
> > >> Jacques Nadeau
> > >> CTO and Co-Founder, Dremio
> > >>
> > >> On Wed, Dec 30, 2015 at 10:48 AM, rahul challapalli <
> > >> challapallira...@gmail.com> wrote:
> > >>
> > >> > I installed the latest master and ran this query. So
> > >> > planner.memory.max_query_memory_per_node should have been the
> default
> > >> > value. I switched back to 1.4.0 branch and this query completed
> > >> > successfully.
> > >> >
> > >> > On Wed, Dec 30, 2015 at 10:37 AM, Abdel Hakim Deneche <
> > >> > adene...@maprtech.com
> > >> > > wrote:
> > >> >
> > >> > > Rahul,
> > >> > >
> > >> > > How much memory was assigned to the sort operator (
> > >> > > planner.memory.max_query_memory_per_node) ?
> > >> > >
> > >> > > On Wed, Dec 30, 2015 at 9:54 AM, rahul challapalli <
> > >> > > challapallira...@gmail.com> wrote:
> > >> > >
> > >> > > > I am seeing an OOM error while executing a simple CTAS query. I
> > >> raised
> > >> > > > DRILL-4324 for this. The query mentioned in the JIRA used to
> > >> complete
> > >> > > > successfully without any issue prior to 1.5. Any idea what could
> > >> have
> > >> > > > caused the regression?
> > >> > > >
> > >> > > > - Rahul
> > >> > > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > --
> > >> > >
> > >> > > Abdelhakim Deneche
> > >> > >
> > >> > > Software Engineer
> > >> > >
> > >> > >   
> > >> > >
> > >> > >
> > >> > > Now Available - Free Hadoop On-Demand Training
> > >> > > <
> > >> > >
> > >> >
> > >>
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > >
> > > Abdelhakim Deneche
> > >
> > > Software Engineer
> > >
> > >   
> > >
> > >
> > > Now Available - Free Hadoop On-Demand Training
> > > <
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > >
> > >
> >
> >
> >
> > --
> >
> > Abdelhakim Deneche
> >
> > Software Engineer
> >
> >   
> >
> >
> > Now Available - Free Hadoop On-Demand Training
> > <
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > >
> >
>

Re: Out Of Memory Error (Possible Regression)

2015-12-30 Thread Jacques Nadeau 
I'll let Steven answer your question directly.

FYI, we are running a regression suite that was forked from the MapR repo a
month or so ago because we had to fix a bunch of things to make it work
with Apache Hadoop. (There was a thread about this back then and we haven't
yet figured out how to merge both suites.) It is possible that he had a
successful run but the failures are happening on items that you've recently
added to your suite.

It is also possible (likely?) that the configuration settings for our
regression clusters are not the same.

--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Wed, Dec 30, 2015 at 12:37 PM, Abdel Hakim Deneche  wrote:

> Steven,
>
> were you able to successfully run the regression tests on the transfer
> patch ? I just tried and saw several queries running out of memory !
>
> On Wed, Dec 30, 2015 at 11:46 AM, Abdel Hakim Deneche <
> adene...@maprtech.com
> > wrote:
>
> > Created DRILL-4236  to
> > keep track of this improvement.
> >
> > On Wed, Dec 30, 2015 at 11:01 AM, Jacques Nadeau 
> > wrote:
> >
> >> Since the accounting changed (more accurate), the termination condition
> >> for
> >> the sort operator will be different than before. In fact, this likely
> will
> >> be sooner since our accounting is much larger than previously (since we
> >> correctly consider the entire allocation rather than simply the used
> >> allocation).
> >>
> >> Hakim,
> >> Steven and I were discussing the need to update the ExternalSort
> operator
> >> to use the new allocator functionality to better manage its memory
> >> envelope. Would you be interested in working on this since you seem to
> be
> >> working with that code the most? Basically, it used to be that there was
> >> no
> >> way the sort operator would be able to correctly detect a memory
> condition
> >> and so it jumped through a bunch of hoops to try to figure out the
> >> termination condition.With the transfer accounting in place, this code
> can
> >> be greatly simplified to just use the current operator memory
> allocation.
> >>
> >> --
> >> Jacques Nadeau
> >> CTO and Co-Founder, Dremio
> >>
> >> On Wed, Dec 30, 2015 at 10:48 AM, rahul challapalli <
> >> challapallira...@gmail.com> wrote:
> >>
> >> > I installed the latest master and ran this query. So
> >> > planner.memory.max_query_memory_per_node should have been the default
> >> > value. I switched back to 1.4.0 branch and this query completed
> >> > successfully.
> >> >
> >> > On Wed, Dec 30, 2015 at 10:37 AM, Abdel Hakim Deneche <
> >> > adene...@maprtech.com
> >> > > wrote:
> >> >
> >> > > Rahul,
> >> > >
> >> > > How much memory was assigned to the sort operator (
> >> > > planner.memory.max_query_memory_per_node) ?
> >> > >
> >> > > On Wed, Dec 30, 2015 at 9:54 AM, rahul challapalli <
> >> > > challapallira...@gmail.com> wrote:
> >> > >
> >> > > > I am seeing an OOM error while executing a simple CTAS query. I
> >> raised
> >> > > > DRILL-4324 for this. The query mentioned in the JIRA used to
> >> complete
> >> > > > successfully without any issue prior to 1.5. Any idea what could
> >> have
> >> > > > caused the regression?
> >> > > >
> >> > > > - Rahul
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > >
> >> > > Abdelhakim Deneche
> >> > >
> >> > > Software Engineer
> >> > >
> >> > >   
> >> > >
> >> > >
> >> > > Now Available - Free Hadoop On-Demand Training
> >> > > <
> >> > >
> >> >
> >>
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >> > > >
> >> > >
> >> >
> >>
> >
> >
> >
> > --
> >
> > Abdelhakim Deneche
> >
> > Software Engineer
> >
> >   
> >
> >
> > Now Available - Free Hadoop On-Demand Training
> > <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
> >
>
>
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
>   
>
>
> Now Available - Free Hadoop On-Demand Training
> <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
>

Re: Out Of Memory Error (Possible Regression)

2015-12-30 Thread Steven Phillips 
I didn't see any tests running out of memory. Which tests are you seeing
with this?

On Wed, Dec 30, 2015 at 1:37 PM, Abdel Hakim Deneche 
wrote:

> Steven,
>
> were you able to successfully run the regression tests on the transfer
> patch ? I just tried and saw several queries running out of memory !
>
> On Wed, Dec 30, 2015 at 11:46 AM, Abdel Hakim Deneche <
> adene...@maprtech.com
> > wrote:
>
> > Created DRILL-4236  to
> > keep track of this improvement.
> >
> > On Wed, Dec 30, 2015 at 11:01 AM, Jacques Nadeau 
> > wrote:
> >
> >> Since the accounting changed (more accurate), the termination condition
> >> for
> >> the sort operator will be different than before. In fact, this likely
> will
> >> be sooner since our accounting is much larger than previously (since we
> >> correctly consider the entire allocation rather than simply the used
> >> allocation).
> >>
> >> Hakim,
> >> Steven and I were discussing the need to update the ExternalSort
> operator
> >> to use the new allocator functionality to better manage its memory
> >> envelope. Would you be interested in working on this since you seem to
> be
> >> working with that code the most? Basically, it used to be that there was
> >> no
> >> way the sort operator would be able to correctly detect a memory
> condition
> >> and so it jumped through a bunch of hoops to try to figure out the
> >> termination condition.With the transfer accounting in place, this code
> can
> >> be greatly simplified to just use the current operator memory
> allocation.
> >>
> >> --
> >> Jacques Nadeau
> >> CTO and Co-Founder, Dremio
> >>
> >> On Wed, Dec 30, 2015 at 10:48 AM, rahul challapalli <
> >> challapallira...@gmail.com> wrote:
> >>
> >> > I installed the latest master and ran this query. So
> >> > planner.memory.max_query_memory_per_node should have been the default
> >> > value. I switched back to 1.4.0 branch and this query completed
> >> > successfully.
> >> >
> >> > On Wed, Dec 30, 2015 at 10:37 AM, Abdel Hakim Deneche <
> >> > adene...@maprtech.com
> >> > > wrote:
> >> >
> >> > > Rahul,
> >> > >
> >> > > How much memory was assigned to the sort operator (
> >> > > planner.memory.max_query_memory_per_node) ?
> >> > >
> >> > > On Wed, Dec 30, 2015 at 9:54 AM, rahul challapalli <
> >> > > challapallira...@gmail.com> wrote:
> >> > >
> >> > > > I am seeing an OOM error while executing a simple CTAS query. I
> >> raised
> >> > > > DRILL-4324 for this. The query mentioned in the JIRA used to
> >> complete
> >> > > > successfully without any issue prior to 1.5. Any idea what could
> >> have
> >> > > > caused the regression?
> >> > > >
> >> > > > - Rahul
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > >
> >> > > Abdelhakim Deneche
> >> > >
> >> > > Software Engineer
> >> > >
> >> > >   
> >> > >
> >> > >
> >> > > Now Available - Free Hadoop On-Demand Training
> >> > > <
> >> > >
> >> >
> >>
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >> > > >
> >> > >
> >> >
> >>
> >
> >
> >
> > --
> >
> > Abdelhakim Deneche
> >
> > Software Engineer
> >
> >   
> >
> >
> > Now Available - Free Hadoop On-Demand Training
> > <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
> >
>
>
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
>   
>
>
> Now Available - Free Hadoop On-Demand Training
> <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
>

Re: Out Of Memory Error (Possible Regression)

2015-12-30 Thread Abdel Hakim Deneche 
Steven,

were you able to successfully run the regression tests on the transfer
patch ? I just tried and saw several queries running out of memory !

On Wed, Dec 30, 2015 at 11:46 AM, Abdel Hakim Deneche  wrote:

> Created DRILL-4236  to
> keep track of this improvement.
>
> On Wed, Dec 30, 2015 at 11:01 AM, Jacques Nadeau 
> wrote:
>
>> Since the accounting changed (more accurate), the termination condition
>> for
>> the sort operator will be different than before. In fact, this likely will
>> be sooner since our accounting is much larger than previously (since we
>> correctly consider the entire allocation rather than simply the used
>> allocation).
>>
>> Hakim,
>> Steven and I were discussing the need to update the ExternalSort operator
>> to use the new allocator functionality to better manage its memory
>> envelope. Would you be interested in working on this since you seem to be
>> working with that code the most? Basically, it used to be that there was
>> no
>> way the sort operator would be able to correctly detect a memory condition
>> and so it jumped through a bunch of hoops to try to figure out the
>> termination condition.With the transfer accounting in place, this code can
>> be greatly simplified to just use the current operator memory allocation.
>>
>> --
>> Jacques Nadeau
>> CTO and Co-Founder, Dremio
>>
>> On Wed, Dec 30, 2015 at 10:48 AM, rahul challapalli <
>> challapallira...@gmail.com> wrote:
>>
>> > I installed the latest master and ran this query. So
>> > planner.memory.max_query_memory_per_node should have been the default
>> > value. I switched back to 1.4.0 branch and this query completed
>> > successfully.
>> >
>> > On Wed, Dec 30, 2015 at 10:37 AM, Abdel Hakim Deneche <
>> > adene...@maprtech.com
>> > > wrote:
>> >
>> > > Rahul,
>> > >
>> > > How much memory was assigned to the sort operator (
>> > > planner.memory.max_query_memory_per_node) ?
>> > >
>> > > On Wed, Dec 30, 2015 at 9:54 AM, rahul challapalli <
>> > > challapallira...@gmail.com> wrote:
>> > >
>> > > > I am seeing an OOM error while executing a simple CTAS query. I
>> raised
>> > > > DRILL-4324 for this. The query mentioned in the JIRA used to
>> complete
>> > > > successfully without any issue prior to 1.5. Any idea what could
>> have
>> > > > caused the regression?
>> > > >
>> > > > - Rahul
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > >
>> > > Abdelhakim Deneche
>> > >
>> > > Software Engineer
>> > >
>> > >   
>> > >
>> > >
>> > > Now Available - Free Hadoop On-Demand Training
>> > > <
>> > >
>> >
>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
>> > > >
>> > >
>> >
>>
>
>
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
>   
>
>
> Now Available - Free Hadoop On-Demand Training
> 
>



-- 

Abdelhakim Deneche

Software Engineer

  


Now Available - Free Hadoop On-Demand Training

Re: Out Of Memory Error (Possible Regression)

2015-12-30 Thread Abdel Hakim Deneche 
Created DRILL-4236  to
keep track of this improvement.

On Wed, Dec 30, 2015 at 11:01 AM, Jacques Nadeau  wrote:

> Since the accounting changed (more accurate), the termination condition for
> the sort operator will be different than before. In fact, this likely will
> be sooner since our accounting is much larger than previously (since we
> correctly consider the entire allocation rather than simply the used
> allocation).
>
> Hakim,
> Steven and I were discussing the need to update the ExternalSort operator
> to use the new allocator functionality to better manage its memory
> envelope. Would you be interested in working on this since you seem to be
> working with that code the most? Basically, it used to be that there was no
> way the sort operator would be able to correctly detect a memory condition
> and so it jumped through a bunch of hoops to try to figure out the
> termination condition.With the transfer accounting in place, this code can
> be greatly simplified to just use the current operator memory allocation.
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Wed, Dec 30, 2015 at 10:48 AM, rahul challapalli <
> challapallira...@gmail.com> wrote:
>
> > I installed the latest master and ran this query. So
> > planner.memory.max_query_memory_per_node should have been the default
> > value. I switched back to 1.4.0 branch and this query completed
> > successfully.
> >
> > On Wed, Dec 30, 2015 at 10:37 AM, Abdel Hakim Deneche <
> > adene...@maprtech.com
> > > wrote:
> >
> > > Rahul,
> > >
> > > How much memory was assigned to the sort operator (
> > > planner.memory.max_query_memory_per_node) ?
> > >
> > > On Wed, Dec 30, 2015 at 9:54 AM, rahul challapalli <
> > > challapallira...@gmail.com> wrote:
> > >
> > > > I am seeing an OOM error while executing a simple CTAS query. I
> raised
> > > > DRILL-4324 for this. The query mentioned in the JIRA used to complete
> > > > successfully without any issue prior to 1.5. Any idea what could have
> > > > caused the regression?
> > > >
> > > > - Rahul
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Abdelhakim Deneche
> > >
> > > Software Engineer
> > >
> > >   
> > >
> > >
> > > Now Available - Free Hadoop On-Demand Training
> > > <
> > >
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > > >
> > >
> >
>



-- 

Abdelhakim Deneche

Software Engineer

  


Now Available - Free Hadoop On-Demand Training

Re: Out Of Memory Error (Possible Regression)

2015-12-30 Thread Jacques Nadeau 
Since the accounting changed (more accurate), the termination condition for
the sort operator will be different than before. In fact, this likely will
be sooner since our accounting is much larger than previously (since we
correctly consider the entire allocation rather than simply the used
allocation).

Hakim,
Steven and I were discussing the need to update the ExternalSort operator
to use the new allocator functionality to better manage its memory
envelope. Would you be interested in working on this since you seem to be
working with that code the most? Basically, it used to be that there was no
way the sort operator would be able to correctly detect a memory condition
and so it jumped through a bunch of hoops to try to figure out the
termination condition.With the transfer accounting in place, this code can
be greatly simplified to just use the current operator memory allocation.

--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Wed, Dec 30, 2015 at 10:48 AM, rahul challapalli <
challapallira...@gmail.com> wrote:

> I installed the latest master and ran this query. So
> planner.memory.max_query_memory_per_node should have been the default
> value. I switched back to 1.4.0 branch and this query completed
> successfully.
>
> On Wed, Dec 30, 2015 at 10:37 AM, Abdel Hakim Deneche <
> adene...@maprtech.com
> > wrote:
>
> > Rahul,
> >
> > How much memory was assigned to the sort operator (
> > planner.memory.max_query_memory_per_node) ?
> >
> > On Wed, Dec 30, 2015 at 9:54 AM, rahul challapalli <
> > challapallira...@gmail.com> wrote:
> >
> > > I am seeing an OOM error while executing a simple CTAS query. I raised
> > > DRILL-4324 for this. The query mentioned in the JIRA used to complete
> > > successfully without any issue prior to 1.5. Any idea what could have
> > > caused the regression?
> > >
> > > - Rahul
> > >
> >
> >
> >
> > --
> >
> > Abdelhakim Deneche
> >
> > Software Engineer
> >
> >   
> >
> >
> > Now Available - Free Hadoop On-Demand Training
> > <
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > >
> >
>

Re: Out Of Memory Error (Possible Regression)

2015-12-30 Thread rahul challapalli 
I installed the latest master and ran this query. So
planner.memory.max_query_memory_per_node should have been the default
value. I switched back to 1.4.0 branch and this query completed
successfully.

On Wed, Dec 30, 2015 at 10:37 AM, Abdel Hakim Deneche  wrote:

> Rahul,
>
> How much memory was assigned to the sort operator (
> planner.memory.max_query_memory_per_node) ?
>
> On Wed, Dec 30, 2015 at 9:54 AM, rahul challapalli <
> challapallira...@gmail.com> wrote:
>
> > I am seeing an OOM error while executing a simple CTAS query. I raised
> > DRILL-4324 for this. The query mentioned in the JIRA used to complete
> > successfully without any issue prior to 1.5. Any idea what could have
> > caused the regression?
> >
> > - Rahul
> >
>
>
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
>   
>
>
> Now Available - Free Hadoop On-Demand Training
> <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
>

Re: Out Of Memory Error (Possible Regression)

2015-12-30 Thread Abdel Hakim Deneche 
Rahul,

How much memory was assigned to the sort operator (
planner.memory.max_query_memory_per_node) ?

On Wed, Dec 30, 2015 at 9:54 AM, rahul challapalli <
challapallira...@gmail.com> wrote:

> I am seeing an OOM error while executing a simple CTAS query. I raised
> DRILL-4324 for this. The query mentioned in the JIRA used to complete
> successfully without any issue prior to 1.5. Any idea what could have
> caused the regression?
>
> - Rahul
>



-- 

Abdelhakim Deneche

Software Engineer

  


Now Available - Free Hadoop On-Demand Training

Out Of Memory Error (Possible Regression)

2015-12-30 Thread rahul challapalli 
I am seeing an OOM error while executing a simple CTAS query. I raised
DRILL-4324 for this. The query mentioned in the JIRA used to complete
successfully without any issue prior to 1.5. Any idea what could have
caused the regression?

- Rahul