[jira] [Commented] (DRILL-3779) NPE during mergeAndSpill operation of external sort
[
https://issues.apache.org/jira/browse/DRILL-3779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14804737#comment-14804737
]
ASF GitHub Bot commented on DRILL-3779:
---
Github user asfgit closed the pull request at:
https://github.com/apache/drill/pull/160
> NPE during mergeAndSpill operation of external sort
> ---
>
> Key: DRILL-3779
> URL: https://issues.apache.org/jira/browse/DRILL-3779
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Affects Versions: 1.1.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: 1.2.0
>
>
> For a query of the following pattern:
> {code}
> alter session set `planner.enable_hashjoin` = false;
> alter session set `planner.width.max_per_node` = 6;
> select
>t1.c1, t1.c2t1.c50
> from t1 left outer join t2 on ..
>left outer join t3 on ...
>left outer join t4 on ...
>left outer join t5 on ...
> where t2.col between t1.c1 and t1.c2
> {code}
> we get an NPE below. The dataset is large so I cannot attach it here. Will
> try to get a repro with smaller data.
> {code}
> java.sql.SQLException: SYSTEM ERROR: NullPointerException
> Fragment 5:0
> [Error Id: 3df8da3d-dee2-45d4-9a0a-abaa0ab3775e on se-node11.se.lab:31010]
> (java.lang.NullPointerException) null
>
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.getBufferSize():523
>
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.mergeAndSpill():461
>
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext():334
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.record.AbstractRecordBatch.next():95
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
>
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():92
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.physical.impl.join.JoinStatus.nextLeft():127
> org.apache.drill.exec.physical.impl.join.JoinStatus.ensureInitial():143
> org.apache.drill.exec.physical.impl.join.MergeJoinBatch.innerNext():171
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.record.AbstractRecordBatch.next():95
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3779) NPE during mergeAndSpill operation of external sort
[
https://issues.apache.org/jira/browse/DRILL-3779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14803088#comment-14803088
]
ASF GitHub Bot commented on DRILL-3779:
---
Github user adeneche commented on the pull request:
https://github.com/apache/drill/pull/160#issuecomment-141121314
two small comments to address, apart from that LGTM,
+1
> NPE during mergeAndSpill operation of external sort
> ---
>
> Key: DRILL-3779
> URL: https://issues.apache.org/jira/browse/DRILL-3779
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Affects Versions: 1.1.0
>Reporter: Aman Sinha
>Assignee: Deneche A. Hakim
> Fix For: 1.2.0
>
>
> For a query of the following pattern:
> {code}
> alter session set `planner.enable_hashjoin` = false;
> alter session set `planner.width.max_per_node` = 6;
> select
>t1.c1, t1.c2t1.c50
> from t1 left outer join t2 on ..
>left outer join t3 on ...
>left outer join t4 on ...
>left outer join t5 on ...
> where t2.col between t1.c1 and t1.c2
> {code}
> we get an NPE below. The dataset is large so I cannot attach it here. Will
> try to get a repro with smaller data.
> {code}
> java.sql.SQLException: SYSTEM ERROR: NullPointerException
> Fragment 5:0
> [Error Id: 3df8da3d-dee2-45d4-9a0a-abaa0ab3775e on se-node11.se.lab:31010]
> (java.lang.NullPointerException) null
>
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.getBufferSize():523
>
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.mergeAndSpill():461
>
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext():334
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.record.AbstractRecordBatch.next():95
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
>
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():92
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.physical.impl.join.JoinStatus.nextLeft():127
> org.apache.drill.exec.physical.impl.join.JoinStatus.ensureInitial():143
> org.apache.drill.exec.physical.impl.join.MergeJoinBatch.innerNext():171
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.record.AbstractRecordBatch.next():95
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3779) NPE during mergeAndSpill operation of external sort
[
https://issues.apache.org/jira/browse/DRILL-3779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14803061#comment-14803061
]
ASF GitHub Bot commented on DRILL-3779:
---
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/160#discussion_r39758626
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/ExternalSortBatch.java
---
@@ -478,28 +498,36 @@ public BatchGroup
mergeAndSpill(LinkedList batchGroups) throws Schem
break;
}
BatchGroup batch = batchGroups.pollLast();
+ assert batch != null;
--- End diff --
please add a message to the assertion, it will make our life easier if this
assertion throws
> NPE during mergeAndSpill operation of external sort
> ---
>
> Key: DRILL-3779
> URL: https://issues.apache.org/jira/browse/DRILL-3779
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Affects Versions: 1.1.0
>Reporter: Aman Sinha
>Assignee: Deneche A. Hakim
> Fix For: 1.2.0
>
>
> For a query of the following pattern:
> {code}
> alter session set `planner.enable_hashjoin` = false;
> alter session set `planner.width.max_per_node` = 6;
> select
>t1.c1, t1.c2t1.c50
> from t1 left outer join t2 on ..
>left outer join t3 on ...
>left outer join t4 on ...
>left outer join t5 on ...
> where t2.col between t1.c1 and t1.c2
> {code}
> we get an NPE below. The dataset is large so I cannot attach it here. Will
> try to get a repro with smaller data.
> {code}
> java.sql.SQLException: SYSTEM ERROR: NullPointerException
> Fragment 5:0
> [Error Id: 3df8da3d-dee2-45d4-9a0a-abaa0ab3775e on se-node11.se.lab:31010]
> (java.lang.NullPointerException) null
>
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.getBufferSize():523
>
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.mergeAndSpill():461
>
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext():334
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.record.AbstractRecordBatch.next():95
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
>
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():92
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.physical.impl.join.JoinStatus.nextLeft():127
> org.apache.drill.exec.physical.impl.join.JoinStatus.ensureInitial():143
> org.apache.drill.exec.physical.impl.join.MergeJoinBatch.innerNext():171
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.record.AbstractRecordBatch.next():95
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3779) NPE during mergeAndSpill operation of external sort
[
https://issues.apache.org/jira/browse/DRILL-3779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14803052#comment-14803052
]
Aman Sinha commented on DRILL-3779:
---
[~adeneche] I am in the process of running all tests but sending PR for review
...
> NPE during mergeAndSpill operation of external sort
> ---
>
> Key: DRILL-3779
> URL: https://issues.apache.org/jira/browse/DRILL-3779
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Affects Versions: 1.1.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: 1.2.0
>
>
> For a query of the following pattern:
> {code}
> alter session set `planner.enable_hashjoin` = false;
> alter session set `planner.width.max_per_node` = 6;
> select
>t1.c1, t1.c2t1.c50
> from t1 left outer join t2 on ..
>left outer join t3 on ...
>left outer join t4 on ...
>left outer join t5 on ...
> where t2.col between t1.c1 and t1.c2
> {code}
> we get an NPE below. The dataset is large so I cannot attach it here. Will
> try to get a repro with smaller data.
> {code}
> java.sql.SQLException: SYSTEM ERROR: NullPointerException
> Fragment 5:0
> [Error Id: 3df8da3d-dee2-45d4-9a0a-abaa0ab3775e on se-node11.se.lab:31010]
> (java.lang.NullPointerException) null
>
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.getBufferSize():523
>
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.mergeAndSpill():461
>
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext():334
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.record.AbstractRecordBatch.next():95
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
>
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():92
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.physical.impl.join.JoinStatus.nextLeft():127
> org.apache.drill.exec.physical.impl.join.JoinStatus.ensureInitial():143
> org.apache.drill.exec.physical.impl.join.MergeJoinBatch.innerNext():171
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.record.AbstractRecordBatch.next():95
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3779) NPE during mergeAndSpill operation of external sort
[
https://issues.apache.org/jira/browse/DRILL-3779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14803055#comment-14803055
]
ASF GitHub Bot commented on DRILL-3779:
---
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/160#discussion_r39757806
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/ExternalSortBatch.java
---
@@ -114,6 +119,12 @@
private final String fileName;
private int firstSpillBatchCount = 0;
+ /**
+ * The copier uses the COPER_BATCH_MEM_LIMIT to estimate the target
--- End diff --
small typo `COPIER_...` instead of `COPER_...`
> NPE during mergeAndSpill operation of external sort
> ---
>
> Key: DRILL-3779
> URL: https://issues.apache.org/jira/browse/DRILL-3779
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Affects Versions: 1.1.0
>Reporter: Aman Sinha
>Assignee: Deneche A. Hakim
> Fix For: 1.2.0
>
>
> For a query of the following pattern:
> {code}
> alter session set `planner.enable_hashjoin` = false;
> alter session set `planner.width.max_per_node` = 6;
> select
>t1.c1, t1.c2t1.c50
> from t1 left outer join t2 on ..
>left outer join t3 on ...
>left outer join t4 on ...
>left outer join t5 on ...
> where t2.col between t1.c1 and t1.c2
> {code}
> we get an NPE below. The dataset is large so I cannot attach it here. Will
> try to get a repro with smaller data.
> {code}
> java.sql.SQLException: SYSTEM ERROR: NullPointerException
> Fragment 5:0
> [Error Id: 3df8da3d-dee2-45d4-9a0a-abaa0ab3775e on se-node11.se.lab:31010]
> (java.lang.NullPointerException) null
>
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.getBufferSize():523
>
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.mergeAndSpill():461
>
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext():334
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.record.AbstractRecordBatch.next():95
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
>
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():92
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.physical.impl.join.JoinStatus.nextLeft():127
> org.apache.drill.exec.physical.impl.join.JoinStatus.ensureInitial():143
> org.apache.drill.exec.physical.impl.join.MergeJoinBatch.innerNext():171
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.record.AbstractRecordBatch.next():95
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3779) NPE during mergeAndSpill operation of external sort
[
https://issues.apache.org/jira/browse/DRILL-3779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14803049#comment-14803049
]
ASF GitHub Bot commented on DRILL-3779:
---
GitHub user amansinha100 opened a pull request:
https://github.com/apache/drill/pull/160
DRILL-3779: Fix NPE in mergeAndSpill(). Add more debug logging messa…
…ges. Make copier per-batch memory limit power of 2.
Get rid of some warnings. Add a few comments.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/amansinha100/incubator-drill external_sort2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/160.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #160
commit 5eac063b5a0c6a149d41a26889d6636dd9bebf1b
Author: Aman Sinha
Date: 2015-09-17T08:29:38Z
DRILL-3779: Fix NPE in mergeAndSpill(). Add more debug logging messages.
Make copier per-batch memory limit power of 2.
Get rid of some warnings. Add a few comments.
> NPE during mergeAndSpill operation of external sort
> ---
>
> Key: DRILL-3779
> URL: https://issues.apache.org/jira/browse/DRILL-3779
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Affects Versions: 1.1.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: 1.2.0
>
>
> For a query of the following pattern:
> {code}
> alter session set `planner.enable_hashjoin` = false;
> alter session set `planner.width.max_per_node` = 6;
> select
>t1.c1, t1.c2t1.c50
> from t1 left outer join t2 on ..
>left outer join t3 on ...
>left outer join t4 on ...
>left outer join t5 on ...
> where t2.col between t1.c1 and t1.c2
> {code}
> we get an NPE below. The dataset is large so I cannot attach it here. Will
> try to get a repro with smaller data.
> {code}
> java.sql.SQLException: SYSTEM ERROR: NullPointerException
> Fragment 5:0
> [Error Id: 3df8da3d-dee2-45d4-9a0a-abaa0ab3775e on se-node11.se.lab:31010]
> (java.lang.NullPointerException) null
>
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.getBufferSize():523
>
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.mergeAndSpill():461
>
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext():334
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.record.AbstractRecordBatch.next():95
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
>
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():92
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.physical.impl.join.JoinStatus.nextLeft():127
> org.apache.drill.exec.physical.impl.join.JoinStatus.ensureInitial():143
> org.apache.drill.exec.physical.impl.join.MergeJoinBatch.innerNext():171
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.record.AbstractRecordBatch.next():95
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
