[
https://issues.apache.org/jira/browse/DRILL-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16507481#comment-16507481
]
ASF GitHub Bot commented on DRILL-6373:
---------------------------------------
vrozov commented on issue #1244: DRILL-6373: Refactor Result Set Loader for
Union, List support
URL: https://github.com/apache/drill/pull/1244#issuecomment-396076908
Changes in partition sender inroduced back in April are not related to the
`ConcurrentModificationException` . The regression is introduced in
`NullableValueVectors.java:110` as it tries to modify `children` collection
while the same collection is iterated over in a different thread. Please take a
look at the stack traces for the competing threads:
```
at
org.apache.drill.exec.record.MaterializedField.addChild(MaterializedField.java:132)
~[vector-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.vector.NullableVarBinaryVector.<init>(NullableVarBinaryVector.java:132)
~[vector-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.expr.BasicTypeHelper.getNewVector(BasicTypeHelper.java:1490)
[vector-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.AbstractMapVector.<init>(AbstractMapVector.java:56)
~[vector-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.MapVector.<init>(MapVector.java:65)
[vector-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.expr.BasicTypeHelper.getNewVector(BasicTypeHelper.java:1275)
[vector-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.expr.BasicTypeHelper.getNewVector(BasicTypeHelper.java:1261)
[vector-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.test.generated.PartitionerGen33655$OutgoingRecordBatch.initializeBatch(PartitionerTemplate.java:381)
[na:na]
at
org.apache.drill.exec.test.generated.PartitionerGen33655.flushOutgoingBatches(PartitionerTemplate.java:173)
[na:na]
at
org.apache.drill.exec.physical.impl.partitionsender.PartitionerDecorator$FlushBatchesHandlingClass.execute(PartitionerDecorator.java:285)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.partitionsender.PartitionerDecorator$PartitionerTask.run(PartitionerDecorator.java:340)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_161]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
```
and
```
at
java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719)
~[na:1.8.0_161]
at
java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:742)
~[na:1.8.0_161]
at
org.apache.drill.exec.record.MaterializedField.withPathAndType(MaterializedField.java:206)
~[vector-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.MaterializedField.clone(MaterializedField.java:185)
~[vector-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.MaterializedField.withPathAndType(MaterializedField.java:207)
~[vector-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.MaterializedField.clone(MaterializedField.java:185)
~[vector-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.AbstractMapVector.<init>(AbstractMapVector.java:49)
~[vector-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.MapVector.<init>(MapVector.java:65)
~[vector-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.expr.BasicTypeHelper.getNewVector(BasicTypeHelper.java:1275)
~[vector-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.expr.BasicTypeHelper.getNewVector(BasicTypeHelper.java:1261)
~[vector-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.test.generated.PartitionerGen33655$OutgoingRecordBatch.initializeBatch(PartitionerTemplate.java:381)
~[na:na]
at
org.apache.drill.exec.test.generated.PartitionerGen33655.flushOutgoingBatches(PartitionerTemplate.java:173)
~[na:na]
at
org.apache.drill.exec.physical.impl.partitionsender.PartitionerDecorator$FlushBatchesHandlingClass.execute(PartitionerDecorator.java:285)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.partitionsender.PartitionerDecorator$PartitionerTask.run(PartitionerDecorator.java:340)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
... 3 common frames omitted
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Refactor the Result Set Loader to prepare for Union, List support
> -----------------------------------------------------------------
>
> Key: DRILL-6373
> URL: https://issues.apache.org/jira/browse/DRILL-6373
> Project: Apache Drill
> Issue Type: Improvement
> Affects Versions: 1.13.0
> Reporter: Paul Rogers
> Assignee: Paul Rogers
> Priority: Major
> Fix For: 1.14.0
>
>
> As the next step in merging the "batch sizing" enhancements, refactor the
> {{ResultSetLoader}} and related classes to prepare for Union and List
> support. This fix follows the refactoring of the column accessors for the
> same purpose. Actual Union and List support is to follow in a separate PR.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
