[jira] [Updated] (DRILL-2119) Drill interpreting a scalar as a map or array in an order by stmt when used with flatten
[ https://issues.apache.org/jira/browse/DRILL-2119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Parth Chandra updated DRILL-2119: - Fix Version/s: 0.9.0 > Drill interpreting a scalar as a map or array in an order by stmt when used > with flatten > > > Key: DRILL-2119 > URL: https://issues.apache.org/jira/browse/DRILL-2119 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Data Types >Reporter: Rahul Challapalli >Assignee: Mehant Baid > Fix For: 0.9.0 > > Attachments: data.json > > > git.commit.id.abbrev=3e33880 > Data Set : > {code} > { > "uid": 1, > "map": { > "rm": [ > { > "mapid": "m1", > "mapvalue": { > "col1": 1, > "col2": [0,1,2,3,4,5] > }, > "rptd": [ > { "a": "foo"}, > { "b": "boo"} > ] > }, > { > "mapid": "m2", > "mapvalue": { > "col1": 0, > "col2": [] > }, > "rptd": [ > { "a": "bar"}, > { "c": 1}, > { "d": 4.5} > ] > } > ] > }, > "lst_lst" : [[1,2,3,4,5],[2,3,4,5,6]] > } > {code} > Query : > {code} > select flatten(rptd) from (select uid, d.map.rm[0].rptd rptd, > flatten(d.lst_lst[1]) lst1 from `temp4.json` d) s1 order by s1.lst1; > {code} > When we remove the flatten from the top level query, things work fine > {code} > select s1.lst1 from (select uid, d.map.rm[0].rptd rptd, flatten(d.lst_lst[1]) > lst1 from `temp4.json` d) s1 order by s1.lst1 > ++ > |lst1| > ++ > | 2 | > | 3 | > | 4 | > | 5 | > | 6 | > ++ > {code} > Log File contents for the error > {code} > java.lang.RuntimeException: java.lang.UnsupportedOperationException: Map, > Array or repeated scalar type should not be used in group by, order by or in > a comparison operator. Drill does not support compare between MAP:REQUIRED > and MAP:REQUIRED. > at > org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:408) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:99) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:89) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:96) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:99) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:89) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:134) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.n
[jira] [Updated] (DRILL-2119) Drill interpreting a scalar as a map or array in an order by stmt when used with flatten
[ https://issues.apache.org/jira/browse/DRILL-2119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Parth Chandra updated DRILL-2119: - Component/s: (was: Execution - Relational Operators) Execution - Data Types > Drill interpreting a scalar as a map or array in an order by stmt when used > with flatten > > > Key: DRILL-2119 > URL: https://issues.apache.org/jira/browse/DRILL-2119 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Data Types >Reporter: Rahul Challapalli >Assignee: Mehant Baid > Attachments: data.json > > > git.commit.id.abbrev=3e33880 > Data Set : > {code} > { > "uid": 1, > "map": { > "rm": [ > { > "mapid": "m1", > "mapvalue": { > "col1": 1, > "col2": [0,1,2,3,4,5] > }, > "rptd": [ > { "a": "foo"}, > { "b": "boo"} > ] > }, > { > "mapid": "m2", > "mapvalue": { > "col1": 0, > "col2": [] > }, > "rptd": [ > { "a": "bar"}, > { "c": 1}, > { "d": 4.5} > ] > } > ] > }, > "lst_lst" : [[1,2,3,4,5],[2,3,4,5,6]] > } > {code} > Query : > {code} > select flatten(rptd) from (select uid, d.map.rm[0].rptd rptd, > flatten(d.lst_lst[1]) lst1 from `temp4.json` d) s1 order by s1.lst1; > {code} > When we remove the flatten from the top level query, things work fine > {code} > select s1.lst1 from (select uid, d.map.rm[0].rptd rptd, flatten(d.lst_lst[1]) > lst1 from `temp4.json` d) s1 order by s1.lst1 > ++ > |lst1| > ++ > | 2 | > | 3 | > | 4 | > | 5 | > | 6 | > ++ > {code} > Log File contents for the error > {code} > java.lang.RuntimeException: java.lang.UnsupportedOperationException: Map, > Array or repeated scalar type should not be used in group by, order by or in > a comparison operator. Drill does not support compare between MAP:REQUIRED > and MAP:REQUIRED. > at > org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:408) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:99) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:89) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:96) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:99) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:89) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:134) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.i
[jira] [Updated] (DRILL-2119) Drill interpreting a scalar as a map or array in an order by stmt when used with flatten
[ https://issues.apache.org/jira/browse/DRILL-2119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Challapalli updated DRILL-2119: - Attachment: data.json Attaching a data set with 2 records. Also when we copy over the same records 5 times we get another exception which looks similar. Data size might have led drill into a different execution path. {code} org.apache.drill.exec.exception.SchemaChangeException: Failure while trying to materialize incoming schema. Errors: Error in expression at index 0. Error: Missing function implementation: [hash(MAP-REQUIRED)]. Full expression: null.. at org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.createPartitioner(PartitionSenderRootExec.java:219) ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext(PartitionSenderRootExec.java:172) ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:57) [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:110) [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] at org.apache.drill.exec.work.WorkManager$RunnableWrapper.run(WorkManager.java:254) [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_71] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71] {code} > Drill interpreting a scalar as a map or array in an order by stmt when used > with flatten > > > Key: DRILL-2119 > URL: https://issues.apache.org/jira/browse/DRILL-2119 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators >Reporter: Rahul Challapalli >Assignee: Mehant Baid > Attachments: data.json > > > git.commit.id.abbrev=3e33880 > Data Set : > {code} > { > "uid": 1, > "map": { > "rm": [ > { > "mapid": "m1", > "mapvalue": { > "col1": 1, > "col2": [0,1,2,3,4,5] > }, > "rptd": [ > { "a": "foo"}, > { "b": "boo"} > ] > }, > { > "mapid": "m2", > "mapvalue": { > "col1": 0, > "col2": [] > }, > "rptd": [ > { "a": "bar"}, > { "c": 1}, > { "d": 4.5} > ] > } > ] > }, > "lst_lst" : [[1,2,3,4,5],[2,3,4,5,6]] > } > {code} > Query : > {code} > select flatten(rptd) from (select uid, d.map.rm[0].rptd rptd, > flatten(d.lst_lst[1]) lst1 from `temp4.json` d) s1 order by s1.lst1; > {code} > When we remove the flatten from the top level query, things work fine > {code} > select s1.lst1 from (select uid, d.map.rm[0].rptd rptd, flatten(d.lst_lst[1]) > lst1 from `temp4.json` d) s1 order by s1.lst1 > ++ > |lst1| > ++ > | 2 | > | 3 | > | 4 | > | 5 | > | 6 | > ++ > {code} > Log File contents for the error > {code} > java.lang.RuntimeException: java.lang.UnsupportedOperationException: Map, > Array or repeated scalar type should not be used in group by, order by or in > a comparison operator. Drill does not support compare between MAP:REQUIRED > and MAP:REQUIRED. > at > org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:408) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:99) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:89) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBa
[jira] [Updated] (DRILL-2119) Drill interpreting a scalar as a map or array in an order by stmt when used with flatten
[ https://issues.apache.org/jira/browse/DRILL-2119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Challapalli updated DRILL-2119: - Assignee: Mehant Baid (was: Jason Altekruse) > Drill interpreting a scalar as a map or array in an order by stmt when used > with flatten > > > Key: DRILL-2119 > URL: https://issues.apache.org/jira/browse/DRILL-2119 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Operators >Reporter: Rahul Challapalli >Assignee: Mehant Baid > > git.commit.id.abbrev=3e33880 > Data Set : > {code} > { > "uid": 1, > "map": { > "rm": [ > { > "mapid": "m1", > "mapvalue": { > "col1": 1, > "col2": [0,1,2,3,4,5] > }, > "rptd": [ > { "a": "foo"}, > { "b": "boo"} > ] > }, > { > "mapid": "m2", > "mapvalue": { > "col1": 0, > "col2": [] > }, > "rptd": [ > { "a": "bar"}, > { "c": 1}, > { "d": 4.5} > ] > } > ] > }, > "lst_lst" : [[1,2,3,4,5],[2,3,4,5,6]] > } > {code} > Query : > {code} > select flatten(rptd) from (select uid, d.map.rm[0].rptd rptd, > flatten(d.lst_lst[1]) lst1 from `temp4.json` d) s1 order by s1.lst1; > {code} > When we remove the flatten from the top level query, things work fine > {code} > select s1.lst1 from (select uid, d.map.rm[0].rptd rptd, flatten(d.lst_lst[1]) > lst1 from `temp4.json` d) s1 order by s1.lst1 > ++ > |lst1| > ++ > | 2 | > | 3 | > | 4 | > | 5 | > | 6 | > ++ > {code} > Log File contents for the error > {code} > java.lang.RuntimeException: java.lang.UnsupportedOperationException: Map, > Array or repeated scalar type should not be used in group by, order by or in > a comparison operator. Drill does not support compare between MAP:REQUIRED > and MAP:REQUIRED. > at > org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:408) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:99) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:89) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:96) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:99) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:89) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:134) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator