Hao Zhu created DRILL-1673:
------------------------------
Summary: Flatten function can not work well with nested arrays.
Key: DRILL-1673
URL: https://issues.apache.org/jira/browse/DRILL-1673
Project: Apache Drill
Issue Type: Bug
Components: Functions - Drill
Affects Versions: 0.7.0
Environment: 0.7.0
Reporter: Hao Zhu
Flatten function failed to scan nested arrays , for example something like ""[[
]]"".
The only difference of JSON files between below 2 tests is
"num":[1,2,3]
VS
"num":[[1,2,3]]
==============Test 1 (Works well):==============
file:
{code}
{"fixed_column":"abc", "list_column":[{"id1":"1","name":"zhu", "num": [1,2,3]},
{"id1":"2","name":"hao", "num": [4,5,6]} ]}
{code}
SQL:
{code}
0: jdbc:drill:zk=local> select t.`fixed_column` as fixed_column,
flatten(t.`list_column`) from
dfs.root.`/Users/hzu/Documents/sharefolder/hp/n2.json` as t;
+--------------+------------+
| fixed_column | EXPR$1 |
+--------------+------------+
| abc | {"id1":"1","name":"zhu","num":[1,2,3]} |
| abc | {"id1":"2","name":"hao","num":[4,5,6]} |
+--------------+------------+
2 rows selected (0.154 seconds)
{code}
==============Test 2 (Failed):==============
file:
{code}
{"fixed_column":"abc", "list_column":[{"id1":"1","name":"zhu", "num":
[[1,2,3]]}, {"id1":"2","name":"hao", "num": [[4,5,6]]} ]}
{code}
SQL:
{code}
0: jdbc:drill:zk=local> select t.`fixed_column` as fixed_column,
flatten(t.`list_column`) from
dfs.root.`/Users/hzu/Documents/sharefolder/hp/n3.json` as t;
+--------------+------------+
| fixed_column | EXPR$1 |
+--------------+------------+
Query failed: Failure while running fragment.[
df28347b-fac1-497d-b9c5-a313ba77aa4d on 10.250.0.115:31010 ]
(java.lang.UnsupportedOperationException)
org.apache.drill.exec.vector.complex.RepeatedListVector$RepeatedListTransferPair.splitAndTransfer():339
org.apache.drill.exec.vector.complex.RepeatedMapVector$SingleMapTransferPair.splitAndTransfer():305
org.apache.drill.exec.test.generated.FlattenerGen22.flattenRecords():93
org.apache.drill.exec.physical.impl.flatten.FlattenRecordBatch.doWork():152
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():89
org.apache.drill.exec.physical.impl.flatten.FlattenRecordBatch.innerNext():118
org.apache.drill.exec.record.AbstractRecordBatch.next():106
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next():124
org.apache.drill.exec.record.AbstractRecordBatch.next():86
org.apache.drill.exec.record.AbstractRecordBatch.next():76
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():52
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():129
org.apache.drill.exec.record.AbstractRecordBatch.next():106
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next():124
org.apache.drill.exec.physical.impl.BaseRootExec.next():67
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():122
org.apache.drill.exec.physical.impl.BaseRootExec.next():57
org.apache.drill.exec.work.fragment.FragmentExecutor.run():105
org.apache.drill.exec.work.WorkManager$RunnableWrapper.run():249
.......():0
java.lang.RuntimeException: java.sql.SQLException: Failure while executing
query.
at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2514)
at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2148)
at sqlline.SqlLine.print(SqlLine.java:1809)
at sqlline.SqlLine$Commands.execute(SqlLine.java:3766)
at sqlline.SqlLine$Commands.sql(SqlLine.java:3663)
at sqlline.SqlLine.dispatch(SqlLine.java:889)
at sqlline.SqlLine.begin(SqlLine.java:763)
at sqlline.SqlLine.start(SqlLine.java:498)
at sqlline.SqlLine.main(SqlLine.java:460)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
