Hello Daniel Becker, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17983 to look at the new patch set (#8). Change subject: IMPALA-10920: Zipping unnest for arrays ...................................................................... IMPALA-10920: Zipping unnest for arrays This patch provides an unnest implementation for arrays where unnesting multiple arrays in one query results the items of the arrays being zipped together instead of joining. There are two different syntaxes introduced for this purpose: 1: ISO:SQL 2016 compliant syntax: SELECT a1.item, a2.item FROM complextypes_arrays t, UNNEST(t.arr1, t.arr2) AS (a1, a2); 2: Postgres compatible syntax: SELECT UNNEST(arr1), UNNEST(arr2) FROM complextypes_arrays; Let me show the expected behaviour through the following example: Inputs: arr1: {1,2,3}, arr2: {11, 12} After running any of the above queries we expect the following output: =============== | arr1 | arr2 | =============== | 1 | 11 | | 2 | 12 | | 3 | NULL | =============== Expected behaviour: - When unnesting multiple arrays with zipping unnest then the 'i'th item of one array will be put next to the 'i'th item of the other arrays in the results. - In case the size of the arrays is not the same then the shorter arrays will be filled with NULL values up to the size of the longest array. On a sidenote, UNNEST is added to Impala's SQL language as a new keyword. This might interfere with use cases where a resource (db, table, column, etc.) is named "UNNEST". Restrictions: - It is not allowed to have WHERE filters on an unnested item of an array in the same SELECT query. E.g. this is not allowed: SELECT arr1.item FROM complextypes_arrays t, UNNEST(t.arr1) WHERE arr1.item < 5; Note, that it is allowed to have an outer SELECT around the one doing unnests and have a filter there on the unnested items. - If there is an outer SELECT filtering on the unnested array's items from the inner SELECT then these predicates won't be pushed down to the SCAN node. They are rather evaluated in the UNNEST node to guarantee result correctness after unnesting. Note, this restriction is only active when there are multiple arrays being unnested, or in other words when zipping unnest logic is required to produce results. - It's not allowed to do a zipping and a (traditional) joining unnest together in one SELECT query. - It's not allowed to perform zipping unnests on arrays from different tables. Testing: - Added a bunch of E2E tests to the test suite to cover both syntaxes. - Did a manual test run on a table with 1000 rows, 3 array columns with size of around 5000 items in each array. I did an unnest on all three arrays in one query to see if there are any crashes or suspicious slowness when running on this scale. Change-Id: Ic58ff6579ecff03962e7a8698edfbe0684ce6cf7 --- M be/src/exec/unnest-node.cc M be/src/exec/unnest-node.h M common/thrift/PlanNodes.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/FromClause.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java A fe/src/main/java/org/apache/impala/analysis/UnnestExpr.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/PlanNode.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/planner/UnnestNode.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java A testdata/ComplexTypesTbl/arrays.orc A testdata/ComplexTypesTbl/arrays.parq M testdata/data/README M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv A testdata/workloads/functional-query/queries/QueryTest/zipping-unnest-in-from-clause.test A testdata/workloads/functional-query/queries/QueryTest/zipping-unnest-in-select-list.test M tests/query_test/test_nested_types.py 30 files changed, 1,406 insertions(+), 136 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/17983/8 -- To view, visit http://gerrit.cloudera.org:8080/17983 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic58ff6579ecff03962e7a8698edfbe0684ce6cf7 Gerrit-Change-Number: 17983 Gerrit-PatchSet: 8 Gerrit-Owner: Gabor Kaszab <gaborkas...@cloudera.com> Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com> Gerrit-Reviewer: Daniel Becker <daniel.bec...@cloudera.com> Gerrit-Reviewer: Gabor Kaszab <gaborkas...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>