Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17638 )
Change subject: IMPALA-9495: Support struct in select list for ORC tables ...................................................................... Patch Set 12: (7 comments) Still digesting deeper details of the FE part. Left some questions first. http://gerrit.cloudera.org:8080/#/c/17638/12/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/17638/12/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@1058 PS12, Line 1058: public void registerSlotDescriptor(SlotDescriptor slotDesc) { Is this used anywhere? If so, it'd be better to move it closer with the existing registerSlotRef() method. http://gerrit.cloudera.org:8080/#/c/17638/12/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java: http://gerrit.cloudera.org:8080/#/c/17638/12/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@336 PS12, Line 336: removeStructChildren(); Do we have test coverage for this? We'd better fully test this. In the initial support of Ranger column-masking we have a similar approach and it's error prone. E.g. when struct children are removed and we forgot to reset SlotRef somewhere so its isAnalyzed_ flag is still true, then it will skip analyze() and won't expand struct children anymore. BTW, can we do this in SlotRef.reset()? http://gerrit.cloudera.org:8080/#/c/17638/12/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@398 PS12, Line 398: expr != null nit: this is covered by the instanceof check http://gerrit.cloudera.org:8080/#/c/17638/12/fe/src/main/java/org/apache/impala/analysis/SlotRef.java File fe/src/main/java/org/apache/impala/analysis/SlotRef.java: http://gerrit.cloudera.org:8080/#/c/17638/12/fe/src/main/java/org/apache/impala/analysis/SlotRef.java@156 PS12, Line 156: if (type_.isStructType() && rootTable instanceof FeFsTable) { I think we also need to abort when 'rootTable' is other kinds of table, e.g. FeHBaseTable. http://gerrit.cloudera.org:8080/#/c/17638/10/testdata/workloads/functional-query/queries/QueryTest/struct-in-select-list.test File testdata/workloads/functional-query/queries/QueryTest/struct-in-select-list.test: http://gerrit.cloudera.org:8080/#/c/17638/10/testdata/workloads/functional-query/queries/QueryTest/struct-in-select-list.test@4 PS10, Line 4: select id, tiny_struct from functional_orc_def.complextypes_structs; > The reason I haven't written "select *" tests is that they won't exercise t I thought "SELECT *" will also include the struct columns as Hive does. Do we plan to support this? http://gerrit.cloudera.org:8080/#/c/17638/12/testdata/workloads/functional-query/queries/QueryTest/struct-in-select-list.test File testdata/workloads/functional-query/queries/QueryTest/struct-in-select-list.test: http://gerrit.cloudera.org:8080/#/c/17638/12/testdata/workloads/functional-query/queries/QueryTest/struct-in-select-list.test@514 PS12, Line 514: order by tiny_struct Could you also add a case for "order by 2"? http://gerrit.cloudera.org:8080/#/c/17638/10/tests/authorization/test_ranger.py File tests/authorization/test_ranger.py: http://gerrit.cloudera.org:8080/#/c/17638/10/tests/authorization/test_ranger.py@999 PS10, Line 999: def test_column_masking(self, vector, unique_name): > Could you add a test here about masking a STRUCT column to NULL? Apart from the previous request, could you add another test that masking primitive colums with the existence with a struct type in the SelectList? E.g. select alltypes, id, str from functional_orc_def.complextypes_structs and the column 'str' has a masking policy. -- To view, visit http://gerrit.cloudera.org:8080/17638 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0fbe56bdcd372b72e99c0195d87a818e7fa4bc3a Gerrit-Change-Number: 17638 Gerrit-PatchSet: 12 Gerrit-Owner: Gabor Kaszab <gaborkas...@cloudera.com> Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com> Gerrit-Reviewer: Daniel Becker <daniel.bec...@cloudera.com> Gerrit-Reviewer: Gabor Kaszab <gaborkas...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Comment-Date: Thu, 26 Aug 2021 08:46:51 +0000 Gerrit-HasComments: Yes
