Hello Anurag Mantripragada, Fang-Yu Rao, Vihang Karajgaonkar, Kurt Deschler, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15108 to look at the new patch set (#10). Change subject: IMPALA-9330: Support resolving unmasked nested columns in masked tables ...................................................................... IMPALA-9330: Support resolving unmasked nested columns in masked tables Column masking policies on primitive columns of a table which contains nested types (though they won't be masked) will cause query failures. To be specifit, if tableA(id int, int_array array<int>) has a masking policy on column "id", all queries on "tableA" will fail, e.g. select id from tableA; select t.id, a.item from tableA t, t.int_array a; Column masking is implemented by wrapping the underlying table/view with a table masking view. However, as we don't support nested types in SelectList, the table masking view can't expose nested columns of the masked table, which causes collection refs not being resolved correctly. This patch fixes the issue by 2 steps: 1) Expose nested columns of the underlying table in the output Type of the table masking view (see InlineViewRef#createTupleDescriptor()). So nested Paths in the original query block can be resolved. 2) For such kind of Paths, resolved them again inside the table masking view. So they can point to the underlying table as what they mean (see Analyzer#resolvePathWithMasking()). TupleDescriptor of such kind of table masking view won't be materialized since the view is simple enough that its query plan is just a ScanNode of the underlying table. The whole query plan can be stitched as if the table is not masked. Note that one day when we support nested columns in SelectList, we may don't need these 2 hacks. This patch also adds some TRACE level loggings to improve debuggability. Column masking is enabled by default after this patch. Test changes in TestRanger.test_column_masking: - Add column masking policy on a table containing nested types. - Add queries on the masked tables. Some queries are borrowed from existing tests for nested types. Tests: - Run CORE tests. Change-Id: I1cc5565c64c1a4a56445b8edde59b1168f387791 --- M be/src/common/global-flags.cc M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/CollectionTableRef.java M fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java M fe/src/main/java/org/apache/impala/analysis/Path.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/authorization/TableMask.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java M testdata/workloads/functional-query/queries/QueryTest/ranger_column_masking.test M tests/authorization/test_ranger.py 16 files changed, 538 insertions(+), 22 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/15108/10 -- To view, visit http://gerrit.cloudera.org:8080/15108 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1cc5565c64c1a4a56445b8edde59b1168f387791 Gerrit-Change-Number: 15108 Gerrit-PatchSet: 10 Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Anurag Mantripragada <anu...@cloudera.com> Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com> Gerrit-Reviewer: Fang-Yu Rao <fangyu....@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Kurt Deschler <kdesc...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Vihang Karajgaonkar <vih...@cloudera.com>