Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20742 )
Change subject: IMPALA-11501: Add flag to allow catalog-cache operations on masked tables ...................................................................... IMPALA-11501: Add flag to allow catalog-cache operations on masked tables REFRESH/INVALIDATE METADATA <table> are the table level catalog-cache operations. In Hive-Ranger plugin, when a table is masked (either by column-masking or row-filtering policy) for a user, the user is unable to perform any modification (insert/delete/update) on the table, i.e. it's considered a read-only user (RANGER-1087, RANGER-1100). However, Hive doesn't have these catalog-cache operations. It's a grey area whether they should be blocked. Before this patch, these catalog-cache operations are considered as modifications on the table so they are also blocked for masked users. Table metadata is required to be loaded so we have the column names to fetch Ranger column masking policies. This causes a performance regression on INVALIDATE METADATA <table> commands since in older versions (e.g. CDH), IM commands don't need to load the table metadata and runs pretty fast. This patch adds a flag, allow_catalog_cache_op_from_masked_users, for coordinators to skip checking masking policies for such statements. When this flag is on, coordinators don't need to load the table metadata thus fix the performance regression as well. Note that Ranger ownership based policies can't be applied correctly when the table is unloaded (so the owner is unknown). REFRESH/INVALIDATE METADATA <table> commands could be denied on owners even if there are Ranger policies allowing the owner's operations. This is a known issue since IMPALA-8228. To ensure a user can always perform these operations, grant REFRESH privilege to them to get rid of the unloaded table issue. This patch also fixes a bug in local catalog mode which only occurs after adding the new flag. The bug is that LocalDb#getTableIfCached() doesn't make good use of the cache. If the table meta is cahced but LocalDb#getTable() hasn't been invoked on the table, getTableIfCached() will always return a LocalIncompleteTable which is missing some table info, e.g. ownership. This causes REFRESH/INVALIDATE statements not able to pass the ownership context to RangerAccessResourceImpl so ownership policies can't be correctly applied. Ideally, LocalDb#getTableIfCached() should return a LocalTable instance if the table is cached. However, in local catalog mode, we don't cache everything that constructs a LocalTable instance. Constructing a LocalTable instance might still trigger external RPCs which should be avoided. As an alternative, this patch checks if the msTable is cached. If it's cached, add it to the LocalIncompleteTable instance so most of the table info can be retrieved, including the ownership string. Tests: - Add e2e tests on both the legacy and local catalog mode. Change-Id: I45935654cbf05a55d740f1b04781022c271f7678 Reviewed-on: http://gerrit.cloudera.org:8080/20742 Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> --- M be/src/service/frontend.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/Privilege.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIncompleteTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M tests/authorization/test_ranger.py 15 files changed, 188 insertions(+), 12 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/20742 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I45935654cbf05a55d740f1b04781022c271f7678 Gerrit-Change-Number: 20742 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com> Gerrit-Reviewer: Fang-Yu Rao <fangyu....@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com>
