Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20742
Change subject: IMPALA-11501: Add flag to allow catalog-cache operations on masked tables ...................................................................... IMPALA-11501: Add flag to allow catalog-cache operations on masked tables REFRESH/INVALIDATE METADATA <table> are the table level catalog-cache operations. In Hive-Ranger plugin, when a table is masked (either by column-masking or row-filtering policy) for a user, the user is unable to perform any modification (insert/delete/update) on the table, i.e. it's considered a read-only user (RANGER-1087, RANGER-1100). However, Hive doesn't have these catalog-cache operations. It's a grey area whether they should be blocked. Before this patch, these catalog-cache operations are considered as modifications on the table so they are also blocked for masked users. Table metadata is required to be loaded so we have the column names to fetch Ranger column masking policies. This causes a performance regression on INVALIDATE METADATA <table> commands since in older versions (e.g. CDH), IM commands don't need to load the table metadata and runs pretty fast. This patch adds a flag, allow_catalog_cache_op_from_masked_users, for coordinators to skip checking masking policies for such statements. When this flag is on, coordinators don't need to load the table metadata thus fix the performance regression as well. Note that Ranger ownership based policies can't be applied correctly when the table is unloaded (so the owner is unknown). REFRESH/INVALIDATE METADATA <table> commands could be denied on owners even if there are Ranger policies allowing the owner's operations. This is a known issue since IMPALA-8228. To ensure a user can always perform these operations, grant REFRESH privilege to them to get rid of the unloaded table issue. This patch also fixes a bug in local catalog mode which only occurs after adding the new flag. The bug is that LocalDb#getTableIfCached() always returns a LocalIncompleteTable regardless whether the table is loaded in the coordinator. LocalIncompleteTable doens't have the owner string, which causes REFRESH/INVALIDATE statements can't pass the ownership context to RangerAccessResourceImpl so ownership policies can't be correctly applied. This patch fixes LocalDb#getTableIfCached() to also lookup in the cache and returns the LocalTable instance if it's loaded. Tests: - Add e2e tests on both the legacy and local catalog mode. Change-Id: I45935654cbf05a55d740f1b04781022c271f7678 --- M be/src/service/frontend.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/Privilege.java M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M tests/authorization/test_ranger.py 13 files changed, 165 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/42/20742/1 -- To view, visit http://gerrit.cloudera.org:8080/20742 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I45935654cbf05a55d740f1b04781022c271f7678 Gerrit-Change-Number: 20742 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com>
