[Impala-ASF-CR] IMPALA-10590: Introduce admission service heartbeat mechanism
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/17194 ) Change subject: IMPALA-10590: Introduce admission service heartbeat mechanism .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/17194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia528d92268cea487ada20b476935a81166f5ad34 Gerrit-Change-Number: 17194 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 18 Mar 2021 03:52:41 + Gerrit-HasComments: No
[Impala-ASF-CR] Revert "IMPALA-10503: testdata load hits hive memory limit errors during hive inserts"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17191 ) Change subject: Revert "IMPALA-10503: testdata load hits hive memory limit errors during hive inserts" .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6980/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17191 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I896c7b2457d537fa1bfe8dc29063da0b7b3df199 Gerrit-Change-Number: 17191 Gerrit-PatchSet: 2 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Thu, 18 Mar 2021 01:44:05 + Gerrit-HasComments: No
[Impala-ASF-CR] Revert "IMPALA-10503: testdata load hits hive memory limit errors during hive inserts"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17191 ) Change subject: Revert "IMPALA-10503: testdata load hits hive memory limit errors during hive inserts" .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17191 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I896c7b2457d537fa1bfe8dc29063da0b7b3df199 Gerrit-Change-Number: 17191 Gerrit-PatchSet: 2 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Thu, 18 Mar 2021 01:44:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10549: Register transactions from external frontend DML
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17122 ) Change subject: IMPALA-10549: Register transactions from external frontend DML .. Patch Set 15: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/17122 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia8863b8d9d281a5d164f10de9c5ee52cf3be63db Gerrit-Change-Number: 17122 Gerrit-PatchSet: 15 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Thu, 18 Mar 2021 01:31:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10549: Register transactions from external frontend DML
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17122 ) Change subject: IMPALA-10549: Register transactions from external frontend DML .. IMPALA-10549: Register transactions from external frontend DML This change registers transactions that were started by an external frontend so that coordinator keepalive can track them properly. Testing: manually tested using DMLs from external frontend Reviewed-by: John Sherman Change-Id: Ia8863b8d9d281a5d164f10de9c5ee52cf3be63db Reviewed-on: http://gerrit.cloudera.org:8080/17122 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/service/frontend.cc M be/src/service/frontend.h M be/src/service/impala-server.cc M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java 5 files changed, 43 insertions(+), 7 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/17122 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ia8863b8d9d281a5d164f10de9c5ee52cf3be63db Gerrit-Change-Number: 17122 Gerrit-PatchSet: 16 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10584: Allow UnpinStream to fail if reservation is not enough
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17195 ) Change subject: IMPALA-10584: Allow UnpinStream to fail if reservation is not enough .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8387/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17195 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I16137b6e423f190f60c3115a06ccd0f77e9f585a Gerrit-Change-Number: 17195 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Thu, 18 Mar 2021 01:07:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10584: Allow UnpinStream to fail if reservation is not enough
Riza Suminto has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17195 Change subject: IMPALA-10584: Allow UnpinStream to fail if reservation is not enough .. IMPALA-10584: Allow UnpinStream to fail if reservation is not enough TestScratchLimit::test_with_unlimited_scratch_limit has been intermittently crashing in ubuntu-16.04-dockerised-tests environment after result spooling is enabled by default in IMPALA-9856. DCHECK violation occurs in ReservationTracker::CheckConsistency() due to BufferedTupleStream wrongly tries to reclaim memory reservation while unpinning the stream. For this bug to surface, all of the following needs to happen: - Stream is in pinned mode. - There are only 2 pages in the stream: 1 read and 1 write. - Stream can not increase reservation anymore either due to memory pressure or low buffer/memory limit. - The stream read page has been fully read and is attached to output RowBatch. But the output RowBatch has not cleaned up yet. - BufferedTupleStream::UnpinStream is invoked. The bug happens because UnpinStream proceeds to NextReadPage where the read page buffer was mistakenly assumed as released. default_page_len_ bytes were added into write_page_reservation_ and subsequently violates the total memory reservation. This patch fixes the bug by disallowing UnpinStream to advance the read iterator if the read page is attached to output RowBatch and there is no more unused reservation. If UnpinStream in turns unable to unpin any page to reclaim memory reservation, TErrorCode::NO_PAGE_UNPINNED will be returned and the stream will stay in pinned mode. In the context of BufferedPlanRootSink, if this error occurs in Send(), the writer will release the lock to give a chance for the reader to read some more rows and clean up the output RowBatch (current_batch_) before retying to add more rows again. Testing: - Add BE test UnpinReadPageMinReservation. - Reenable result spooling in TestScratchLimit that was disabled in IMPALA-10559. - Pass core tests. Change-Id: I16137b6e423f190f60c3115a06ccd0f77e9f585a --- M be/src/exec/buffered-plan-root-sink.cc M be/src/runtime/buffered-tuple-stream-test.cc M be/src/runtime/buffered-tuple-stream.cc M be/src/runtime/buffered-tuple-stream.h M be/src/runtime/spillable-row-batch-queue.cc M be/src/runtime/spillable-row-batch-queue.h M common/thrift/generate_error_codes.py M tests/query_test/test_scratch_limit.py 8 files changed, 200 insertions(+), 47 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/17195/1 -- To view, visit http://gerrit.cloudera.org:8080/17195 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I16137b6e423f190f60c3115a06ccd0f77e9f585a Gerrit-Change-Number: 17195 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto
[Impala-ASF-CR] IMPALA-10518: Add ImpalaServer interface to retrieve executor membership.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17181 ) Change subject: IMPALA-10518: Add ImpalaServer interface to retrieve executor membership. .. IMPALA-10518: Add ImpalaServer interface to retrieve executor membership. This patch adds an interface to ImpalaServer to retrieve the current executor membership snapshot from impalad for use by an external frontend. This involves sending a thrift request to impalad and receiving a thrift response. Refactored some code in exec-env into a separate function in the impala namespace which makes it easier to populate the needed information for an external frontend. Testing: - Ran selected tests for sanity check (no impact is expected since this is adding a new interface): - Frontend tests (PlannerTest, CardinalityTest) - Backend tests under custom_cluster/test_executor_groups.py - Manually tested with external frontend to ensure it gets the executor membership snapshot. Change-Id: Ie89b71f4555c368869ee7b9d6341756c60af12b5 Reviewed-on: http://gerrit.cloudera.org:8080/17181 Reviewed-by: Thomas Tauber-Marshall Tested-by: Impala Public Jenkins --- M be/src/rpc/hs2-http-test.cc M be/src/runtime/exec-env.cc M be/src/scheduling/cluster-membership-mgr.cc M be/src/scheduling/cluster-membership-mgr.h M be/src/service/impala-hs2-server.cc M be/src/service/impala-server.h M common/thrift/ImpalaService.thrift M tests/hs2/test_hs2.py 8 files changed, 124 insertions(+), 39 deletions(-) Approvals: Thomas Tauber-Marshall: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/17181 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie89b71f4555c368869ee7b9d6341756c60af12b5 Gerrit-Change-Number: 17181 Gerrit-PatchSet: 6 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10518: Add ImpalaServer interface to retrieve executor membership.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17181 ) Change subject: IMPALA-10518: Add ImpalaServer interface to retrieve executor membership. .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/17181 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie89b71f4555c368869ee7b9d6341756c60af12b5 Gerrit-Change-Number: 17181 Gerrit-PatchSet: 5 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Thu, 18 Mar 2021 00:39:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10579: Avoid infinite loop from remote file iterator
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/17192 )
Change subject: IMPALA-10579: Avoid infinite loop from remote file iterator
..
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/17192/1/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
File fe/src/main/java/org/apache/impala/common/FileSystemUtil.java:
http://gerrit.cloudera.org:8080/#/c/17192/1/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@837
PS1, Line 837: IMPALA-1057
IMPALA-10579
http://gerrit.cloudera.org:8080/#/c/17192/1/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@840
PS1, Line 840: if (last_iter_had_exception) {
: throw ex;
: }
I think if the iterator encounters two absence subdirectories continuely, we
will throw exception here. E.g. when there are three concurrent INSERTs on the
table, the last finished INSERT could fail by this.
Could you loop test_insert_stress.py several times to test?
--
To view, visit http://gerrit.cloudera.org:8080/17192
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8b60aab0286390ced364188a24ee78f99073730
Gerrit-Change-Number: 17192
Gerrit-PatchSet: 1
Gerrit-Owner: Kurt Deschler
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Comment-Date: Thu, 18 Mar 2021 00:38:49 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10577: Add retrying of AdmitQuery
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17188 ) Change subject: IMPALA-10577: Add retrying of AdmitQuery .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/17188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8bc0cac666bbd613a1143c0e2c4f84d3b0ad003a Gerrit-Change-Number: 17188 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 18 Mar 2021 00:36:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10577: Add retrying of AdmitQuery
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17188 ) Change subject: IMPALA-10577: Add retrying of AdmitQuery .. IMPALA-10577: Add retrying of AdmitQuery This patch adds retries of the AdmitQuery rpc by coordinators. This helps to ensure that if an admissiond goes down and is restarted or is temporarily unreachable, queries won't fail. The retries are done with backoff and jitter to avoid overloading the admissiond in these scenarios. A new flag, --admission_max_retry_time_s, is added to control how long queries will continue retrying before giving up. The AdmitQuery rpc is made idempotent - if a query is submitted with the same query id as one the admissiond already knows about, AdmitQuery will return OK without submitting the query to be scheduled again. Testing: - Added a custom cluster test that checks that queries won't fail when the admissiond goes down. Change-Id: I8bc0cac666bbd613a1143c0e2c4f84d3b0ad003a Reviewed-on: http://gerrit.cloudera.org:8080/17188 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/scheduling/admission-control-service.cc M be/src/scheduling/remote-admission-control-client.cc M be/src/scheduling/remote-admission-control-client.h M common/protobuf/admission_control_service.proto M tests/custom_cluster/test_admission_controller.py 5 files changed, 137 insertions(+), 28 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/17188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8bc0cac666bbd613a1143c0e2c4f84d3b0ad003a Gerrit-Change-Number: 17188 Gerrit-PatchSet: 4 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-9234: Support Ranger row filtering policies
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16976 ) Change subject: IMPALA-9234: Support Ranger row filtering policies .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8386/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16976 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I580517be241225ca15e45686381b78890178d7cc Gerrit-Change-Number: 16976 Gerrit-PatchSet: 11 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 18 Mar 2021 00:35:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9234: Support Ranger row filtering policies
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/16976 ) Change subject: IMPALA-9234: Support Ranger row filtering policies .. Patch Set 11: Code-Review+2 I feel comfortable with the current patch to move ahead. There's good discussion here and possible follow-up enhancements but it makes sense to get this core patch in. -- To view, visit http://gerrit.cloudera.org:8080/16976 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I580517be241225ca15e45686381b78890178d7cc Gerrit-Change-Number: 16976 Gerrit-PatchSet: 11 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 18 Mar 2021 00:24:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9234: Support Ranger row filtering policies
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/16976 )
Change subject: IMPALA-9234: Support Ranger row filtering policies
..
Patch Set 11:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/16976/9/fe/src/main/java/org/apache/impala/authorization/TableMask.java
File fe/src/main/java/org/apache/impala/authorization/TableMask.java:
http://gerrit.cloudera.org:8080/#/c/16976/9/fe/src/main/java/org/apache/impala/authorization/TableMask.java@109
PS9, Line 109: // Parse the row filter string to AST by using it in a fake
query.
> > Note that only the WHERE clause is returned. So "SELECT *" is used for si
Done. I think "SELECT 1" is the same complexity since we just do parsing (not
analyzing) here. But maybe "SELECT 1" can avoid the above confusion.
http://gerrit.cloudera.org:8080/#/c/16976/9/fe/src/main/java/org/apache/impala/authorization/TableMask.java@111
PS9, Line 111: SelectStmt selectStmt = (SelectStmt) Parser.parse(stmtSql);
> I would think that the admin user who has access to the row filter definition
> should see the parse exception details such that he can fix it but for the
> normal user there should be some level of obscuring of the error message
> ..otherwise it would reveal the filter details and in any case the normal
> user won't be able to fix it by himself.
No, the admin user can't see the error if the policy doesn't apply to him/her.
Usually the policy is set for normal users. I think the process of adding a row
filter should be:
1) Admin user adds a new table containing sensitive data.
2) Admin user adds a row-filtering policy for it in Ranger Admin WebUI. Note
that Ranger treates the row filter as a string and won't validate it. The admin
user should also choose what roles/groups/users the policy applies to.
3) Admin user switchs to target users and run some queries on the table. Verify
that the row filter works as expected. At this point, a detailed error exposing
the row filter helps to debug the malformed row filter string.
4) Admin user publishs the table (notify the target users about this new table).
5) If the normal users encounter any errors, they should reach out to the admin
user for help. They usually don't have permission to edit the row filter policy.
> BTW, in the above execution I assume you were the privileged user. But it
> sounds like the behavior would have been the same if you were not.
Good point! I think this is a problem that we resolve the column/row masking
before privilege checks. I'll file a follow-up JIRA for this.
> I think the Hive message is also problematic ..it just happened to show less
> for this case but what if the row filter subquery had some syntax issue ?
Here's another example for syntax issue. Adding a row filter "id = (select from
functional.alltypessmall)" on table functional.alltypestiny for my username,
then launch beeline
0: jdbc:hive2://localhost:11050> select * from functional.alltypestiny;
Error: Error while compiling statement: FAILED: SemanticException
org.apache.hadoop.hive.ql.parse.ParseException: line 1:487 cannot recognize
input near '(' 'select' 'from' in expression specification
(state=42000,code=4)
You can play around with Hive by launching it using
"testdata/bin/run-hive-server.sh -with_ranger".
> EDIT: I see later that you mention both Impala and Hive will show the full
> rewritten query in the EXPLAIN plan and the query profile. This contains the
> expanded row filter even for the normal user .. so I am not sure if the parse
> exception is that relevant. I would be curious to know how other products
> like Spark deal with such things. The row filter could have things like
> Salary or Social Security number predicate etc which should ideally be hidden.
Yeah, I will do some investigations.
--
To view, visit http://gerrit.cloudera.org:8080/16976
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I580517be241225ca15e45686381b78890178d7cc
Gerrit-Change-Number: 16976
Gerrit-PatchSet: 11
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Thu, 18 Mar 2021 00:16:03 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9234: Support Ranger row filtering policies
Hello Aman Sinha, Fang-Yu Rao, Tim Armstrong, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16976 to look at the new patch set (#11). Change subject: IMPALA-9234: Support Ranger row filtering policies .. IMPALA-9234: Support Ranger row filtering policies Ranger row filtering policies provide customized expressions to filter out rows for specific users when reading from a table. This patch adds support for this feature. A new feature flag, enable_row_filtering, is added to disable this experimental feature. It defaults to be true so the feature is enabled by default. Enabling row-filtering requires --enable_column_masking=true since it depends on the column masking implementation. Note that row filtering policies take effects prior to any column masking policies, because column masking policies apply on result data. Implementation: The existing table masking view infrastructure can be extended to support row filtering. Currently when analyzing a table with column masking policies, we replace the TableRef with an InlineViewRef which contains a SelectStmt wrapping the columns with masking expressions. This patch adds the row filtering expressions to the WhereClause of the SelectStmt. Limitations: - Expressions using subqueries are not supported (IMPALA-10483). - Row filtering policies on nested tables will not be applied when nested collection columns are used directly in the FROM clause. This will leak data so we forbid such kinds of queries until IMPALA-10484 is resolved. Tests: - Add FE test for error message when disabling row filtering. - Add e2e test with row filtering policies. - Add e2e test with column masking and row filtering policies both take place. - Verified audits in a CDP cluster with Ranger and Solr set up. Change-Id: I580517be241225ca15e45686381b78890178d7cc --- M be/src/common/global-flags.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/TableMask.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationContext.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/util/AuthorizationUtil.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M testdata/workloads/functional-query/queries/QueryTest/ranger_column_masking.test A testdata/workloads/functional-query/queries/QueryTest/ranger_column_masking_and_row_filtering.test A testdata/workloads/functional-query/queries/QueryTest/ranger_row_filtering.test M tests/authorization/test_ranger.py 23 files changed, 1,005 insertions(+), 113 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/16976/11 -- To view, visit http://gerrit.cloudera.org:8080/16976 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I580517be241225ca15e45686381b78890178d7cc Gerrit-Change-Number: 16976 Gerrit-PatchSet: 11 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-10512: ALTER TABLE ADD PARTITION should bump the write id for ACID tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17081 ) Change subject: IMPALA-10512: ALTER TABLE ADD PARTITION should bump the write id for ACID tables .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6976/ -- To view, visit http://gerrit.cloudera.org:8080/17081 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iad247008b7c206db00516326c1447bd00a9b34bd Gerrit-Change-Number: 17081 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 17 Mar 2021 23:21:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10590: Introduce admission service heartbeat mechanism
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17194 ) Change subject: IMPALA-10590: Introduce admission service heartbeat mechanism .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8385/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia528d92268cea487ada20b476935a81166f5ad34 Gerrit-Change-Number: 17194 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 17 Mar 2021 22:40:23 + Gerrit-HasComments: No
[Impala-ASF-CR] Revert "IMPALA-10503: testdata load hits hive memory limit errors during hive inserts"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17191 ) Change subject: Revert "IMPALA-10503: testdata load hits hive memory limit errors during hive inserts" .. Patch Set 1: Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6975/ -- To view, visit http://gerrit.cloudera.org:8080/17191 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I896c7b2457d537fa1bfe8dc29063da0b7b3df199 Gerrit-Change-Number: 17191 Gerrit-PatchSet: 1 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 17 Mar 2021 22:32:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10590: Introduce admission service heartbeat mechanism
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17194 ) Change subject: IMPALA-10590: Introduce admission service heartbeat mechanism .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/17194/1/tests/custom_cluster/test_admission_controller.py File tests/custom_cluster/test_admission_controller.py: http://gerrit.cloudera.org:8080/#/c/17194/1/tests/custom_cluster/test_admission_controller.py@1358 PS1, Line 1358: @ flake8: E303 too many blank lines (2) -- To view, visit http://gerrit.cloudera.org:8080/17194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia528d92268cea487ada20b476935a81166f5ad34 Gerrit-Change-Number: 17194 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 17 Mar 2021 22:21:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10590: Introduce admission service heartbeat mechanism
Thomas Tauber-Marshall has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17194 Change subject: IMPALA-10590: Introduce admission service heartbeat mechanism .. IMPALA-10590: Introduce admission service heartbeat mechanism Currently, if a ReleaseQuery rpc fails, it's possible for the admission service to think that some resources are still being used that are actually free. This patch fixes the issue by introducing a periodic heartbeat rpc from coordinators to the admission service which contains a list of queries registered at that coordinator. If there is a query that the admission service thinks is running but is not included in the heartbeat, the admission service can conclude that the query must have already completed and release its resources. Testing: - Added a test that uses a debug action to simulate ReleaseQuery rpcs failing and checks that query resources are released properly. Change-Id: Ia528d92268cea487ada20b476935a81166f5ad34 --- M be/src/scheduling/admission-control-service.cc M be/src/scheduling/admission-control-service.h M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/local-admission-control-client.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M common/protobuf/admission_control_service.proto M tests/custom_cluster/test_admission_controller.py 9 files changed, 232 insertions(+), 42 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/94/17194/1 -- To view, visit http://gerrit.cloudera.org:8080/17194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ia528d92268cea487ada20b476935a81166f5ad34 Gerrit-Change-Number: 17194 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10558: Implement ds theta exclude() function
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17153 ) Change subject: IMPALA-10558: Implement ds_theta_exclude() function .. IMPALA-10558: Implement ds_theta_exclude() function This function receives two strings that are serialized Apache DataSketches Theta sketches. Computes the a-not-b set operation given two sketches of same or different column. Example: select ds_theta_estimate(ds_theta_exclude(sketch1, sketch2)) from sketch_tbl; +---+ | ds_theta_estimate(ds_theta_exclude(sketch1, sketch2)) | +---+ | 5 | +---+ Change-Id: I05119fd8c652c07ff248a99e44b0da3541e46ca3 Reviewed-on: http://gerrit.cloudera.org:8080/17153 Reviewed-by: Gabor Kaszab Tested-by: Impala Public Jenkins --- M be/src/exprs/datasketches-common.cc M be/src/exprs/datasketches-functions-ir.cc M be/src/exprs/datasketches-functions.h M common/function-registry/impala_functions.py M testdata/workloads/functional-query/queries/QueryTest/datasketches-theta.test 5 files changed, 169 insertions(+), 0 deletions(-) Approvals: Gabor Kaszab: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/17153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I05119fd8c652c07ff248a99e44b0da3541e46ca3 Gerrit-Change-Number: 17153 Gerrit-PatchSet: 6 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10558: Implement ds theta exclude() function
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17153 ) Change subject: IMPALA-10558: Implement ds_theta_exclude() function .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/17153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I05119fd8c652c07ff248a99e44b0da3541e46ca3 Gerrit-Change-Number: 17153 Gerrit-PatchSet: 5 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 17 Mar 2021 22:14:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10579: Avoid infinite loop from remote file iterator
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17192 ) Change subject: IMPALA-10579: Avoid infinite loop from remote file iterator .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8384/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17192 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia8b60aab0286390ced364188a24ee78f99073730 Gerrit-Change-Number: 17192 Gerrit-PatchSet: 1 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 17 Mar 2021 20:02:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10549: Register transactions from external frontend DML
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17122 ) Change subject: IMPALA-10549: Register transactions from external frontend DML .. Patch Set 15: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6979/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17122 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia8863b8d9d281a5d164f10de9c5ee52cf3be63db Gerrit-Change-Number: 17122 Gerrit-PatchSet: 15 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 17 Mar 2021 19:47:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10549: Register transactions from external frontend DML
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17122 ) Change subject: IMPALA-10549: Register transactions from external frontend DML .. Patch Set 15: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17122 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia8863b8d9d281a5d164f10de9c5ee52cf3be63db Gerrit-Change-Number: 17122 Gerrit-PatchSet: 15 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 17 Mar 2021 19:47:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10549: Register transactions from external frontend DML
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/17122 ) Change subject: IMPALA-10549: Register transactions from external frontend DML .. Patch Set 14: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17122 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia8863b8d9d281a5d164f10de9c5ee52cf3be63db Gerrit-Change-Number: 17122 Gerrit-PatchSet: 14 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 17 Mar 2021 19:46:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10579: Avoid infinite loop from remote file iterator
Kurt Deschler has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17192 Change subject: IMPALA-10579: Avoid infinite loop from remote file iterator .. IMPALA-10579: Avoid infinite loop from remote file iterator This patch adds a flag to FileSystemUtil.FilterIterator.hasNext() to avoid getting into an infinite loop if the remote iterator throws repeated FileNotFoundException exceptions. Since these exceptions do not contain information regarding which file is missing, the new check simply re-throws the exception if the remote hasNext() call throws twice in a row. Testing: Tested manually by temporarily adding explicit exceptions in the problem loop Change-Id: Ia8b60aab0286390ced364188a24ee78f99073730 --- M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java 1 file changed, 10 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/17192/1 -- To view, visit http://gerrit.cloudera.org:8080/17192 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ia8b60aab0286390ced364188a24ee78f99073730 Gerrit-Change-Number: 17192 Gerrit-PatchSet: 1 Gerrit-Owner: Kurt Deschler
[Impala-ASF-CR] IMPALA-10518: Add ImpalaServer interface to retrieve executor membership.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17181 ) Change subject: IMPALA-10518: Add ImpalaServer interface to retrieve executor membership. .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6978/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17181 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie89b71f4555c368869ee7b9d6341756c60af12b5 Gerrit-Change-Number: 17181 Gerrit-PatchSet: 5 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 17 Mar 2021 18:55:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10518: Add ImpalaServer interface to retrieve executor membership.
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/17181 ) Change subject: IMPALA-10518: Add ImpalaServer interface to retrieve executor membership. .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17181 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie89b71f4555c368869ee7b9d6341756c60af12b5 Gerrit-Change-Number: 17181 Gerrit-PatchSet: 5 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 17 Mar 2021 18:55:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10577: Add retrying of AdmitQuery
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17188 ) Change subject: IMPALA-10577: Add retrying of AdmitQuery .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6977/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8bc0cac666bbd613a1143c0e2c4f84d3b0ad003a Gerrit-Change-Number: 17188 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 17 Mar 2021 18:53:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10512: ALTER TABLE ADD PARTITION should bump the write id for ACID tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17081 ) Change subject: IMPALA-10512: ALTER TABLE ADD PARTITION should bump the write id for ACID tables .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6976/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17081 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iad247008b7c206db00516326c1447bd00a9b34bd Gerrit-Change-Number: 17081 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 17 Mar 2021 17:44:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10512: ALTER TABLE ADD PARTITION should bump the write id for ACID tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17081 ) Change subject: IMPALA-10512: ALTER TABLE ADD PARTITION should bump the write id for ACID tables .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17081 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iad247008b7c206db00516326c1447bd00a9b34bd Gerrit-Change-Number: 17081 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 17 Mar 2021 17:44:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10512: ALTER TABLE ADD PARTITION should bump the write id for ACID tables
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/17081 ) Change subject: IMPALA-10512: ALTER TABLE ADD PARTITION should bump the write id for ACID tables .. Patch Set 4: Thanks for the review, everyone! -- To view, visit http://gerrit.cloudera.org:8080/17081 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iad247008b7c206db00516326c1447bd00a9b34bd Gerrit-Change-Number: 17081 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 17 Mar 2021 17:44:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10512: ALTER TABLE ADD PARTITION should bump the write id for ACID tables
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/17081 ) Change subject: IMPALA-10512: ALTER TABLE ADD PARTITION should bump the write id for ACID tables .. Patch Set 4: Code-Review+2 +1 Bumping up to +2 using Csaba's vote from earlier. -- To view, visit http://gerrit.cloudera.org:8080/17081 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iad247008b7c206db00516326c1447bd00a9b34bd Gerrit-Change-Number: 17081 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 17 Mar 2021 17:23:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10512: ALTER TABLE ADD PARTITION should bump the write id for ACID tables
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/17081 ) Change subject: IMPALA-10512: ALTER TABLE ADD PARTITION should bump the write id for ACID tables .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/17081/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/17081/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3145 PS3, Line 3145: addHmsPartitionsInTransaction > alterTableRecoverPartitions is not supported for transactional tables in Im Thanks for the clarification. I just wanted to make sure we are covered here. -- To view, visit http://gerrit.cloudera.org:8080/17081 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iad247008b7c206db00516326c1447bd00a9b34bd Gerrit-Change-Number: 17081 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 17 Mar 2021 16:54:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Revert "IMPALA-10503: testdata load hits hive memory limit errors during hive inserts"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17191 ) Change subject: Revert "IMPALA-10503: testdata load hits hive memory limit errors during hive inserts" .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6975/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17191 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I896c7b2457d537fa1bfe8dc29063da0b7b3df199 Gerrit-Change-Number: 17191 Gerrit-PatchSet: 1 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 17 Mar 2021 16:50:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9234: Support Ranger row filtering policies
Aman Sinha has posted comments on this change. (
http://gerrit.cloudera.org:8080/16976 )
Change subject: IMPALA-9234: Support Ranger row filtering policies
..
Patch Set 10:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/16976/9/fe/src/main/java/org/apache/impala/authorization/TableMask.java
File fe/src/main/java/org/apache/impala/authorization/TableMask.java:
http://gerrit.cloudera.org:8080/#/c/16976/9/fe/src/main/java/org/apache/impala/authorization/TableMask.java@109
PS9, Line 109: String stmtSql = String.format("SELECT * FROM %s.%s WHERE
%s",
> Note that only the WHERE clause is returned. So "SELECT *" is used for
> simplicity.
That's a good point...but in that case could we just say SELECT 1 ? Not a big
deal.
Right, I was thinking about IMPALA-9223 in my comment above.
http://gerrit.cloudera.org:8080/#/c/16976/9/fe/src/main/java/org/apache/impala/authorization/TableMask.java@111
PS9, Line 111: SelectStmt selectStmt = (SelectStmt) Parser.parse(stmtSql);
> Yes, good question. For example if table functional.alltypestiny has an ill
I would think that the admin user who has access to the row filter definition
should see the parse exception details such that he can fix it but for the
normal user there should be some level of obscuring of the error message
..otherwise it would reveal the filter details and in any case the normal user
won't be able to fix it by himself.
BTW, in the above execution I assume you were the privileged user. But it
sounds like the behavior would have been the same if you were not.
I am ok if you want to create an enhancement JIRA for this and see what makes
sense for Impala. It is probably not a blocker for the feature. I think the
Hive message is also problematic ..it just happened to show less for this case
but what if the row filter subquery had some syntax issue ?
EDIT: I see later that you mention both Impala and Hive will show the full
rewritten query in the EXPLAIN plan and the query profile. This contains the
expanded row filter even for the normal user .. so I am not sure if the parse
exception is that relevant. I would be curious to know how other products like
Spark deal with such things. The row filter could have things like Salary or
Social Security number predicate etc which should ideally be hidden.
--
To view, visit http://gerrit.cloudera.org:8080/16976
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I580517be241225ca15e45686381b78890178d7cc
Gerrit-Change-Number: 16976
Gerrit-PatchSet: 10
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 17 Mar 2021 16:01:21 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10558: Implement ds theta exclude() function
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17153 ) Change subject: IMPALA-10558: Implement ds_theta_exclude() function .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6974/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I05119fd8c652c07ff248a99e44b0da3541e46ca3 Gerrit-Change-Number: 17153 Gerrit-PatchSet: 5 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 17 Mar 2021 15:50:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10558: Implement ds theta exclude() function
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/17153 ) Change subject: IMPALA-10558: Implement ds_theta_exclude() function .. Patch Set 5: Code-Review+2 Thanks for the changes! -- To view, visit http://gerrit.cloudera.org:8080/17153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I05119fd8c652c07ff248a99e44b0da3541e46ca3 Gerrit-Change-Number: 17153 Gerrit-PatchSet: 5 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 17 Mar 2021 15:50:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10558: Implement ds theta exclude() function
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17153 ) Change subject: IMPALA-10558: Implement ds_theta_exclude() function .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8383/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I05119fd8c652c07ff248a99e44b0da3541e46ca3 Gerrit-Change-Number: 17153 Gerrit-PatchSet: 5 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 17 Mar 2021 15:35:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10455: Reorder Maven repositories for cleaner mirror semantics
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/17020 ) Change subject: IMPALA-10455: Reorder Maven repositories for cleaner mirror semantics .. Patch Set 2: Code-Review+1 Hi Joe, nice improvement. LGTM! -- To view, visit http://gerrit.cloudera.org:8080/17020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7046c7ec5391833e98ee6a463fb8c08b6a04cb26 Gerrit-Change-Number: 17020 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Wed, 17 Mar 2021 14:21:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10558: Implement ds theta exclude() function
Fucun Chu has posted comments on this change. ( http://gerrit.cloudera.org:8080/17153 ) Change subject: IMPALA-10558: Implement ds_theta_exclude() function .. Patch Set 5: (4 comments) Thanks for the review! Addressed the comments. http://gerrit.cloudera.org:8080/#/c/17153/4/be/src/exprs/datasketches-common.cc File be/src/exprs/datasketches-common.cc: http://gerrit.cloudera.org:8080/#/c/17153/4/be/src/exprs/datasketches-common.cc@54 PS4, Line 54: bool DeserializeDsSketch( > Could you please comment that this is a specialization of the template Dese Done http://gerrit.cloudera.org:8080/#/c/17153/4/be/src/exprs/datasketches-functions-ir.cc File be/src/exprs/datasketches-functions-ir.cc: http://gerrit.cloudera.org:8080/#/c/17153/4/be/src/exprs/datasketches-functions-ir.cc@148 PS4, Line 148: > nit: I don't think this comment and the one below adds much. The comment ab Done http://gerrit.cloudera.org:8080/#/c/17153/3/be/src/exprs/datasketches-functions.h File be/src/exprs/datasketches-functions.h: http://gerrit.cloudera.org:8080/#/c/17153/3/be/src/exprs/datasketches-functions.h@74 PS3, Line 74: sketches. If they ar > nit: "...sketches. If they are not..." Done http://gerrit.cloudera.org:8080/#/c/17153/4/testdata/workloads/functional-query/queries/QueryTest/datasketches-theta.test File testdata/workloads/functional-query/queries/QueryTest/datasketches-theta.test: http://gerrit.cloudera.org:8080/#/c/17153/4/testdata/workloads/functional-query/queries/QueryTest/datasketches-theta.test@398 PS4, Line 398: ch. : create table ske > Does this mean that with this test A and B has no common items so the resul Done -- To view, visit http://gerrit.cloudera.org:8080/17153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I05119fd8c652c07ff248a99e44b0da3541e46ca3 Gerrit-Change-Number: 17153 Gerrit-PatchSet: 5 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 17 Mar 2021 14:20:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10558: Implement ds theta exclude() function
Fucun Chu has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/17153 ) Change subject: IMPALA-10558: Implement ds_theta_exclude() function .. IMPALA-10558: Implement ds_theta_exclude() function This function receives two strings that are serialized Apache DataSketches Theta sketches. Computes the a-not-b set operation given two sketches of same or different column. Example: select ds_theta_estimate(ds_theta_exclude(sketch1, sketch2)) from sketch_tbl; +---+ | ds_theta_estimate(ds_theta_exclude(sketch1, sketch2)) | +---+ | 5 | +---+ Change-Id: I05119fd8c652c07ff248a99e44b0da3541e46ca3 --- M be/src/exprs/datasketches-common.cc M be/src/exprs/datasketches-functions-ir.cc M be/src/exprs/datasketches-functions.h M common/function-registry/impala_functions.py M testdata/workloads/functional-query/queries/QueryTest/datasketches-theta.test 5 files changed, 169 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/17153/5 -- To view, visit http://gerrit.cloudera.org:8080/17153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I05119fd8c652c07ff248a99e44b0da3541e46ca3 Gerrit-Change-Number: 17153 Gerrit-PatchSet: 5 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9234: Support Ranger row filtering policies
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/16976 )
Change subject: IMPALA-9234: Support Ranger row filtering policies
..
Patch Set 10:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/16976/9/fe/src/main/java/org/apache/impala/authorization/TableMask.java
File fe/src/main/java/org/apache/impala/authorization/TableMask.java:
http://gerrit.cloudera.org:8080/#/c/16976/9/fe/src/main/java/org/apache/impala/authorization/TableMask.java@109
PS9, Line 109: String stmtSql = String.format("SELECT * FROM %s.%s WHERE
%s",
> I think you mean IMPALA-9223. It's commented in InlineViewRef.createTableMa
Update: I'm going to fix IMPALA-9661 so don't need IMPALA-9223
--
To view, visit http://gerrit.cloudera.org:8080/16976
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I580517be241225ca15e45686381b78890178d7cc
Gerrit-Change-Number: 16976
Gerrit-PatchSet: 10
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 17 Mar 2021 09:09:29 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9234: Support Ranger row filtering policies
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16976 ) Change subject: IMPALA-9234: Support Ranger row filtering policies .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8382/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16976 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I580517be241225ca15e45686381b78890178d7cc Gerrit-Change-Number: 16976 Gerrit-PatchSet: 10 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 17 Mar 2021 08:28:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10558: Implement ds theta exclude() function
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/17153 ) Change subject: IMPALA-10558: Implement ds_theta_exclude() function .. Patch Set 4: (4 comments) http://gerrit.cloudera.org:8080/#/c/17153/4/be/src/exprs/datasketches-common.cc File be/src/exprs/datasketches-common.cc: http://gerrit.cloudera.org:8080/#/c/17153/4/be/src/exprs/datasketches-common.cc@54 PS4, Line 54: bool DeserializeDsSketch( Could you please comment that this is a specialization of the template DeserializeDsSketch() for theta sketches? http://gerrit.cloudera.org:8080/#/c/17153/4/be/src/exprs/datasketches-functions-ir.cc File be/src/exprs/datasketches-functions-ir.cc: http://gerrit.cloudera.org:8080/#/c/17153/4/be/src/exprs/datasketches-functions-ir.cc@148 PS4, Line 148: // A is not null nit: I don't think this comment and the one below adds much. The comment above explains well the functionality here. http://gerrit.cloudera.org:8080/#/c/17153/3/be/src/exprs/datasketches-functions.h File be/src/exprs/datasketches-functions.h: http://gerrit.cloudera.org:8080/#/c/17153/3/be/src/exprs/datasketches-functions.h@74 PS3, Line 74: sketch. If it is not nit: "...sketches. If they are not..." http://gerrit.cloudera.org:8080/#/c/17153/4/testdata/workloads/functional-query/queries/QueryTest/datasketches-theta.test File testdata/workloads/functional-query/queries/QueryTest/datasketches-theta.test: http://gerrit.cloudera.org:8080/#/c/17153/4/testdata/workloads/functional-query/queries/QueryTest/datasketches-theta.test@398 PS4, Line 398: check that it is equal to the : # estimate of a. Does this mean that with this test A and B has no common items so the result of A-not-B equals to the cardinality of A? If yes, do you think it would make sense to add a test where there are some common items between the two input sets but not all the items are common? -- To view, visit http://gerrit.cloudera.org:8080/17153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I05119fd8c652c07ff248a99e44b0da3541e46ca3 Gerrit-Change-Number: 17153 Gerrit-PatchSet: 4 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 17 Mar 2021 08:12:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9234: Support Ranger row filtering policies
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/16976 )
Change subject: IMPALA-9234: Support Ranger row filtering policies
..
Patch Set 10:
(4 comments)
Thank Aman!
http://gerrit.cloudera.org:8080/#/c/16976/9/be/src/util/backend-gflag-util.cc
File be/src/util/backend-gflag-util.cc:
http://gerrit.cloudera.org:8080/#/c/16976/9/be/src/util/backend-gflag-util.cc@158
PS9, Line 158: DEFINE_bool(enable_row_filtering, true,
> Since enabling this flag depends on enabling column masking, it would be us
Done
http://gerrit.cloudera.org:8080/#/c/16976/9/fe/src/main/java/org/apache/impala/authorization/TableMask.java
File fe/src/main/java/org/apache/impala/authorization/TableMask.java:
http://gerrit.cloudera.org:8080/#/c/16976/9/fe/src/main/java/org/apache/impala/authorization/TableMask.java@109
PS9, Line 109: String stmtSql = String.format("SELECT * FROM %s.%s WHERE
%s",
> My understanding is the SELECT * will need to be changed to only project co
I think you mean IMPALA-9223. It's commented in
InlineViewRef.createTableMaskView() where we actually create the table mask
view.
This query is just for parsing the row filter string into AST. Note that only
the WHERE clause is returned. So "SELECT *" is used for simplicity.
http://gerrit.cloudera.org:8080/#/c/16976/9/fe/src/main/java/org/apache/impala/authorization/TableMask.java@111
PS9, Line 111: SelectStmt selectStmt = (SelectStmt) Parser.parse(stmtSql);
> Since this will throw an AnalysisException (or ParseException) for a malfor
Yes, good question. For example if table functional.alltypestiny has an illegal
row filter like "10 id = int_col", query on it will fail as
[localhost:21050] default> select * from functional.alltypestiny;
Query: select * from functional.alltypestiny
Query submitted at: 2021-03-17 14:46:27 (Coordinator:
http://quanlong-OptiPlex-BJ:25000)
ERROR: ParseException: Syntax error in line 1:
SELECT * FROM functional.alltypestiny WHERE 10 id = int_col
^
Encountered: IDENTIFIER
Expected: AND, BETWEEN, DIV, EXCEPT, GROUP, HAVING, ILIKE, IN, INTERSECT,
IREGEXP, IS, LIKE, LIMIT, ||, MINUS, NOT, OFFSET, OR, ORDER, REGEXP, RLIKE,
UNION
CAUSED BY: Exception: Syntax error
I tried it in Hive as well, the failure is similar:
0: jdbc:hive2://localhost:11050> select * from functional.alltypestiny;
Error: Error while compiling statement: FAILED: SemanticException
org.apache.hadoop.hive.ql.parse.ParseException: line 1:484 missing ) at 'id'
near 'id'
line 1:487 missing EOF at '=' near 'id' (state=42000,code=4)
We will expose more details than Hive since we have a more friendly error
message. But it can help user to correct the error. I'm not sure whether we
should swallow the details. I think it's undefined on the error behavior for
illegal row filter. Do you think we should file a JIRA for both Hive and Impala?
http://gerrit.cloudera.org:8080/#/c/16976/9/testdata/workloads/functional-query/queries/QueryTest/ranger_row_filtering.test
File
testdata/workloads/functional-query/queries/QueryTest/ranger_row_filtering.test:
http://gerrit.cloudera.org:8080/#/c/16976/9/testdata/workloads/functional-query/queries/QueryTest/ranger_row_filtering.test@1
PS9, Line 1:
> Thanks for adding these tests. I had a couple of suggestions to the test c
Good questions!
1. The test at line 67 is a related test. Currently we don't have a formatted
error message. The raw cause is exposed.
2. Done for WITH clause. Borrowing some tests in ranger_column_masking.test.
For the VIEWs, the tests on alltypes_view and masked_view are related. Should I
add more?
3. Currently both Hive and Impala will show the real plan containing the
masking expressions. So Impala runtime profile will has them as well. I think
it's ok to have them, which helps the end user to understand the query results.
--
To view, visit http://gerrit.cloudera.org:8080/16976
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I580517be241225ca15e45686381b78890178d7cc
Gerrit-Change-Number: 16976
Gerrit-PatchSet: 10
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 17 Mar 2021 08:08:25 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9234: Support Ranger row filtering policies
Hello Aman Sinha, Fang-Yu Rao, Tim Armstrong, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16976 to look at the new patch set (#10). Change subject: IMPALA-9234: Support Ranger row filtering policies .. IMPALA-9234: Support Ranger row filtering policies Ranger row filtering policies provide customized expressions to filter out rows for specific users when reading from a table. This patch adds support for this feature. A new feature flag, enable_row_filtering, is added to disable this experimental feature. It defaults to be true so the feature is enabled by default. Enabling row-filtering requires --enable_column_masking=true since it depends on the column masking implementation. Note that row filtering policies take effects prior to any column masking policies, because column masking policies apply on result data. Implementation: The existing table masking view infrastructure can be extended to support row filtering. Currently when analyzing a table with column masking policies, we replace the TableRef with an InlineViewRef which contains a SelectStmt wrapping the columns with masking expressions. This patch adds the row filtering expressions to the WhereClause of the SelectStmt. Limitations: - Expressions using subqueries are not supported (IMPALA-10483). - Row filtering policies on nested tables will not be applied when nested collection columns are used directly in the FROM clause. This will leak data so we forbid such kinds of queries until IMPALA-10484 is resolved. Tests: - Add FE test for error message when disabling row filtering. - Add e2e test with row filtering policies. - Add e2e test with column masking and row filtering policies both take place. - Verified audits in a CDP cluster with Ranger and Solr set up. Change-Id: I580517be241225ca15e45686381b78890178d7cc --- M be/src/common/global-flags.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/TableMask.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationContext.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/util/AuthorizationUtil.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M testdata/workloads/functional-query/queries/QueryTest/ranger_column_masking.test A testdata/workloads/functional-query/queries/QueryTest/ranger_column_masking_and_row_filtering.test A testdata/workloads/functional-query/queries/QueryTest/ranger_row_filtering.test M tests/authorization/test_ranger.py 23 files changed, 1,005 insertions(+), 113 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/16976/10 -- To view, visit http://gerrit.cloudera.org:8080/16976 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I580517be241225ca15e45686381b78890178d7cc Gerrit-Change-Number: 16976 Gerrit-PatchSet: 10 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong
