[Impala-ASF-CR] Bump up GBN to 13088653
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17424 ) Change subject: Bump up GBN to 13088653 .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/17424 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3ad6311f0ef1a06391b70d58cdb4e51e08de5ea Gerrit-Change-Number: 17424 Gerrit-PatchSet: 1 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 12 May 2021 05:55:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10401: Enable Ranger audit logs to log4j
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/17416 ) Change subject: IMPALA-10401: Enable Ranger audit logs to log4j .. Patch Set 3: (1 comment) > Patch Set 2: Code-Review+2 > > (2 comments) > > Carry on Csaba's +2 and Fang-Yu's +1 Thanks Quanlong! http://gerrit.cloudera.org:8080/#/c/17416/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17416/1//COMMIT_MSG@16 PS1, Line 16: catalogd, cooridnator > Yes, e.g. for test_ranger.py::TestRanger::test_hive_with_ranger_setup, I ca Thanks Quanlong! -- To view, visit http://gerrit.cloudera.org:8080/17416 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e6850882594f31dc2250fe205fa6cad26a30571 Gerrit-Change-Number: 17416 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 12 May 2021 01:30:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10401: Enable Ranger audit logs to log4j
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17416 ) Change subject: IMPALA-10401: Enable Ranger audit logs to log4j .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7132/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17416 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e6850882594f31dc2250fe205fa6cad26a30571 Gerrit-Change-Number: 17416 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 12 May 2021 01:16:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10401: Enable Ranger audit logs to log4j
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17416 ) Change subject: IMPALA-10401: Enable Ranger audit logs to log4j .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17416 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e6850882594f31dc2250fe205fa6cad26a30571 Gerrit-Change-Number: 17416 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 12 May 2021 01:16:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10401: Enable Ranger audit logs to log4j
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/17416 )
Change subject: IMPALA-10401: Enable Ranger audit logs to log4j
..
Patch Set 2: Code-Review+2
(2 comments)
Carry on Csaba's +2 and Fang-Yu's +1
http://gerrit.cloudera.org:8080/#/c/17416/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/17416/1//COMMIT_MSG@10
PS1, Line 10: provides us
> nit: provides us with
Done
http://gerrit.cloudera.org:8080/#/c/17416/1//COMMIT_MSG@16
PS1, Line 16: catalogd, cooridnator
> According to https://github.com/apache/impala/blame/master/bin/create-test-
Yes, e.g. for test_ranger.py::TestRanger::test_hive_with_ranger_setup, I can
see related logs in hive-server2.log
2021-05-12T09:10:23,084 INFO [org.apache.ranger.audit.queue.AuditBatchQueue0]
audit.log4j:
{"repoType":3,"repo":"test_impala","reqUser":"quanlong","evtTime":"2021-05-12
09:10:21.974","access":"SHOW_ROLES","resType":"@null","action":"select","result":1,"agent":"hiveServer2","policy":-1,"enforcer":"ranger-acl","reqData":"show
roles","agentHost":"quanlong-OptiPlex-BJ","logType":"RangerAudit","id":"da27161a-f988-4772-86b5-b399ab233b04-0","seq_num":1,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"test-cluster"}
2021-05-12T09:10:23,085 INFO [org.apache.ranger.audit.queue.AuditBatchQueue0]
audit.log4j:
{"repoType":3,"repo":"test_impala","reqUser":"quanlong","evtTime":"2021-05-12
09:10:21.977","access":"CUSTOM","resource":"functional/alltypestiny/id","resType":"@column","action":"select","result":1,"agent":"hiveServer2","policy":43,"enforcer":"ranger-acl","sess":"dd77a65c-dbc7-4f22-aeee-f12d7fe4f4f6","cliType":"HIVESERVER2","cliIP":"127.0.0.1","reqData":"select
id from
functional.alltypestiny","agentHost":"quanlong-OptiPlex-BJ","logType":"RangerAudit","id":"605cfc4d-7c25-4f59-97d6-a826cfb0c7ea-0","seq_num":1,"event_count":1,"event_dur_ms":0,"tags":[],"additional_info":"{\"remote-ip-address\":127.0.0.1,
\"forwarded-ip-addresses\":[]","cluster_name":"test-cluster","policy_version":1}
--
To view, visit http://gerrit.cloudera.org:8080/17416
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9e6850882594f31dc2250fe205fa6cad26a30571
Gerrit-Change-Number: 17416
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Comment-Date: Wed, 12 May 2021 01:16:03 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10401: Enable Ranger audit logs to log4j
Hello Fang-Yu Rao, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17416 to look at the new patch set (#2). Change subject: IMPALA-10401: Enable Ranger audit logs to log4j .. IMPALA-10401: Enable Ranger audit logs to log4j This patch enables audit logging to log4j for Ranger plugins, which provides us with more info in debugging flaky test failures. The audits are logged by Log4JAuditDestination. Each line shows which policy takes effect and whether the access is allowed. For FE tests, the log file is in logs/fe_tests/FeSupport.INFO. For e2e and custom cluster tests, the audits are logged in log files of processes that have ranger plugins (e.g. catalogd, cooridnator). Tests: - Manually verify the audit logs exist. Change-Id: I9e6850882594f31dc2250fe205fa6cad26a30571 --- M fe/src/test/resources/ranger-hive-audit.xml 1 file changed, 5 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/16/17416/2 -- To view, visit http://gerrit.cloudera.org:8080/17416 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9e6850882594f31dc2250fe205fa6cad26a30571 Gerrit-Change-Number: 17416 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] Bump up GBN to 13088653
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/17424 ) Change subject: Bump up GBN to 13088653 .. Patch Set 1: Aman just gave a +2 for my patch at https://gerrit.cloudera.org/c/17339/. So I think this patch could be rebased against the master branch after IMPALA-10675 (Exclude hive-exec from ranger-plugins-audit) is merged. :-) -- To view, visit http://gerrit.cloudera.org:8080/17424 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3ad6311f0ef1a06391b70d58cdb4e51e08de5ea Gerrit-Change-Number: 17424 Gerrit-PatchSet: 1 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 12 May 2021 00:48:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10675: Exclude hive-exec from ranger-plugins-audit
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17339 ) Change subject: IMPALA-10675: Exclude hive-exec from ranger-plugins-audit .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7131/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17339 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I459360028e12dabbdbaff2250f1b712b6f165f5c Gerrit-Change-Number: 17339 Gerrit-PatchSet: 2 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 12 May 2021 00:46:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10675: Exclude hive-exec from ranger-plugins-audit
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17339 ) Change subject: IMPALA-10675: Exclude hive-exec from ranger-plugins-audit .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17339 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I459360028e12dabbdbaff2250f1b712b6f165f5c Gerrit-Change-Number: 17339 Gerrit-PatchSet: 2 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 12 May 2021 00:46:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10675: Exclude hive-exec from ranger-plugins-audit
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/17339 ) Change subject: IMPALA-10675: Exclude hive-exec from ranger-plugins-audit .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17339 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I459360028e12dabbdbaff2250f1b712b6f165f5c Gerrit-Change-Number: 17339 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 12 May 2021 00:44:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10413: fix impalad crashes when canceling the retrying query
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16911 ) Change subject: IMPALA-10413: fix impalad crashes when canceling the retrying query .. Patch Set 15: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7130/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16911 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4fd7228acd0a70d33859029052239f9b9f795e5d Gerrit-Change-Number: 16911 Gerrit-PatchSet: 15 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Wed, 12 May 2021 00:44:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10413: fix impalad crashes when canceling the retrying query
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16911 ) Change subject: IMPALA-10413: fix impalad crashes when canceling the retrying query .. Patch Set 15: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16911 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4fd7228acd0a70d33859029052239f9b9f795e5d Gerrit-Change-Number: 16911 Gerrit-PatchSet: 15 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Wed, 12 May 2021 00:44:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10413: fix impalad crashes when canceling the retrying query
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/16911 ) Change subject: IMPALA-10413: fix impalad crashes when canceling the retrying query .. Patch Set 14: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16911 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4fd7228acd0a70d33859029052239f9b9f795e5d Gerrit-Change-Number: 16911 Gerrit-PatchSet: 14 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Wed, 12 May 2021 00:43:30 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump up GBN to 13088653
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/17424 ) Change subject: Bump up GBN to 13088653 .. Patch Set 1: Code-Review+1 Thanks Vihang for working on this! I actually had a patch that excludes hive-exec from ranger-audit-plugin at https://gerrit.cloudera.org/c/17339/. So after your patch is merged, I will abandon mine. -- To view, visit http://gerrit.cloudera.org:8080/17424 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3ad6311f0ef1a06391b70d58cdb4e51e08de5ea Gerrit-Change-Number: 17424 Gerrit-PatchSet: 1 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 12 May 2021 00:43:27 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump up GBN to 13088653
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17424 ) Change subject: Bump up GBN to 13088653 .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8701/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17424 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3ad6311f0ef1a06391b70d58cdb4e51e08de5ea Gerrit-Change-Number: 17424 Gerrit-PatchSet: 1 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 12 May 2021 00:23:43 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump up GBN to 13088653
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17424 ) Change subject: Bump up GBN to 13088653 .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7129/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/17424 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3ad6311f0ef1a06391b70d58cdb4e51e08de5ea Gerrit-Change-Number: 17424 Gerrit-PatchSet: 1 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 12 May 2021 00:02:35 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump up GBN to 13088653
Vihang Karajgaonkar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17424 Change subject: Bump up GBN to 13088653 .. Bump up GBN to 13088653 This change bumps up the CDP_BUILD_NUMBER to 13088653 which includes additional metastore thrift API changes needed for Catalogd's metastore endpoint to be functional. Additionally, this adds a exclusion in the ranger-audit-plugin dependency because it leaks hive-exec jar in the classpath which causes thrift errors during compilation. The compilation error occur because hive-exec leaks 0.9.3 thrift while impala now uses 0.11.0 version of thrift. Testing: 1. Verified that code builds locally. 2. Run core tests. Change-Id: Ie3ad6311f0ef1a06391b70d58cdb4e51e08de5ea --- M bin/impala-config.sh M fe/pom.xml 2 files changed, 20 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/17424/1 -- To view, visit http://gerrit.cloudera.org:8080/17424 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ie3ad6311f0ef1a06391b70d58cdb4e51e08de5ea Gerrit-Change-Number: 17424 Gerrit-PatchSet: 1 Gerrit-Owner: Vihang Karajgaonkar
[native-toolchain-CR] IMPALA-10678: Bump Kudu version to f486f0813a
Wenzhe Zhou has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17423 Change subject: IMPALA-10678: Bump Kudu version to f486f0813a .. IMPALA-10678: Bump Kudu version to f486f0813a This pulls in the latest version of Kudu which includes the patches for KUDU-1884 (Make Kerberos principal customizable). Testing: - Successfully built native-toolchain on Jenkins without publishing dependencies to S3. Change-Id: Ieafa9e5301297cd11032ea9cd5dbfa3fd4fa971a --- M buildall.sh 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/23/17423/1 -- To view, visit http://gerrit.cloudera.org:8080/17423 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ieafa9e5301297cd11032ea9cd5dbfa3fd4fa971a Gerrit-Change-Number: 17423 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-10413: fix impalad crashes when canceling the retrying query
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/16911 ) Change subject: IMPALA-10413: fix impalad crashes when canceling the retrying query .. Patch Set 14: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/16911 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4fd7228acd0a70d33859029052239f9b9f795e5d Gerrit-Change-Number: 16911 Gerrit-PatchSet: 14 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Tue, 11 May 2021 20:09:39 + Gerrit-HasComments: No
[Impala-ASF-CR] Upgrading Python cryptography library
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/17323 ) Change subject: Upgrading Python cryptography library .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/17323/2/infra/python/deps/adls-requirements.txt File infra/python/deps/adls-requirements.txt: http://gerrit.cloudera.org:8080/#/c/17323/2/infra/python/deps/adls-requirements.txt@24 PS2, Line 24: cryptography==3.2 > I don't have one at hand, but I will look into how to run the tests with AD SGTM! :-) -- To view, visit http://gerrit.cloudera.org:8080/17323 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1d35b73dc245781aa36282c6a268390152b63f05 Gerrit-Change-Number: 17323 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 11 May 2021 19:35:33 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10684: Report invalid parameter setting errors during HS2 OpenSession
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/17363 ) Change subject: IMPALA-10684: Report invalid parameter setting errors during HS2 OpenSession .. Patch Set 2: > (3 comments) > > Currently with hs2, any parameters supplied during OpenSession that > fail the parsing check are logged and discarded. This is bad > because the user may have made a simple mistake such as quoting the > parameter value. The user would be able to determine that the > parameter was not set using the set command or looking for the > error in the log. Further confusing matters, the set statement does > not run this check values either - parameters can be set to invalid > values and are not reported until a query is executed. I agree that silently dropping the invalid query options is a bad behavior. My concern is that this has been the behavior for 5+ years, and I have to assume that some users may rely on it. That reliance could be embedded in code, and it is not always easy to modify immediately. Returning an error here could cause existing use cases to be unable to open a session. One option is to use the SUCCESS_WITH_INFO status to return a message saying that some query options were ignored. Another option is we could have a startup parameter that enables the strict mode, though I don't know how we would stage enabling it. -- To view, visit http://gerrit.cloudera.org:8080/17363 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I19a859bf5226feb903017a66f199ca5da916f217 Gerrit-Change-Number: 17363 Gerrit-PatchSet: 2 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Comment-Date: Tue, 11 May 2021 19:25:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10282: Implement ds cpc sketch() and ds cpc estimate() functions
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16656 ) Change subject: IMPALA-10282: Implement ds_cpc_sketch() and ds_cpc_estimate() functions .. IMPALA-10282: Implement ds_cpc_sketch() and ds_cpc_estimate() functions These functions can be used to get cardinality estimates of data using CPC algorithm from Apache DataSketches. ds_cpc_sketch() receives a dataset, e.g. a column from a table, and returns a serialized CPC sketch in string format. This can be written to a table or be fed directly to ds_cpc_estimate() that returns the cardinality estimate for that sketch. Similar to the HLL sketch, the primary use-case for the CPC sketch is for counting distinct values as a stream, and then merging multiple sketches together for a total distinct count. For more details about Apache DataSketches' CPC see: http://datasketches.apache.org/docs/CPC/CPC.html Figures-of-Merit Comparison of the HLL and CPC Sketches see: https://datasketches.apache.org/docs/DistinctCountMeritComparisons.html Testing: - Added some tests running estimates for small datasets where the amount of data is small enough to get the correct results. - Ran manual tests on tpch_parquet.lineitem to compare perfomance with ndv(). Depending on data characteristics ndv() appears 2x-3x faster. CPC gives closer estimate than current ndv(). CPC is more accurate than HLL in some cases Change-Id: I731e66fbadc74bc339c973f4d9337db9b7dd715a Reviewed-on: http://gerrit.cloudera.org:8080/16656 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exprs/aggregate-functions-ir.cc M be/src/exprs/aggregate-functions.h M be/src/exprs/datasketches-common.cc M be/src/exprs/datasketches-common.h M be/src/exprs/datasketches-functions-ir.cc M be/src/exprs/datasketches-functions.h M common/function-registry/impala_functions.py M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java M testdata/data/README A testdata/data/cpc_sketches_from_hive.parquet A testdata/workloads/functional-query/queries/QueryTest/datasketches-cpc.test M tests/query_test/test_datasketches.py 12 files changed, 398 insertions(+), 0 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16656 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I731e66fbadc74bc339c973f4d9337db9b7dd715a Gerrit-Change-Number: 16656 Gerrit-PatchSet: 10 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10282: Implement ds cpc sketch() and ds cpc estimate() functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16656 ) Change subject: IMPALA-10282: Implement ds_cpc_sketch() and ds_cpc_estimate() functions .. Patch Set 9: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16656 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I731e66fbadc74bc339c973f4d9337db9b7dd715a Gerrit-Change-Number: 16656 Gerrit-PatchSet: 9 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 11 May 2021 18:07:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10684: Report invalid parameter setting errors during HS2 OpenSession
Kurt Deschler has posted comments on this change. ( http://gerrit.cloudera.org:8080/17363 ) Change subject: IMPALA-10684: Report invalid parameter setting errors during HS2 OpenSession .. Patch Set 2: More on the mechanics.. In Beeswax server, set options are stored in beeswax::Query.configuration which is a list of strings that can contain anything. Beeswax options are parsed during impala::ImpalaServer::QueryToTQueryContext. With HS2, OpenSession parses the options directly into TQueryOptions so illegal values cannot be preserved. -- To view, visit http://gerrit.cloudera.org:8080/17363 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I19a859bf5226feb903017a66f199ca5da916f217 Gerrit-Change-Number: 17363 Gerrit-PatchSet: 2 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Comment-Date: Tue, 11 May 2021 17:33:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10502: Handle CREATE/DROP events correctly
Sourabh Goyal has posted comments on this change. (
http://gerrit.cloudera.org:8080/17308 )
Change subject: IMPALA-10502: Handle CREATE/DROP events correctly
..
Patch Set 5:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/17308/5/fe/src/main/java/org/apache/impala/catalog/events/DeleteEventLog.java
File fe/src/main/java/org/apache/impala/catalog/events/DeleteEventLog.java:
http://gerrit.cloudera.org:8080/#/c/17308/5/fe/src/main/java/org/apache/impala/catalog/events/DeleteEventLog.java@106
PS5, Line 106: eventLog_ = new TreeMap<>(eventLog_.tailMap(eventId + 1));
nit: Add a debug log statement when doing garbage collection ?
http://gerrit.cloudera.org:8080/#/c/17308/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/17308/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@744
PS5, Line 744: LOG.debug("EventId: {} Table was not added since it
already exists in catalog",
nit: Log table name as well?
http://gerrit.cloudera.org:8080/#/c/17308/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@2257
PS5, Line 2257: .getNextMetastoreEvents(catalog_, eventId,
Does it make more sense to either
use the existing metastore client created at line no. 2247
or
close it so that it is returned to the pool and let
MetastoreEventsProcessor.getNextMetastoreEvents() get a client from pool ?
http://gerrit.cloudera.org:8080/#/c/17308/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@2260
PS5, Line 2260: .equals(event.getEventType()) ||
DropTableEvent.DROP_TABLE_EVENT_TYPE
Why are we filtering DROP_TABLE_EVENT_TYPE here? Isn't it being handled at line
no. 2548?
http://gerrit.cloudera.org:8080/#/c/17308/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3093
PS5, Line 3093: msClient.getHiveClient().createTable(newTable);
Just confirming: For any ddl operations, HMS adds entry to NotificationEvents
table before returning the response to the client? If not, we may have an issue
when executing getNextMetastoreEvents().
--
To view, visit http://gerrit.cloudera.org:8080/17308
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia2c5e96b48abac015240f20295b3ec3b1d71f24a
Gerrit-Change-Number: 17308
Gerrit-PatchSet: 5
Gerrit-Owner: Vihang Karajgaonkar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Sourabh Goyal
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Tue, 11 May 2021 17:09:32 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] Upgrading Python cryptography library
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/17323 ) Change subject: Upgrading Python cryptography library .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/17323/2/infra/python/deps/adls-requirements.txt File infra/python/deps/adls-requirements.txt: http://gerrit.cloudera.org:8080/#/c/17323/2/infra/python/deps/adls-requirements.txt@24 PS2, Line 24: cryptography==3.2 > I don't actually have an ADLS environment set up. Do you? I don't have one at hand, but I will look into how to run the tests with ADLS. Is it ok if I'll run the tests some time this week and will report back? -- To view, visit http://gerrit.cloudera.org:8080/17323 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1d35b73dc245781aa36282c6a268390152b63f05 Gerrit-Change-Number: 17323 Gerrit-PatchSet: 3 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 11 May 2021 14:39:53 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Upgrading Python cryptography library
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/17323 ) Change subject: Upgrading Python cryptography library .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/17323/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17323/2//COMMIT_MSG@7 PS2, Line 7: crptography > typo, missing first y Done http://gerrit.cloudera.org:8080/#/c/17323/2/infra/python/deps/adls-requirements.txt File infra/python/deps/adls-requirements.txt: http://gerrit.cloudera.org:8080/#/c/17323/2/infra/python/deps/adls-requirements.txt@24 PS2, Line 24: cryptography==3.2 > about testing: this change only affects ADLS environments, which is not cov I don't actually have an ADLS environment set up. Do you? -- To view, visit http://gerrit.cloudera.org:8080/17323 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1d35b73dc245781aa36282c6a268390152b63f05 Gerrit-Change-Number: 17323 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 11 May 2021 13:37:16 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Upgrading Python cryptography library
Hello Joe McDonnell, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17323 to look at the new patch set (#3). Change subject: Upgrading Python cryptography library .. Upgrading Python cryptography library For release notes, see https://cryptography.io/en/latest/changelog/ Change-Id: I1d35b73dc245781aa36282c6a268390152b63f05 --- M infra/python/deps/adls-requirements.txt 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/17323/3 -- To view, visit http://gerrit.cloudera.org:8080/17323 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1d35b73dc245781aa36282c6a268390152b63f05 Gerrit-Change-Number: 17323 Gerrit-PatchSet: 3 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-10680: Replace StringToFloatInternal using fast double parser library
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17389 ) Change subject: IMPALA-10680: Replace StringToFloatInternal using fast_double_parser library .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8700/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17389 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic105ad38a2fcbf2fb4e8ae8af6d9a8e251a9c141 Gerrit-Change-Number: 17389 Gerrit-PatchSet: 3 Gerrit-Owner: Amogh Margoor Gerrit-Reviewer: Amogh Margoor Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 11 May 2021 11:59:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10282: Implement ds cpc sketch() and ds cpc estimate() functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16656 ) Change subject: IMPALA-10282: Implement ds_cpc_sketch() and ds_cpc_estimate() functions .. Patch Set 9: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16656 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I731e66fbadc74bc339c973f4d9337db9b7dd715a Gerrit-Change-Number: 16656 Gerrit-PatchSet: 9 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 11 May 2021 12:21:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10282: Implement ds cpc sketch() and ds cpc estimate() functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16656 ) Change subject: IMPALA-10282: Implement ds_cpc_sketch() and ds_cpc_estimate() functions .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7128/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16656 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I731e66fbadc74bc339c973f4d9337db9b7dd715a Gerrit-Change-Number: 16656 Gerrit-PatchSet: 9 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 11 May 2021 12:21:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10282: Implement ds cpc sketch() and ds cpc estimate() functions
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/16656 ) Change subject: IMPALA-10282: Implement ds_cpc_sketch() and ds_cpc_estimate() functions .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16656 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I731e66fbadc74bc339c973f4d9337db9b7dd715a Gerrit-Change-Number: 16656 Gerrit-PatchSet: 8 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 11 May 2021 12:19:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10653: [DOCS] LDAP configuration option changes
Csaba Ringhofer has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17403 ) Change subject: IMPALA-10653: [DOCS] LDAP configuration option changes .. IMPALA-10653: [DOCS] LDAP configuration option changes This commit updates the LDAP docs with the new flag changes introduced mainly in IMPALA-2563 and IMPALA-10161. Change-Id: Ic82c5dcd46fbd09264ae9a85d65e4044b6576800 Reviewed-on: http://gerrit.cloudera.org:8080/17403 Tested-by: Impala Public Jenkins Reviewed-by: Csaba Ringhofer --- M docs/topics/impala_ldap.xml 1 file changed, 298 insertions(+), 10 deletions(-) Approvals: Impala Public Jenkins: Verified Csaba Ringhofer: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/17403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic82c5dcd46fbd09264ae9a85d65e4044b6576800 Gerrit-Change-Number: 17403 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate
[Impala-ASF-CR] IMPALA-10653: [DOCS] LDAP configuration option changes
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/17403 ) Change subject: IMPALA-10653: [DOCS] LDAP configuration option changes .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic82c5dcd46fbd09264ae9a85d65e4044b6576800 Gerrit-Change-Number: 17403 Gerrit-PatchSet: 2 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Tue, 11 May 2021 11:57:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10413: fix impalad crashes when canceling the retrying query
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16911 ) Change subject: IMPALA-10413: fix impalad crashes when canceling the retrying query .. Patch Set 14: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8699/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16911 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4fd7228acd0a70d33859029052239f9b9f795e5d Gerrit-Change-Number: 16911 Gerrit-PatchSet: 14 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Tue, 11 May 2021 11:55:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10680: Replace StringToFloatInternal using fast double parser library
Amogh Margoor has posted comments on this change. ( http://gerrit.cloudera.org:8080/17389 ) Change subject: IMPALA-10680: Replace StringToFloatInternal using fast_double_parser library .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/17389/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17389/2//COMMIT_MSG@13 PS2, Line 13: speed > It would be nice to have some measurements, for example by adding a large n Sure, will try to measure it. Theoretically, this should not improve performance but just improve the accuracy without perf penalty. http://gerrit.cloudera.org:8080/#/c/17389/2//COMMIT_MSG@17 PS2, Line 17: 2. Ran existing Backend tests for correctness. > Can you add some tests to https://github.com/apache/impala/blob/master/be/s I have added them now. Thanks for the suggestion. -- To view, visit http://gerrit.cloudera.org:8080/17389 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic105ad38a2fcbf2fb4e8ae8af6d9a8e251a9c141 Gerrit-Change-Number: 17389 Gerrit-PatchSet: 3 Gerrit-Owner: Amogh Margoor Gerrit-Reviewer: Amogh Margoor Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 11 May 2021 11:44:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10680: Replace StringToFloatInternal using fast double parser library
Amogh Margoor has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/17389 ) Change subject: IMPALA-10680: Replace StringToFloatInternal using fast_double_parser library .. IMPALA-10680: Replace StringToFloatInternal using fast_double_parser library StringToFloatInternal is used to parse string into float. It had logic to ensure it is faster than standard functions like strod in many cases, but it was not as accurate. We are replacing it by a third party library named fast_double_parser which is both fast and doesn't sacrifise the accurancy for speed. Testing: 1. Added test to check for accuracy improvement. 2. Ran existing Backend tests for correctness. Change-Id: Ic105ad38a2fcbf2fb4e8ae8af6d9a8e251a9c141 --- M be/src/exprs/expr-test.cc M be/src/util/string-parser.h M testdata/workloads/functional-query/queries/QueryTest/values.test 3 files changed, 39 insertions(+), 67 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/17389/3 -- To view, visit http://gerrit.cloudera.org:8080/17389 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic105ad38a2fcbf2fb4e8ae8af6d9a8e251a9c141 Gerrit-Change-Number: 17389 Gerrit-PatchSet: 3 Gerrit-Owner: Amogh Margoor Gerrit-Reviewer: Amogh Margoor Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10413: fix impalad crashes when canceling the retrying query
Xianqing He has uploaded a new patch set (#14). ( http://gerrit.cloudera.org:8080/16911 ) Change subject: IMPALA-10413: fix impalad crashes when canceling the retrying query .. IMPALA-10413: fix impalad crashes when canceling the retrying query The crash happens when canceling the retrying query. If the original query was unregistered while the new query was being created, it will call HandleRetryFailure to abort the new query. But the status is ok, so when calling Status::AddDetail impalad will crash. After the WaitAsync interface called and before the retry_request_state moved to retried_client_request_state_ , if abort the new retry query, retry_request_state need to call Finalize, otherwise the wait-thread will leak. In some cases like canceled the original query or closed the session we may not create the new query, so we also check whether the query is retried. Tests: Add test in tests/custom_cluster/test_query_retries.py and manually tested 100 times to make sure that there was no Impalad crash Change-Id: I4fd7228acd0a70d33859029052239f9b9f795e5d --- M be/src/runtime/query-driver.cc M be/src/service/client-request-state.h M be/src/service/impala-server.cc M tests/custom_cluster/test_query_retries.py 4 files changed, 106 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/16911/14 -- To view, visit http://gerrit.cloudera.org:8080/16911 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4fd7228acd0a70d33859029052239f9b9f795e5d Gerrit-Change-Number: 16911 Gerrit-PatchSet: 14 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Xianqing He
[Impala-ASF-CR] IMPALA-10653: [DOCS] LDAP configuration option changes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17403 ) Change subject: IMPALA-10653: [DOCS] LDAP configuration option changes .. Patch Set 2: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/630/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/17403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic82c5dcd46fbd09264ae9a85d65e4044b6576800 Gerrit-Change-Number: 17403 Gerrit-PatchSet: 2 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Tue, 11 May 2021 07:36:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10653: [DOCS] LDAP configuration option changes
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/17403 ) Change subject: IMPALA-10653: [DOCS] LDAP configuration option changes .. Patch Set 1: (1 comment) Thank you for the review Csaba, fixed the nit. http://gerrit.cloudera.org:8080/#/c/17403/1/docs/topics/impala_ldap.xml File docs/topics/impala_ldap.xml: http://gerrit.cloudera.org:8080/#/c/17403/1/docs/topics/impala_ldap.xml@479 PS1, Line 479: replace > nit: replaced Done -- To view, visit http://gerrit.cloudera.org:8080/17403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic82c5dcd46fbd09264ae9a85d65e4044b6576800 Gerrit-Change-Number: 17403 Gerrit-PatchSet: 1 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Tue, 11 May 2021 07:29:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10653: [DOCS] LDAP configuration option changes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17403 ) Change subject: IMPALA-10653: [DOCS] LDAP configuration option changes .. Patch Set 2: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/630/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/17403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic82c5dcd46fbd09264ae9a85d65e4044b6576800 Gerrit-Change-Number: 17403 Gerrit-PatchSet: 2 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 11 May 2021 07:29:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10653: [DOCS] LDAP configuration option changes
Hello Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17403 to look at the new patch set (#2). Change subject: IMPALA-10653: [DOCS] LDAP configuration option changes .. IMPALA-10653: [DOCS] LDAP configuration option changes This commit updates the LDAP docs with the new flag changes introduced mainly in IMPALA-2563 and IMPALA-10161. Change-Id: Ic82c5dcd46fbd09264ae9a85d65e4044b6576800 --- M docs/topics/impala_ldap.xml 1 file changed, 298 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/17403/2 -- To view, visit http://gerrit.cloudera.org:8080/17403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic82c5dcd46fbd09264ae9a85d65e4044b6576800 Gerrit-Change-Number: 17403 Gerrit-PatchSet: 2 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10401: Enable Ranger audit logs to log4j
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/17416 ) Change subject: IMPALA-10401: Enable Ranger audit logs to log4j .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17416 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e6850882594f31dc2250fe205fa6cad26a30571 Gerrit-Change-Number: 17416 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 11 May 2021 06:44:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10413: fix impalad crashes when canceling the retrying query
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/16911 ) Change subject: IMPALA-10413: fix impalad crashes when canceling the retrying query .. Patch Set 12: Code-Review+1 (2 comments) Thanks for updating the patch! I'll see if Wenzhe want to have a final look. Left some minor comments on the test. http://gerrit.cloudera.org:8080/#/c/16911/12/tests/custom_cluster/test_query_retries.py File tests/custom_cluster/test_query_retries.py: http://gerrit.cloudera.org:8080/#/c/16911/12/tests/custom_cluster/test_query_retries.py@746 PS12, Line 746: == True nit: Don't need "== True" for bool. http://gerrit.cloudera.org:8080/#/c/16911/12/tests/custom_cluster/test_query_retries.py@774 PS12, Line 774: assert self.cluster.get_first_impalad().service.get_num_in_flight_queries() == 1 This may be flaky since we check it immediately. Could you add a wait before this line? e.g. self.wait_for_state(handle, self.client.QUERY_STATES['RUNNING'], 60) -- To view, visit http://gerrit.cloudera.org:8080/16911 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4fd7228acd0a70d33859029052239f9b9f795e5d Gerrit-Change-Number: 16911 Gerrit-PatchSet: 12 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Tue, 11 May 2021 06:35:49 + Gerrit-HasComments: Yes
