[Impala-ASF-CR] IMPALA-13061: Create query live as external table
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21401 ) Change subject: IMPALA-13061: Create query live as external table .. IMPALA-13061: Create query live as external table Impala determines whether a managed table is transactional based on the 'transactional' table property. It assumes any managed table with transactional=true returns non-null getValidWriteIds. When 'default_transactional_type=insert_only' is set at startup (via default_query_options), impala_query_live is created as a managed table with transactional=true, but SystemTables don't implement getValidWriteIds and are not meant to be transactional. DataSourceTable has a similar problem, and when a JDBC table is created setJdbcDataSourceProperties sets transactional=false. This patch uses CREATE EXTERNAL TABLE sys.impala_Query_live so that it is not created as a managed table and 'transactional' is not set. That avoids creating a SystemTable that Impala can't read (it encounters an IllegalStateException). Change-Id: Ie60a2bd03fabc63c85bcd9fa2489e9d47cd2aa65 Reviewed-on: http://gerrit.cloudera.org:8080/21401 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/service/workload-management.cc M tests/custom_cluster/test_query_live.py 2 files changed, 33 insertions(+), 5 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/21401 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie60a2bd03fabc63c85bcd9fa2489e9d47cd2aa65 Gerrit-Change-Number: 21401 Gerrit-PatchSet: 6 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-13061: Create query live as external table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21401 ) Change subject: IMPALA-13061: Create query live as external table .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21401 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie60a2bd03fabc63c85bcd9fa2489e9d47cd2aa65 Gerrit-Change-Number: 21401 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Wed, 08 May 2024 05:29:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13058: Init first arrival time and completion time with -1
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21405 ) Change subject: IMPALA-13058: Init first_arrival_time_ and completion_time_ with -1 .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21405 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1176e2118bb03414ab35049f50009ff0e8c63f58 Gerrit-Change-Number: 21405 Gerrit-PatchSet: 5 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 08 May 2024 04:39:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13058: Init first arrival time and completion time with -1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21405 ) Change subject: IMPALA-13058: Init first_arrival_time_ and completion_time_ with -1 .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10617/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/21405 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1176e2118bb03414ab35049f50009ff0e8c63f58 Gerrit-Change-Number: 21405 Gerrit-PatchSet: 5 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 08 May 2024 04:43:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13058: Init first arrival time and completion time with -1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21405 ) Change subject: IMPALA-13058: Init first_arrival_time_ and completion_time_ with -1 .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/16127/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21405 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1176e2118bb03414ab35049f50009ff0e8c63f58 Gerrit-Change-Number: 21405 Gerrit-PatchSet: 5 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 08 May 2024 04:30:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13058: Init first arrival time and completion time with -1
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21405 ) Change subject: IMPALA-13058: Init first_arrival_time_ and completion_time_ with -1 .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/21405/4/be/src/service/client-request-state.cc File be/src/service/client-request-state.cc: http://gerrit.cloudera.org:8080/#/c/21405/4/be/src/service/client-request-state.cc@123 PS4, Line 123: profile_->AddChild(summary_profile_); > I'd like to revert this back to query_events_->Start(), since it already mo Done. This change still pass test_basic_filters in ARM. -- To view, visit http://gerrit.cloudera.org:8080/21405 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1176e2118bb03414ab35049f50009ff0e8c63f58 Gerrit-Change-Number: 21405 Gerrit-PatchSet: 5 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 08 May 2024 04:07:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13058: Init first arrival time and completion time with -1
Hello Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21405 to look at the new patch set (#5). Change subject: IMPALA-13058: Init first_arrival_time_ and completion_time_ with -1 .. IMPALA-13058: Init first_arrival_time_ and completion_time_ with -1 Impala run over ARM machine shows 'arch_sys_counter' clock source being used rather than more precise 'tsc'. This cause MonotonicStopwatch::Now() to use 'CLOCK_MONOTONIC_COARSE' rather than 'CLOCK_MONOTONIC'. This is what printed near the beginning of impalad log: I0506 13:49:15.429359 355337 init.cc:600] OS distribution: Red Hat Enterprise Linux 8.8 (Ootpa) OS version: Linux version 4.18.0-477.15.1.el8_8.aarch64 ... Clock: clocksource: 'arch_sys_counter', clockid_t: CLOCK_MONOTONIC_COARSE This difference in clock source causes test failure in test_runtime_filters.py::TestRuntimeFilters::test_basic_filters. This patch fixes the issue by initializing first_arrival_time_ and completion_time_ fields of Coordinator::FilterState with -1 and accept 0 as valid value for those fields. query_events_ initialization and start are also moved to the beginning of ClientRequestState's contructor. Testing: - Tweak row_regex pattern in runtime_filters.test. - Loop and pass test_runtime_filters.py in exhaustive mode 3 times in ARM machine. Change-Id: I1176e2118bb03414ab35049f50009ff0e8c63f58 --- M be/src/runtime/coordinator-filter-state.h M be/src/runtime/coordinator.cc M be/src/service/client-request-state.cc M testdata/workloads/functional-query/queries/QueryTest/runtime_filters.test 4 files changed, 21 insertions(+), 18 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/21405/5 -- To view, visit http://gerrit.cloudera.org:8080/21405 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1176e2118bb03414ab35049f50009ff0e8c63f58 Gerrit-Change-Number: 21405 Gerrit-PatchSet: 5 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-13058: Init first arrival time and completion time with -1
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21405 ) Change subject: IMPALA-13058: Init first_arrival_time_ and completion_time_ with -1 .. Patch Set 4: (1 comment) > Patch Set 4: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10615/ ubuntu-20.04-from-scratch job fail doing apt-get install 15:54:13 + sudo apt-get --yes install git 15:54:13 E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 2804 (apt-get) 15:54:13 E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it? http://gerrit.cloudera.org:8080/#/c/21405/4/be/src/service/client-request-state.cc File be/src/service/client-request-state.cc: http://gerrit.cloudera.org:8080/#/c/21405/4/be/src/service/client-request-state.cc@123 PS4, Line 123: query_events_->Start(query_events_start_time); I'd like to revert this back to query_events_->Start(), since it already moved to beginning of constructor and calling MonotonicStopWatch::Now() twice feels overkill. -- To view, visit http://gerrit.cloudera.org:8080/21405 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1176e2118bb03414ab35049f50009ff0e8c63f58 Gerrit-Change-Number: 21405 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 08 May 2024 04:04:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13058: Init first arrival time and completion time with -1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21405 ) Change subject: IMPALA-13058: Init first_arrival_time_ and completion_time_ with -1 .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10615/ -- To view, visit http://gerrit.cloudera.org:8080/21405 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1176e2118bb03414ab35049f50009ff0e8c63f58 Gerrit-Change-Number: 21405 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 08 May 2024 03:58:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12607: Bump the GBN and fetch events specific to the db/table from the metastore
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20979 ) Change subject: IMPALA-12607: Bump the GBN and fetch events specific to the db/table from the metastore .. Patch Set 10: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10614/ -- To view, visit http://gerrit.cloudera.org:8080/20979 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6aecd5108b31c24e6e2c6f9fba6d4d44a3b00729 Gerrit-Change-Number: 20979 Gerrit-PatchSet: 10 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Wed, 08 May 2024 01:11:18 + Gerrit-HasComments: No
[Impala-ASF-CR](branch-3.4.2) IMPALA-12362: (part-4/4) Refactor linux packaging related cmake files.
Hello Impala Public Jenkins, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/21410 to review the following change. Change subject: IMPALA-12362: (part-4/4) Refactor linux packaging related cmake files. .. IMPALA-12362: (part-4/4) Refactor linux packaging related cmake files. Independent linux packaging related content to package/CMakeLists.txt to make it more clearly. This patch also add LICENSE and NOTICE file in the final package. Testing: - Manually deploy package on Ubuntu22.04 and verify it. Backport note for 3.4.x: - Resolved conflicts in CMakeLists.txt and modified package/CMakeLists.txt accordingly. Change-Id: If3914dcda69f81a735cdf70d76c59fa09454777b Reviewed-on: http://gerrit.cloudera.org:8080/20263 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M .gitignore M CMakeLists.txt M NOTICE.txt A package/CMakeLists.txt 4 files changed, 136 insertions(+), 101 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/21410/1 -- To view, visit http://gerrit.cloudera.org:8080/21410 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: branch-3.4.2 Gerrit-MessageType: newchange Gerrit-Change-Id: If3914dcda69f81a735cdf70d76c59fa09454777b Gerrit-Change-Number: 21410 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Xiang Yang
[Impala-ASF-CR] IMPALA-13061: Create query live as external table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21401 ) Change subject: IMPALA-13061: Create query live as external table .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21401 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie60a2bd03fabc63c85bcd9fa2489e9d47cd2aa65 Gerrit-Change-Number: 21401 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Wed, 08 May 2024 00:24:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13061: Create query live as external table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21401 ) Change subject: IMPALA-13061: Create query live as external table .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10616/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/21401 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie60a2bd03fabc63c85bcd9fa2489e9d47cd2aa65 Gerrit-Change-Number: 21401 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Wed, 08 May 2024 00:24:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/21382 )
Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
..
Patch Set 8:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/21382/7/be/src/util/jwt-util.cc
File be/src/util/jwt-util.cc:
http://gerrit.cloudera.org:8080/#/c/21382/7/be/src/util/jwt-util.cc@320
PS7, Line 320:
: Status status;
: JWTPublicKey* jwt_pub_key = nullptr;
: try {
: if (algorithm == "hs256") {
: jwt_pub_key = new HS256JWTPublicKey(algorithm,
it_k->second);
: } else if (algorithm == "hs
What's the reason to remove x5c?
http://gerrit.cloudera.org:8080/#/c/21382/8/be/src/util/jwt-util.cc
File be/src/util/jwt-util.cc:
http://gerrit.cloudera.org:8080/#/c/21382/8/be/src/util/jwt-util.cc@356
PS8, Line 356: if (it_n == kv_map.end() || it_e == kv_map.end()) {
: return Status("'n' and 'e' properties are required");
: } else if (it_n->second.empty() || it_e->second.empty()) {
: return Status("'n' and 'e' properties must be a non-empty
string");
: }
Are "n" and "e" optional when 'x5c' is provided?
http://gerrit.cloudera.org:8080/#/c/21382/8/be/src/util/jwt-util.cc@381
PS8, Line 381: return Status(Substitute("Invalid x5c certificate"));
add { } around return statement.
Should we free bio before return?
http://gerrit.cloudera.org:8080/#/c/21382/8/be/src/util/jwt-util.cc@407
PS8, Line 407:
nit: extra indent spaces.
--
To view, visit http://gerrit.cloudera.org:8080/21382
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74
Gerrit-Change-Number: 21382
Gerrit-PatchSet: 8
Gerrit-Owner: gaurav singh
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
Gerrit-Comment-Date: Wed, 08 May 2024 00:12:45 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12934: Added Calcite parsing files to Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21194 ) Change subject: IMPALA-12934: Added Calcite parsing files to Impala .. Patch Set 10: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If756b5ea8beb85661a30fb5d029e74ebb6719767 Gerrit-Change-Number: 21194 Gerrit-PatchSet: 10 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Wed, 08 May 2024 00:11:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13058: Init first arrival time and completion time with -1
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21405 ) Change subject: IMPALA-13058: Init first_arrival_time_ and completion_time_ with -1 .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21405 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1176e2118bb03414ab35049f50009ff0e8c63f58 Gerrit-Change-Number: 21405 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 07 May 2024 22:49:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13058: Init first arrival time and completion time with -1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21405 ) Change subject: IMPALA-13058: Init first_arrival_time_ and completion_time_ with -1 .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10615/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/21405 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1176e2118bb03414ab35049f50009ff0e8c63f58 Gerrit-Change-Number: 21405 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 07 May 2024 22:53:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13058: Init first arrival time and completion time with -1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21405 ) Change subject: IMPALA-13058: Init first_arrival_time_ and completion_time_ with -1 .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21405 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1176e2118bb03414ab35049f50009ff0e8c63f58 Gerrit-Change-Number: 21405 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 07 May 2024 22:53:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21382 ) Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/16126/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 9 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Tue, 07 May 2024 22:50:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13058: Init first arrival time and completion time with -1
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21405 ) Change subject: IMPALA-13058: Init first_arrival_time_ and completion_time_ with -1 .. Patch Set 3: (4 comments) http://gerrit.cloudera.org:8080/#/c/21405/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21405/2//COMMIT_MSG@19 PS2, Line 19: cause > nit: causes Done http://gerrit.cloudera.org:8080/#/c/21405/2//COMMIT_MSG@21 PS2, Line 21: fix > nit: fixes Done http://gerrit.cloudera.org:8080/#/c/21405/2//COMMIT_MSG@22 PS2, Line 22: wi > nit: are Removed. http://gerrit.cloudera.org:8080/#/c/21405/2//COMMIT_MSG@25 PS2, Line 25: ar > nit: are Done -- To view, visit http://gerrit.cloudera.org:8080/21405 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1176e2118bb03414ab35049f50009ff0e8c63f58 Gerrit-Change-Number: 21405 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 07 May 2024 22:42:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13058: Init first arrival time and completion time with -1
Hello Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21405 to look at the new patch set (#3). Change subject: IMPALA-13058: Init first_arrival_time_ and completion_time_ with -1 .. IMPALA-13058: Init first_arrival_time_ and completion_time_ with -1 Impala run over ARM machine shows 'arch_sys_counter' clock source being used rather than more precise 'tsc'. This cause MonotonicStopwatch::Now() to use 'CLOCK_MONOTONIC_COARSE' rather than 'CLOCK_MONOTONIC'. This is what printed near the beginning of impalad log: I0506 13:49:15.429359 355337 init.cc:600] OS distribution: Red Hat Enterprise Linux 8.8 (Ootpa) OS version: Linux version 4.18.0-477.15.1.el8_8.aarch64 ... Clock: clocksource: 'arch_sys_counter', clockid_t: CLOCK_MONOTONIC_COARSE This difference in clock source causes test failure in test_runtime_filters.py::TestRuntimeFilters::test_basic_filters. This patch fixes the issue by initializing first_arrival_time_ and completion_time_ fields of Coordinator::FilterState with -1 and accept 0 as valid value for those fields. query_events_ initialization and start are also moved to the beginning of ClientRequestState's contructor. Testing: - Tweak row_regex pattern in runtime_filters.test. - Loop and pass test_runtime_filters.py in exhaustive mode 3 times in ARM machine. Change-Id: I1176e2118bb03414ab35049f50009ff0e8c63f58 --- M be/src/runtime/coordinator-filter-state.h M be/src/runtime/coordinator.cc M be/src/service/client-request-state.cc M testdata/workloads/functional-query/queries/QueryTest/runtime_filters.test 4 files changed, 22 insertions(+), 18 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/21405/3 -- To view, visit http://gerrit.cloudera.org:8080/21405 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1176e2118bb03414ab35049f50009ff0e8c63f58 Gerrit-Change-Number: 21405 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-13058: Init first arrival time and completion time with -1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21405 ) Change subject: IMPALA-13058: Init first_arrival_time_ and completion_time_ with -1 .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/16125/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21405 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1176e2118bb03414ab35049f50009ff0e8c63f58 Gerrit-Change-Number: 21405 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 07 May 2024 22:38:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13058: Init first arrival time and completion time with -1
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21405 ) Change subject: IMPALA-13058: Init first_arrival_time_ and completion_time_ with -1 .. Patch Set 2: Code-Review+1 (4 comments) Thanks to fix this issue. http://gerrit.cloudera.org:8080/#/c/21405/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21405/2//COMMIT_MSG@19 PS2, Line 19: cause nit: causes http://gerrit.cloudera.org:8080/#/c/21405/2//COMMIT_MSG@21 PS2, Line 21: fix nit: fixes http://gerrit.cloudera.org:8080/#/c/21405/2//COMMIT_MSG@22 PS2, Line 22: is nit: are http://gerrit.cloudera.org:8080/#/c/21405/2//COMMIT_MSG@25 PS2, Line 25: is nit: are -- To view, visit http://gerrit.cloudera.org:8080/21405 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1176e2118bb03414ab35049f50009ff0e8c63f58 Gerrit-Change-Number: 21405 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 07 May 2024 22:34:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12935: First pass on Calcite planner functions
Steve Carlin has posted comments on this change. (
http://gerrit.cloudera.org:8080/21357 )
Change subject: IMPALA-12935: First pass on Calcite planner functions
..
Patch Set 7:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21357/7/java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/FunctionSignature.java
File
java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/FunctionSignature.java:
http://gerrit.cloudera.org:8080/#/c/21357/7/java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/FunctionSignature.java@61
PS7, Line 61: abstract public class FunctionSignature {
> This is something I wish I had a good answer for and I'll have to figure th
Ok, I started implementing it using the Impala resolver, and I hit an issue...
In the FunctionDetailStatics module, there is a mapping of Calcite function
names to Impala function names. One of the unit tests that failed was with the
"cast" statement.
Impala knows each of its casting functions by a very specific name, like
"casttobigint", "casttoint", etc...
At resolving time for validation, we'd have to put in some special logic to
find the right function. This is one situation where I would prefer to use
Calcite's validation as opposed to Impala's. There's a bunch of work and
coding outside of the existing Impala logic to get the right type, and it would
be a "cast" specific logic.
Then, if we used the Impala resolver, we'd still have to get it back to a
Calcite operator. We could theoretically use a generic SqlKind.OTHER_FUNCTION
operator, but this would cause us to lose some Calcite optimization code
depending on the right type. So I don't think that would be right. Which
means we'd have to map each "casttobigint", etc.. function back to a Calcite
operator.
This is still doable, and perhaps this is the route to go. But then we also
need a translator after optimization which brings it back to the Impala
function from the Calcite operator. I was able to reuse the "function" logic
in its current iteration, but for the cast case, we would again need some
special logic to map Calcite to Impala.
I think the logic I encoded here seems pretty generic. I can't say I will be
able to get away with never special casing things. And indeed, the
FunctionDetailStatics object is a special case. But that's just a simple map
lumping all "cast" functions together, and I think the code is pretty small
there.
It's probably worth noting that "case" will prove to be a problem in the future
because the Impala signature is different from the Calcite signature. But
either way, special case logic will be needed for this.
I hope this makes sense. I'm open for discussion here. I can try to code it
the other way and see what it looks like, but I feel like the code here is
fairly compact? And generic?
--
To view, visit http://gerrit.cloudera.org:8080/21357
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2dd4e402d69ee10547abeeafe893164ffd789b88
Gerrit-Change-Number: 21357
Gerrit-PatchSet: 7
Gerrit-Owner: Steve Carlin
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Steve Carlin
Gerrit-Comment-Date: Tue, 07 May 2024 22:28:52 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21382 to look at the new patch set (#9). Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys This enables the jwt verification using the x5c certificate(s) in the RSA jwks keys. The x5c claim can be part of the jwks either as a string or an array. This patch only supports a single x5c certificate per jwk. If the "x5c" is present and "alg" is not present, then "alg" is extracted from the "x5c" certificate using the signature algorithm. However, if "x5c" is not preseent, then "alg" is a mandatory field on jwk. Current mapping of signature algorithm string => algorithm: sha224WithRSAEncryption => rs224 sha256WithRSAEncryption => rs256 sha384WithRSAEncryption => rs384 sha512WithRSAEncryption => rs512 If "x5c" is present, then it is given priority over other mandatory fields like "n", "e" to construct the public key. Testing: * added unit test case VerifyJwtTokenWithx5cCertificate to verify jwt with x5c certificate. * added unit test case VerifyJwtTokenWithx5cCertificateWithoutAlg to verify jwt with x5c certificate without "alg". Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 --- M be/src/util/jwt-util-test.cc M be/src/util/jwt-util.cc 2 files changed, 256 insertions(+), 16 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/9 -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 9 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21382 ) Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. Patch Set 9: (3 comments) http://gerrit.cloudera.org:8080/#/c/21382/9/be/src/util/jwt-util-test.cc File be/src/util/jwt-util-test.cc: http://gerrit.cloudera.org:8080/#/c/21382/9/be/src/util/jwt-util-test.cc@325 PS9, Line 325: std::string rsa_pub_key_jwk_n_1 = "5dIMi_SgqaF7CZbwWgVLCUwILxYW4LAY6cU-ptsb9H4LRgwcIGoj77jJwpU1P5GJCm_HNRk5DHnSqfWHDOex1k5Pcqhk8ukAZzDMWwCWDcFkOA26-Kikgugtys2MLPwasr_DgvTQDsqiW7XaeIjm0Y8mnrfjy018sLrtNsbckYNwftWgDjYFFQ8kubuezUg-KxGfq8N9DXtXaEgpVpjA6hHe9svHI8d3gKp9B3AMUkOjDTJjZO_zPBUA9w0zNRH9BuaB8iSMO1pmPoMbg_N_Oq_wpLMCDc2nTMDmz5U0nQDfAUc3nba6oG_g_yuKYts4QoriFboxV-jP4bBr4-4NjPRPTEfIhLh1gmPX60CiEfiUx9w9bJ6CaetKiqGudagc57BK_UT9rrRp4jwqt_iWPmV9CSvL5ebYkmacujdMkW0ZmN1y3QOXykc4XLAd3lK5k7a_csI2V-y5ekDL1MonLmxk6I4aiRUG77r76KbPT6AjFxRN8enCdkIT6IvPgb1HWIrK7YwxXvmIK4ELzzGvwqTqQySQxLNklUXGrgmTlHaiwsGcpTbltAoCI1j_JffT-5dcxnk_FST4ZgAWMjzPkbTWA2pgJVDgqkaoM_4D4xHjHrpUE7x9ZQKgEwAF9aH7ZauqOFaKkTrNjN3gF6j4b7CwXk5gqG_uXGvPOzJHD-s"; line too long (720 > 90) http://gerrit.cloudera.org:8080/#/c/21382/9/be/src/util/jwt-util-test.cc@329 PS9, Line 329: std::string rsa_pub_key_jwk_x5c = R"(MIIE2jCCAsICAQEwDQYJKoZIhvcNAQELBQAwMzELMAkGA1UEBhMCVVMxEDAOBgNVBAoMB0pXVC1DUFAxEjAQBgNVBAMMCWxvY2FsaG9zdDAeFw0yMzEyMjIxMzIzNTdaFw0zMzEyMTkxMzIzNTdaMDMxCzAJBgNVBAYTAlVTMRAwDgYDVQQKDAdKV1QtQ1BQMRIwEAYDVQQDDAlsb2NhbGhvc3QwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDl0gyL9KCpoXsJlvBaBUsJTAgvFhbgsBjpxT6m2xv0fgtGDBwgaiPvuMnClTU/kYkKb8c1GTkMedKp9YcM57HWTk9yqGTy6QBnMMxbAJYNwWQ4Dbr4qKSC6C3KzYws/Bqyv8OC9NAOyqJbtdp4iObRjyaet+PLTXywuu02xtyRg3B+1aAONgUVDyS5u57NSD4rEZ+rw30Ne1doSClWmMDqEd72y8cjx3eAqn0HcAxSQ6MNMmNk7/M8FQD3DTM1Ef0G5oHyJIw7WmY+gxuD8386r/CkswINzadMwObPlTSdAN8BRzedtrqgb+D/K4pi2zhCiuIVujFX6M/hsGvj7g2M9E9MR8iEuHWCY9frQKIR+JTH3D1snoJp60qKoa51qBznsEr9RP2utGniPCq3+JY+ZX0JK8vl5tiSZpy6N0yRbRmY3XLdA5fKRzhcsB3eUrmTtr9ywjZX7Ll6QMvUyicubGTojhqJFQbvuvvops9PoCMXFE3x6cJ2QhPoi8+BvUdYisrtjDFe+YgrgQvPMa/CpOpDJJDEs2SVRcauCZOUdqLCwZylNuW0CgIjWP8l99P7l1zGeT8VJPhmABYyPM+RtNYDamAlUOCqRqgz/gPjEeMeulQTvH1lAqATAAX1oftlq6o4VoqROs2M3eAXqPhvsLBeTmCob+5ca887MkcP6wIDAQABMA0GCSqGSIb3DQEBCwUAA4ICAQBW2kREK4hlzxCDqykxrwfbQpiPwrbFmn+3RDJla+pI4L3wrvYT1nU96guFIU3zKnbMzqwPMRUCUjadr2jKxAmMWxCd/ThHQB+ne5xTvx7/6RVQfGjyMCG/SZtSH8/aO7ILNRtPT+SL5ZZwezaqv6gD89tSXB/w/0pYXy70wDuU17KCrTsKSISWGJ1cKi5l2R/m/ZaGjcV8U8NcFepF2bX3u/i0zhaqOqjiwrSEt7fWGDLabPs6n7GtfibZROEDZ/h0JrDINC+6mSfTOYAMJvGjeHA3H/NvzqR+CJgpXGCqElqVuBF0HdxPmwRRBoZC/BLIEcz0VHmB4rcpfaV47TZT+J+04fHYp4Y1S0u112CDrDe+61cDrnbDHC7aGX0G93pYSBKAB1e3LLc9rXQgf2F0pRtFB3rgZA9MtJ+TL7DUvY4VXJNq3v7UolIdldYRdk21YqAS2Hp0fivvFoEk2P/WbwDEErxR0FkZ/JQoI9FMJ9AvDxa4MsFFtlQVInfD2HUu+nhnuEAA8R6L+F2XqhfLY/H7H31iFBK6UCuqptED71VwWHqfBsAPRhLXAqGco7Ln2dzioyj0QdwJqQQIqigltSYtXxfIMLW0BekQ5yln7QTxnZlobkPHUW9s3NK+OMLuKCzVREzjic/aioQP3cRBMXkG2deMwrk3aX8yJuz4gA==)"; line too long (1706 > 90) http://gerrit.cloudera.org:8080/#/c/21382/9/be/src/util/jwt-util-test.cc@329 PS9, Line 329: std::string rsa_pub_key_jwk_x5c = R"(MIIE2jCCAsICAQEwDQYJKoZIhvcNAQELBQAwMzELMAkGA1UEBhMCVVMxEDAOBgNVBAoMB0pXVC1DUFAxEjAQBgNVBAMMCWxvY2FsaG9zdDAeFw0yMzEyMjIxMzIzNTdaFw0zMzEyMTkxMzIzNTdaMDMxCzAJBgNVBAYTAlVTMRAwDgYDVQQKDAdKV1QtQ1BQMRIwEAYDVQQDDAlsb2NhbGhvc3QwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDl0gyL9KCpoXsJlvBaBUsJTAgvFhbgsBjpxT6m2xv0fgtGDBwgaiPvuMnClTU/kYkKb8c1GTkMedKp9YcM57HWTk9yqGTy6QBnMMxbAJYNwWQ4Dbr4qKSC6C3KzYws/Bqyv8OC9NAOyqJbtdp4iObRjyaet+PLTXywuu02xtyRg3B+1aAONgUVDyS5u57NSD4rEZ+rw30Ne1doSClWmMDqEd72y8cjx3eAqn0HcAxSQ6MNMmNk7/M8FQD3DTM1Ef0G5oHyJIw7WmY+gxuD8386r/CkswINzadMwObPlTSdAN8BRzedtrqgb+D/K4pi2zhCiuIVujFX6M/hsGvj7g2M9E9MR8iEuHWCY9frQKIR+JTH3D1snoJp60qKoa51qBznsEr9RP2utGniPCq3+JY+ZX0JK8vl5tiSZpy6N0yRbRmY3XLdA5fKRzhcsB3eUrmTtr9ywjZX7Ll6QMvUyicubGTojhqJFQbvuvvops9PoCMXFE3x6cJ2QhPoi8+BvUdYisrtjDFe+YgrgQvPMa/CpOpDJJDEs2SVRcauCZOUdqLCwZylNuW0CgIjWP8l99P7l1zGeT8VJPhmABYyPM+RtNYDamAlUOCqRqgz/gPjEeMeulQTvH1lAqATAAX1oftlq6o4VoqROs2M3eAXqPhvsLBeTmCob+5ca887MkcP6wIDAQABMA0GCSqGSIb3DQEBCwUAA4ICAQBW2kREK4hlzxCDqykxrwfbQpiPwrbFmn+3RDJla+pI4L3wrvYT1nU96guFIU3zKnbMzqwPMRUCUjadr2jKxAmMWxCd/ThHQB+ne5xTvx7/6RVQfGjyMCG/SZtSH8/aO7ILNRtPT+SL5ZZwezaqv6gD89tSXB/w/0pYXy70wDuU17KCrTsKSISWGJ1cKi5l2R/m/ZaGjcV8U8NcFepF2bX3u/i0zhaqOqjiwrSEt7fWGDLabPs6n7GtfibZROEDZ/h0JrDINC+6mSfTOYAMJvGjeHA3H/NvzqR+CJgpXGCqElqVuBF0HdxPmwRRBoZC/BLIEcz0VHmB4rcpfaV47TZT+J+04fHYp4Y1S0u112CDrDe+61cDrnbDHC7aGX0G93pYSBKAB1e3LLc9rXQgf2F0pRtFB3rgZA9MtJ+TL7DUvY4VXJNq3v7UolIdldYRdk21YqAS2Hp0fivvFoEk2P/WbwDEErxR0FkZ/JQoI9FMJ9AvDxa4MsFFtlQVInfD2HUu+nhnuEAA8R6L+F2XqhfLY/H7H31iFBK6UCuqptED71VwWHqfBsAPRhLXAqGco7Ln2dzioyj0QdwJqQQIqigltSYtXxfIMLW0BekQ5yln7QTxnZlobkPHUW9s3NK+OMLuKCzVREzjic/aioQP3cRBMXkG2deMwrk3aX8yJuz4gA==)"; tab used for whitespace -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit
[Impala-ASF-CR] IMPALA-13058: Init first arrival time and completion time with -1
Hello Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21405 to look at the new patch set (#2). Change subject: IMPALA-13058: Init first_arrival_time_ and completion_time_ with -1 .. IMPALA-13058: Init first_arrival_time_ and completion_time_ with -1 Impala run over ARM machine shows 'arch_sys_counter' clock source being used rather than more precise 'tsc'. This cause MonotonicStopwatch::Now() to use 'CLOCK_MONOTONIC_COARSE' rather than 'CLOCK_MONOTONIC'. This is what printed near the beginning of impalad log: I0506 13:49:15.429359 355337 init.cc:600] OS distribution: Red Hat Enterprise Linux 8.8 (Ootpa) OS version: Linux version 4.18.0-477.15.1.el8_8.aarch64 ... Clock: clocksource: 'arch_sys_counter', clockid_t: CLOCK_MONOTONIC_COARSE This difference in clock source cause test failure in test_runtime_filters.py::TestRuntimeFilters::test_basic_filters. This patch fix the issue by initializing first_arrival_time_ and completion_time_ fields of Coordinator::FilterState is with -1 and accept 0 as valid value for those fields. query_events_ initialization and start is also moved to the beginning of ClientRequestState's contructor. Testing: - Tweak row_regex pattern in runtime_filters.test. - Loop and pass test_runtime_filters.py in exhaustive mode 3 times in ARM machine. Change-Id: I1176e2118bb03414ab35049f50009ff0e8c63f58 --- M be/src/runtime/coordinator-filter-state.h M be/src/runtime/coordinator.cc M be/src/service/client-request-state.cc M testdata/workloads/functional-query/queries/QueryTest/runtime_filters.test 4 files changed, 22 insertions(+), 18 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/21405/2 -- To view, visit http://gerrit.cloudera.org:8080/21405 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1176e2118bb03414ab35049f50009ff0e8c63f58 Gerrit-Change-Number: 21405 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21383 ) Change subject: IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/16124/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I156d1f0c694b91ba34be70bc53ae9bacf924b3b9 Gerrit-Change-Number: 21383 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 07 May 2024 22:17:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21383 ) Change subject: IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote .. Patch Set 3: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I156d1f0c694b91ba34be70bc53ae9bacf924b3b9 Gerrit-Change-Number: 21383 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 07 May 2024 22:09:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21383 ) Change subject: IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/21383/2/be/src/service/data-stream-service.cc File be/src/service/data-stream-service.cc: http://gerrit.cloudera.org:8080/#/c/21383/2/be/src/service/data-stream-service.cc@141 PS2, Line 141: bool query_found = false; > nit: could be useful as a hidden config option. Done http://gerrit.cloudera.org:8080/#/c/21383/2/tests/custom_cluster/test_runtime_filter_aggregation.py File tests/custom_cluster/test_runtime_filter_aggregation.py: http://gerrit.cloudera.org:8080/#/c/21383/2/tests/custom_cluster/test_runtime_filter_aggregation.py@115 PS2, Line 115: #The JOIN BUILD fragment in first and third impalads immediately receive EOS > typo: anc -> and Done -- To view, visit http://gerrit.cloudera.org:8080/21383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I156d1f0c694b91ba34be70bc53ae9bacf924b3b9 Gerrit-Change-Number: 21383 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 07 May 2024 21:53:08 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote
Hello Csaba Ringhofer, Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21383 to look at the new patch set (#3). Change subject: IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote .. IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote It is possible to have UpdateFilterFromRemote RPC arrive to an impalad executor before QueryState of the destination query is created or complete initialization. This patch add wait mechanism in UpdateFilterFromRemote RPC endpoint to wait for few miliseconds until QueryState exist and complete initialization. The wait time is fixed at 500ms. If wait time passed and QueryState still not found or initialized, UpdateFilterFromRemote RPC is deemed fail and query execution move on without complete filter. Testing: - Add test_runtime_filter_aggregation.py::TestLateQueryStateInit - Pass exhastive runs of test_runtime_filter_aggregation.py Change-Id: I156d1f0c694b91ba34be70bc53ae9bacf924b3b9 --- M be/src/runtime/query-exec-mgr.cc M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/runtime/runtime-filter-bank.cc M be/src/runtime/runtime-filter-bank.h M be/src/scheduling/scheduler.cc M be/src/service/data-stream-service.cc M be/src/util/network-util.h M common/protobuf/data_stream_service.proto M tests/custom_cluster/test_runtime_filter_aggregation.py 10 files changed, 171 insertions(+), 25 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/21383/3 -- To view, visit http://gerrit.cloudera.org:8080/21383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I156d1f0c694b91ba34be70bc53ae9bacf924b3b9 Gerrit-Change-Number: 21383 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21382 ) Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/16123/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 8 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Tue, 07 May 2024 21:51:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21383 ) Change subject: IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote .. Patch Set 2: Code-Review+1 (2 comments) http://gerrit.cloudera.org:8080/#/c/21383/2/be/src/service/data-stream-service.cc File be/src/service/data-stream-service.cc: http://gerrit.cloudera.org:8080/#/c/21383/2/be/src/service/data-stream-service.cc@141 PS2, Line 141: int32_t min_wait_time_ms = 500; nit: could be useful as a hidden config option. http://gerrit.cloudera.org:8080/#/c/21383/2/tests/custom_cluster/test_runtime_filter_aggregation.py File tests/custom_cluster/test_runtime_filter_aggregation.py: http://gerrit.cloudera.org:8080/#/c/21383/2/tests/custom_cluster/test_runtime_filter_aggregation.py@115 PS2, Line 115: #The JOIN BUILD fragment in first anc third impalads immediately receive EOS typo: anc -> and -- To view, visit http://gerrit.cloudera.org:8080/21383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I156d1f0c694b91ba34be70bc53ae9bacf924b3b9 Gerrit-Change-Number: 21383 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 07 May 2024 21:25:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21382 ) Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. Patch Set 8: (8 comments) http://gerrit.cloudera.org:8080/#/c/21382/8/be/src/util/jwt-util-test.cc File be/src/util/jwt-util-test.cc: http://gerrit.cloudera.org:8080/#/c/21382/8/be/src/util/jwt-util-test.cc@325 PS8, Line 325: std::string rsa_pub_key_jwk_n_1 = "5dIMi_SgqaF7CZbwWgVLCUwILxYW4LAY6cU-ptsb9H4LRgwcIGoj77jJwpU1P5GJCm_HNRk5DHnSqfWHDOex1k5Pcqhk8ukAZzDMWwCWDcFkOA26-Kikgugtys2MLPwasr_DgvTQDsqiW7XaeIjm0Y8mnrfjy018sLrtNsbckYNwftWgDjYFFQ8kubuezUg-KxGfq8N9DXtXaEgpVpjA6hHe9svHI8d3gKp9B3AMUkOjDTJjZO_zPBUA9w0zNRH9BuaB8iSMO1pmPoMbg_N_Oq_wpLMCDc2nTMDmz5U0nQDfAUc3nba6oG_g_yuKYts4QoriFboxV-jP4bBr4-4NjPRPTEfIhLh1gmPX60CiEfiUx9w9bJ6CaetKiqGudagc57BK_UT9rrRp4jwqt_iWPmV9CSvL5ebYkmacujdMkW0ZmN1y3QOXykc4XLAd3lK5k7a_csI2V-y5ekDL1MonLmxk6I4aiRUG77r76KbPT6AjFxRN8enCdkIT6IvPgb1HWIrK7YwxXvmIK4ELzzGvwqTqQySQxLNklUXGrgmTlHaiwsGcpTbltAoCI1j_JffT-5dcxnk_FST4ZgAWMjzPkbTWA2pgJVDgqkaoM_4D4xHjHrpUE7x9ZQKgEwAF9aH7ZauqOFaKkTrNjN3gF6j4b7CwXk5gqG_uXGvPOzJHD-s"; line too long (720 > 90) http://gerrit.cloudera.org:8080/#/c/21382/8/be/src/util/jwt-util-test.cc@329 PS8, Line 329: std::string rsa_pub_key_jwk_x5c = R"(MIIE2jCCAsICAQEwDQYJKoZIhvcNAQELBQAwMzELMAkGA1UEBhMCVVMxEDAOBgNVBAoMB0pXVC1DUFAxEjAQBgNVBAMMCWxvY2FsaG9zdDAeFw0yMzEyMjIxMzIzNTdaFw0zMzEyMTkxMzIzNTdaMDMxCzAJBgNVBAYTAlVTMRAwDgYDVQQKDAdKV1QtQ1BQMRIwEAYDVQQDDAlsb2NhbGhvc3QwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDl0gyL9KCpoXsJlvBaBUsJTAgvFhbgsBjpxT6m2xv0fgtGDBwgaiPvuMnClTU/kYkKb8c1GTkMedKp9YcM57HWTk9yqGTy6QBnMMxbAJYNwWQ4Dbr4qKSC6C3KzYws/Bqyv8OC9NAOyqJbtdp4iObRjyaet+PLTXywuu02xtyRg3B+1aAONgUVDyS5u57NSD4rEZ+rw30Ne1doSClWmMDqEd72y8cjx3eAqn0HcAxSQ6MNMmNk7/M8FQD3DTM1Ef0G5oHyJIw7WmY+gxuD8386r/CkswINzadMwObPlTSdAN8BRzedtrqgb+D/K4pi2zhCiuIVujFX6M/hsGvj7g2M9E9MR8iEuHWCY9frQKIR+JTH3D1snoJp60qKoa51qBznsEr9RP2utGniPCq3+JY+ZX0JK8vl5tiSZpy6N0yRbRmY3XLdA5fKRzhcsB3eUrmTtr9ywjZX7Ll6QMvUyicubGTojhqJFQbvuvvops9PoCMXFE3x6cJ2QhPoi8+BvUdYisrtjDFe+YgrgQvPMa/CpOpDJJDEs2SVRcauCZOUdqLCwZylNuW0CgIjWP8l99P7l1zGeT8VJPhmABYyPM+RtNYDamAlUOCqRqgz/gPjEeMeulQTvH1lAqATAAX1oftlq6o4VoqROs2M3eAXqPhvsLBeTmCob+5ca887MkcP6wIDAQABMA0GCSqGSIb3DQEBCwUAA4ICAQBW2kREK4hlzxCDqykxrwfbQpiPwrbFmn+3RDJla+pI4L3wrvYT1nU96guFIU3zKnbMzqwPMRUCUjadr2jKxAmMWxCd/ThHQB+ne5xTvx7/6RVQfGjyMCG/SZtSH8/aO7ILNRtPT+SL5ZZwezaqv6gD89tSXB/w/0pYXy70wDuU17KCrTsKSISWGJ1cKi5l2R/m/ZaGjcV8U8NcFepF2bX3u/i0zhaqOqjiwrSEt7fWGDLabPs6n7GtfibZROEDZ/h0JrDINC+6mSfTOYAMJvGjeHA3H/NvzqR+CJgpXGCqElqVuBF0HdxPmwRRBoZC/BLIEcz0VHmB4rcpfaV47TZT+J+04fHYp4Y1S0u112CDrDe+61cDrnbDHC7aGX0G93pYSBKAB1e3LLc9rXQgf2F0pRtFB3rgZA9MtJ+TL7DUvY4VXJNq3v7UolIdldYRdk21YqAS2Hp0fivvFoEk2P/WbwDEErxR0FkZ/JQoI9FMJ9AvDxa4MsFFtlQVInfD2HUu+nhnuEAA8R6L+F2XqhfLY/H7H31iFBK6UCuqptED71VwWHqfBsAPRhLXAqGco7Ln2dzioyj0QdwJqQQIqigltSYtXxfIMLW0BekQ5yln7QTxnZlobkPHUW9s3NK+OMLuKCzVREzjic/aioQP3cRBMXkG2deMwrk3aX8yJuz4gA==)"; line too long (1706 > 90) http://gerrit.cloudera.org:8080/#/c/21382/8/be/src/util/jwt-util-test.cc@329 PS8, Line 329: std::string rsa_pub_key_jwk_x5c = R"(MIIE2jCCAsICAQEwDQYJKoZIhvcNAQELBQAwMzELMAkGA1UEBhMCVVMxEDAOBgNVBAoMB0pXVC1DUFAxEjAQBgNVBAMMCWxvY2FsaG9zdDAeFw0yMzEyMjIxMzIzNTdaFw0zMzEyMTkxMzIzNTdaMDMxCzAJBgNVBAYTAlVTMRAwDgYDVQQKDAdKV1QtQ1BQMRIwEAYDVQQDDAlsb2NhbGhvc3QwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDl0gyL9KCpoXsJlvBaBUsJTAgvFhbgsBjpxT6m2xv0fgtGDBwgaiPvuMnClTU/kYkKb8c1GTkMedKp9YcM57HWTk9yqGTy6QBnMMxbAJYNwWQ4Dbr4qKSC6C3KzYws/Bqyv8OC9NAOyqJbtdp4iObRjyaet+PLTXywuu02xtyRg3B+1aAONgUVDyS5u57NSD4rEZ+rw30Ne1doSClWmMDqEd72y8cjx3eAqn0HcAxSQ6MNMmNk7/M8FQD3DTM1Ef0G5oHyJIw7WmY+gxuD8386r/CkswINzadMwObPlTSdAN8BRzedtrqgb+D/K4pi2zhCiuIVujFX6M/hsGvj7g2M9E9MR8iEuHWCY9frQKIR+JTH3D1snoJp60qKoa51qBznsEr9RP2utGniPCq3+JY+ZX0JK8vl5tiSZpy6N0yRbRmY3XLdA5fKRzhcsB3eUrmTtr9ywjZX7Ll6QMvUyicubGTojhqJFQbvuvvops9PoCMXFE3x6cJ2QhPoi8+BvUdYisrtjDFe+YgrgQvPMa/CpOpDJJDEs2SVRcauCZOUdqLCwZylNuW0CgIjWP8l99P7l1zGeT8VJPhmABYyPM+RtNYDamAlUOCqRqgz/gPjEeMeulQTvH1lAqATAAX1oftlq6o4VoqROs2M3eAXqPhvsLBeTmCob+5ca887MkcP6wIDAQABMA0GCSqGSIb3DQEBCwUAA4ICAQBW2kREK4hlzxCDqykxrwfbQpiPwrbFmn+3RDJla+pI4L3wrvYT1nU96guFIU3zKnbMzqwPMRUCUjadr2jKxAmMWxCd/ThHQB+ne5xTvx7/6RVQfGjyMCG/SZtSH8/aO7ILNRtPT+SL5ZZwezaqv6gD89tSXB/w/0pYXy70wDuU17KCrTsKSISWGJ1cKi5l2R/m/ZaGjcV8U8NcFepF2bX3u/i0zhaqOqjiwrSEt7fWGDLabPs6n7GtfibZROEDZ/h0JrDINC+6mSfTOYAMJvGjeHA3H/NvzqR+CJgpXGCqElqVuBF0HdxPmwRRBoZC/BLIEcz0VHmB4rcpfaV47TZT+J+04fHYp4Y1S0u112CDrDe+61cDrnbDHC7aGX0G93pYSBKAB1e3LLc9rXQgf2F0pRtFB3rgZA9MtJ+TL7DUvY4VXJNq3v7UolIdldYRdk21YqAS2Hp0fivvFoEk2P/WbwDEErxR0FkZ/JQoI9FMJ9AvDxa4MsFFtlQVInfD2HUu+nhnuEAA8R6L+F2XqhfLY/H7H31iFBK6UCuqptED71VwWHqfBsAPRhLXAqGco7Ln2dzioyj0QdwJqQQIqigltSYtXxfIMLW0BekQ5yln7QTxnZlobkPHUW9s3NK+OMLuKCzVREzjic/aioQP3cRBMXkG2deMwrk3aX8yJuz4gA==)"; tab used for whitespace http://gerrit.cloudera.org:8080/#/c/21382/8/be/src/util/jwt-util-test.cc@393 PS8, Line 393:
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
gaurav singh has posted comments on this change. (
http://gerrit.cloudera.org:8080/21382 )
Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
..
Patch Set 8:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/21382/6/be/src/util/jwt-util.cc
File be/src/util/jwt-util.cc:
http://gerrit.cloudera.org:8080/#/c/21382/6/be/src/util/jwt-util.cc@226
PS6, Line 226: }
> nit: extra indent spaces
Done
http://gerrit.cloudera.org:8080/#/c/21382/6/be/src/util/jwt-util.cc@256
PS6, Line 256:
\
: for (size_t i = 0; i < json_value.Size() && i <
MAX_X5C_CERTIFICATES; i++) { \
:
> The for loop block should be indented with 4 spaces.
Done. Thx.
http://gerrit.cloudera.org:8080/#/c/21382/6/be/src/util/jwt-util.cc@323
PS6, Line 323: try {
> Is it possible there is no 'k' property when 'x5c' is provided?
Its not specified in the RFC:
https://datatracker.ietf.org/doc/html/rfc7517#section-4.4
It only says that x5c can be optional.
--
To view, visit http://gerrit.cloudera.org:8080/21382
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74
Gerrit-Change-Number: 21382
Gerrit-PatchSet: 8
Gerrit-Owner: gaurav singh
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
Gerrit-Comment-Date: Tue, 07 May 2024 21:28:01 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21382 to look at the new patch set (#8). Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys .. IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys This enables the jwt verification using the x5c certificate(s) in the RSA jwks keys. The x5c claim can be part of the jwks either as a string or an array. This patch only supports a single x5c certificate per jwk. If the "x5c" is present and "alg" is not present, then "alg" is extracted from the "x5c" certificate using the signature algorithm. However, if "x5c" is not preseent, then "alg" is a mandatory field on jwk. Current mapping of signature algorithm string => algorithm: sha224WithRSAEncryption => rs224 sha256WithRSAEncryption => rs256 sha384WithRSAEncryption => rs384 sha512WithRSAEncryption => rs512 If "x5c" is present, then it is given priority over other mandatory fields like "n", "e" to construct the public key. Testing: * added unit test case VerifyJwtTokenWithx5cCertificate to verify jwt with x5c certificate. * added unit test case VerifyJwtTokenWithx5cCertificateWithoutAlg to verify jwt with x5c certificate without "alg". Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 --- M be/src/util/jwt-util-test.cc M be/src/util/jwt-util.cc 2 files changed, 276 insertions(+), 16 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/8 -- To view, visit http://gerrit.cloudera.org:8080/21382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74 Gerrit-Change-Number: 21382 Gerrit-PatchSet: 8 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-13038: Support profile tab for imported query profiles
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21400 ) Change subject: IMPALA-13038: Support profile tab for imported query profiles .. Patch Set 2: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/21400/2/www/query_profile.tmpl File www/query_profile.tmpl: http://gerrit.cloudera.org:8080/#/c/21400/2/www/query_profile.tmpl@53 PS2, Line 53: nit: indent with two spaces for this function block to keep consistent. -- To view, visit http://gerrit.cloudera.org:8080/21400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddcf2e285abbf42f97bde19014be076ccd6374bc Gerrit-Change-Number: 21400 Gerrit-PatchSet: 2 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 07 May 2024 21:22:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11499: Refactor UrlEncode function to handle special characters
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21131 ) Change subject: IMPALA-11499: Refactor UrlEncode function to handle special characters .. Patch Set 26: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/16122/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88c4aba5d811dfcec809583d0c16fcbc0ca730fb Gerrit-Change-Number: 21131 Gerrit-PatchSet: 26 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zihao Ye Gerrit-Comment-Date: Tue, 07 May 2024 21:16:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11499: Refactor UrlEncode function to handle special characters
pranav.lo...@cloudera.com has uploaded a new patch set (#26). ( http://gerrit.cloudera.org:8080/21131 ) Change subject: IMPALA-11499: Refactor UrlEncode function to handle special characters .. IMPALA-11499: Refactor UrlEncode function to handle special characters An error came from an issue with URL encoding, where certain Unicode characters were being incorrectly encoded due to their UTF-8 representation matching characters in the set of characters to escape. For example, the string '运', which consists of three bytes 0xe8 0xbf 0x90 was wrongly getting encoded into '\E8%FFBF\90', because the middle byte matched one of the two bytes that represented the "\u00FF" literal. Inclusion of "\u00FF" was likely a mistake from the beginning and it should have been '\x7F'. The patch makes three key changes: 1. Before the change, the set of characters that need to be escaped was stored as a string. The current patch uses an unordered_set instead. 2. '\xFF', which is an invalid UTF-8 byte and whose inclusion was erroneous from the beginning, is replaced with '\x7F', which is a control character for DELETE, ensuring consistency and correctness in URL encoding. 3. The list of characters to be escaped is extended to match the current list in Hive. Testing: Tests on both traditional Hive tables and Iceberg tables are included in unicode-column-name.test, insert.test, coding-util-test.cc and test_insert.py. Change-Id: I88c4aba5d811dfcec809583d0c16fcbc0ca730fb --- M be/src/util/coding-util-test.cc M be/src/util/coding-util.cc M testdata/workloads/functional-query/queries/QueryTest/insert.test M testdata/workloads/functional-query/queries/QueryTest/unicode-column-name.test M tests/query_test/test_insert.py 5 files changed, 152 insertions(+), 20 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/21131/26 -- To view, visit http://gerrit.cloudera.org:8080/21131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I88c4aba5d811dfcec809583d0c16fcbc0ca730fb Gerrit-Change-Number: 21131 Gerrit-PatchSet: 26 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zihao Ye
[Impala-ASF-CR] IMPALA-11499: Refactor UrlEncode function to handle special characters
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21131 ) Change subject: IMPALA-11499: Refactor UrlEncode function to handle special characters .. Patch Set 26: (1 comment) http://gerrit.cloudera.org:8080/#/c/21131/26/tests/query_test/test_insert.py File tests/query_test/test_insert.py: http://gerrit.cloudera.org:8080/#/c/21131/26/tests/query_test/test_insert.py@343 PS26, Line 343: = flake8: E225 missing whitespace around operator -- To view, visit http://gerrit.cloudera.org:8080/21131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88c4aba5d811dfcec809583d0c16fcbc0ca730fb Gerrit-Change-Number: 21131 Gerrit-PatchSet: 26 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zihao Ye Gerrit-Comment-Date: Tue, 07 May 2024 20:51:22 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11499: Refactor UrlEncode function to handle special characters
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21131 ) Change subject: IMPALA-11499: Refactor UrlEncode function to handle special characters .. Patch Set 25: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88c4aba5d811dfcec809583d0c16fcbc0ca730fb Gerrit-Change-Number: 21131 Gerrit-PatchSet: 25 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zihao Ye Gerrit-Comment-Date: Tue, 07 May 2024 20:34:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12607: Bump the GBN and fetch events specific to the db/table from the metastore
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20979 ) Change subject: IMPALA-12607: Bump the GBN and fetch events specific to the db/table from the metastore .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/16121/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20979 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6aecd5108b31c24e6e2c6f9fba6d4d44a3b00729 Gerrit-Change-Number: 20979 Gerrit-PatchSet: 10 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 07 May 2024 20:29:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11499: Refactor UrlEncode function to handle special characters
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21131 ) Change subject: IMPALA-11499: Refactor UrlEncode function to handle special characters .. Patch Set 25: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/16120/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88c4aba5d811dfcec809583d0c16fcbc0ca730fb Gerrit-Change-Number: 21131 Gerrit-PatchSet: 25 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zihao Ye Gerrit-Comment-Date: Tue, 07 May 2024 20:21:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11499: Refactor UrlEncode function to handle special characters
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21131 ) Change subject: IMPALA-11499: Refactor UrlEncode function to handle special characters .. Patch Set 24: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/16119/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/21131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88c4aba5d811dfcec809583d0c16fcbc0ca730fb Gerrit-Change-Number: 21131 Gerrit-PatchSet: 24 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zihao Ye Gerrit-Comment-Date: Tue, 07 May 2024 20:18:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12935: First pass on Calcite planner functions
Steve Carlin has posted comments on this change. (
http://gerrit.cloudera.org:8080/21357 )
Change subject: IMPALA-12935: First pass on Calcite planner functions
..
Patch Set 7:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/21357/7/java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/FunctionSignature.java
File
java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/FunctionSignature.java:
http://gerrit.cloudera.org:8080/#/c/21357/7/java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/FunctionSignature.java@61
PS7, Line 61: abstract public class FunctionSignature {
> Impala has logic for this type of matching as part of its Function class. I
This is something I wish I had a good answer for and I'll have to figure this
out. This class has gone through some iterations for me and I started
development on it a few years back.
Upon looking at it, probably the reason I created it this way is because there
is no "Function" class here. It deals with types and name only. But of
course, the "Function" is just the types and the name (and the db, which is
included in the name).
I think it makes sense to keep this class to keep the relationship between
Calcite and Impala Functions, but perhaps the resolving portion should be fixed
up for reuse. I'll need to investigate this.
http://gerrit.cloudera.org:8080/#/c/21357/7/java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaOperatorTable.java
File
java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaOperatorTable.java:
http://gerrit.cloudera.org:8080/#/c/21357/7/java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaOperatorTable.java@50
PS7, Line 50: public class ImpalaOperatorTable extends
ReflectiveSqlOperatorTable {
> If I'm understanding the Calcite code, this is primarily for sets of functi
Ok, this is gonna be a long-winded explanation. And perhaps there is another
approach we can take. But here is my thought process:
Calcite has a lot of built in logic when they resolve functions embedded in
their code. Their focus was to match standard SQL, not really to handle
non-standard SQL functions.
With Calcite's logic, they have an SqlKind built into their resolver, and their
different functions have different SqlKind values
https://javadoc.io/doc/org.apache.calcite/calcite-core/latest/index.html.
Embedded throughout their code are case statements that base their logic on the
SqlKind for the function. This is included with some of their optimization
rules.
So if we were to use purely our function resolver, we would have to map
individual functions to their appropriate SqlKind. I don't know if this is a
good thing or a bad thing, but I guess I went the "lazy" route and figured
let's let Calcite do the resolving and they will produce the right SqlKind
type. It also seems to fit in more with the Calcite way, imo if we ever
decided to standardize Impala a bit more?
Another potential issue is that Calcite and Impala might not necessarily be 1:1
on functions. There might be some standard SQL syntax that validates just fine
on Calcite, translates to Impala, but not in a 1:1 fashion. This could result
in some awkwardness at validation time. I'm not sure I have an example here,
so perhaps this isn't true.
You did ask about UDFs. While I haven't coded that yet, my gut instinct on how
this would be done is through Chained operator tables. Calcite allows lookups
to go through multiple order chained operator tables. Each UDF would be in its
own operator table object and chained on when doing the lookup.
You asked about the existing Impala lookup functionality, which I commented on
in another comment of yours.
--
To view, visit http://gerrit.cloudera.org:8080/21357
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2dd4e402d69ee10547abeeafe893164ffd789b88
Gerrit-Change-Number: 21357
Gerrit-PatchSet: 7
Gerrit-Owner: Steve Carlin
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Steve Carlin
Gerrit-Comment-Date: Tue, 07 May 2024 20:04:35 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13058: Init first arrival time and completion time with -1
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21405 ) Change subject: IMPALA-13058: Init first_arrival_time_ and completion_time_ with -1 .. Patch Set 1: | Maybe a better solution is to Start query_events_ at start_time_us() It's better to keep consistent. Could we run exhaustive test on ARM to verify the fixing? -- To view, visit http://gerrit.cloudera.org:8080/21405 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1176e2118bb03414ab35049f50009ff0e8c63f58 Gerrit-Change-Number: 21405 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 07 May 2024 20:02:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12607: Bump the GBN and fetch events specific to the db/table from the metastore
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/20979 ) Change subject: IMPALA-12607: Bump the GBN and fetch events specific to the db/table from the metastore .. Patch Set 10: (2 comments) http://gerrit.cloudera.org:8080/#/c/20979/9/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java: http://gerrit.cloudera.org:8080/#/c/20979/9/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@466 PS9, Line 466: // db_name and table_name. So it makes sense to fetch all the events and filter > Can you add a comment on why the ACID/external cases are different? Done http://gerrit.cloudera.org:8080/#/c/20979/9/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/20979/9/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@4017 PS9, Line 4017: 3 > testInsertIntoTransactionalTable also does an insert - this doesn't lead to Yeah, correct. db_name/table_name will be null, so we won't fetch those events here. -- To view, visit http://gerrit.cloudera.org:8080/20979 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6aecd5108b31c24e6e2c6f9fba6d4d44a3b00729 Gerrit-Change-Number: 20979 Gerrit-PatchSet: 10 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 07 May 2024 20:00:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12607: Bump the GBN and fetch events specific to the db/table from the metastore
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20979 ) Change subject: IMPALA-12607: Bump the GBN and fetch events specific to the db/table from the metastore .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10614/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/20979 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6aecd5108b31c24e6e2c6f9fba6d4d44a3b00729 Gerrit-Change-Number: 20979 Gerrit-PatchSet: 10 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 07 May 2024 20:00:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12607: Bump the GBN and fetch events specific to the db/table from the metastore
Hello Quanlong Huang, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20979 to look at the new patch set (#10). Change subject: IMPALA-12607: Bump the GBN and fetch events specific to the db/table from the metastore .. IMPALA-12607: Bump the GBN and fetch events specific to the db/table from the metastore Bump the GBN to 49623641 to leverage HIVE-27499, so that Impala can directly fetch the latest events specific to the db/table from the metastore, instead of fetching the events from metastore and then filtering in the cache matching the DbName/TableName. Implementation Details: Currently when a DDL/DML is performed in Impala, we fetch all the events from metastore based on current eventId and then filter them in Impala which can be a bottleneck if the events count is huge. This can be optimized by including db name and/or table name in the notification event request object and then filter by event type in impala. This can provide performance boost on tables that generate a lot of events. Testing: 1) Did some tests in local cluster 2) Added a test case in MetaStoreEventsProcessorTest Change-Id: I6aecd5108b31c24e6e2c6f9fba6d4d44a3b00729 --- M bin/impala-config.sh M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/catalog/metastore/CatalogHmsSyncToLatestEventIdTest.java 4 files changed, 197 insertions(+), 33 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/20979/10 -- To view, visit http://gerrit.cloudera.org:8080/20979 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6aecd5108b31c24e6e2c6f9fba6d4d44a3b00729 Gerrit-Change-Number: 20979 Gerrit-PatchSet: 10 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-11499: Refactor UrlEncode function to handle special characters
pranav.lo...@cloudera.com has posted comments on this change. (
http://gerrit.cloudera.org:8080/21131 )
Change subject: IMPALA-11499: Refactor UrlEncode function to handle special
characters
..
Patch Set 25:
(5 comments)
> Uploaded patch set 25.
http://gerrit.cloudera.org:8080/#/c/21131/23/be/src/util/coding-util-test.cc
File be/src/util/coding-util-test.cc:
http://gerrit.cloudera.org:8080/#/c/21131/23/be/src/util/coding-util-test.cc@94
PS23, Line 94: string test_path = "/home/impala/directory/";
> These strings could also be extracted into variables.
Done
http://gerrit.cloudera.org:8080/#/c/21131/23/tests/query_test/test_insert.py
File tests/query_test/test_insert.py:
http://gerrit.cloudera.org:8080/#/c/21131/23/tests/query_test/test_insert.py@320
PS23, Line 320: is
> "is a"
Done
http://gerrit.cloudera.org:8080/#/c/21131/23/tests/query_test/test_insert.py@320
PS23, Line 320: the result
> "the result line..."
Done
http://gerrit.cloudera.org:8080/#/c/21131/23/tests/query_test/test_insert.py@321
PS23, Line 321: we only verify that the value is present in str
> "we only verify that the value is present in string representation of the w
Done
http://gerrit.cloudera.org:8080/#/c/21131/24/tests/query_test/test_insert.py
File tests/query_test/test_insert.py:
http://gerrit.cloudera.org:8080/#/c/21131/24/tests/query_test/test_insert.py@301
PS24, Line 301: "create table {}(i int) partitioned by (p string)
stored as parquet".format(tbl))
I've added a new test for iceberg, since there are a lot of differences in how
we create, insert and how show partition works in iceberg, I've skipped the
helper function to improve the readabilty. The only thing that was left common
were a few asserts and the strings. I also think that if we're to add any new
tests for iceberg, it'd be easier and more convenient if it'll have a separate
function.
I've an optimized version for the same as well and have shared the same with
you, but I think it just gets more complicated when we'll need to handle
iceberg and parquet tables separately because of their show partition
evaluation.
--
To view, visit http://gerrit.cloudera.org:8080/21131
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I88c4aba5d811dfcec809583d0c16fcbc0ca730fb
Gerrit-Change-Number: 21131
Gerrit-PatchSet: 25
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Daniel Becker
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Zihao Ye
Gerrit-Comment-Date: Tue, 07 May 2024 19:56:02 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11499: Refactor UrlEncode function to handle special characters
pranav.lo...@cloudera.com has uploaded a new patch set (#25). ( http://gerrit.cloudera.org:8080/21131 ) Change subject: IMPALA-11499: Refactor UrlEncode function to handle special characters .. IMPALA-11499: Refactor UrlEncode function to handle special characters An error came from an issue with URL encoding, where certain Unicode characters were being incorrectly encoded due to their UTF-8 representation matching characters in the set of characters to escape. For example, the string '运', which consists of three bytes 0xe8 0xbf 0x90 was wrongly getting encoded into '\E8%FFBF\90', because the middle byte matched one of the two bytes that represented the "\u00FF" literal. Inclusion of "\u00FF" was likely a mistake from the beginning and it should have been '\x7F'. The patch makes three key changes: 1. Before the change, the set of characters that need to be escaped was stored as a string. The current patch uses an unordered_set instead. 2. '\xFF', which is an invalid UTF-8 byte and whose inclusion was erroneous from the beginning, is replaced with '\x7F', which is a control character for DELETE, ensuring consistency and correctness in URL encoding. 3. The list of characters to be escaped is extended to match the current list in Hive. Testing: Tests on both traditional Hive tables and Iceberg tables are included in unicode-column-name.test, insert.test, coding-util-test.cc and test_insert.py. Change-Id: I88c4aba5d811dfcec809583d0c16fcbc0ca730fb --- M be/src/util/coding-util-test.cc M be/src/util/coding-util.cc M testdata/workloads/functional-query/queries/QueryTest/insert.test M testdata/workloads/functional-query/queries/QueryTest/unicode-column-name.test M tests/query_test/test_insert.py 5 files changed, 141 insertions(+), 20 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/21131/25 -- To view, visit http://gerrit.cloudera.org:8080/21131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I88c4aba5d811dfcec809583d0c16fcbc0ca730fb Gerrit-Change-Number: 21131 Gerrit-PatchSet: 25 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zihao Ye
[Impala-ASF-CR] IMPALA-11499: Refactor UrlEncode function to handle special characters
pranav.lo...@cloudera.com has uploaded a new patch set (#24). ( http://gerrit.cloudera.org:8080/21131 ) Change subject: IMPALA-11499: Refactor UrlEncode function to handle special characters .. IMPALA-11499: Refactor UrlEncode function to handle special characters An error came from an issue with URL encoding, where certain Unicode characters were being incorrectly encoded due to their UTF-8 representation matching characters in the set of characters to escape. For example, the string '运', which consists of three bytes 0xe8 0xbf 0x90 was wrongly getting encoded into '\E8%FFBF\90', because the middle byte matched one of the two bytes that represented the "\u00FF" literal. Inclusion of "\u00FF" was likely a mistake from the beginning and it should have been '\x7F'. The patch makes three key changes: 1. Before the change, the set of characters that need to be escaped was stored as a string. The current patch uses an unordered_set instead. 2. '\xFF', which is an invalid UTF-8 byte and whose inclusion was erroneous from the beginning, is replaced with '\x7F', which is a control character for DELETE, ensuring consistency and correctness in URL encoding. 3. The list of characters to be escaped is extended to match the current list in Hive. Testing: Tests on both traditional Hive tables and Iceberg tables are included in unicode-column-name.test, insert.test, coding-util-test.cc and test_insert.py. Change-Id: I88c4aba5d811dfcec809583d0c16fcbc0ca730fb --- M be/src/util/coding-util-test.cc M be/src/util/coding-util.cc M testdata/workloads/functional-query/queries/QueryTest/insert.test M testdata/workloads/functional-query/queries/QueryTest/unicode-column-name.test M tests/query_test/test_insert.py 5 files changed, 141 insertions(+), 20 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/21131/24 -- To view, visit http://gerrit.cloudera.org:8080/21131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I88c4aba5d811dfcec809583d0c16fcbc0ca730fb Gerrit-Change-Number: 21131 Gerrit-PatchSet: 24 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zihao Ye
[Impala-ASF-CR] IMPALA-13018: Fix test tpcds q80a for JDBC table
Abhishek Rawat has posted comments on this change. (
http://gerrit.cloudera.org:8080/21409 )
Change subject: IMPALA-13018: Fix test_tpcds_q80a for JDBC table
..
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/21409/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21409/1//COMMIT_MSG@7
PS1, Line 7: IMPALA-13018: Fix test_tpcds_q80a for JDBC table
Maybe update the title to something generic, since this applies to other
queries also and not just this particular query.
Maybe:
"Block push down of conjuncts with expressions on base column values for jdbc
tables"
http://gerrit.cloudera.org:8080/#/c/21409/1//COMMIT_MSG@17
PS1, Line 17: But casting to Date/Timestamp for a column cannot be pushed down
to JDBC
I'm wondering if this applies to other non Date/Timestamp types also? Can we do
proper pushdown if TBinaryPredicate expects TColumnDesc to be one of the
operands.
struct TBinaryPredicate {
// Column on which the predicate is applied. Always set.
1: optional TColumnDesc col
// Comparison operator. Always set.
2: optional TComparisonOp op
// Value on the right side of the binary predicate. Always set.
3: optional Data.TColumnValue value
}
Basically, if you have predicate like following, can it be properly pushed down?
CAST (stringCol AS INT) >= 1234
--
To view, visit http://gerrit.cloudera.org:8080/21409
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iabd7e28b8d5f11f25a000dc4c9ab65895056b572
Gerrit-Change-Number: 21409
Gerrit-PatchSet: 1
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Tue, 07 May 2024 19:35:23 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13018: Fix test tpcds q80a for JDBC table
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21409 ) Change subject: IMPALA-13018: Fix test_tpcds_q80a for JDBC table .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/21409/1/fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java File fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java: http://gerrit.cloudera.org:8080/#/c/21409/1/fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java@300 PS1, Line 300: castExpr.getType().isDateOrTimeType() > nit: Just want to double check, is CastExpr OK for types other than Date/Ti Other numeric and string types are ok. For future release, we need to add TExpr to TBinaryPredicate. -- To view, visit http://gerrit.cloudera.org:8080/21409 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iabd7e28b8d5f11f25a000dc4c9ab65895056b572 Gerrit-Change-Number: 21409 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 07 May 2024 19:31:01 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13018: Fix test tpcds q80a for JDBC table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21409 ) Change subject: IMPALA-13018: Fix test_tpcds_q80a for JDBC table .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/16118/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21409 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iabd7e28b8d5f11f25a000dc4c9ab65895056b572 Gerrit-Change-Number: 21409 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 07 May 2024 19:22:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13061: Create query live as external table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21401 ) Change subject: IMPALA-13061: Create query live as external table .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/16117/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21401 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie60a2bd03fabc63c85bcd9fa2489e9d47cd2aa65 Gerrit-Change-Number: 21401 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 07 May 2024 19:20:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13018: Fix test tpcds q80a for JDBC table
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21409 ) Change subject: IMPALA-13018: Fix test_tpcds_q80a for JDBC table .. Patch Set 1: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/21409/1/fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java File fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java: http://gerrit.cloudera.org:8080/#/c/21409/1/fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java@300 PS1, Line 300: castExpr.getType().isDateOrTimeType() nit: Just want to double check, is CastExpr OK for types other than Date/Timestamp? Or should it be avoided entirely regardless of expression type? -- To view, visit http://gerrit.cloudera.org:8080/21409 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iabd7e28b8d5f11f25a000dc4c9ab65895056b572 Gerrit-Change-Number: 21409 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 07 May 2024 19:12:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12934: Added Calcite parsing files to Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21194 ) Change subject: IMPALA-12934: Added Calcite parsing files to Impala .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10613/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/21194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If756b5ea8beb85661a30fb5d029e74ebb6719767 Gerrit-Change-Number: 21194 Gerrit-PatchSet: 10 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Tue, 07 May 2024 19:10:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13061: Create query live as external table
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21401 ) Change subject: IMPALA-13061: Create query live as external table .. Patch Set 4: Code-Review+2 (2 comments) http://gerrit.cloudera.org:8080/#/c/21401/3/tests/custom_cluster/test_query_live.py File tests/custom_cluster/test_query_live.py: http://gerrit.cloudera.org:8080/#/c/21401/3/tests/custom_cluster/test_query_live.py@124 PS3, Line 124: 'select tables_queried from sys.impala_query_live where query_id = "' : + result.query_id + '"') > Can you add assertion for other table properties of impala_query_live here? Done http://gerrit.cloudera.org:8080/#/c/21401/3/tests/custom_cluster/test_query_live.py@175 PS3, Line 175: "--use_local_catalog=true", : catalogd_args="--enable_workload_mgmt " : > Switched to CREATE EXTERNAL TABLE, I think that makes sense for System Tabl Done -- To view, visit http://gerrit.cloudera.org:8080/21401 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie60a2bd03fabc63c85bcd9fa2489e9d47cd2aa65 Gerrit-Change-Number: 21401 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 07 May 2024 19:03:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13061: Create query live as external table
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/21401 ) Change subject: IMPALA-13061: Create query live as external table .. Patch Set 4: Code-Review+1 Looks good! -- To view, visit http://gerrit.cloudera.org:8080/21401 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie60a2bd03fabc63c85bcd9fa2489e9d47cd2aa65 Gerrit-Change-Number: 21401 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 07 May 2024 18:59:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13018: Fix test tpcds q80a for JDBC table
Wenzhe Zhou has uploaded this change for review. (
http://gerrit.cloudera.org:8080/21409
Change subject: IMPALA-13018: Fix test_tpcds_q80a for JDBC table
..
IMPALA-13018: Fix test_tpcds_q80a for JDBC table
The query of q80a consists BETWEEN with casting to timestamp in where
clause like:
d_date between cast('2000-08-23' as timestamp)
and (cast('2000-08-23' as timestamp) + interval 30 days)
Between predicate does cast all exprs to compatible types. Planner
generates predicates for DataSourceScanNode as:
CAST(d_date AS TIMESTAMP) >= TIMESTAMP '2000-08-23 00:00:00',
CAST(d_date AS TIMESTAMP) <= TIMESTAMP '2000-09-22 00:00:00'
But casting to Date/Timestamp for a column cannot be pushed down to JDBC
table now. This patch fixes the issue by not adding such conjuncts to
offered predicate list for JDBC table.
Testing:
- Passed all tpcds queries for JDBC tables, including q80a.
- Passed core test
Change-Id: Iabd7e28b8d5f11f25a000dc4c9ab65895056b572
---
M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M tests/query_test/test_tpcds_queries.py
2 files changed, 18 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/09/21409/1
--
To view, visit http://gerrit.cloudera.org:8080/21409
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iabd7e28b8d5f11f25a000dc4c9ab65895056b572
Gerrit-Change-Number: 21409
Gerrit-PatchSet: 1
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Anonymous Coward
[Impala-ASF-CR] IMPALA-13061: Create query live as external table
Jason Fehr has removed a vote on this change. Change subject: IMPALA-13061: Create query live as external table .. Removed Code-Review+1 by Jason Fehr -- To view, visit http://gerrit.cloudera.org:8080/21401 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: Ie60a2bd03fabc63c85bcd9fa2489e9d47cd2aa65 Gerrit-Change-Number: 21401 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-13061: Create query live as external table
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/21401 ) Change subject: IMPALA-13061: Create query live as external table .. Patch Set 4: Code-Review+1 Looks good! -- To view, visit http://gerrit.cloudera.org:8080/21401 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie60a2bd03fabc63c85bcd9fa2489e9d47cd2aa65 Gerrit-Change-Number: 21401 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 07 May 2024 18:58:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13061: Create query live as external table
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21401 ) Change subject: IMPALA-13061: Create query live as external table .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/21401/3/tests/custom_cluster/test_query_live.py File tests/custom_cluster/test_query_live.py: http://gerrit.cloudera.org:8080/#/c/21401/3/tests/custom_cluster/test_query_live.py@175 PS3, Line 175: "--use_local_catalog=true", : catalogd_args="--enable_workload_mgmt " : > I can't actually find a way to detect "transactional" from "DESCRIBE EXTEND Switched to CREATE EXTERNAL TABLE, I think that makes sense for System Tables because Hive can't manage them, and with these options Hive was converting them to external table with purge=true. -- To view, visit http://gerrit.cloudera.org:8080/21401 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie60a2bd03fabc63c85bcd9fa2489e9d47cd2aa65 Gerrit-Change-Number: 21401 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 07 May 2024 18:57:58 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13061: Create query live as external table
Hello Quanlong Huang, Andrew Sherman, Riza Suminto, Jason Fehr, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21401 to look at the new patch set (#4). Change subject: IMPALA-13061: Create query live as external table .. IMPALA-13061: Create query live as external table Impala determines whether a managed table is transactional based on the 'transactional' table property. It assumes any managed table with transactional=true returns non-null getValidWriteIds. When 'default_transactional_type=insert_only' is set at startup (via default_query_options), impala_query_live is created as a managed table with transactional=true, but SystemTables don't implement getValidWriteIds and are not meant to be transactional. DataSourceTable has a similar problem, and when a JDBC table is created setJdbcDataSourceProperties sets transactional=false. This patch uses CREATE EXTERNAL TABLE sys.impala_Query_live so that it is not created as a managed table and 'transactional' is not set. That avoids creating a SystemTable that Impala can't read (it encounters an IllegalStateException). Change-Id: Ie60a2bd03fabc63c85bcd9fa2489e9d47cd2aa65 --- M be/src/service/workload-management.cc M tests/custom_cluster/test_query_live.py 2 files changed, 33 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/21401/4 -- To view, visit http://gerrit.cloudera.org:8080/21401 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie60a2bd03fabc63c85bcd9fa2489e9d47cd2aa65 Gerrit-Change-Number: 21401 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-13051: Speed up, refactor query log tests
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21358 ) Change subject: IMPALA-13051: Speed up, refactor query log tests .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21358 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1127ef041a3e024bf2b262767d56ec5f29bf3855 Gerrit-Change-Number: 21358 Gerrit-PatchSet: 6 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 07 May 2024 18:24:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13061: Set transactional�lse on query live table
Michael Smith has posted comments on this change. (
http://gerrit.cloudera.org:8080/21401 )
Change subject: IMPALA-13061: Set transactional=false on query live table
..
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21401/3/tests/custom_cluster/test_query_live.py
File tests/custom_cluster/test_query_live.py:
http://gerrit.cloudera.org:8080/#/c/21401/3/tests/custom_cluster/test_query_live.py@175
PS3, Line 175: esult = self.client.execute("select * from functional.alltypes",
: fetch_profile_after_close=True)
:
> Can you also assert that 'transactional'='false' in table properties via "D
I can't actually find a way to detect "transactional" from "DESCRIBE EXTENDED"
output. But maybe the right thing to do is "CREATE EXTERNAL TABLE", which
ensures its not a managed table.
--
To view, visit http://gerrit.cloudera.org:8080/21401
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie60a2bd03fabc63c85bcd9fa2489e9d47cd2aa65
Gerrit-Change-Number: 21401
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jason Fehr
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Riza Suminto
Gerrit-Comment-Date: Tue, 07 May 2024 17:52:00 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13058: Init first arrival time and completion time with -1
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21405 ) Change subject: IMPALA-13058: Init first_arrival_time_ and completion_time_ with -1 .. Patch Set 1: > Patch Set 1: > > This patch only help bypassing the flaky test, but does not address the > underlying issue with RuntimeProfile::EventSequence::Start() from: > > be/src/service/client-request-state.cc:140: query_events_->Start(); > Maybe a better solution is to Start query_events_ at start_time_us() query_events_->Start(start_time_us() * 1000); -- To view, visit http://gerrit.cloudera.org:8080/21405 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1176e2118bb03414ab35049f50009ff0e8c63f58 Gerrit-Change-Number: 21405 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 07 May 2024 17:25:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12934: Added Calcite parsing files to Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21194 ) Change subject: IMPALA-12934: Added Calcite parsing files to Impala .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/16115/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If756b5ea8beb85661a30fb5d029e74ebb6719767 Gerrit-Change-Number: 21194 Gerrit-PatchSet: 10 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Tue, 07 May 2024 17:03:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13061: Set transactional�lse on query live table
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/21401 ) Change subject: IMPALA-13061: Set transactional=false on query live table .. Patch Set 3: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21401 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie60a2bd03fabc63c85bcd9fa2489e9d47cd2aa65 Gerrit-Change-Number: 21401 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 07 May 2024 16:59:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12935: First pass on Calcite planner functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21357 ) Change subject: IMPALA-12935: First pass on Calcite planner functions .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/16116/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21357 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2dd4e402d69ee10547abeeafe893164ffd789b88 Gerrit-Change-Number: 21357 Gerrit-PatchSet: 8 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Tue, 07 May 2024 17:03:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12934: Added Calcite parsing files to Impala
Steve Carlin has posted comments on this change. (
http://gerrit.cloudera.org:8080/21194 )
Change subject: IMPALA-12934: Added Calcite parsing files to Impala
..
Patch Set 9:
(2 comments)
I saw your long comment and your follow-up comment. I assume from your
follow-up comment that we should stick with option 2?
My take was this: If we were purely making changes to config.fmpp, option 1
makes more sense. But since Parser.jj has changes, I figured it makes sense to
have all of the parsing information in front of us as opposed to going back and
forth. I don't have a super strong feeling about it, so let me know if we
should file a Jira to at least revisit what we want to do.
As for putting it under the codegen directory...As you can see, I made the
change, and I suppose it makes sense. It's not really a "java" file, so to put
it under src/main/java has a slightly wrong feel to it. My one comment though
is this: When I first started the Impala project, I did a lot of my searching
within a bash shell under the src/main/java/... directory. THe "codegen"
directory feels to me like it's in a side little nook. It took me a little
extra time to find that code when I was trying to understand how the code
works. So I'm not in love with having this code organization. But now that
I'm used to it, and now that I suppose many other projects work this way, it
does make sense to me to put it in the location you suggested.
http://gerrit.cloudera.org:8080/#/c/21194/9/java/calcite-planner/pom.xml
File java/calcite-planner/pom.xml:
http://gerrit.cloudera.org:8080/#/c/21194/9/java/calcite-planner/pom.xml@106
PS9, Line 106:
: org.apache.maven.plugins
: maven-resources-plugin
:
:
: copy-resources
: generate-sources
:
: copy-resources
:
:
:
${project.build.directory}/codegen
:
:
:
src/main/java/org/apache/impala/calcite/parserimpl/codegen
: false
:
:
:
:
:
:
> After a second look, I think I would put the Parser.jj and config.fmpp in s
I did have to put Parser.jj under a separate directory "templates" because it
needs the "jj" files in its own separate directory. It's very sensitive...even
the existence of a ".Parser.jj.swp" file in that directory will be picked up
and cause problems. But this also matches the directory structure in Calcite.
http://gerrit.cloudera.org:8080/#/c/21194/9/java/calcite-planner/src/main/java/org/apache/impala/calcite/parserimpl/codegen/config.fmpp
File
java/calcite-planner/src/main/java/org/apache/impala/calcite/parserimpl/codegen/config.fmpp:
http://gerrit.cloudera.org:8080/#/c/21194/9/java/calcite-planner/src/main/java/org/apache/impala/calcite/parserimpl/codegen/config.fmpp@21
PS9, Line 21: package: "org.apache.calcite.sql.parser.impl",
> Let's make this somewhere in org.apache.impala.
Done
--
To view, visit http://gerrit.cloudera.org:8080/21194
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If756b5ea8beb85661a30fb5d029e74ebb6719767
Gerrit-Change-Number: 21194
Gerrit-PatchSet: 9
Gerrit-Owner: Steve Carlin
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Steve Carlin
Gerrit-Comment-Date: Tue, 07 May 2024 16:51:48 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13061: Set transactional�lse on query live table
Riza Suminto has posted comments on this change. (
http://gerrit.cloudera.org:8080/21401 )
Change subject: IMPALA-13061: Set transactional=false on query live table
..
Patch Set 3:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/21401/3/tests/custom_cluster/test_query_live.py
File tests/custom_cluster/test_query_live.py:
http://gerrit.cloudera.org:8080/#/c/21401/3/tests/custom_cluster/test_query_live.py@124
PS3, Line 124: describe_ext_result = self.execute_query('describe extended
sys.impala_query_live')
: assert len(describe_ext_result.data) == 82
Can you add assertion for other table properties of impala_query_live here?
http://gerrit.cloudera.org:8080/#/c/21401/3/tests/custom_cluster/test_query_live.py@175
PS3, Line 175: esult = self.client.execute("select * from functional.alltypes",
: fetch_profile_after_close=True)
:
Can you also assert that 'transactional'='false' in table properties via
"DESCRIBE EXTENDED" query?
--
To view, visit http://gerrit.cloudera.org:8080/21401
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie60a2bd03fabc63c85bcd9fa2489e9d47cd2aa65
Gerrit-Change-Number: 21401
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jason Fehr
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Riza Suminto
Gerrit-Comment-Date: Tue, 07 May 2024 16:51:38 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13061: Set transactional�lse on query live table
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21401 ) Change subject: IMPALA-13061: Set transactional=false on query live table .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/21401/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21401/1//COMMIT_MSG@10 PS1, Line 10: 'transactional' table property. It assumes any table with > Is the point here is that if default_transactional_type=insert_only then th Done -- To view, visit http://gerrit.cloudera.org:8080/21401 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie60a2bd03fabc63c85bcd9fa2489e9d47cd2aa65 Gerrit-Change-Number: 21401 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 07 May 2024 16:43:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13061: Set transactional�lse on query live table
Hello Quanlong Huang, Andrew Sherman, Riza Suminto, Jason Fehr, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21401 to look at the new patch set (#3). Change subject: IMPALA-13061: Set transactional=false on query live table .. IMPALA-13061: Set transactional=false on query live table Impala determines whether a table is transactional based solely on the 'transactional' table property. It assumes any table with transactional=true returns non-null getValidWriteIds. When 'default_transactional_type=insert_only' is set at startup (via default_query_options), impala_query_live is created with transactional=true, but SystemTables don't implement getValidWriteIds and are not meant to be transactional. DataSourceTable has a similar problem, and when a JDBC table is created setJdbcDataSourceProperties sets transactional=false. This patch sets 'transactional'='false' in tblproperties for impala_query_live to avoid creating a table that Impala can't read (it encounters an IllegalStateException). Change-Id: Ie60a2bd03fabc63c85bcd9fa2489e9d47cd2aa65 --- M be/src/service/workload-management.cc M tests/custom_cluster/test_query_live.py 2 files changed, 18 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/21401/3 -- To view, visit http://gerrit.cloudera.org:8080/21401 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie60a2bd03fabc63c85bcd9fa2489e9d47cd2aa65 Gerrit-Change-Number: 21401 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-13061: Set transactional�lse on query live table
Hello Quanlong Huang, Andrew Sherman, Riza Suminto, Jason Fehr, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21401 to look at the new patch set (#2). Change subject: IMPALA-13061: Set transactional=false on query live table .. IMPALA-13061: Set transactional=false on query live table Impala determines whether a table is transactional based solely on the 'transactional' table property. It assumes any table with transactional=true returns non-null getValidWriteIds. When 'default_transactional_type=insert_only' is set at startup (via default_query_options), impala_query_live is created with transactional=true, but SystemTables don't implement getValidWriteIds and are not meant to be transactional. DataSourceTables have a similar problem, and when a JDBC table is created setJdbcDataSourceProperties sets transactional=false. This patch sets 'transactional'='false' in tblproperties for impala_query_live to avoid creating a table that Impala can't read (it encounters an IllegalStateException). Change-Id: Ie60a2bd03fabc63c85bcd9fa2489e9d47cd2aa65 --- M be/src/service/workload-management.cc M tests/custom_cluster/test_query_live.py 2 files changed, 18 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/21401/2 -- To view, visit http://gerrit.cloudera.org:8080/21401 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie60a2bd03fabc63c85bcd9fa2489e9d47cd2aa65 Gerrit-Change-Number: 21401 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-12935: First pass on Calcite planner functions
Hello Aman Sinha, Joe McDonnell, Csaba Ringhofer, Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21357 to look at the new patch set (#8). Change subject: IMPALA-12935: First pass on Calcite planner functions .. IMPALA-12935: First pass on Calcite planner functions This commit handles the first pass on getting functions to work through the Calcite planner. Only basic functions will work with this commit. Implicit conversions for parameters are not yet supported. Custom UDFs are also not supported yet. The functions are loaded in CalciteJniFrontend from the Impala "builtin" database. A "FunctionSignature" is created for each builtin function. Each function is loaded into Calcite's "ImpalaOperatorTable" object which is used in the validation process. Each Impala function maps to a Calcite "ImpalaOperator" object. For function names that are not an exact match with a Calcite operator, an entry is found in FunctionDetailStatics to do the conversion. When the Calcite validator tries to validate the function, it gets the return type through the "ImpalaOperator.inferReturnType()" method. In this method, a "FunctionSignatureForLookup" signature is created for looking up the stored signatures. The "Lookup" signature does not have a return type (yet), so the matching of signatures will be based on matching name and parameters. As mentioned above, implicit conversion is not yet supported in this commit; this will be added later. After validation is complete, the functions will be in a Calcite format. After the rest of compilation (relnode conversion, optimization) is complete, the function needs to be converted back into Impala form (the Expr object) to eventually get it into its thrift request. In this commit, all functions are converted into Expr starting in the ImpalaProjectRel, since this is the RelNode where functions do their thing. The RexCallConverter and RexLiteralConverter get called via the CreateExprVisitor for this conversion. Since Calcite is providing the analysis portion of the planning, there is no need to go through Impala's Analyzer object. However, the Impala planner requires Expr objects to be analyzed. To get around this, the AnalyzedFunctionCallExpr and AnalyzedNullLiteral objects exist which analyze the expression in the constructor. While this could potentially be combined with the existing FunctionCallExpr and NullLiteral objects, this fits in with the general plan to avoid changing "fe" Impala code as much as we can until much later in the commit cycle. Also, there will be other Analyzed*Expr classes created in the future, but this commit is intended for basic function call expressions only. One minor change to the parser is added with this commit. Calcite parser does not have acknowledge the "string" datatype, so this has been added here in Parser.jj and config.fmpp. Change-Id: I2dd4e402d69ee10547abeeafe893164ffd789b88 --- M java/calcite-planner/src/main/codegen/config.fmpp M java/calcite-planner/src/main/codegen/templates/Parser.jj A java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/AnalyzedFunctionCallExpr.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/AnalyzedNullLiteral.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/FunctionDetailStatics.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/FunctionResolver.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/FunctionSignature.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/FunctionSignatureForLookup.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/FunctionSignatureForStorage.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/RexCallConverter.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/RexLiteralConverter.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaAggregateOperator.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaHelperOperator.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaOperatorTable.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaScalarOperator.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaProjectRel.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/util/CreateExprVisitor.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteJniFrontend.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteValidator.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeConverter.java M testdata/workloads/functional-query/queries/QueryTes
[Impala-ASF-CR] IMPALA-12934: Added Calcite parsing files to Impala
Hello Aman Sinha, Joe McDonnell, Csaba Ringhofer, Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21194 to look at the new patch set (#10). Change subject: IMPALA-12934: Added Calcite parsing files to Impala .. IMPALA-12934: Added Calcite parsing files to Impala Adding the framework to create our own parsing syntax for Impala using the base Calcite Parser.jj file. The Parser.jj file here was grabbed from Calcite 1.36. So with this commit, we are using the same parsing analysis as Calcite 1.36. Any changes made on top of the Parser.jj file or the config.fmpp file in the future are Impala specific changes, so a diff can be done from this commit to see all the Impala parsing changes. The config.fmpp file was grabbed from Calcite 1.36 default_config.fmpp. The Calcite intention of the config.fmpp file is to allow markup of variables in the Parser.jj file. So it is always preferable to modify the default_config.fmpp file when possible. Our version is grabbed from https://github.com/apache/calcite/blob/main/core/src/main/codegen/config.fmpp and slightly modified with the class name to make it compile for Impala. There's no unit test needed since there is no functional change. The Calcite planner will eventually make changes in the ".jj" file to support the differences between the Impala parser and the Calcite parser. Change-Id: If756b5ea8beb85661a30fb5d029e74ebb6719767 --- M java/calcite-planner/pom.xml A java/calcite-planner/src/main/codegen/config.fmpp A java/calcite-planner/src/main/codegen/templates/Parser.jj M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteQueryParser.java 4 files changed, 9,616 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/94/21194/10 -- To view, visit http://gerrit.cloudera.org:8080/21194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If756b5ea8beb85661a30fb5d029e74ebb6719767 Gerrit-Change-Number: 21194 Gerrit-PatchSet: 10 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Steve Carlin
[Impala-ASF-CR] IMPALA-13058: Init first arrival time and completion time with -1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21405 ) Change subject: IMPALA-13058: Init first_arrival_time_ and completion_time_ with -1 .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/16114/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21405 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1176e2118bb03414ab35049f50009ff0e8c63f58 Gerrit-Change-Number: 21405 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 07 May 2024 16:26:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13058: Init first arrival time and completion time with -1
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21405 ) Change subject: IMPALA-13058: Init first_arrival_time_ and completion_time_ with -1 .. Patch Set 1: This patch only help bypassing the flaky test, but does not address the underlying issue with RuntimeProfile::EventSequence::Start() from: be/src/service/client-request-state.cc:140: query_events_->Start(); https://issues.apache.org/jira/browse/IMPALA-13058?focusedCommentId=17844060&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17844060 -- To view, visit http://gerrit.cloudera.org:8080/21405 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1176e2118bb03414ab35049f50009ff0e8c63f58 Gerrit-Change-Number: 21405 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 07 May 2024 16:05:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13058: Init first arrival time and completion time with -1
Riza Suminto has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21405 Change subject: IMPALA-13058: Init first_arrival_time_ and completion_time_ with -1 .. IMPALA-13058: Init first_arrival_time_ and completion_time_ with -1 Before this patch, first_arrival_time_ and completion_time_ of Coordinator::FilterState is initialized with 0. This patch change the initialization value to -1 and accept 0 as valid value for those fields. Testing: - Tweak row_regex pattern in runtime_filters.test. - Pass test_runtime_filters.py in exhaustive mode. Change-Id: I1176e2118bb03414ab35049f50009ff0e8c63f58 --- M be/src/runtime/coordinator-filter-state.h M be/src/runtime/coordinator.cc M testdata/workloads/functional-query/queries/QueryTest/runtime_filters.test 3 files changed, 11 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/21405/1 -- To view, visit http://gerrit.cloudera.org:8080/21405 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I1176e2118bb03414ab35049f50009ff0e8c63f58 Gerrit-Change-Number: 21405 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto
[Impala-ASF-CR] IMPALA-11499: Refactor UrlEncode function to handle special characters
Daniel Becker has posted comments on this change. (
http://gerrit.cloudera.org:8080/21131 )
Change subject: IMPALA-11499: Refactor UrlEncode function to handle special
characters
..
Patch Set 23:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21131/23/tests/query_test/test_insert.py
File tests/query_test/test_insert.py:
http://gerrit.cloudera.org:8080/#/c/21131/23/tests/query_test/test_insert.py@301
PS23, Line 301: "create table {}(i int) partitioned by (p string)
stored as parquet".format(tbl))
One more thing: you should run this test with an Iceberg table too. You could
extract the common parts (mostly everything except for the create and insert
statements) into a helper function.
--
To view, visit http://gerrit.cloudera.org:8080/21131
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I88c4aba5d811dfcec809583d0c16fcbc0ca730fb
Gerrit-Change-Number: 21131
Gerrit-PatchSet: 23
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Daniel Becker
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Zihao Ye
Gerrit-Comment-Date: Tue, 07 May 2024 14:19:54 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21160 ) Change subject: IMPALA-12921: Support locally built Ranger .. Patch Set 10: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21160 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27 Gerrit-Change-Number: 21160 Gerrit-PatchSet: 10 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 07 May 2024 14:03:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11499: Refactor UrlEncode function to handle special characters
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/21131 ) Change subject: IMPALA-11499: Refactor UrlEncode function to handle special characters .. Patch Set 23: (2 comments) http://gerrit.cloudera.org:8080/#/c/21131/23/tests/query_test/test_insert.py File tests/query_test/test_insert.py: http://gerrit.cloudera.org:8080/#/c/21131/23/tests/query_test/test_insert.py@320 PS23, Line 320: result line "the result line..." http://gerrit.cloudera.org:8080/#/c/21131/23/tests/query_test/test_insert.py@321 PS23, Line 321: the values present inside it are just verified. "we only verify that the value is present in string representation of the whole row" -- To view, visit http://gerrit.cloudera.org:8080/21131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88c4aba5d811dfcec809583d0c16fcbc0ca730fb Gerrit-Change-Number: 21131 Gerrit-PatchSet: 23 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zihao Ye Gerrit-Comment-Date: Tue, 07 May 2024 12:46:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11499: Refactor UrlEncode function to handle special characters
Daniel Becker has posted comments on this change. (
http://gerrit.cloudera.org:8080/21131 )
Change subject: IMPALA-11499: Refactor UrlEncode function to handle special
characters
..
Patch Set 23:
(2 comments)
Just a few nits.
http://gerrit.cloudera.org:8080/#/c/21131/23/be/src/util/coding-util-test.cc
File be/src/util/coding-util-test.cc:
http://gerrit.cloudera.org:8080/#/c/21131/23/be/src/util/coding-util-test.cc@94
PS23, Line 94: TestUrl("/home/impala/directory/",
"%2Fhome%2Fimpala%2Fdirectory%2F", false);
These strings could also be extracted into variables.
http://gerrit.cloudera.org:8080/#/c/21131/23/tests/query_test/test_insert.py
File tests/query_test/test_insert.py:
http://gerrit.cloudera.org:8080/#/c/21131/23/tests/query_test/test_insert.py@320
PS23, Line 320: are
"is a"
--
To view, visit http://gerrit.cloudera.org:8080/21131
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I88c4aba5d811dfcec809583d0c16fcbc0ca730fb
Gerrit-Change-Number: 21131
Gerrit-PatchSet: 23
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Daniel Becker
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Zihao Ye
Gerrit-Comment-Date: Tue, 07 May 2024 12:24:14 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13038: Support profile tab for imported query profiles
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21400 ) Change subject: IMPALA-13038: Support profile tab for imported query profiles .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/16113/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddcf2e285abbf42f97bde19014be076ccd6374bc Gerrit-Change-Number: 21400 Gerrit-PatchSet: 2 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 07 May 2024 12:05:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13038: Support profile tab for imported query profiles
Surya Hebbar has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/21400 ) Change subject: IMPALA-13038: Support profile tab for imported query profiles .. IMPALA-13038: Support profile tab for imported query profiles For query profile imports currently the following tabs are supported. - Query Statement - Query Timeline - Query Text Plan With the current patch "Query Profile" tab will also be supported. In the "QueryProfileHandler", "query_id" is now added before verifying its existence in the query log as in "QuerySummaryHandler" and others. On loading the imported "Query Profile" page, query profile download section and server's non-existing query id alerts are removed. All unsupported navbar tabs are removed and current tab is set to active. The query profile is retrieved from the indexedDB's "imported_queries" database. Then query profile is passed onto "profileToString" method, which converts the profile into indented text for displaying on the profile page. Each profile and its child profiles are printed in the following order with the right indentation(fields are skipped, if they do not exist). Profile name: - Info strings: - Event sequences: - Offset: - Events: - Child profile(recursive): - Counters: Change-Id: Iddcf2e285abbf42f97bde19014be076ccd6374bc --- M be/src/service/impala-http-handler.cc M www/query_plan_text.tmpl M www/query_profile.tmpl M www/query_stmt.tmpl M www/query_timeline.tmpl 5 files changed, 81 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/21400/2 -- To view, visit http://gerrit.cloudera.org:8080/21400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iddcf2e285abbf42f97bde19014be076ccd6374bc Gerrit-Change-Number: 21400 Gerrit-PatchSet: 2 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21160 ) Change subject: IMPALA-12921: Support locally built Ranger .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/16112/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21160 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27 Gerrit-Change-Number: 21160 Gerrit-PatchSet: 10 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 07 May 2024 09:08:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21160 ) Change subject: IMPALA-12921: Support locally built Ranger .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/16111/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21160 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27 Gerrit-Change-Number: 21160 Gerrit-PatchSet: 9 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 07 May 2024 09:04:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/20221 ) Change subject: IMPALA-11871: Skip permissions loading and check on HDFS if Ranger is enabled .. Patch Set 11: Hi all, please let me know if there are additional comments on this patch. Thank you very much for the help! -- To view, visit http://gerrit.cloudera.org:8080/20221 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id33c400fbe0c918b6b65d713b09009512835a4c9 Gerrit-Change-Number: 20221 Gerrit-PatchSet: 11 Gerrit-Owner: Halim Kim Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Halim Kim Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 07 May 2024 08:53:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/21160 ) Change subject: IMPALA-12921: Support locally built Ranger .. Patch Set 10: (2 comments) Hi all, I have revised patch set 7 according to Quanlong's comments. Please ignore patch sets 8 and 9 since I made some mistakes when addressing Quanlong's comments. Let me know if there is any additional suggestion. I plan to collect the list of passed authorization-related tests when Apache Ranger is used with this patch and will update on this when the results are available. http://gerrit.cloudera.org:8080/#/c/21160/7//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21160/7//COMMIT_MSG@64 PS7, Line 64: - Verified that this patch passed the core tests when CDP Ranger is > Do you mean using Apache Ranger? That would be great! Thanks Quanlong! I only verified that this patch passed the core tests with CDP Ranger. I will clarify this in the commit message of the next patch. I could try to collect the list of passed authorization-related tests when the patched Apache Ranger (with RANGER-4771.diff on top of RANGER-4745) is used. http://gerrit.cloudera.org:8080/#/c/21160/7/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java: http://gerrit.cloudera.org:8080/#/c/21160/7/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@500 PS7, Line 500: req.setResource(resource); > It seems using the empty constructor and setting the fields later is a more Thanks Quanlong! I will revise this accordingly in the next patch. -- To view, visit http://gerrit.cloudera.org:8080/21160 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27 Gerrit-Change-Number: 21160 Gerrit-PatchSet: 10 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 07 May 2024 08:49:26 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21160 ) Change subject: IMPALA-12921: Support locally built Ranger .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10612/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/21160 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27 Gerrit-Change-Number: 21160 Gerrit-PatchSet: 10 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 07 May 2024 08:50:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Hello Quanlong Huang, Aman Sinha, Joe McDonnell, Impala Public Jenkins, John
Sherman,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21160
to look at the new patch set (#10).
Change subject: IMPALA-12921: Support locally built Ranger
..
IMPALA-12921: Support locally built Ranger
This patch does the following.
1. Add the support for locally built Ranger.
2. Resolve IMPALA-12985 by switching to the new constructor when
instantiating RangerAccessRequestImpl.
The instructions on how to build Ranger locally and point
Impala to the locally built Ranger server are provided as below.
Suppose the Ranger project is under the folder $RANGER_SRC_DIR. We could
execute the following to build Ranger. By default, the compressed
tarball is produced under $RANGER_SRC_DIR/target.
mvn clean compile -B -nsu -DskipCheck=true -Dcheckstyle.skip=true \
package install -DskipITs -DskipTests -Dmaven.javadoc.skip=true
After building Ranger, we need to build Impala's Java code so that
Impala's Java code could consume the locally produced Ranger classes. We
will need to export the following environment variables before building
Impala. This prevents bootstrap_toolchain.py from trying to download the
compressed Ranger tarball. Moreover, we need to apply
IMPALA-12921_pom_patching.diff on Impala if the locally built Ranger
pulls in an incompatible version of hive-storage-api.
1. export RANGER_VERSION_OVERRIDE=\
$(mvn -f $RANGER_SRC_DIR/pom.xml -q help:evaluate \
-Dexpression=project.version -DforceStdout)
2. export RANGER_HOME_OVERRIDE=$RANGER_SRC_DIR/target/\
ranger-${RANGER_VERSION_OVERRIDE}-admin
It then suffices to execute the following to point
Impala to the locally built Ranger server before starting Impala.
1. source $IMPALA_HOME/bin/impala-config.sh
2. tar zxv -f $RANGER_SRC_DIR/target/\
ranger-${IMPALA_RANGER_VERSION}-admin.tar.gz \
-C $RANGER_SRC_DIR/target/
3. $IMPALA_HOME/bin/create-test-configuration.sh
4. $IMPALA_HOME/bin/create-test-configuration.sh \
-create_ranger_policy_db
5. $IMPALA_HOME/testdata/bin/run-ranger.sh
(run-all.sh has to be executed instead if other underlying services
have not been started)
6. $IMPALA_HOME/testdata/bin/setup-ranger.sh
Testing:
- Manually verified that we could point Impala to a locally built
Apache Ranger on the master branch (with tip being RANGER-4745).
- Manually verified that with the Ranger patch in RANGER-4771.diff,
we could manage the policy repository via GRANT/REVOKE statements.
- Verified that this patch passed the core tests when CDP Ranger is
used.
Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27
---
M README-build.md
M bin/bootstrap_toolchain.py
M bin/create-test-configuration.sh
M bin/impala-config.sh
M bin/rat_exclude_files.txt
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java
M testdata/bin/setup-ranger.sh
A testdata/cluster/ranger/IMPALA-12921_pom_patching.diff
A testdata/cluster/ranger/RANGER-4771.diff
A testdata/cluster/ranger/README
R testdata/cluster/ranger/setup/all_database_policy_revised.json.template
M testdata/cluster/ranger/setup/impala_user_non_owner.json.template
M testdata/cluster/ranger/setup/impala_user_owner.json.template
14 files changed, 142 insertions(+), 16 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/21160/10
--
To view, visit http://gerrit.cloudera.org:8080/21160
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27
Gerrit-Change-Number: 21160
Gerrit-PatchSet: 10
Gerrit-Owner: Fang-Yu Rao
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: John Sherman
Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Hello Quanlong Huang, Aman Sinha, Joe McDonnell, Impala Public Jenkins, John
Sherman,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21160
to look at the new patch set (#9).
Change subject: IMPALA-12921: Support locally built Ranger
..
IMPALA-12921: Support locally built Ranger
This patch does the following.
1. Add the support for locally built Ranger.
2. Resolve IMPALA-12985 by switching to the new constructor when
instantiating RangerAccessRequestImpl.
The instructions on how to build Ranger locally and point
Impala to the locally built Ranger server are provided as below.
Suppose the Ranger project is under the folder $RANGER_SRC_DIR. We could
execute the following to build Ranger. By default, the compressed
tarball is produced under $RANGER_SRC_DIR/target.
mvn clean compile -B -nsu -DskipCheck=true -Dcheckstyle.skip=true \
package install -DskipITs -DskipTests -Dmaven.javadoc.skip=true
After building Ranger, we need to build Impala's Java code so that
Impala's Java code could consume the locally produced Ranger classes. We
will need to export the following environment variables before building
Impala. This prevents bootstrap_toolchain.py from trying to download the
compressed Ranger tarball. Moreover, we need to apply
IMPALA-12921_pom_patching.diff on Impala if the locally built Ranger
pulls in an incompatible version of hive-storage-api.
1. export RANGER_VERSION_OVERRIDE=\
$(mvn -f $RANGER_SRC_DIR/pom.xml -q help:evaluate \
-Dexpression=project.version -DforceStdout)
2. export RANGER_HOME_OVERRIDE=$RANGER_SRC_DIR/target/\
ranger-${RANGER_VERSION_OVERRIDE}-admin
It then suffices to execute the following to point
Impala to the locally built Ranger server before starting Impala.
1. source $IMPALA_HOME/bin/impala-config.sh
2. tar zxv -f $RANGER_SRC_DIR/target/\
ranger-${IMPALA_RANGER_VERSION}-admin.tar.gz \
-C $RANGER_SRC_DIR/target/
3. $IMPALA_HOME/bin/create-test-configuration.sh
4. $IMPALA_HOME/bin/create-test-configuration.sh \
-create_ranger_policy_db
5. $IMPALA_HOME/testdata/bin/run-ranger.sh
(run-all.sh has to be executed instead if other underlying services
have not been started)
6. $IMPALA_HOME/testdata/bin/setup-ranger.sh
Testing:
- Manually verified that we could point Impala to a locally built
Apache Ranger on the master branch (with tip being RANGER-4745).
- Manually verified that with the Ranger patch in RANGER-4771.diff,
we could manage the policy repository via GRANT/REVOKE statements.
- Verified that this patch passed the core tests when CDP Ranger is
used.
Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27
---
M README-build.md
M bin/bootstrap_toolchain.py
M bin/create-test-configuration.sh
M bin/impala-config.sh
M bin/rat_exclude_files.txt
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java
M testdata/bin/setup-ranger.sh
A testdata/cluster/ranger/IMPALA-12921_pom_patching.diff
A testdata/cluster/ranger/RANGER-4771.diff
A testdata/cluster/ranger/README
R testdata/cluster/ranger/setup/all_database_policy_revised.json.template
M testdata/cluster/ranger/setup/impala_user_non_owner.json.template
M testdata/cluster/ranger/setup/impala_user_owner.json.template
14 files changed, 142 insertions(+), 16 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/21160/9
--
To view, visit http://gerrit.cloudera.org:8080/21160
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27
Gerrit-Change-Number: 21160
Gerrit-PatchSet: 9
Gerrit-Owner: Fang-Yu Rao
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: John Sherman
Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21160 ) Change subject: IMPALA-12921: Support locally built Ranger .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/16110/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21160 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27 Gerrit-Change-Number: 21160 Gerrit-PatchSet: 8 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 07 May 2024 08:35:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger
Hello Quanlong Huang, Aman Sinha, Joe McDonnell, Impala Public Jenkins, John
Sherman,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21160
to look at the new patch set (#8).
Change subject: IMPALA-12921: Support locally built Ranger
..
IMPALA-12921: Support locally built Ranger
This patch does the following.
1. Add the support for locally built Ranger.
2. Resolve IMPALA-12985 by switching to the new constructor when
instantiating RangerAccessRequestImpl.
The instructions on how to build Ranger locally and point
Impala to the locally built Ranger server are provided as below.
Suppose the Ranger project is under the folder $RANGER_SRC_DIR. We could
execute the following to build Ranger. By default, the compressed
tarball is produced under $RANGER_SRC_DIR/target.
mvn clean compile -B -nsu -DskipCheck=true -Dcheckstyle.skip=true \
package install -DskipITs -DskipTests -Dmaven.javadoc.skip=true
After building Ranger, we need to build Impala's Java code so that
Impala's Java code could consume the locally produced Ranger classes. We
will need to export the following environment variables before building
Impala. This prevents bootstrap_toolchain.py from trying to download the
compressed Ranger tarball. Moreover, we need to apply
IMPALA-12921_pom_patching.diff on Impala if the locally built Ranger
pulls in an incompatible version of hive-storage-api.
1. export RANGER_VERSION_OVERRIDE=\
$(mvn -f $RANGER_SRC_DIR/pom.xml -q help:evaluate \
-Dexpression=project.version -DforceStdout)
2. export RANGER_HOME_OVERRIDE=$RANGER_SRC_DIR/target/\
ranger-${RANGER_VERSION_OVERRIDE}-admin
It then suffices to execute the following to point
Impala to the locally built Ranger server before starting Impala.
1. source $IMPALA_HOME/bin/impala-config.sh
2. tar zxv -f $RANGER_SRC_DIR/target/\
ranger-${IMPALA_RANGER_VERSION}-admin.tar.gz \
-C $RANGER_SRC_DIR/target/
3. $IMPALA_HOME/bin/create-test-configuration.sh
4. $IMPALA_HOME/bin/create-test-configuration.sh \
-create_ranger_policy_db
5. $IMPALA_HOME/testdata/bin/run-ranger.sh
(run-all.sh has to be executed instead if other underlying services
have not been started)
6. $IMPALA_HOME/testdata/bin/setup-ranger.sh
Testing:
- Manually verified that we could point Impala to a locally built
Apache Ranger on the master branch (with tip being RANGER-4745).
- Manually verified that with the Ranger patch in RANGER-4771.diff,
we could manage the policy repository via GRANT/REVOKE statements.
- Verified that this patch passed the core tests when CDP Ranger is
used.
Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27
---
M README-build.md
M bin/bootstrap_toolchain.py
M bin/create-test-configuration.sh
M bin/impala-config.sh
M bin/rat_exclude_files.txt
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java
M testdata/bin/setup-ranger.sh
A testdata/cluster/ranger/IMPALA-12921_pom_patching.diff
A testdata/cluster/ranger/RANGER-4771.diff
A testdata/cluster/ranger/README
R testdata/cluster/ranger/setup/all_database_policy_revised.json.template
M testdata/cluster/ranger/setup/impala_user_non_owner.json.template
M testdata/cluster/ranger/setup/impala_user_owner.json.template
14 files changed, 134 insertions(+), 13 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/21160/8
--
To view, visit http://gerrit.cloudera.org:8080/21160
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27
Gerrit-Change-Number: 21160
Gerrit-PatchSet: 8
Gerrit-Owner: Fang-Yu Rao
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: John Sherman
Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-12607: Bump the GBN and fetch events specific to the db/table from the metastore
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/20979 ) Change subject: IMPALA-12607: Bump the GBN and fetch events specific to the db/table from the metastore .. Patch Set 9: Code-Review+1 (2 comments) http://gerrit.cloudera.org:8080/#/c/20979/9/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java: http://gerrit.cloudera.org:8080/#/c/20979/9/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@466 PS9, Line 466: metaDataFilter = new MetaDataFilter(getTableNotificationEventFilter(tbl)); Can you add a comment on why the ACID/external cases are different? http://gerrit.cloudera.org:8080/#/c/20979/9/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/20979/9/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@4017 PS9, Line 4017: 3 testInsertIntoTransactionalTable also does an insert - this doesn't lead to new events, because transaction events are not specific to db, right? -- To view, visit http://gerrit.cloudera.org:8080/20979 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6aecd5108b31c24e6e2c6f9fba6d4d44a3b00729 Gerrit-Change-Number: 20979 Gerrit-PatchSet: 9 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 07 May 2024 07:22:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13057: Incorporate tuple/slot information into tuple cache key
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21398 ) Change subject: IMPALA-13057: Incorporate tuple/slot information into tuple cache key .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/16109/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f5278e9dbb976cbebdc6a21a6e66bc90ce06c6c Gerrit-Change-Number: 21398 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 07 May 2024 07:00:14 + Gerrit-HasComments: No
