date:20240508

Quanlong Huang has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20979 )

Change subject: IMPALA-12607: Bump the GBN and fetch events specific to the 
db/table from the metastore
..


Patch Set 11:

> Patch Set 11: Verified-1
>
> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10620/

There are some failures when building on Apache Hive (USE_APACHE_HIVE=true): 
https://jenkins.impala.io/job/all-build-options-ub2004/1176/consoleFull

19:27:02 [ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.11.0:compile (default-compile) 
on project impala-frontend: Compilation failure: Compilation failure:
19:27:02 [ERROR] 
/home/ubuntu/Impala/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:[397,25]
 cannot find symbol
19:27:02 [ERROR]   symbol:   method setCatName(java.lang.String)
19:27:02 [ERROR]   location: variable eventRequest of type 
org.apache.hadoop.hive.metastore.api.NotificationEventRequest
19:27:02 [ERROR] 
/home/ubuntu/Impala/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:[401,25]
 cannot find symbol
19:27:02 [ERROR]   symbol:   method setDbName(java.lang.String)
19:27:02 [ERROR]   location: variable eventRequest of type 
org.apache.hadoop.hive.metastore.api.NotificationEventRequest
19:27:02 [ERROR] 
/home/ubuntu/Impala/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:[405,25]
 cannot find symbol
19:27:02 [ERROR]   symbol:   method 
setTableNames(java.util.List)
19:27:02 [ERROR]   location: variable eventRequest of type 
org.apache.hadoop.hive.metastore.api.NotificationEventRequest

We need to wrap these usages into MetastoreShim.


-- 
To view, visit http://gerrit.cloudera.org:8080/20979
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6aecd5108b31c24e6e2c6f9fba6d4d44a3b00729
Gerrit-Change-Number: 20979
Gerrit-PatchSet: 11
Gerrit-Owner: Sai Hemanth Gantasala 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Sai Hemanth Gantasala 
Gerrit-Comment-Date: Thu, 09 May 2024 02:34:04 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12934: Added Calcite parsing files to Impala

2024-05-08 Thread Joe McDonnell (Code Review)

Joe McDonnell has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/21194 )

Change subject: IMPALA-12934: Added Calcite parsing files to Impala
..

IMPALA-12934: Added Calcite parsing files to Impala

Adding the framework to create our own parsing syntax for Impala using
the base Calcite Parser.jj file.

The Parser.jj file here was grabbed from Calcite 1.36. So with this commit,
we are using the same parsing analysis as Calcite 1.36. Any changes made
on top of the Parser.jj file or the config.fmpp file in the future are Impala
specific changes, so a diff can be done from this commit to see all the Impala
parsing changes.

The config.fmpp file was grabbed from Calcite 1.36 default_config.fmpp. The
Calcite intention of the config.fmpp file is to allow markup of variables in
the Parser.jj file. So it is always preferable to modify the
default_config.fmpp file when possible. Our version is grabbed from
https://github.com/apache/calcite/blob/main/core/src/main/codegen/config.fmpp
and slightly modified with the class name to make it compile for Impala.

There's no unit test needed since there is no functional change. The Calcite
planner will eventually make changes in the ".jj" file to support the
differences
between the Impala parser and the Calcite parser.
Change-Id: If756b5ea8beb85661a30fb5d029e74ebb6719767
Reviewed-on: http://gerrit.cloudera.org:8080/21194
Tested-by: Impala Public Jenkins
Reviewed-by: Joe McDonnell
---
M java/calcite-planner/pom.xml
A java/calcite-planner/src/main/codegen/config.fmpp
A java/calcite-planner/src/main/codegen/templates/Parser.jj
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteQueryParser.java
4 files changed, 9,616 insertions(+), 1 deletion(-)

Approvals:
Impala Public Jenkins: Verified
Joe McDonnell: Looks good to me, approved

--
To view, visit http://gerrit.cloudera.org:8080/21194
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: If756b5ea8beb85661a30fb5d029e74ebb6719767
Gerrit-Change-Number: 21194
Gerrit-PatchSet: 11
Gerrit-Owner: Steve Carlin
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Steve Carlin

[Impala-ASF-CR] IMPALA-12934: Added Calcite parsing files to Impala

2024-05-08 Thread Joe McDonnell (Code Review)

Joe McDonnell has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21194 )

Change subject: IMPALA-12934: Added Calcite parsing files to Impala
..


Patch Set 10: Code-Review+2

I think long term we will be happier with Option 1, particularly when we need 
to upgrade Calcite, but we can address that later.


--
To view, visit http://gerrit.cloudera.org:8080/21194
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If756b5ea8beb85661a30fb5d029e74ebb6719767
Gerrit-Change-Number: 21194
Gerrit-PatchSet: 10
Gerrit-Owner: Steve Carlin 
Gerrit-Reviewer: Aman Sinha 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Michael Smith 
Gerrit-Reviewer: Steve Carlin 
Gerrit-Comment-Date: Thu, 09 May 2024 01:12:41 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12607: Bump the GBN and fetch events specific to the db/table from the metastore

2024-05-08 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20979 )

Change subject: IMPALA-12607: Bump the GBN and fetch events specific to the 
db/table from the metastore
..


Patch Set 11: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10620/


--
To view, visit http://gerrit.cloudera.org:8080/20979
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6aecd5108b31c24e6e2c6f9fba6d4d44a3b00729
Gerrit-Change-Number: 20979
Gerrit-PatchSet: 11
Gerrit-Owner: Sai Hemanth Gantasala 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Sai Hemanth Gantasala 
Gerrit-Comment-Date: Wed, 08 May 2024 22:58:14 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote

2024-05-08 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21383 )

Change subject: IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote
..


Patch Set 6:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/16145/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/21383
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I156d1f0c694b91ba34be70bc53ae9bacf924b3b9
Gerrit-Change-Number: 21383
Gerrit-PatchSet: 6
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Jason Fehr 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Michael Smith 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Comment-Date: Wed, 08 May 2024 22:11:38 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote

2024-05-08 Thread Jason Fehr (Code Review)

Jason Fehr has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21383 )

Change subject: IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote
..


Patch Set 6: Code-Review+1


--
To view, visit http://gerrit.cloudera.org:8080/21383
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I156d1f0c694b91ba34be70bc53ae9bacf924b3b9
Gerrit-Change-Number: 21383
Gerrit-PatchSet: 6
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Jason Fehr 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Michael Smith 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Comment-Date: Wed, 08 May 2024 22:06:21 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys

2024-05-08 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21382 )

Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
..


Patch Set 14:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/16144/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/21382
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74
Gerrit-Change-Number: 21382
Gerrit-PatchSet: 14
Gerrit-Owner: gaurav singh 
Gerrit-Reviewer: Abhishek Rawat 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Reviewer: gaurav singh 
Gerrit-Comment-Date: Wed, 08 May 2024 22:03:59 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys

2024-05-08 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21382 )

Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
..


Patch Set 13:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/16143/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/21382
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74
Gerrit-Change-Number: 21382
Gerrit-PatchSet: 13
Gerrit-Owner: gaurav singh 
Gerrit-Reviewer: Abhishek Rawat 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Reviewer: gaurav singh 
Gerrit-Comment-Date: Wed, 08 May 2024 22:01:28 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11499: Refactor UrlEncode function to handle special characters

Quanlong Huang has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21131 )

Change subject: IMPALA-11499: Refactor UrlEncode function to handle special 
characters
..


Patch Set 27: Code-Review+1


--
To view, visit http://gerrit.cloudera.org:8080/21131
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I88c4aba5d811dfcec809583d0c16fcbc0ca730fb
Gerrit-Change-Number: 21131
Gerrit-PatchSet: 27
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Anonymous Coward 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Smith 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zihao Ye 
Gerrit-Comment-Date: Wed, 08 May 2024 21:48:13 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote

Riza Suminto has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21383 )

Change subject: IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote
..


Patch Set 6:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/21383/5/be/src/util/network-util.h
File be/src/util/network-util.h:

http://gerrit.cloudera.org:8080/#/c/21383/5/be/src/util/network-util.h@118
PS5, Line 118: /// Compare function for two NetworkAddressPB.
> Please add unit tests in network-util-test.cc either in this change or a fo
Done



--
To view, visit http://gerrit.cloudera.org:8080/21383
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I156d1f0c694b91ba34be70bc53ae9bacf924b3b9
Gerrit-Change-Number: 21383
Gerrit-PatchSet: 6
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Jason Fehr 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Michael Smith 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Comment-Date: Wed, 08 May 2024 21:47:44 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote

Hello Kurt Deschler, Jason Fehr, Csaba Ringhofer, Michael Smith, Impala Public 
Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/21383

to look at the new patch set (#6).

Change subject: IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote
..

IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote

It is possible to have UpdateFilterFromRemote RPC arrive to an impalad
executor before QueryState of the destination query is created or
complete initialization. This patch add wait mechanism in
UpdateFilterFromRemote RPC endpoint to wait for few miliseconds until
QueryState exist and complete initialization.

The wait time is fixed at 500ms, with exponential sleep period in
between. If wait time passed and QueryState still not found or
initialized, UpdateFilterFromRemote RPC is deemed fail and query
execution move on without complete filter.

Testing:
- Add BE tests in network-util-test.cc
- Add test_runtime_filter_aggregation.py::TestLateQueryStateInit
- Pass exhastive runs of test_runtime_filter_aggregation.py

Change-Id: I156d1f0c694b91ba34be70bc53ae9bacf924b3b9
---
M be/src/runtime/query-exec-mgr.cc
M be/src/runtime/query-state.cc
M be/src/runtime/query-state.h
M be/src/runtime/runtime-filter-bank.cc
M be/src/runtime/runtime-filter-bank.h
M be/src/scheduling/scheduler.cc
M be/src/service/data-stream-service.cc
M be/src/util/network-util-test.cc
M be/src/util/network-util.cc
M be/src/util/network-util.h
M common/protobuf/data_stream_service.proto
M tests/custom_cluster/test_runtime_filter_aggregation.py
12 files changed, 295 insertions(+), 28 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/21383/6
--
To view, visit http://gerrit.cloudera.org:8080/21383
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I156d1f0c694b91ba34be70bc53ae9bacf924b3b9
Gerrit-Change-Number: 21383
Gerrit-PatchSet: 6
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Jason Fehr 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Michael Smith 
Gerrit-Reviewer: Riza Suminto

[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys

Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/21382

to look at the new patch set (#14).

Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
..

IMPALA-12559: Support x5c Parameter for RSA JSON
Web Keys

This enables the jwt verification using the x5c
certificate(s) in the RSA jwks keys. The x5c claim can be
part of the jwks either as a string or an array.
This patch only supports a single x5c certificate per
jwk.

If the "x5c" is present and "alg" is not present,
then "alg" is extracted from the "x5c" certificate using the
signature algorithm. However, if "x5c" is not preseent, then
"alg" is a mandatory field on jwk.

Current mapping of signature algorithm string => algorithm:

sha224WithRSAEncryption => rs224
sha256WithRSAEncryption => rs256
sha384WithRSAEncryption => rs384
sha512WithRSAEncryption => rs512

If "x5c" is present, then it is given priority over other
mandatory fields like "n", "e" to construct the public key.

Testing:
* added unit test VerifyJwtTokenWithx5cCertificate to
verify jwt with x5c certificate.
* added unit test VerifyJwtTokenWithx5cCertificateWithoutAlg
to verify jwt with x5c certificate without "alg".
* added e2e test testJwtAuthWithJwksX5cHttpUrl to verify
jwt with x5c certificate.

Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74
---
M be/src/util/jwt-util-test.cc
M be/src/util/jwt-util.cc
M fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java
M testdata/jwt/jwks_rs256.json
4 files changed, 378 insertions(+), 22 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/14
--
To view, visit http://gerrit.cloudera.org:8080/21382
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74
Gerrit-Change-Number: 21382
Gerrit-PatchSet: 14
Gerrit-Owner: gaurav singh 
Gerrit-Reviewer: Abhishek Rawat 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Reviewer: gaurav singh

[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys

gaurav singh has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21382 )

Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
..


Patch Set 13:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/21382/6//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/21382/6//COMMIT_MSG@22
PS6, Line 22:
> Please add end-end test case.
Added e2e test testJwtAuthWithJwksX5cHttpUrl



--
To view, visit http://gerrit.cloudera.org:8080/21382
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74
Gerrit-Change-Number: 21382
Gerrit-PatchSet: 13
Gerrit-Owner: gaurav singh 
Gerrit-Reviewer: Abhishek Rawat 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Reviewer: gaurav singh 
Gerrit-Comment-Date: Wed, 08 May 2024 21:37:25 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys

Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/21382

to look at the new patch set (#13).

Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
..

IMPALA-12559: Support x5c Parameter for RSA JSON
Web Keys

This enables the jwt verification using the x5c
certificate(s) in the RSA jwks keys. The x5c claim can be
part of the jwks either as a string or an array.
This patch only supports a single x5c certificate per
jwk.

If the "x5c" is present and "alg" is not present,
then "alg" is extracted from the "x5c" certificate using the
signature algorithm. However, if "x5c" is not preseent, then
"alg" is a mandatory field on jwk.

Current mapping of signature algorithm string => algorithm:

sha224WithRSAEncryption => rs224
sha256WithRSAEncryption => rs256
sha384WithRSAEncryption => rs384
sha512WithRSAEncryption => rs512

If "x5c" is present, then it is given priority over other
mandatory fields like "n", "e" to construct the public key.

Testing:
* added unit test VerifyJwtTokenWithx5cCertificate to
verify jwt with x5c certificate.
* added unit test VerifyJwtTokenWithx5cCertificateWithoutAlg
to verify jwt with x5c certificate without "alg".
* added e2e test testJwtAuthWithJwksX5cHttpUrl to verify
jwt with x5c certificate.

Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74
---
M be/src/util/jwt-util-test.cc
M be/src/util/jwt-util.cc
M fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java
M testdata/jwt/jwks_rs256.json
4 files changed, 378 insertions(+), 22 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/13
--
To view, visit http://gerrit.cloudera.org:8080/21382
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74
Gerrit-Change-Number: 21382
Gerrit-PatchSet: 13
Gerrit-Owner: gaurav singh 
Gerrit-Reviewer: Abhishek Rawat 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Reviewer: gaurav singh

[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys

2024-05-08 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21382 )

Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
..


Patch Set 13:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/21382/13/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java
File fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java:

http://gerrit.cloudera.org:8080/#/c/21382/13/fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java@72
PS13, Line 72:   String jwtTokenX5c_ =
line has trailing whitespace



--
To view, visit http://gerrit.cloudera.org:8080/21382
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74
Gerrit-Change-Number: 21382
Gerrit-PatchSet: 13
Gerrit-Owner: gaurav singh 
Gerrit-Reviewer: Abhishek Rawat 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Reviewer: gaurav singh 
Gerrit-Comment-Date: Wed, 08 May 2024 21:37:52 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10451: Fix avro table loading failures caused by HIVE-24157

2024-05-08 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21413 )

Change subject: IMPALA-10451: Fix avro table loading failures caused by 
HIVE-24157
..


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10621/ 
DRY_RUN=true


--
To view, visit http://gerrit.cloudera.org:8080/21413
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3e2a47d60d4079fece9c04091258215f3d6a7b52
Gerrit-Change-Number: 21413
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Comment-Date: Wed, 08 May 2024 21:34:26 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10451: Fix avro table loading failures caused by HIVE-24157

2024-05-08 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21413 )

Change subject: IMPALA-10451: Fix avro table loading failures caused by 
HIVE-24157
..


Patch Set 1:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/16142/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/21413
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3e2a47d60d4079fece9c04091258215f3d6a7b52
Gerrit-Change-Number: 21413
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Comment-Date: Wed, 08 May 2024 21:24:31 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10451: Fix avro table loading failures caused by HIVE-24157

2024-05-08 Thread Joe McDonnell (Code Review)

Joe McDonnell has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/21413


Change subject: IMPALA-10451: Fix avro table loading failures caused by 
HIVE-24157
..

IMPALA-10451: Fix avro table loading failures caused by HIVE-24157

HIVE-24157 introduces a restriction to prohibit casting DATE/TIMESTAMP
types to and from NUMERIC. It's enabled by default and can be turned off
by set hive.strict.timestamp.conversion=false.

This restriction breaks the data loading on avro_coldef and
avro_extra_coldef tables, which results in empty data set and finally
fails TestAvroSchemaResolution.test_avro_schema_resolution.

This patch explicitly disables the restriction in loading these two avro
tables.

The Hive version currently used for development does not have HIVE-24157,
but upstream Hive does have it. Adding hive.strict.timestamp.conversion
does not cause problems for Hive versions that don't have HIVE-24157.

Tests:
 - Run the data loading and test_avro_schema_resolution locally using
   a Hive that has HIVE-24157.
 - Run CORE tests
 - Run data loading with a Hive that doesn't have HIVE-24157.

Change-Id: I3e2a47d60d4079fece9c04091258215f3d6a7b52
---
M testdata/avro_schema_resolution/create_table.sql
1 file changed, 4 insertions(+), 0 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/21413/1
--
To view, visit http://gerrit.cloudera.org:8080/21413
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I3e2a47d60d4079fece9c04091258215f3d6a7b52
Gerrit-Change-Number: 21413
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell

[Impala-ASF-CR] IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote

2024-05-08 Thread Jason Fehr (Code Review)

Jason Fehr has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21383 )

Change subject: IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote
..


Patch Set 5: Code-Review+1

(2 comments)

http://gerrit.cloudera.org:8080/#/c/21383/4/be/src/service/data-stream-service.cc
File be/src/service/data-stream-service.cc:

http://gerrit.cloudera.org:8080/#/c/21383/4/be/src/service/data-stream-service.cc@159
PS4, Line 159: } else if (qs->is_initialized()) {
> It is more likely, but if IsTerminalState() is true, there is no need to pr
Done


http://gerrit.cloudera.org:8080/#/c/21383/5/be/src/util/network-util.h
File be/src/util/network-util.h:

http://gerrit.cloudera.org:8080/#/c/21383/5/be/src/util/network-util.h@118
PS5, Line 118: /// Compare function for two NetworkAddressPB.
Please add unit tests in network-util-test.cc either in this change or a 
follow-up change.



--
To view, visit http://gerrit.cloudera.org:8080/21383
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I156d1f0c694b91ba34be70bc53ae9bacf924b3b9
Gerrit-Change-Number: 21383
Gerrit-PatchSet: 5
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Jason Fehr 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Michael Smith 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Comment-Date: Wed, 08 May 2024 20:46:30 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote

Michael Smith has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21383 )

Change subject: IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote
..


Patch Set 5: Code-Review+1

(1 comment)

http://gerrit.cloudera.org:8080/#/c/21383/5/be/src/runtime/query-state.h
File be/src/runtime/query-state.h:

http://gerrit.cloudera.org:8080/#/c/21383/5/be/src/runtime/query-state.h@328
PS5, Line 328: BackendExecState exec_state = backend_exec_state_;
> backend_exec_state_ can only transition to terminal state once and wont rev
Ack



--
To view, visit http://gerrit.cloudera.org:8080/21383
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I156d1f0c694b91ba34be70bc53ae9bacf924b3b9
Gerrit-Change-Number: 21383
Gerrit-PatchSet: 5
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Jason Fehr 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Michael Smith 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Comment-Date: Wed, 08 May 2024 20:41:53 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote

Riza Suminto has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21383 )

Change subject: IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote
..


Patch Set 5:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/21383/5/be/src/runtime/query-state.h
File be/src/runtime/query-state.h:

http://gerrit.cloudera.org:8080/#/c/21383/5/be/src/runtime/query-state.h@328
PS5, Line 328: BackendExecState exec_state = backend_exec_state_;
> Shouldn't this be atomic if we need to worry about it's value changing? Or
backend_exec_state_ can only transition to terminal state once and wont revert 
back.

This is just to ensure that we are comparing 1 constant value for 3 times below 
rather than checking against backend_exec_state_ that might be modified in 
between three checks (say, backend_exec_state_lock_ is changing from EXECUTING 
to FINISHED when in the middle of backend_exec_state_ == ERROR check).
I don't think obtaining backend_exec_state_lock_ is necessary because it will 
release the lock anyway when applying runtime filter update. Existing codes in 
query-state.cc also call IsTerminalState() without holding lock.



--
To view, visit http://gerrit.cloudera.org:8080/21383
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I156d1f0c694b91ba34be70bc53ae9bacf924b3b9
Gerrit-Change-Number: 21383
Gerrit-PatchSet: 5
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Jason Fehr 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Michael Smith 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Comment-Date: Wed, 08 May 2024 20:35:34 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote

2024-05-08 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21383 )

Change subject: IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote
..


Patch Set 5:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/16141/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/21383
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I156d1f0c694b91ba34be70bc53ae9bacf924b3b9
Gerrit-Change-Number: 21383
Gerrit-PatchSet: 5
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Jason Fehr 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Michael Smith 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Comment-Date: Wed, 08 May 2024 20:34:37 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote

Michael Smith has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21383 )

Change subject: IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote
..


Patch Set 5:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/21383/5/be/src/runtime/query-state.h
File be/src/runtime/query-state.h:

http://gerrit.cloudera.org:8080/#/c/21383/5/be/src/runtime/query-state.h@328
PS5, Line 328: BackendExecState exec_state = backend_exec_state_;
Shouldn't this be atomic if we need to worry about it's value changing? Or take 
backend_exec_state_lock_?



--
To view, visit http://gerrit.cloudera.org:8080/21383
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I156d1f0c694b91ba34be70bc53ae9bacf924b3b9
Gerrit-Change-Number: 21383
Gerrit-PatchSet: 5
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Jason Fehr 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Michael Smith 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Comment-Date: Wed, 08 May 2024 20:21:25 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote

Riza Suminto has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21383 )

Change subject: IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote
..


Patch Set 5:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/21383/4/be/src/scheduling/scheduler.cc
File be/src/scheduling/scheduler.cc:

http://gerrit.cloudera.org:8080/#/c/21383/4/be/src/scheduling/scheduler.cc@341
PS4, Line 341:   if (UNLIKELY(FLAGS_sort_runtime_filter_aggregator_candidates)) 
{
> Would it be worthwhile to wrap this condition with UNLIKELY since the flag
Done


http://gerrit.cloudera.org:8080/#/c/21383/4/be/src/service/data-stream-service.cc
File be/src/service/data-stream-service.cc:

http://gerrit.cloudera.org:8080/#/c/21383/4/be/src/service/data-stream-service.cc@159
PS4, Line 159: } else if (qs->is_initialized()) {
> Is this condition the more likely to happen?  If so, please make it the if
It is more likely, but if IsTerminalState() is true, there is no need to 
process filter update at all.
Therefore, it is checked first in the earlier branch.

Removed IsCancelled() check since IsTerminalState() already include 
BackendExecState::CANCELLED checking.


http://gerrit.cloudera.org:8080/#/c/21383/4/be/src/util/network-util.h
File be/src/util/network-util.h:

http://gerrit.cloudera.org:8080/#/c/21383/4/be/src/util/network-util.h@124
PS4, Line 124: if (lhs.has_uds_address()) {
> What if only one has_uds_address? Seems like they should be unequal then.
Done.



--
To view, visit http://gerrit.cloudera.org:8080/21383
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I156d1f0c694b91ba34be70bc53ae9bacf924b3b9
Gerrit-Change-Number: 21383
Gerrit-PatchSet: 5
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Jason Fehr 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Michael Smith 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Comment-Date: Wed, 08 May 2024 20:17:36 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote

Hello Kurt Deschler, Jason Fehr, Csaba Ringhofer, Michael Smith, Impala Public 
Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/21383

to look at the new patch set (#5).

Change subject: IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote
..

IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote

It is possible to have UpdateFilterFromRemote RPC arrive to an impalad
executor before QueryState of the destination query is created or
complete initialization. This patch add wait mechanism in
UpdateFilterFromRemote RPC endpoint to wait for few miliseconds until
QueryState exist and complete initialization.

The wait time is fixed at 500ms, with exponential sleep period in
between. If wait time passed and QueryState still not found or
initialized, UpdateFilterFromRemote RPC is deemed fail and query
execution move on without complete filter.

Testing:
- Add test_runtime_filter_aggregation.py::TestLateQueryStateInit
- Pass exhastive runs of test_runtime_filter_aggregation.py

Change-Id: I156d1f0c694b91ba34be70bc53ae9bacf924b3b9
---
M be/src/runtime/query-exec-mgr.cc
M be/src/runtime/query-state.cc
M be/src/runtime/query-state.h
M be/src/runtime/runtime-filter-bank.cc
M be/src/runtime/runtime-filter-bank.h
M be/src/scheduling/scheduler.cc
M be/src/service/data-stream-service.cc
M be/src/util/network-util.h
M common/protobuf/data_stream_service.proto
M tests/custom_cluster/test_runtime_filter_aggregation.py
10 files changed, 190 insertions(+), 25 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/21383/5
--
To view, visit http://gerrit.cloudera.org:8080/21383
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I156d1f0c694b91ba34be70bc53ae9bacf924b3b9
Gerrit-Change-Number: 21383
Gerrit-PatchSet: 5
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Jason Fehr 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Michael Smith 
Gerrit-Reviewer: Riza Suminto

[Impala-ASF-CR] IMPALA-13038: Support profile tab for imported query profiles

2024-05-08 Thread Kurt Deschler (Code Review)

Kurt Deschler has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21400 )

Change subject: IMPALA-13038: Support profile tab for imported query profiles
..


Patch Set 4:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/21400/3/www/query_profile.tmpl
File www/query_profile.tmpl:

http://gerrit.cloudera.org:8080/#/c/21400/3/www/query_profile.tmpl@62
PS3, Line 62:   result = `${result}${indent}Offset: 
${eventSeq.offset}\n`;
> Check performance of these repeated appends. It may be better to accumulate
Is this approach actually faster? Concatenation is only efficient if the string 
doesn't reserve space. Please check javascript docs and measure performance.


http://gerrit.cloudera.org:8080/#/c/21400/4/www/scripts/util.js
File www/scripts/util.js:

http://gerrit.cloudera.org:8080/#/c/21400/4/www/scripts/util.js@105
PS4, Line 105:   return 
window.location.search.substring(10,window.location.search.indexOf("&"));
Should parse out using the query_id tag. These attributes can appear in any 
order in URLs.



--
To view, visit http://gerrit.cloudera.org:8080/21400
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iddcf2e285abbf42f97bde19014be076ccd6374bc
Gerrit-Change-Number: 21400
Gerrit-PatchSet: 4
Gerrit-Owner: Surya Hebbar 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Surya Hebbar 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Comment-Date: Wed, 08 May 2024 20:02:08 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11499: Refactor UrlEncode function to handle special characters

2024-05-08 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21131 )

Change subject: IMPALA-11499: Refactor UrlEncode function to handle special 
characters
..


Patch Set 27:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/16140/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/21131
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I88c4aba5d811dfcec809583d0c16fcbc0ca730fb
Gerrit-Change-Number: 21131
Gerrit-PatchSet: 27
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Anonymous Coward 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Smith 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zihao Ye 
Gerrit-Comment-Date: Wed, 08 May 2024 18:39:34 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys

2024-05-08 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21382 )

Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
..


Patch Set 12:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/16139/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/21382
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74
Gerrit-Change-Number: 21382
Gerrit-PatchSet: 12
Gerrit-Owner: gaurav singh 
Gerrit-Reviewer: Abhishek Rawat 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Reviewer: gaurav singh 
Gerrit-Comment-Date: Wed, 08 May 2024 18:35:47 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote

Michael Smith has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21383 )

Change subject: IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote
..


Patch Set 4:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/21383/4/be/src/util/network-util.h
File be/src/util/network-util.h:

http://gerrit.cloudera.org:8080/#/c/21383/4/be/src/util/network-util.h@124
PS4, Line 124: comp = lhs.uds_address().compare(rhs.uds_address());
What if only one has_uds_address? Seems like they should be unequal then.



--
To view, visit http://gerrit.cloudera.org:8080/21383
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I156d1f0c694b91ba34be70bc53ae9bacf924b3b9
Gerrit-Change-Number: 21383
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Jason Fehr 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Michael Smith 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Comment-Date: Wed, 08 May 2024 18:19:28 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote

2024-05-08 Thread Jason Fehr (Code Review)

Jason Fehr has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21383 )

Change subject: IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote
..


Patch Set 4: Code-Review+1

(2 comments)

Looks good, couple of questions that may not need code changes depending on the 
answer.

http://gerrit.cloudera.org:8080/#/c/21383/4/be/src/scheduling/scheduler.cc
File be/src/scheduling/scheduler.cc:

http://gerrit.cloudera.org:8080/#/c/21383/4/be/src/scheduling/scheduler.cc@341
PS4, Line 341:   if (FLAGS_sort_runtime_filter_aggregator_candidates) {
Would it be worthwhile to wrap this condition with UNLIKELY since the flag is 
intended for testing?


http://gerrit.cloudera.org:8080/#/c/21383/4/be/src/service/data-stream-service.cc
File be/src/service/data-stream-service.cc:

http://gerrit.cloudera.org:8080/#/c/21383/4/be/src/service/data-stream-service.cc@159
PS4, Line 159: } else if (qs->is_initialized()) {
Is this condition the more likely to happen?  If so, please make it the if 
condition and the existing if condition becomes the else if:

if (qs->is_initialized()) {
  ...
} else if (qs.get() == nullptr || qs->IsCancelled() || qs->IsTerminalState()) {
  ...
}



--
To view, visit http://gerrit.cloudera.org:8080/21383
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I156d1f0c694b91ba34be70bc53ae9bacf924b3b9
Gerrit-Change-Number: 21383
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Jason Fehr 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Michael Smith 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Comment-Date: Wed, 08 May 2024 18:18:22 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11499: Refactor UrlEncode function to handle special characters

Michael Smith has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21131 )

Change subject: IMPALA-11499: Refactor UrlEncode function to handle special 
characters
..


Patch Set 27: Verified+1 Code-Review+1


--
To view, visit http://gerrit.cloudera.org:8080/21131
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I88c4aba5d811dfcec809583d0c16fcbc0ca730fb
Gerrit-Change-Number: 21131
Gerrit-PatchSet: 27
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Anonymous Coward 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Smith 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zihao Ye 
Gerrit-Comment-Date: Wed, 08 May 2024 18:15:52 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11499: Refactor UrlEncode function to handle special characters

pranav.lo...@cloudera.com has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21131 )

Change subject: IMPALA-11499: Refactor UrlEncode function to handle special 
characters
..


Patch Set 27:

(1 comment)

> Uploaded patch set 27.

http://gerrit.cloudera.org:8080/#/c/21131/26/tests/query_test/test_insert.py
File tests/query_test/test_insert.py:

http://gerrit.cloudera.org:8080/#/c/21131/26/tests/query_test/test_insert.py@343
PS26, Line 343:
> This.
Done



--
To view, visit http://gerrit.cloudera.org:8080/21131
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I88c4aba5d811dfcec809583d0c16fcbc0ca730fb
Gerrit-Change-Number: 21131
Gerrit-PatchSet: 27
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Anonymous Coward 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Smith 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zihao Ye 
Gerrit-Comment-Date: Wed, 08 May 2024 18:12:34 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys

gaurav singh has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21382 )

Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
..


Patch Set 12:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/21382/10/be/src/util/jwt-util.cc
File be/src/util/jwt-util.cc:

http://gerrit.cloudera.org:8080/#/c/21382/10/be/src/util/jwt-util.cc@403
PS10, Line 403: al
> nit: extra spaces
Done



--
To view, visit http://gerrit.cloudera.org:8080/21382
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74
Gerrit-Change-Number: 21382
Gerrit-PatchSet: 12
Gerrit-Owner: gaurav singh 
Gerrit-Reviewer: Abhishek Rawat 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Reviewer: gaurav singh 
Gerrit-Comment-Date: Wed, 08 May 2024 18:12:11 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11499: Refactor UrlEncode function to handle special characters

pranav.lo...@cloudera.com has uploaded a new patch set (#27). ( 
http://gerrit.cloudera.org:8080/21131 )

Change subject: IMPALA-11499: Refactor UrlEncode function to handle special 
characters
..

IMPALA-11499: Refactor UrlEncode function to handle special characters

An error came from an issue with URL encoding, where certain Unicode
characters were being incorrectly encoded due to their UTF-8
representation matching characters in the set of characters to escape.
For example, the string '运', which consists of three bytes
0xe8 0xbf 0x90 was wrongly getting encoded into '\E8%FFBF\90',
because the middle byte matched one of the two bytes that
represented the "\u00FF" literal. Inclusion of "\u00FF" was likely
a mistake from the beginning and it should have been '\x7F'.

The patch makes three key changes:
1. Before the change, the set of characters that need to be escaped
was stored as a string. The current patch uses an unordered_set
instead.

2. '\xFF', which is an invalid UTF-8 byte and whose inclusion was
erroneous from the beginning, is replaced with '\x7F', which is a
control character for DELETE, ensuring consistency and correctness in
URL encoding.

3. The list of characters to be escaped is extended to match the
current list in Hive.

Testing: Tests on both traditional Hive tables and Iceberg tables
are included in unicode-column-name.test, insert.test,
coding-util-test.cc and test_insert.py.

Change-Id: I88c4aba5d811dfcec809583d0c16fcbc0ca730fb
---
M be/src/util/coding-util-test.cc
M be/src/util/coding-util.cc
M testdata/workloads/functional-query/queries/QueryTest/insert.test
M testdata/workloads/functional-query/queries/QueryTest/unicode-column-name.test
M tests/query_test/test_insert.py
5 files changed, 152 insertions(+), 20 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/21131/27
--
To view, visit http://gerrit.cloudera.org:8080/21131
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I88c4aba5d811dfcec809583d0c16fcbc0ca730fb
Gerrit-Change-Number: 21131
Gerrit-PatchSet: 27
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Anonymous Coward 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Smith 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zihao Ye

[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys

Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/21382

to look at the new patch set (#12).

Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
..

IMPALA-12559: Support x5c Parameter for RSA JSON
Web Keys

This enables the jwt verification using the x5c
certificate(s) in the RSA jwks keys. The x5c claim can be
part of the jwks either as a string or an array.
This patch only supports a single x5c certificate per
jwk.

If the "x5c" is present and "alg" is not present,
then "alg" is extracted from the "x5c" certificate using the
signature algorithm. However, if "x5c" is not preseent, then
"alg" is a mandatory field on jwk.

Current mapping of signature algorithm string => algorithm:

sha224WithRSAEncryption => rs224
sha256WithRSAEncryption => rs256
sha384WithRSAEncryption => rs384
sha512WithRSAEncryption => rs512

If "x5c" is present, then it is given priority over other
mandatory fields like "n", "e" to construct the public key.

Testing:
* added unit test case VerifyJwtTokenWithx5cCertificate to
verify jwt with x5c certificate.
* added unit test case VerifyJwtTokenWithx5cCertificateWithoutAlg
to verify jwt with x5c certificate without "alg".

Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74
---
M be/src/util/jwt-util-test.cc
M be/src/util/jwt-util.cc
2 files changed, 286 insertions(+), 16 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/12
--
To view, visit http://gerrit.cloudera.org:8080/21382
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74
Gerrit-Change-Number: 21382
Gerrit-PatchSet: 12
Gerrit-Owner: gaurav singh 
Gerrit-Reviewer: Abhishek Rawat 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Reviewer: gaurav singh

[Impala-ASF-CR] IMPALA-12607: Bump the GBN and fetch events specific to the db/table from the metastore

2024-05-08 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20979 )

Change subject: IMPALA-12607: Bump the GBN and fetch events specific to the 
db/table from the metastore
..


Patch Set 11:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/16138/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/20979
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6aecd5108b31c24e6e2c6f9fba6d4d44a3b00729
Gerrit-Change-Number: 20979
Gerrit-PatchSet: 11
Gerrit-Owner: Sai Hemanth Gantasala 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Sai Hemanth Gantasala 
Gerrit-Comment-Date: Wed, 08 May 2024 18:06:57 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11499: Refactor UrlEncode function to handle special characters

Michael Smith has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21131 )

Change subject: IMPALA-11499: Refactor UrlEncode function to handle special 
characters
..


Patch Set 26:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/21131/26/tests/query_test/test_insert.py
File tests/query_test/test_insert.py:

http://gerrit.cloudera.org:8080/#/c/21131/26/tests/query_test/test_insert.py@343
PS26, Line 343: =
> flake8: E225 missing whitespace around operator
This.



--
To view, visit http://gerrit.cloudera.org:8080/21131
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I88c4aba5d811dfcec809583d0c16fcbc0ca730fb
Gerrit-Change-Number: 21131
Gerrit-PatchSet: 26
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Anonymous Coward 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Smith 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zihao Ye 
Gerrit-Comment-Date: Wed, 08 May 2024 18:01:35 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys

2024-05-08 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21382 )

Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
..


Patch Set 11:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/16137/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/21382
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74
Gerrit-Change-Number: 21382
Gerrit-PatchSet: 11
Gerrit-Owner: gaurav singh 
Gerrit-Reviewer: Abhishek Rawat 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Reviewer: gaurav singh 
Gerrit-Comment-Date: Wed, 08 May 2024 17:50:00 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12607: Bump the GBN and fetch events specific to the db/table from the metastore

2024-05-08 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20979 )

Change subject: IMPALA-12607: Bump the GBN and fetch events specific to the 
db/table from the metastore
..


Patch Set 11:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10620/ 
DRY_RUN=true


--
To view, visit http://gerrit.cloudera.org:8080/20979
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6aecd5108b31c24e6e2c6f9fba6d4d44a3b00729
Gerrit-Change-Number: 20979
Gerrit-PatchSet: 11
Gerrit-Owner: Sai Hemanth Gantasala 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Sai Hemanth Gantasala 
Gerrit-Comment-Date: Wed, 08 May 2024 17:42:04 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12607: Bump the GBN and fetch events specific to the db/table from the metastore

2024-05-08 Thread Sai Hemanth Gantasala (Code Review)

Hello Quanlong Huang, Csaba Ringhofer, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/20979

to look at the new patch set (#11).

Change subject: IMPALA-12607: Bump the GBN and fetch events specific to the 
db/table from the metastore
..

IMPALA-12607: Bump the GBN and fetch events specific to the db/table
from the metastore

Bump the GBN to 49623641 to leverage HIVE-27499, so that Impala can
directly fetch the latest events specific to the db/table from the
metastore, instead of fetching the events from metastore and then
filtering in the cache matching the DbName/TableName.

Implementation Details:
Currently when a DDL/DML is performed in Impala, we fetch all the
events from metastore based on current eventId and then filter them in
Impala which can be a bottleneck if the events count is huge. This can
be optimized by including db name and/or table name in the notification
event request object and then filter by event type in impala. This can
provide performance boost on tables that generate a lot of events.

Note:
Also included ShowUtils class in hive-minimal-exec jar as it is
required in the current build version

Testing:
1) Did some tests in local cluster
2) Added a test case in MetaStoreEventsProcessorTest

Change-Id: I6aecd5108b31c24e6e2c6f9fba6d4d44a3b00729
---
M bin/impala-config.sh
M 
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M 
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M 
fe/src/test/java/org/apache/impala/catalog/metastore/CatalogHmsSyncToLatestEventIdTest.java
M java/shaded-deps/hive-exec/pom.xml
5 files changed, 199 insertions(+), 33 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/20979/11
--
To view, visit http://gerrit.cloudera.org:8080/20979
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6aecd5108b31c24e6e2c6f9fba6d4d44a3b00729
Gerrit-Change-Number: 20979
Gerrit-PatchSet: 11
Gerrit-Owner: Sai Hemanth Gantasala 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Sai Hemanth Gantasala

[Impala-ASF-CR] IMPALA-13038: Support profile tab for imported query profiles

2024-05-08 Thread Wenzhe Zhou (Code Review)

Wenzhe Zhou has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21400 )

Change subject: IMPALA-13038: Support profile tab for imported query profiles
..


Patch Set 4:

Tried the patch set 4. Timeline tab shows nothing for an imported json profile. 
Other three tabs work fine.


--
To view, visit http://gerrit.cloudera.org:8080/21400
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iddcf2e285abbf42f97bde19014be076ccd6374bc
Gerrit-Change-Number: 21400
Gerrit-PatchSet: 4
Gerrit-Owner: Surya Hebbar 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Surya Hebbar 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Comment-Date: Wed, 08 May 2024 17:37:48 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys

2024-05-08 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21382 )

Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
..


Patch Set 11:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/21382/11/be/src/util/jwt-util-test.cc
File be/src/util/jwt-util-test.cc:

http://gerrit.cloudera.org:8080/#/c/21382/11/be/src/util/jwt-util-test.cc@337
PS11, Line 337: std::string rsa_pub_key_jwk_x5c = 
R"(MIIE2jCCAsICAQEwDQYJKoZIhvcNAQELBQAwMzELMAkGA1UEBhMCVVMxEDAOBgNVBAoMB0pXVC1DUFAxEjAQBgNVBAMMCWxvY2FsaG9zdDAeFw0yMzEyMjIxMzIzNTdaFw0zMzEyMTkxMzIzNTdaMDMxCzAJBgNVBAYTAlVTMRAwDgYDVQQKDAdKV1QtQ1BQMRIwEAYDVQQDDAlsb2NhbGhvc3QwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDl0gyL9KCpoXsJlvBaBUsJTAgvFhbgsBjpxT6m2xv0fgtGDBwgaiPvuMnClTU/kYkKb8c1GTkMedKp9YcM57HWTk9yqGTy6QBnMMxbAJYNwWQ4Dbr4qKSC6C3KzYws/Bqyv8OC9NAOyqJbtdp4iObRjyaet+PLTXywuu02xtyRg3B+1aAONgUVDyS5u57NSD4rEZ+rw30Ne1doSClWmMDqEd72y8cjx3eAqn0HcAxSQ6MNMmNk7/M8FQD3DTM1Ef0G5oHyJIw7WmY+gxuD8386r/CkswINzadMwObPlTSdAN8BRzedtrqgb+D/K4pi2zhCiuIVujFX6M/hsGvj7g2M9E9MR8iEuHWCY9frQKIR+JTH3D1snoJp60qKoa51qBznsEr9RP2utGniPCq3+JY+ZX0JK8vl5tiSZpy6N0yRbRmY3XLdA5fKRzhcsB3eUrmTtr9ywjZX7Ll6QMvUyicubGTojhqJFQbvuvvops9PoCMXFE3x6cJ2QhPoi8+BvUdYisrtjDFe+YgrgQvPMa/CpOpDJJDEs2SVRcauCZOUdqLCwZylNuW0CgIjWP8l99P7l1zGeT8VJPhmABYyPM+RtNYDamAlUOCqRqgz/gPjEeMeulQTvH1lAqATAAX1oftlq6o4VoqROs2M3eAXqPhvsLBeTmCob+5ca887MkcP6wIDAQABMA0GCSqGSIb3DQEBCwUAA4ICAQBW2kREK4hlzxCDqykxrwfbQpiPwrbFmn+3RDJla+pI4L3wrvYT1nU96guFIU3zKnbMzqwPMRUCUjadr2jKxAmMWxCd/ThHQB+ne5xTvx7/6RVQfGjyMCG/SZtSH8/aO7ILNRtPT+SL5ZZwezaqv6gD89tSXB/w/0pYXy70wDuU17KCrTsKSISWGJ1cKi5l2R/m/ZaGjcV8U8NcFepF2bX3u/i0zhaqOqjiwrSEt7fWGDLabPs6n7GtfibZROEDZ/h0JrDINC+6mSfTOYAMJvGjeHA3H/NvzqR+CJgpXGCqElqVuBF0HdxPmwRRBoZC/BLIEcz0VHmB4rcpfaV47TZT+J+04fHYp4Y1S0u112CDrDe+61cDrnbDHC7aGX0G93pYSBKAB1e3LLc9rXQgf2F0pRtFB3rgZA9MtJ+TL7DUvY4VXJNq3v7UolIdldYRdk21YqAS2Hp0fivvFoEk2P/WbwDEErxR0FkZ/JQoI9FMJ9AvDxa4MsFFtlQVInfD2HUu+nhnuEAA8R6L+F2XqhfLY/H7H31iFBK6UCuqptED71VwWHqfBsAPRhLXAqGco7Ln2dzioyj0QdwJqQQIqigltSYtXxfIMLW0BekQ5yln7QTxnZlobkPHUW9s3NK+OMLuKCzVREzjic/aioQP3cRBMXkG2deMwrk3aX8yJuz4gA==)";
line too long (1704 > 90)



--
To view, visit http://gerrit.cloudera.org:8080/21382
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74
Gerrit-Change-Number: 21382
Gerrit-PatchSet: 11
Gerrit-Owner: gaurav singh 
Gerrit-Reviewer: Abhishek Rawat 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Reviewer: gaurav singh 
Gerrit-Comment-Date: Wed, 08 May 2024 17:26:20 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys

Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/21382

to look at the new patch set (#11).

Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
..

IMPALA-12559: Support x5c Parameter for RSA JSON
Web Keys

This enables the jwt verification using the x5c
certificate(s) in the RSA jwks keys. The x5c claim can be
part of the jwks either as a string or an array.
This patch only supports a single x5c certificate per
jwk.

If the "x5c" is present and "alg" is not present,
then "alg" is extracted from the "x5c" certificate using the
signature algorithm. However, if "x5c" is not preseent, then
"alg" is a mandatory field on jwk.

Current mapping of signature algorithm string => algorithm:

sha224WithRSAEncryption => rs224
sha256WithRSAEncryption => rs256
sha384WithRSAEncryption => rs384
sha512WithRSAEncryption => rs512

If "x5c" is present, then it is given priority over other
mandatory fields like "n", "e" to construct the public key.

Testing:
* added unit test case VerifyJwtTokenWithx5cCertificate to
verify jwt with x5c certificate.
* added unit test case VerifyJwtTokenWithx5cCertificateWithoutAlg
to verify jwt with x5c certificate without "alg".

Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74
---
M be/src/util/jwt-util-test.cc
M be/src/util/jwt-util.cc
2 files changed, 267 insertions(+), 16 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/11
--
To view, visit http://gerrit.cloudera.org:8080/21382
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74
Gerrit-Change-Number: 21382
Gerrit-PatchSet: 11
Gerrit-Owner: gaurav singh 
Gerrit-Reviewer: Abhishek Rawat 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Reviewer: gaurav singh

[Impala-ASF-CR] IMPALA-13038: Support profile tab for imported query profiles

2024-05-08 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21400 )

Change subject: IMPALA-13038: Support profile tab for imported query profiles
..


Patch Set 4:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/16136/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/21400
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iddcf2e285abbf42f97bde19014be076ccd6374bc
Gerrit-Change-Number: 21400
Gerrit-PatchSet: 4
Gerrit-Owner: Surya Hebbar 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Surya Hebbar 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Comment-Date: Wed, 08 May 2024 17:02:04 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13038: Support profile tab for imported query profiles

Surya Hebbar has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21400 )

Change subject: IMPALA-13038: Support profile tab for imported query profiles
..


Patch Set 4:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/21400/3/be/src/service/impala-http-handler.cc
File be/src/service/impala-http-handler.cc:

http://gerrit.cloudera.org:8080/#/c/21400/3/be/src/service/impala-http-handler.cc@307
PS3, Line 307:   if (!parse_status.ok()) {
> Move these after parse_status error check.
Done


http://gerrit.cloudera.org:8080/#/c/21400/3/www/query_profile.tmpl
File www/query_profile.tmpl:

http://gerrit.cloudera.org:8080/#/c/21400/3/www/query_profile.tmpl@62
PS3, Line 62:   result = `${result}${indent}Offset: 
${eventSeq.offset}\n`;
> Check performance of these repeated appends. It may be better to accumulate
Done. I have further optimized these by reducing number of concatenations.

Currently, JSON profiles upto 250MBs do not pose a problem as they can be 
stored in indexedDB.

Much larger profiles have to be stored after compression. This can be 
implemented through Compression Stream APIs available in recent browsers(2023 
and afterwards).

I am planning to implement this in the next patch.


http://gerrit.cloudera.org:8080/#/c/21400/3/www/query_profile.tmpl@113
PS3, Line 113: };
> Is there a tag that can be checked here? Please add a comment of where this
Done



--
To view, visit http://gerrit.cloudera.org:8080/21400
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iddcf2e285abbf42f97bde19014be076ccd6374bc
Gerrit-Change-Number: 21400
Gerrit-PatchSet: 4
Gerrit-Owner: Surya Hebbar 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Surya Hebbar 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Comment-Date: Wed, 08 May 2024 16:38:55 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13038: Support profile tab for imported query profiles

Surya Hebbar has uploaded a new patch set (#4). ( 
http://gerrit.cloudera.org:8080/21400 )

Change subject: IMPALA-13038: Support profile tab for imported query profiles
..

IMPALA-13038: Support profile tab for imported query profiles

For query profile imports currently the following tabs are supported.
 - Query Statement
 - Query Timeline
 - Query Text Plan

With the current patch "Query Profile" tab will also be supported.

In the "QueryProfileHandler", "query_id" is now added before verifying
its existence in the query log as in "QuerySummaryHandler" and others.

"getQueryID" function has been added to "util.js", as it is helpful
across multiple query pages for retrieving the query ID into JS scripts,
before the page loads up.

On loading the imported "Query Profile" page, query profile download
section and server's non-existing query ID alerts are removed.
All unsupported navbar tabs are removed and current tab is set to active.

The query profile is retrieved from the indexedDB's "imported_queries"
database. Then query profile is passed onto "profileToString" function,
which converts the profile into indented text for displaying on the
profile page.

Each profile and its child profiles are printed in the following order
with the right indentation(fields are skipped, if they do not exist).

Profile name:
  - Info strings:
  - Event sequences:
- Offset:
- Events:
  - Child profile(recursive):
  - Counters:

Change-Id: Iddcf2e285abbf42f97bde19014be076ccd6374bc
---
M be/src/service/impala-http-handler.cc
M www/query_plan_text.tmpl
M www/query_profile.tmpl
M www/query_stmt.tmpl
M www/query_timeline.tmpl
M www/scripts/util.js
6 files changed, 94 insertions(+), 30 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/21400/4
--
To view, visit http://gerrit.cloudera.org:8080/21400
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iddcf2e285abbf42f97bde19014be076ccd6374bc
Gerrit-Change-Number: 21400
Gerrit-PatchSet: 4
Gerrit-Owner: Surya Hebbar 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Surya Hebbar 
Gerrit-Reviewer: Wenzhe Zhou

[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys

2024-05-08 Thread Wenzhe Zhou (Code Review)

Wenzhe Zhou has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21382 )

Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
..


Patch Set 10:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/21382/6//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/21382/6//COMMIT_MSG@22
PS6, Line 22:
> Could you add test case in fe/test/java/org/apache/impala/customcluster/Jwt
Please add end-end test case.


http://gerrit.cloudera.org:8080/#/c/21382/10/be/src/util/jwt-util.cc
File be/src/util/jwt-util.cc:

http://gerrit.cloudera.org:8080/#/c/21382/10/be/src/util/jwt-util.cc@403
PS10, Line 403:
nit: extra spaces



--
To view, visit http://gerrit.cloudera.org:8080/21382
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74
Gerrit-Change-Number: 21382
Gerrit-PatchSet: 10
Gerrit-Owner: gaurav singh 
Gerrit-Reviewer: Abhishek Rawat 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Reviewer: gaurav singh 
Gerrit-Comment-Date: Wed, 08 May 2024 16:18:52 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote

2024-05-08 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21383 )

Change subject: IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote
..


Patch Set 4:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/16135/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/21383
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I156d1f0c694b91ba34be70bc53ae9bacf924b3b9
Gerrit-Change-Number: 21383
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Michael Smith 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Comment-Date: Wed, 08 May 2024 15:47:09 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys

2024-05-08 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21382 )

Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
..


Patch Set 10:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/16134/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/21382
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74
Gerrit-Change-Number: 21382
Gerrit-PatchSet: 10
Gerrit-Owner: gaurav singh 
Gerrit-Reviewer: Abhishek Rawat 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Reviewer: gaurav singh 
Gerrit-Comment-Date: Wed, 08 May 2024 15:34:30 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote

Riza Suminto has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21383 )

Change subject: IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote
..


Patch Set 4:

(6 comments)

http://gerrit.cloudera.org:8080/#/c/21383/3/be/src/scheduling/scheduler.cc
File be/src/scheduling/scheduler.cc:

http://gerrit.cloudera.org:8080/#/c/21383/3/be/src/scheduling/scheduler.cc@59
PS3, Line 59: DEFINE_bool_hidden(sort_runtime_filter_aggregator_candidates, 
false,
> candidates
Done


http://gerrit.cloudera.org:8080/#/c/21383/3/be/src/scheduling/scheduler.cc@59
PS3, Line 59: sort_runtime_filter_aggregator_candidate
> typo: _candidates
Done


http://gerrit.cloudera.org:8080/#/c/21383/3/be/src/service/data-stream-service.cc
File be/src/service/data-stream-service.cc:

http://gerrit.cloudera.org:8080/#/c/21383/3/be/src/service/data-stream-service.cc@143
PS3, Line 143: sleep_duration_ms
> Maybe even less than 20ms. Especially if these sleeps can stack with comple
Changed to begin with 2ms sleep, and then keep doubling for next iteration.


http://gerrit.cloudera.org:8080/#/c/21383/3/be/src/util/network-util.h
File be/src/util/network-util.h:

http://gerrit.cloudera.org:8080/#/c/21383/3/be/src/util/network-util.h@123
PS3, Line 123:   if (comp == 0 && lhs.has_uds_address() && 
rhs.has_uds_address()) {
> Check has_uds_address since it is optional field.
Done


http://gerrit.cloudera.org:8080/#/c/21383/3/tests/custom_cluster/test_runtime_filter_aggregation.py
File tests/custom_cluster/test_runtime_filter_aggregation.py:

http://gerrit.cloudera.org:8080/#/c/21383/3/tests/custom_cluster/test_runtime_filter_aggregation.py@89
PS3, Line 89: cls._init_delay]
> nit: +2 indentations on broken lines
Done


http://gerrit.cloudera.org:8080/#/c/21383/3/tests/custom_cluster/test_runtime_filter_aggregation.py@121
PS3, Line 121: all_blocked = 'UpdateFilterFromRemote RPC called with 
remaining wait time'
> Could we check in the profile that the runtime filter did not arrive in the
Done



--
To view, visit http://gerrit.cloudera.org:8080/21383
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I156d1f0c694b91ba34be70bc53ae9bacf924b3b9
Gerrit-Change-Number: 21383
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Michael Smith 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Comment-Date: Wed, 08 May 2024 15:28:13 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote

Hello Kurt Deschler, Csaba Ringhofer, Michael Smith, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/21383

to look at the new patch set (#4).

Change subject: IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote
..

IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote

It is possible to have UpdateFilterFromRemote RPC arrive to an impalad
executor before QueryState of the destination query is created or
complete initialization. This patch add wait mechanism in
UpdateFilterFromRemote RPC endpoint to wait for few miliseconds until
QueryState exist and complete initialization.

The wait time is fixed at 500ms, with exponential sleep period in
between. If wait time passed and QueryState still not found or
initialized, UpdateFilterFromRemote RPC is deemed fail and query
execution move on without complete filter.

Testing:
- Add test_runtime_filter_aggregation.py::TestLateQueryStateInit
- Pass exhastive runs of test_runtime_filter_aggregation.py

Change-Id: I156d1f0c694b91ba34be70bc53ae9bacf924b3b9
---
M be/src/runtime/query-exec-mgr.cc
M be/src/runtime/query-state.cc
M be/src/runtime/query-state.h
M be/src/runtime/runtime-filter-bank.cc
M be/src/runtime/runtime-filter-bank.h
M be/src/scheduling/scheduler.cc
M be/src/service/data-stream-service.cc
M be/src/util/network-util.h
M common/protobuf/data_stream_service.proto
M tests/custom_cluster/test_runtime_filter_aggregation.py
10 files changed, 180 insertions(+), 25 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/21383/4
--
To view, visit http://gerrit.cloudera.org:8080/21383
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I156d1f0c694b91ba34be70bc53ae9bacf924b3b9
Gerrit-Change-Number: 21383
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Michael Smith 
Gerrit-Reviewer: Riza Suminto

[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys

gaurav singh has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21382 )

Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
..


Patch Set 10:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/21382/8/be/src/util/jwt-util.cc
File be/src/util/jwt-util.cc:

http://gerrit.cloudera.org:8080/#/c/21382/8/be/src/util/jwt-util.cc@356
PS8, Line 356: if (it_n == kv_map.end() || it_e == kv_map.end()) {
 : return Status("'n' and 'e' properties are required");
 :   } else if (it_n->second.empty() || it_e->second.empty()) {
 : return Status("'n' and 'e' properties must be a non-empty 
string");
 :   }
> Are "n" and "e" optional when 'x5c' is provided?
As per the RFC document, they are not optional when x5c is provided.


http://gerrit.cloudera.org:8080/#/c/21382/8/be/src/util/jwt-util.cc@381
PS8, Line 381: O_puts(bio, data);
> add { } around return statement.
good point. The man page for PEM_read_bio_X509() doesn't mention whether it 
frees it not and also no examples in openssl docs. I think its good to free it 
if PEM_read_bio_X509 errors out.


http://gerrit.cloudera.org:8080/#/c/21382/8/be/src/util/jwt-util.cc@407
PS8, Line 407: pu
> nit: extra indent spaces.
Done



--
To view, visit http://gerrit.cloudera.org:8080/21382
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74
Gerrit-Change-Number: 21382
Gerrit-PatchSet: 10
Gerrit-Owner: gaurav singh 
Gerrit-Reviewer: Abhishek Rawat 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Reviewer: gaurav singh 
Gerrit-Comment-Date: Wed, 08 May 2024 15:11:02 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys

Hello Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/21382

to look at the new patch set (#10).

Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
..

IMPALA-12559: Support x5c Parameter for RSA JSON
Web Keys

This enables the jwt verification using the x5c
certificate(s) in the RSA jwks keys. The x5c claim can be
part of the jwks either as a string or an array.
This patch only supports a single x5c certificate per
jwk.

If the "x5c" is present and "alg" is not present,
then "alg" is extracted from the "x5c" certificate using the
signature algorithm. However, if "x5c" is not preseent, then
"alg" is a mandatory field on jwk.

Current mapping of signature algorithm string => algorithm:

sha224WithRSAEncryption => rs224
sha256WithRSAEncryption => rs256
sha384WithRSAEncryption => rs384
sha512WithRSAEncryption => rs512

If "x5c" is present, then it is given priority over other
mandatory fields like "n", "e" to construct the public key.

Testing:
* added unit test case VerifyJwtTokenWithx5cCertificate to
verify jwt with x5c certificate.
* added unit test case VerifyJwtTokenWithx5cCertificateWithoutAlg
to verify jwt with x5c certificate without "alg".

Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74
---
M be/src/util/jwt-util-test.cc
M be/src/util/jwt-util.cc
2 files changed, 259 insertions(+), 16 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/21382/10
--
To view, visit http://gerrit.cloudera.org:8080/21382
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74
Gerrit-Change-Number: 21382
Gerrit-PatchSet: 10
Gerrit-Owner: gaurav singh 
Gerrit-Reviewer: Abhishek Rawat 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Reviewer: gaurav singh

[Impala-ASF-CR] IMPALA-11499: Refactor UrlEncode function to handle special characters

2024-05-08 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21131 )

Change subject: IMPALA-11499: Refactor UrlEncode function to handle special 
characters
..


Patch Set 26: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/21131
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I88c4aba5d811dfcec809583d0c16fcbc0ca730fb
Gerrit-Change-Number: 21131
Gerrit-PatchSet: 26
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Anonymous Coward 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Smith 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zihao Ye 
Gerrit-Comment-Date: Wed, 08 May 2024 15:12:37 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys

2024-05-08 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21382 )

Change subject: IMPALA-12559: Support x5c Parameter for RSA JSON Web Keys
..


Patch Set 10:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/21382/10/be/src/util/jwt-util-test.cc
File be/src/util/jwt-util-test.cc:

http://gerrit.cloudera.org:8080/#/c/21382/10/be/src/util/jwt-util-test.cc@325
PS10, Line 325: std::string rsa_pub_key_jwk_n_1 = 
"5dIMi_SgqaF7CZbwWgVLCUwILxYW4LAY6cU-ptsb9H4LRgwcIGoj77jJwpU1P5GJCm_HNRk5DHnSqfWHDOex1k5Pcqhk8ukAZzDMWwCWDcFkOA26-Kikgugtys2MLPwasr_DgvTQDsqiW7XaeIjm0Y8mnrfjy018sLrtNsbckYNwftWgDjYFFQ8kubuezUg-KxGfq8N9DXtXaEgpVpjA6hHe9svHI8d3gKp9B3AMUkOjDTJjZO_zPBUA9w0zNRH9BuaB8iSMO1pmPoMbg_N_Oq_wpLMCDc2nTMDmz5U0nQDfAUc3nba6oG_g_yuKYts4QoriFboxV-jP4bBr4-4NjPRPTEfIhLh1gmPX60CiEfiUx9w9bJ6CaetKiqGudagc57BK_UT9rrRp4jwqt_iWPmV9CSvL5ebYkmacujdMkW0ZmN1y3QOXykc4XLAd3lK5k7a_csI2V-y5ekDL1MonLmxk6I4aiRUG77r76KbPT6AjFxRN8enCdkIT6IvPgb1HWIrK7YwxXvmIK4ELzzGvwqTqQySQxLNklUXGrgmTlHaiwsGcpTbltAoCI1j_JffT-5dcxnk_FST4ZgAWMjzPkbTWA2pgJVDgqkaoM_4D4xHjHrpUE7x9ZQKgEwAF9aH7ZauqOFaKkTrNjN3gF6j4b7CwXk5gqG_uXGvPOzJHD-s";
line too long (720 > 90)


http://gerrit.cloudera.org:8080/#/c/21382/10/be/src/util/jwt-util-test.cc@329
PS10, Line 329: std::string rsa_pub_key_jwk_x5c = 
R"(MIIE2jCCAsICAQEwDQYJKoZIhvcNAQELBQAwMzELMAkGA1UEBhMCVVMxEDAOBgNVBAoMB0pXVC1DUFAxEjAQBgNVBAMMCWxvY2FsaG9zdDAeFw0yMzEyMjIxMzIzNTdaFw0zMzEyMTkxMzIzNTdaMDMxCzAJBgNVBAYTAlVTMRAwDgYDVQQKDAdKV1QtQ1BQMRIwEAYDVQQDDAlsb2NhbGhvc3QwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDl0gyL9KCpoXsJlvBaBUsJTAgvFhbgsBjpxT6m2xv0fgtGDBwgaiPvuMnClTU/kYkKb8c1GTkMedKp9YcM57HWTk9yqGTy6QBnMMxbAJYNwWQ4Dbr4qKSC6C3KzYws/Bqyv8OC9NAOyqJbtdp4iObRjyaet+PLTXywuu02xtyRg3B+1aAONgUVDyS5u57NSD4rEZ+rw30Ne1doSClWmMDqEd72y8cjx3eAqn0HcAxSQ6MNMmNk7/M8FQD3DTM1Ef0G5oHyJIw7WmY+gxuD8386r/CkswINzadMwObPlTSdAN8BRzedtrqgb+D/K4pi2zhCiuIVujFX6M/hsGvj7g2M9E9MR8iEuHWCY9frQKIR+JTH3D1snoJp60qKoa51qBznsEr9RP2utGniPCq3+JY+ZX0JK8vl5tiSZpy6N0yRbRmY3XLdA5fKRzhcsB3eUrmTtr9ywjZX7Ll6QMvUyicubGTojhqJFQbvuvvops9PoCMXFE3x6cJ2QhPoi8+BvUdYisrtjDFe+YgrgQvPMa/CpOpDJJDEs2SVRcauCZOUdqLCwZylNuW0CgIjWP8l99P7l1zGeT8VJPhmABYyPM+RtNYDamAlUOCqRqgz/gPjEeMeulQTvH1lAqATAAX1oftlq6o4VoqROs2M3eAXqPhvsLBeTmCob+5ca887MkcP6wIDAQABMA0GCSqGSIb3DQEBCwUAA4ICAQBW2kREK4hlzxCDqykxrwfbQpiPwrbFmn+3RDJla+pI4L3wrvYT1nU96guFIU3zKnbMzqwPMRUCUjadr2jKxAmMWxCd/ThHQB+ne5xTvx7/6RVQfGjyMCG/SZtSH8/aO7ILNRtPT+SL5ZZwezaqv6gD89tSXB/w/0pYXy70wDuU17KCrTsKSISWGJ1cKi5l2R/m/ZaGjcV8U8NcFepF2bX3u/i0zhaqOqjiwrSEt7fWGDLabPs6n7GtfibZROEDZ/h0JrDINC+6mSfTOYAMJvGjeHA3H/NvzqR+CJgpXGCqElqVuBF0HdxPmwRRBoZC/BLIEcz0VHmB4rcpfaV47TZT+J+04fHYp4Y1S0u112CDrDe+61cDrnbDHC7aGX0G93pYSBKAB1e3LLc9rXQgf2F0pRtFB3rgZA9MtJ+TL7DUvY4VXJNq3v7UolIdldYRdk21YqAS2Hp0fivvFoEk2P/WbwDEErxR0FkZ/JQoI9FMJ9AvDxa4MsFFtlQVInfD2HUu+nhnuEAA8R6L+F2XqhfLY/H7H31iFBK6UCuqptED71VwWHqfBsAPRhLXAqGco7Ln2dzioyj0QdwJqQQIqigltSYtXxfIMLW0BekQ5yln7QTxnZlobkPHUW9s3NK+OMLuKCzVREzjic/aioQP3cRBMXkG2deMwrk3aX8yJuz4gA==)";
line too long (1704 > 90)



--
To view, visit http://gerrit.cloudera.org:8080/21382
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I70be6f9f54190544aa005b2644e2ed8db6f6bb74
Gerrit-Change-Number: 21382
Gerrit-PatchSet: 10
Gerrit-Owner: gaurav singh 
Gerrit-Reviewer: Abhishek Rawat 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Reviewer: gaurav singh 
Gerrit-Comment-Date: Wed, 08 May 2024 15:11:30 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger

2024-05-08 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21160 )

Change subject: IMPALA-12921: Support locally built Ranger
..


Patch Set 11: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/21160
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27
Gerrit-Change-Number: 21160
Gerrit-PatchSet: 11
Gerrit-Owner: Fang-Yu Rao 
Gerrit-Reviewer: Aman Sinha 
Gerrit-Reviewer: Fang-Yu Rao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: John Sherman 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Comment-Date: Wed, 08 May 2024 14:52:47 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote

2024-05-08 Thread Kurt Deschler (Code Review)

Kurt Deschler has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21383 )

Change subject: IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote
..


Patch Set 3:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/21383/3/be/src/scheduling/scheduler.cc
File be/src/scheduling/scheduler.cc:

http://gerrit.cloudera.org:8080/#/c/21383/3/be/src/scheduling/scheduler.cc@59
PS3, Line 59: DEFINE_bool_hidden(sort_runtime_filter_aggregator_cadidates, 
false,
candidates


http://gerrit.cloudera.org:8080/#/c/21383/3/be/src/service/data-stream-service.cc
File be/src/service/data-stream-service.cc:

http://gerrit.cloudera.org:8080/#/c/21383/3/be/src/service/data-stream-service.cc@143
PS3, Line 143: sleep_duration_ms
> Isn't this too much, especially at the beginning? I would vote a short slee
Maybe even less than 20ms. Especially if these sleeps can stack with complex 
plan.


http://gerrit.cloudera.org:8080/#/c/21383/3/be/src/util/network-util.h
File be/src/util/network-util.h:

http://gerrit.cloudera.org:8080/#/c/21383/3/be/src/util/network-util.h@123
PS3, Line 123:   if (comp == 0) comp = 
lhs.uds_address().compare(rhs.uds_address());
Check has_uds_address since it is optional field.



--
To view, visit http://gerrit.cloudera.org:8080/21383
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I156d1f0c694b91ba34be70bc53ae9bacf924b3b9
Gerrit-Change-Number: 21383
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Michael Smith 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Comment-Date: Wed, 08 May 2024 14:45:04 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote

2024-05-08 Thread Csaba Ringhofer (Code Review)

Csaba Ringhofer has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21383 )

Change subject: IMPALA-13040: Add waiting mechanism in UpdateFilterFromRemote
..


Patch Set 3:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/21383/3/be/src/scheduling/scheduler.cc
File be/src/scheduling/scheduler.cc:

http://gerrit.cloudera.org:8080/#/c/21383/3/be/src/scheduling/scheduler.cc@59
PS3, Line 59: sort_runtime_filter_aggregator_cadidates
typo: _candidates


http://gerrit.cloudera.org:8080/#/c/21383/3/be/src/service/data-stream-service.cc
File be/src/service/data-stream-service.cc:

http://gerrit.cloudera.org:8080/#/c/21383/3/be/src/service/data-stream-service.cc@143
PS3, Line 143: sleep_duration_ms
Isn't this too much, especially at the beginning? I would vote a short sleep, 
e.g. 20ms on the first sleep.


http://gerrit.cloudera.org:8080/#/c/21383/3/tests/custom_cluster/test_runtime_filter_aggregation.py
File tests/custom_cluster/test_runtime_filter_aggregation.py:

http://gerrit.cloudera.org:8080/#/c/21383/3/tests/custom_cluster/test_runtime_filter_aggregation.py@89
PS3, Line 89:   cls, 'debug_action',
nit: +2 indentations on broken lines


http://gerrit.cloudera.org:8080/#/c/21383/3/tests/custom_cluster/test_runtime_filter_aggregation.py@121
PS3, Line 121: all_blocked = 'UpdateFilterFromRemote RPC called with 
remaining wait time'
Could we check in the profile that the runtime filter did not arrive in the end?



--
To view, visit http://gerrit.cloudera.org:8080/21383
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I156d1f0c694b91ba34be70bc53ae9bacf924b3b9
Gerrit-Change-Number: 21383
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Smith 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Comment-Date: Wed, 08 May 2024 14:33:15 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13038: Support profile tab for imported query profiles

2024-05-08 Thread Kurt Deschler (Code Review)

Kurt Deschler has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21400 )

Change subject: IMPALA-13038: Support profile tab for imported query profiles
..


Patch Set 3:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/21400/3/be/src/service/impala-http-handler.cc
File be/src/service/impala-http-handler.cc:

http://gerrit.cloudera.org:8080/#/c/21400/3/be/src/service/impala-http-handler.cc@307
PS3, Line 307:   Value query_id_val(PrintId(unique_id).c_str(), 
document->GetAllocator());
Move these after parse_status error check.


http://gerrit.cloudera.org:8080/#/c/21400/3/www/query_profile.tmpl
File www/query_profile.tmpl:

http://gerrit.cloudera.org:8080/#/c/21400/3/www/query_profile.tmpl@62
PS3, Line 62:   result += `${indent}Offset: ${eventSeq.offset}\n`;
Check performance of these repeated appends. It may be better to accumulate the 
sections into separate strings then append at the return.


http://gerrit.cloudera.org:8080/#/c/21400/3/www/query_profile.tmpl@113
PS3, Line 113: query.id = query.id.substring(10, query.id.indexOf("&"));
Is there a tag that can be checked here? Please add a comment of where this 
URL/string comes from and what the format is.



--
To view, visit http://gerrit.cloudera.org:8080/21400
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iddcf2e285abbf42f97bde19014be076ccd6374bc
Gerrit-Change-Number: 21400
Gerrit-PatchSet: 3
Gerrit-Owner: Surya Hebbar 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Surya Hebbar 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Comment-Date: Wed, 08 May 2024 14:09:26 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12709: Add support for hierarchical metastore event processing

2024-05-08 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21031 )

Change subject: IMPALA-12709: Add support for hierarchical metastore event 
processing
..


Patch Set 14:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/16133/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/21031
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I76d8a739f9db6d40f01028bfd786a85d83f9e5d6
Gerrit-Change-Number: 21031
Gerrit-PatchSet: 14
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Anonymous Coward 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Wed, 08 May 2024 13:51:16 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches

2024-05-08 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21412 )

Change subject: IMPALA-13034: Add logs and counters for HTTP profile requests 
blocking client fetches
..


Patch Set 2:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/16132/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/21412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc
Gerrit-Change-Number: 21412
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Comment-Date: Wed, 08 May 2024 13:44:29 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12709: Add support for hierarchical metastore event processing

2024-05-08 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21031 )

Change subject: IMPALA-12709: Add support for hierarchical metastore event 
processing
..


Patch Set 13:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/16131/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/21031
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I76d8a739f9db6d40f01028bfd786a85d83f9e5d6
Gerrit-Change-Number: 21031
Gerrit-PatchSet: 13
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Anonymous Coward 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Wed, 08 May 2024 13:33:06 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12709: Add support for hierarchical metastore event processing

Hello cclive1...@gmail.com, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/21031

to look at the new patch set (#14).

Change subject: IMPALA-12709: Add support for hierarchical metastore event 
processing
..

IMPALA-12709: Add support for hierarchical metastore event processing

At present, metastore event processor is single threaded. Notification
events are processed sequentially with a maximum limit of 1000 events
fetched and processed in a single batch. Multiple locks are used to
address the concurrency issues that may arise when catalog DDL
operation processing and metastore event processing tries to
access/update the catalog objects concurrently. Waiting for a lock or
file metadata loading of a table can slow the event processing and can
affect the processing of other events following it. Those events may
not be dependent on the previous event. Altogether it takes a very
long time to synchronize all the HMS events.

Existing metastore event processing is turned into multi-level
event processing with enable_hierarchical_event_processing flag. It
is not enabled by default. Idea is to segregate the events based on
their dependency, maintain the order of events as they occur within
the dependency and process them independently as much as possible:
1. All the events of a table are processed in the same order they
   have occurred.
2. Events of different tables are processed in parallel.
3. When a database is altered, all the table events relating to
   the database that occurred after the alter db event are processed
   only after the alter database event is processed.

Added a new hms_event_polling_interval_ms flag to support millisecond
precision event polling interval since the current flag in seconds.

Testing:
 - Executed existing end to end tests.
 - Added end-to-end test with enable_hierarchical_event_processing.
 - Added event processing performance tests. They are marked to skip.

Change-Id: I76d8a739f9db6d40f01028bfd786a85d83f9e5d6
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M be/src/util/event-metrics.cc
M be/src/util/event-metrics.h
M common/thrift/BackendGflags.thrift
M common/thrift/JniCatalog.thrift
M common/thrift/metrics.json
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/TableWriteId.java
A fe/src/main/java/org/apache/impala/catalog/events/DBBarrierEvent.java
A fe/src/main/java/org/apache/impala/catalog/events/DBEventExecutor.java
M fe/src/main/java/org/apache/impala/catalog/events/ExternalEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M 
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/events/NoOpEventProcessor.java
A fe/src/main/java/org/apache/impala/catalog/events/TableEventExecutor.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
M fe/src/main/java/org/apache/impala/util/DebugUtils.java
M fe/src/test/java/org/apache/impala/catalog/CatalogTableWriteIdTest.java
M 
fe/src/test/java/org/apache/impala/catalog/events/EventsProcessorStressTest.java
M 
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M 
fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java
M 
fe/src/test/java/org/apache/impala/catalog/metastore/CatalogHmsSyncToLatestEventIdTest.java
A tests/custom_cluster/test_event_processing_perf.py
M tests/util/event_processor_utils.py
29 files changed, 1,688 insertions(+), 100 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/21031/14
--
To view, visit http://gerrit.cloudera.org:8080/21031
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I76d8a739f9db6d40f01028bfd786a85d83f9e5d6
Gerrit-Change-Number: 21031
Gerrit-PatchSet: 14
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Anonymous Coward 
Gerrit-Reviewer: Impala Public Jenkins

[Impala-ASF-CR] IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches

Hello Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/21412

to look at the new patch set (#2).

Change subject: IMPALA-13034: Add logs and counters for HTTP profile requests 
blocking client fetches
..

IMPALA-13034: Add logs and counters for HTTP profile requests blocking client 
fetches

There are several endpoints in WebUI that can dump a query profile:
/query_profile, /query_profile_encoded, /query_profile_plain_text,
/query_profile_json. The HTTP handler thread goes into
ImpalaServer::GetRuntimeProfileOutput() which acquires lock of the
ClientRequestState. This could block client requests in fetching query
results.

To help identify this issue, this patch adds warning logs when such
profile dumping requests run slow and the query is still in-flight. Also
adds a profile counter, GetInFlightProfileTimeStats, for the summary
stats of this time. Dumping the profiles after the query is archived
(e.g. closed) won't be tracked.

Logs for slow http responses are also added. The thresholds are defined
by two new flags, slow_profile_dump_warning_threshold_ms, and
slow_http_response_warning_threshold_ms.

Note that dumping the profile in-flight won't always block the query,
e.g. if there are no client fetch requests or if the coordinator
fragment is idle waiting for executor fragment instances. So a long time
shown in GetInFlightProfileTimeStats doesn't mean it's hitting the
issue.

To better identify this issue, this patch adds another profile counter,
ClientFetchLockWaitTimer, as the cumulative time client fetch requests
waiting for locks.

Also fixes false positive logs for complaining invalid query handles.
Such logs are added in GetQueryHandle() when the query is not found in
the active query map, but it could still exist in the query log. This
removes the logs in GetQueryHandle() and lets the callers decide whether
to log the error.

Tests:
 - Added e2e test
 - Ran CORE tests

Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc
---
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-beeswax-server.cc
M be/src/service/impala-hs2-server.cc
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/util/webserver.cc
M tests/query_test/test_observability.py
8 files changed, 103 insertions(+), 13 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/21412/2
--
To view, visit http://gerrit.cloudera.org:8080/21412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc
Gerrit-Change-Number: 21412
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins

[Impala-ASF-CR] IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches

Quanlong Huang has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21412 )

Change subject: IMPALA-13034: Add logs and counters for HTTP profile requests 
blocking client fetches
..


Patch Set 2:

(5 comments)

http://gerrit.cloudera.org:8080/#/c/21412/1/tests/query_test/test_observability.py
File tests/query_test/test_observability.py:

http://gerrit.cloudera.org:8080/#/c/21412/1/tests/query_test/test_observability.py@972
PS1, Line 972:
> flake8: W605 invalid escape sequence '\('
Done


http://gerrit.cloudera.org:8080/#/c/21412/1/tests/query_test/test_observability.py@972
PS1, Line 972: )
> flake8: W605 invalid escape sequence '\)'
Done


http://gerrit.cloudera.org:8080/#/c/21412/1/tests/query_test/test_observability.py@985
PS1, Line 985: i
> flake8: F841 local variable 'page' is assigned to but never used
Done


http://gerrit.cloudera.org:8080/#/c/21412/1/tests/query_test/test_observability.py@990
PS1, Line 990: (
> flake8: W605 invalid escape sequence '\d'
Done


http://gerrit.cloudera.org:8080/#/c/21412/1/tests/query_test/test_observability.py@990
PS1, Line 990: )
> flake8: W605 invalid escape sequence '\)'
Done



--
To view, visit http://gerrit.cloudera.org:8080/21412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc
Gerrit-Change-Number: 21412
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Comment-Date: Wed, 08 May 2024 13:21:11 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12709: Add support for hierarchical metastore event processing

2024-05-08 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21031 )

Change subject: IMPALA-12709: Add support for hierarchical metastore event 
processing
..


Patch Set 13:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/21031/13/tests/custom_cluster/test_event_processing_perf.py
File tests/custom_cluster/test_event_processing_perf.py:

http://gerrit.cloudera.org:8080/#/c/21031/13/tests/custom_cluster/test_event_processing_perf.py@30
PS13, Line 30: @pytest.mark.skip(reason="Catalogd event processing performance 
tests")
flake8: E302 expected 2 blank lines, found 1


http://gerrit.cloudera.org:8080/#/c/21031/13/tests/custom_cluster/test_event_processing_perf.py@221
PS13, Line 221: (
flake8: E225 missing whitespace around operator



--
To view, visit http://gerrit.cloudera.org:8080/21031
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I76d8a739f9db6d40f01028bfd786a85d83f9e5d6
Gerrit-Change-Number: 21031
Gerrit-PatchSet: 13
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Anonymous Coward 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Wed, 08 May 2024 13:09:40 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12709: Add support for hierarchical metastore event processing

Hello cclive1...@gmail.com, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/21031

to look at the new patch set (#13).

Change subject: IMPALA-12709: Add support for hierarchical metastore event 
processing
..

IMPALA-12709: Add support for hierarchical metastore event processing

At present, metastore event processor is single threaded. Notification
events are processed sequentially with a maximum limit of 1000 events
fetched and processed in a single batch. Multiple locks are used to
address the concurrency issues that may arise when catalog DDL
operation processing and metastore event processing tries to
access/update the catalog objects concurrently. Waiting for a lock or
file metadata loading of a table can slow the event processing and can
affect the processing of other events following it. Those events may
not be dependent on the previous event. Altogether it takes a very
long time to synchronize all the HMS events.

Existing metastore event processing is turned into multi-level
event processing with enable_hierarchical_event_processing flag. It
is not enabled by default. Idea is to segregate the events based on
their dependency, maintain the order of events as they occur within
the dependency and process them independently as much as possible:
1. All the events of a table are processed in the same order they
   have occurred.
2. Events of different tables are processed in parallel.
3. When a database is altered, all the table events relating to
   the database that occurred after the alter db event are processed
   only after the alter database event is processed.

Added a new hms_event_polling_interval_ms flag to support millisecond
precision event polling interval since the current flag in seconds.

Testing:
 - Executed existing end to end tests.
 - Added end-to-end test with enable_hierarchical_event_processing.
 - Added event processing performance tests. They are marked to skip.

Change-Id: I76d8a739f9db6d40f01028bfd786a85d83f9e5d6
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M be/src/util/event-metrics.cc
M be/src/util/event-metrics.h
M common/thrift/BackendGflags.thrift
M common/thrift/JniCatalog.thrift
M common/thrift/metrics.json
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/TableWriteId.java
A fe/src/main/java/org/apache/impala/catalog/events/DBBarrierEvent.java
A fe/src/main/java/org/apache/impala/catalog/events/DBEventExecutor.java
M fe/src/main/java/org/apache/impala/catalog/events/ExternalEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M 
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/events/NoOpEventProcessor.java
A fe/src/main/java/org/apache/impala/catalog/events/TableEventExecutor.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
M fe/src/main/java/org/apache/impala/util/DebugUtils.java
M fe/src/test/java/org/apache/impala/catalog/CatalogTableWriteIdTest.java
M 
fe/src/test/java/org/apache/impala/catalog/events/EventsProcessorStressTest.java
M 
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M 
fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java
M 
fe/src/test/java/org/apache/impala/catalog/metastore/CatalogHmsSyncToLatestEventIdTest.java
A tests/custom_cluster/test_event_processing_perf.py
M tests/util/event_processor_utils.py
29 files changed, 1,687 insertions(+), 100 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/21031/13
--
To view, visit http://gerrit.cloudera.org:8080/21031
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I76d8a739f9db6d40f01028bfd786a85d83f9e5d6
Gerrit-Change-Number: 21031
Gerrit-PatchSet: 13
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Anonymous Coward 
Gerrit-Reviewer: Impala Public Jenkins

[Impala-ASF-CR] IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches

2024-05-08 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21412 )

Change subject: IMPALA-13034: Add logs and counters for HTTP profile requests 
blocking client fetches
..


Patch Set 1:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/16130/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/21412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc
Gerrit-Change-Number: 21412
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Wed, 08 May 2024 12:19:17 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches

Quanlong Huang has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/21412


Change subject: IMPALA-13034: Add logs and counters for HTTP profile requests 
blocking client fetches
..

IMPALA-13034: Add logs and counters for HTTP profile requests blocking client 
fetches

There are several endpoints in WebUI that can dump a query profile:
/query_profile, /query_profile_encoded, /query_profile_plain_text,
/query_profile_json. The HTTP handler thread goes into
ImpalaServer::GetRuntimeProfileOutput() which acquires lock of the
ClientRequestState. This could block client requests in fetching query
results.

To help identify this issue, this patch adds warning logs when such
profile dumping requests run slow and the query is still in-flight. Also
adds a profile counter, GetInFlightProfileTimeStats, for the summary
stats of this time. Dumping the profiles after the query is archived
(e.g. closed) won't be tracked.

Logs for slow http responses are also added. The thresholds are defined
by two new flags, slow_profile_dump_warning_threshold_ms, and
slow_http_response_warning_threshold_ms.

Note that dumping the profile in-flight won't always block the query,
e.g. if there are no client fetch requests or if the coordinator
fragment is idle waiting for executor fragment instances. So a long time
shown in GetInFlightProfileTimeStats doesn't mean it's hitting the
issue.

To better identify this issue, this patch adds another profile counter,
ClientFetchLockWaitTimer, as the cumulative time client fetch requests
waiting for locks.

Also fixes false positive logs for complaining invalid query handles.
Such logs are added in GetQueryHandle() when the query is not found in
the active query map, but it could still exist in the query log. This
removes the logs in GetQueryHandle() and lets the callers decide whether
to log the error.

Tests:
 - Added e2e test
 - Ran CORE tests

Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc
---
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-beeswax-server.cc
M be/src/service/impala-hs2-server.cc
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/util/webserver.cc
M tests/query_test/test_observability.py
8 files changed, 103 insertions(+), 13 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/21412/1
--
To view, visit http://gerrit.cloudera.org:8080/21412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc
Gerrit-Change-Number: 21412
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang

[Impala-ASF-CR] IMPALA-13034: Add logs and counters for HTTP profile requests blocking client fetches

2024-05-08 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21412 )

Change subject: IMPALA-13034: Add logs and counters for HTTP profile requests 
blocking client fetches
..


Patch Set 1:

(5 comments)

http://gerrit.cloudera.org:8080/#/c/21412/1/tests/query_test/test_observability.py
File tests/query_test/test_observability.py:

http://gerrit.cloudera.org:8080/#/c/21412/1/tests/query_test/test_observability.py@972
PS1, Line 972: \
flake8: W605 invalid escape sequence '\('


http://gerrit.cloudera.org:8080/#/c/21412/1/tests/query_test/test_observability.py@972
PS1, Line 972: \
flake8: W605 invalid escape sequence '\)'


http://gerrit.cloudera.org:8080/#/c/21412/1/tests/query_test/test_observability.py@985
PS1, Line 985: p
flake8: F841 local variable 'page' is assigned to but never used


http://gerrit.cloudera.org:8080/#/c/21412/1/tests/query_test/test_observability.py@990
PS1, Line 990: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/21412/1/tests/query_test/test_observability.py@990
PS1, Line 990: \
flake8: W605 invalid escape sequence '\)'



--
To view, visit http://gerrit.cloudera.org:8080/21412
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I538ebe914f70f460bc8412770a8f7a1cc8b505dc
Gerrit-Change-Number: 21412
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Wed, 08 May 2024 11:56:39 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11499: Refactor UrlEncode function to handle special characters

2024-05-08 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21131 )

Change subject: IMPALA-11499: Refactor UrlEncode function to handle special 
characters
..


Patch Set 26:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10619/ 
DRY_RUN=true


--
To view, visit http://gerrit.cloudera.org:8080/21131
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I88c4aba5d811dfcec809583d0c16fcbc0ca730fb
Gerrit-Change-Number: 21131
Gerrit-PatchSet: 26
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Anonymous Coward 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Smith 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zihao Ye 
Gerrit-Comment-Date: Wed, 08 May 2024 10:43:04 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger

2024-05-08 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21160 )

Change subject: IMPALA-12921: Support locally built Ranger
..


Patch Set 11:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/16129/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/21160
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27
Gerrit-Change-Number: 21160
Gerrit-PatchSet: 11
Gerrit-Owner: Fang-Yu Rao 
Gerrit-Reviewer: Aman Sinha 
Gerrit-Reviewer: Fang-Yu Rao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: John Sherman 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Comment-Date: Wed, 08 May 2024 09:55:59 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger

2024-05-08 Thread Fang-Yu Rao (Code Review)

Fang-Yu Rao has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21160 )

Change subject: IMPALA-12921: Support locally built Ranger
..


Patch Set 11:

(1 comment)

In patch set 11 I added some steps to create the user 'non_owner_2' on the 
Ranger server since in Apache Ranger, the Ranger server also checks the 
existence of the grantee in the GRANT/REVOKE statements.

In addition, I plan to add a startup flag to the catalog server so that we 
could specify whether the resource type of 'storage-type' is supported by the 
Ranger service. Without this, some basic statements like GRANT ALL ON SERVER 
could not even be supported if Apache Ranger is used.

http://gerrit.cloudera.org:8080/#/c/21160/7//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/21160/7//COMMIT_MSG@64
PS7, Line 64:Note that the resource type of 'storage-type' is not supported
> Thanks Quanlong!
After running some authorization-related tests, e.g., RangerAuditLogTest, I 
realized that the resource type of 'storage-type' is not supported in Apache 
Ranger yet. Hence the support for storage handler privileges added in 
IMPALA-10436 is not supported.

This also affects the ALL privilege on SERVER since this the ALL privilege on 
SERVER also implies the RWSTORAGE privilege on all storage handler URI's.

It may be a good idea to add a startup flag to the catalog server so that we 
could specify whether the resource type of 'storage-type' is supported by the 
Ranger service so the catalog server won't send the requests to grant/revoke 
privileges on the storage handler URI's.



--
To view, visit http://gerrit.cloudera.org:8080/21160
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27
Gerrit-Change-Number: 21160
Gerrit-PatchSet: 11
Gerrit-Owner: Fang-Yu Rao 
Gerrit-Reviewer: Aman Sinha 
Gerrit-Reviewer: Fang-Yu Rao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: John Sherman 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Comment-Date: Wed, 08 May 2024 09:47:46 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger

2024-05-08 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21160 )

Change subject: IMPALA-12921: Support locally built Ranger
..


Patch Set 11:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10618/ 
DRY_RUN=true


--
To view, visit http://gerrit.cloudera.org:8080/21160
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27
Gerrit-Change-Number: 21160
Gerrit-PatchSet: 11
Gerrit-Owner: Fang-Yu Rao 
Gerrit-Reviewer: Aman Sinha 
Gerrit-Reviewer: Fang-Yu Rao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: John Sherman 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Comment-Date: Wed, 08 May 2024 09:48:29 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13058: Init first arrival time and completion time with -1

2024-05-08 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/21405 )

Change subject: IMPALA-13058: Init first_arrival_time_ and completion_time_ 
with -1
..

IMPALA-13058: Init first_arrival_time_ and completion_time_ with -1

Impala run over ARM machine shows 'arch_sys_counter' clock source being
used rather than more precise 'tsc'. This cause
MonotonicStopwatch::Now() to use 'CLOCK_MONOTONIC_COARSE' rather than
'CLOCK_MONOTONIC'. This is what printed near the beginning of impalad
log:

I0506 13:49:15.429359 355337 init.cc:600] OS distribution: Red Hat Enterprise 
Linux 8.8 (Ootpa)
OS version: Linux version 4.18.0-477.15.1.el8_8.aarch64 ...
Clock: clocksource: 'arch_sys_counter', clockid_t: CLOCK_MONOTONIC_COARSE

This difference in clock source causes test failure in
test_runtime_filters.py::TestRuntimeFilters::test_basic_filters. This
patch fixes the issue by initializing first_arrival_time_ and
completion_time_ fields of Coordinator::FilterState with -1 and accept 0
as valid value for those fields.

query_events_ initialization and start are also moved to the beginning
of ClientRequestState's contructor.

Testing:
- Tweak row_regex pattern in runtime_filters.test.
- Loop and pass test_runtime_filters.py in exhaustive mode 3 times
  in ARM machine.

Change-Id: I1176e2118bb03414ab35049f50009ff0e8c63f58
Reviewed-on: http://gerrit.cloudera.org:8080/21405
Reviewed-by: Wenzhe Zhou 
Tested-by: Impala Public Jenkins 
---
M be/src/runtime/coordinator-filter-state.h
M be/src/runtime/coordinator.cc
M be/src/service/client-request-state.cc
M testdata/workloads/functional-query/queries/QueryTest/runtime_filters.test
4 files changed, 21 insertions(+), 18 deletions(-)

Approvals:
  Wenzhe Zhou: Looks good to me, approved
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/21405
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I1176e2118bb03414ab35049f50009ff0e8c63f58
Gerrit-Change-Number: 21405
Gerrit-PatchSet: 6
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Reviewer: Wenzhe Zhou

[Impala-ASF-CR] IMPALA-13058: Init first arrival time and completion time with -1

2024-05-08 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21405 )

Change subject: IMPALA-13058: Init first_arrival_time_ and completion_time_ 
with -1
..


Patch Set 5: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/21405
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1176e2118bb03414ab35049f50009ff0e8c63f58
Gerrit-Change-Number: 21405
Gerrit-PatchSet: 5
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Comment-Date: Wed, 08 May 2024 09:46:29 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12921: Support locally built Ranger

2024-05-08 Thread Fang-Yu Rao (Code Review)

Hello Quanlong Huang, Aman Sinha, Joe McDonnell, Impala Public Jenkins, John
Sherman,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/21160

to look at the new patch set (#11).

Change subject: IMPALA-12921: Support locally built Ranger
..

IMPALA-12921: Support locally built Ranger

This patch does the following.
1. Add the support for locally built Ranger.
2. Resolve IMPALA-12985 by switching to the new constructor when
instantiating RangerAccessRequestImpl.

The instructions on how to build Ranger locally and point
Impala to the locally built Ranger server are provided as below.

Suppose the Ranger project is under the folder $RANGER_SRC_DIR. We could
execute the following to build Ranger. By default, the compressed
tarball is produced under $RANGER_SRC_DIR/target.

mvn clean compile -B -nsu -DskipCheck=true -Dcheckstyle.skip=true \
package install -DskipITs -DskipTests -Dmaven.javadoc.skip=true

After building Ranger, we need to build Impala's Java code so that
Impala's Java code could consume the locally produced Ranger classes. We
will need to export the following environment variables before building
Impala. This prevents bootstrap_toolchain.py from trying to download the
compressed Ranger tarball. Moreover, we need to apply
IMPALA-12921_pom_patching.diff on Impala if the locally built Ranger
pulls in an incompatible version of hive-storage-api.

1. export RANGER_VERSION_OVERRIDE=\
$(mvn -f $RANGER_SRC_DIR/pom.xml -q help:evaluate \
-Dexpression=project.version -DforceStdout)

2. export RANGER_HOME_OVERRIDE=$RANGER_SRC_DIR/target/\
ranger-${RANGER_VERSION_OVERRIDE}-admin

It then suffices to execute the following to point
Impala to the locally built Ranger server before starting Impala.

1. source $IMPALA_HOME/bin/impala-config.sh

2. tar zxv -f $RANGER_SRC_DIR/target/\
ranger-${IMPALA_RANGER_VERSION}-admin.tar.gz \
-C $RANGER_SRC_DIR/target/

3. $IMPALA_HOME/bin/create-test-configuration.sh

4. $IMPALA_HOME/bin/create-test-configuration.sh \
-create_ranger_policy_db

5. $IMPALA_HOME/testdata/bin/run-ranger.sh
(run-all.sh has to be executed instead if other underlying services
have not been started)

6. $IMPALA_HOME/testdata/bin/setup-ranger.sh

Testing:
- Manually verified that we could point Impala to a locally built
Apache Ranger on the master branch (with tip being RANGER-4745).
- Manually verified that with the Ranger patch in RANGER-4771.diff,
we could manage the policy repository via GRANT/REVOKE statements.
Note that the resource type of 'storage-type' is not supported
in Apache Ranger yet and thus the support for storage handler
privileges added in IMPALA-10436 is not supported. The ALL privilege
on SERVER is also affected in that this privilege on SERVER also
applies to all storage handler URI's.
- Verified that this patch passed the core tests when CDP Ranger is
used.

Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27
---
M README-build.md
M bin/bootstrap_toolchain.py
M bin/create-test-configuration.sh
M bin/impala-config.sh
M bin/rat_exclude_files.txt
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java
M testdata/bin/setup-ranger.sh
A testdata/cluster/ranger/IMPALA-12921_pom_patching.diff
A testdata/cluster/ranger/RANGER-4771.diff
A testdata/cluster/ranger/README
R testdata/cluster/ranger/setup/all_database_policy_revised.json.template
A testdata/cluster/ranger/setup/impala_group_non_owner_2.json
M testdata/cluster/ranger/setup/impala_user_non_owner.json.template
A testdata/cluster/ranger/setup/impala_user_non_owner_2.json.template
M testdata/cluster/ranger/setup/impala_user_owner.json.template
16 files changed, 183 insertions(+), 16 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/21160/11
--
To view, visit http://gerrit.cloudera.org:8080/21160
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27
Gerrit-Change-Number: 21160
Gerrit-PatchSet: 11
Gerrit-Owner: Fang-Yu Rao
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: John Sherman
Gerrit-Reviewer: Quanlong Huang

[Impala-ASF-CR] IMPALA-13038: Support profile tab for imported query profiles

2024-05-08 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21400 )

Change subject: IMPALA-13038: Support profile tab for imported query profiles
..


Patch Set 3:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/16128/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/21400
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iddcf2e285abbf42f97bde19014be076ccd6374bc
Gerrit-Change-Number: 21400
Gerrit-PatchSet: 3
Gerrit-Owner: Surya Hebbar 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Surya Hebbar 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Comment-Date: Wed, 08 May 2024 07:32:31 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13038: Support profile tab for imported query profiles

2024-05-08 Thread Wenzhe Zhou (Code Review)

Wenzhe Zhou has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21400 )

Change subject: IMPALA-13038: Support profile tab for imported query profiles
..


Patch Set 3: Code-Review+1


--
To view, visit http://gerrit.cloudera.org:8080/21400
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iddcf2e285abbf42f97bde19014be076ccd6374bc
Gerrit-Change-Number: 21400
Gerrit-PatchSet: 3
Gerrit-Owner: Surya Hebbar 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Surya Hebbar 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Comment-Date: Wed, 08 May 2024 07:13:11 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13038: Support profile tab for imported query profiles

Surya Hebbar has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21400 )

Change subject: IMPALA-13038: Support profile tab for imported query profiles
..


Patch Set 3:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/21400/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/21400/1//COMMIT_MSG@14
PS1, Line 14: With the current patch "Query Profile" tab will also be supported.
> I messed up my local environment. It did show the "Query Profile" tab after
Done


http://gerrit.cloudera.org:8080/#/c/21400/1//COMMIT_MSG@21
PS1, Line 21:  active.
> nit: wrap around long line.
Done


http://gerrit.cloudera.org:8080/#/c/21400/2/www/query_profile.tmpl
File www/query_profile.tmpl:

http://gerrit.cloudera.org:8080/#/c/21400/2/www/query_profile.tmpl@53
PS2, Line 53:   le
> nit: indent with two spaces for this function block to keep consistent.
Done



--
To view, visit http://gerrit.cloudera.org:8080/21400
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iddcf2e285abbf42f97bde19014be076ccd6374bc
Gerrit-Change-Number: 21400
Gerrit-PatchSet: 3
Gerrit-Owner: Surya Hebbar 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Surya Hebbar 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Comment-Date: Wed, 08 May 2024 07:09:01 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13038: Support profile tab for imported query profiles