[Impala-ASF-CR] IMPALA-10904: Add some perf tools to the repository

2021-09-10 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17834 )

Change subject: IMPALA-10904: Add some perf tools to the repository
..


Patch Set 5: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/17834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib3da696b939204d23c5285dcf1bf6ee3a3738415
Gerrit-Change-Number: 17834
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Fri, 10 Sep 2021 22:29:20 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-10904: Add some perf tools to the repository

2021-09-10 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/17834 )

Change subject: IMPALA-10904: Add some perf tools to the repository
..

IMPALA-10904: Add some perf tools to the repository

This patch adds two perf tools to the repository. Both can be used to
generate flame graphs (https://www.brendangregg.com/flamegraphs.html).

perf-record.sh:
It samples the CPU stack traces for the entire system, or for a
specific PID until the user hits Ctrl+C. It can be useful if the
developer wants to take a look at what Impala is doing.
The resulting flame graph is written to an SVG file.

perf-query.sh:
It takes a query string as a parameter and passes it to the impala
shell to execute. While the query is executing the script samples
the CPU stack traces for the entire system. The resulting flame
graph is written to an SVG file.
E.g.:
perf-query.sh "select count(*) from tpch.lineitem group by l_returnflag"

Change-Id: Ib3da696b939204d23c5285dcf1bf6ee3a3738415
Reviewed-on: http://gerrit.cloudera.org:8080/17834
Reviewed-by: Impala Public Jenkins 
Tested-by: Impala Public Jenkins 
---
A bin/perf_tools/perf-query.sh
A bin/perf_tools/perf-record.sh
2 files changed, 172 insertions(+), 0 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

--
To view, visit http://gerrit.cloudera.org:8080/17834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ib3da696b939204d23c5285dcf1bf6ee3a3738415
Gerrit-Change-Number: 17834
Gerrit-PatchSet: 6
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Reviewer: Zoltan Borok-Nagy 


[Impala-ASF-CR] IMPALA-10876: Support to download JWKS from given URL

2021-09-10 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17802 )

Change subject: IMPALA-10876: Support to download JWKS from given URL
..


Patch Set 3:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/9451/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/17802
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic6ac8cf0010c13db30219776d1d275709bf211df
Gerrit-Change-Number: 17802
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Comment-Date: Fri, 10 Sep 2021 20:56:12 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-10876: Support to download JWKS from given URL

2021-09-10 Thread Wenzhe Zhou (Code Review)
Wenzhe Zhou has uploaded a new patch set (#3). ( 
http://gerrit.cloudera.org:8080/17802 )

Change subject: IMPALA-10876: Support to download JWKS from given URL
..

IMPALA-10876: Support to download JWKS from given URL

This patch added supporting to download JWKS from a given URL.
We call libcurl's APIs to download file from the given URL. curl was
added to native-toolchain. This patch modified bootstrap_toolchain.py
and makefiles to integrate libcurl.

Added end-end JWT authentication test cases with JWKS specified as
HTTP/HTTPS URL.

Testing:
 - Passed core run.

Change-Id: Ic6ac8cf0010c13db30219776d1d275709bf211df
---
M CMakeLists.txt
M be/CMakeLists.txt
M be/src/rpc/authentication.cc
M be/src/service/impala-server.cc
M be/src/util/jwt-util-internal.h
M be/src/util/jwt-util.cc
M be/src/util/jwt-util.h
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M bin/rat_exclude_files.txt
A cmake_modules/FindCurl.cmake
M fe/src/test/java/org/apache/impala/customcluster/CustomClusterRunner.java
M fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java
M fe/src/test/java/org/apache/impala/customcluster/JwtWebserverTest.java
M fe/src/test/java/org/apache/impala/customcluster/LdapHS2Test.java
M fe/src/test/java/org/apache/impala/customcluster/LdapWebserverTest.java
R www/jwt/jwks_rs256.json
17 files changed, 396 insertions(+), 39 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/17802/3
--
To view, visit http://gerrit.cloudera.org:8080/17802
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic6ac8cf0010c13db30219776d1d275709bf211df
Gerrit-Change-Number: 17802
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou 
Gerrit-Reviewer: Impala Public Jenkins 


[Impala-ASF-CR] IMPALA-10907: Refactor MetastoreEvents class

2021-09-10 Thread Csaba Ringhofer (Code Review)
Csaba Ringhofer has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17833 )

Change subject: IMPALA-10907: Refactor MetastoreEvents class
..


Patch Set 1:

(2 comments)

Thanks for the refactor! I also found MatastoreEvent.java too long.

http://gerrit.cloudera.org:8080/#/c/17833/1/fe/src/main/java/org/apache/impala/catalog/events/AddPartitionEvent.java
File fe/src/main/java/org/apache/impala/catalog/events/AddPartitionEvent.java:

http://gerrit.cloudera.org:8080/#/c/17833/1/fe/src/main/java/org/apache/impala/catalog/events/AddPartitionEvent.java@1
PS1, Line 1: package org.apache.impala.catalog.events;
Licenses are missing (which have led to the broken build)


http://gerrit.cloudera.org:8080/#/c/17833/1/fe/src/main/java/org/apache/impala/catalog/events/AddPartitionEvent.java@35
PS1, Line 35: 
Preconditions.checkState(eventType_.equals(MetastoreEventType.ADD_PARTITION));
: if (event.getMessage() == null) {
:   throw new IllegalStateException(debugString("Event message 
is null"));
: }
: try {
:   AddPartitionMessage addPartitionMessage_ =
:   MetastoreEventsProcessor.getMessageDeserializer()
:   .getAddPartitionMessage(event.getMessage());
optional idea for improvement:
I think that the amount of code could be reduced by moving this logic (and its 
friends in other classes) to the factory. The constructors could take and 
AddPartitionMessage/similar argument.

The Exception-> MetastoreNotificationException logic could be probably also 
moved to the factory.



--
To view, visit http://gerrit.cloudera.org:8080/17833
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I71edd17e6705f81cd402f9aecc86ef350a76
Gerrit-Change-Number: 17833
Gerrit-PatchSet: 1
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Fri, 10 Sep 2021 16:50:16 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-10904: Add some perf tools to the repository

2021-09-10 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17834 )

Change subject: IMPALA-10904: Add some perf tools to the repository
..


Patch Set 4:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/9450/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/17834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib3da696b939204d23c5285dcf1bf6ee3a3738415
Gerrit-Change-Number: 17834
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Fri, 10 Sep 2021 16:34:56 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-10904: Add some perf tools to the repository

2021-09-10 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17834 )

Change subject: IMPALA-10904: Add some perf tools to the repository
..


Patch Set 5: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/17834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib3da696b939204d23c5285dcf1bf6ee3a3738415
Gerrit-Change-Number: 17834
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Fri, 10 Sep 2021 16:15:47 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-10904: Add some perf tools to the repository

2021-09-10 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17834 )

Change subject: IMPALA-10904: Add some perf tools to the repository
..


Patch Set 5:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7467/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/17834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib3da696b939204d23c5285dcf1bf6ee3a3738415
Gerrit-Change-Number: 17834
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Fri, 10 Sep 2021 16:15:48 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster

2021-09-10 Thread Csaba Ringhofer (Code Review)
Csaba Ringhofer has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17793 )

Change subject: IMPALA-10870: Add Apache Hive 3.1.2 to the minicluster
..


Patch Set 7: Code-Review+1

(1 comment)

Just bumped into this - looks great!

http://gerrit.cloudera.org:8080/#/c/17793/7//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/17793/7//COMMIT_MSG@43
PS7, Line 43: so that a new metastore db is created and the Apache Hive 3.1.2 
schema is
nit: wrap at 72



--
To view, visit http://gerrit.cloudera.org:8080/17793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1978909589ecacb15d32d874e97f050a85adf1f6
Gerrit-Change-Number: 17793
Gerrit-PatchSet: 7
Gerrit-Owner: Fucun Chu 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Laszlo Gaal 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Comment-Date: Fri, 10 Sep 2021 16:14:57 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-10904: Add some perf tools to the repository

2021-09-10 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17834 )

Change subject: IMPALA-10904: Add some perf tools to the repository
..


Patch Set 4:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/17834/4/bin/perf_tools/perf-query.sh
File bin/perf_tools/perf-query.sh:

http://gerrit.cloudera.org:8080/#/c/17834/4/bin/perf_tools/perf-query.sh@48
PS4, Line 48:   echo "Please use 'git clone 
https://github.com/brendangregg/FlameGraph' in ${IMPALA_TOOLCHAIN}"
line too long (97 > 90)


http://gerrit.cloudera.org:8080/#/c/17834/4/bin/perf_tools/perf-record.sh
File bin/perf_tools/perf-record.sh:

http://gerrit.cloudera.org:8080/#/c/17834/4/bin/perf_tools/perf-record.sh@41
PS4, Line 41:   echo "Please use 'git clone 
https://github.com/brendangregg/FlameGraph' in ${IMPALA_TOOLCHAIN}"
line too long (97 > 90)



--
To view, visit http://gerrit.cloudera.org:8080/17834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib3da696b939204d23c5285dcf1bf6ee3a3738415
Gerrit-Change-Number: 17834
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Fri, 10 Sep 2021 16:12:48 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-10904: Add some perf tools to the repository

2021-09-10 Thread Zoltan Borok-Nagy (Code Review)
Zoltan Borok-Nagy has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17834 )

Change subject: IMPALA-10904: Add some perf tools to the repository
..


Patch Set 4: Code-Review+2

Fixed a comment. Carry +2

Thanks for the review!


--
To view, visit http://gerrit.cloudera.org:8080/17834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib3da696b939204d23c5285dcf1bf6ee3a3738415
Gerrit-Change-Number: 17834
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Fri, 10 Sep 2021 16:12:51 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-10904: Add some perf tools to the repository

2021-09-10 Thread Zoltan Borok-Nagy (Code Review)
Hello Riza Suminto, Joe McDonnell, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/17834

to look at the new patch set (#4).

Change subject: IMPALA-10904: Add some perf tools to the repository
..

IMPALA-10904: Add some perf tools to the repository

This patch adds two perf tools to the repository. Both can be used to
generate flame graphs (https://www.brendangregg.com/flamegraphs.html).

perf-record.sh:
It samples the CPU stack traces for the entire system, or for a
specific PID until the user hits Ctrl+C. It can be useful if the
developer wants to take a look at what Impala is doing.
The resulting flame graph is written to an SVG file.

perf-query.sh:
It takes a query string as a parameter and passes it to the impala
shell to execute. While the query is executing the script samples
the CPU stack traces for the entire system. The resulting flame
graph is written to an SVG file.
E.g.:
perf-query.sh "select count(*) from tpch.lineitem group by l_returnflag"

Change-Id: Ib3da696b939204d23c5285dcf1bf6ee3a3738415
---
A bin/perf_tools/perf-query.sh
A bin/perf_tools/perf-record.sh
2 files changed, 172 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/17834/4
--
To view, visit http://gerrit.cloudera.org:8080/17834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib3da696b939204d23c5285dcf1bf6ee3a3738415
Gerrit-Change-Number: 17834
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Reviewer: Zoltan Borok-Nagy 


[Impala-ASF-CR] IMPALA-10904: Add some perf tools to the repository

2021-09-10 Thread Joe McDonnell (Code Review)
Joe McDonnell has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17834 )

Change subject: IMPALA-10904: Add some perf tools to the repository
..


Patch Set 3: Code-Review+2

This looks really cool!


--
To view, visit http://gerrit.cloudera.org:8080/17834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib3da696b939204d23c5285dcf1bf6ee3a3738415
Gerrit-Change-Number: 17834
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Fri, 10 Sep 2021 16:02:23 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-10904: Add some perf tools to the repository

2021-09-10 Thread Riza Suminto (Code Review)
Riza Suminto has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17834 )

Change subject: IMPALA-10904: Add some perf tools to the repository
..


Patch Set 3: Code-Review+1

Looks good to me. Thanks Zoltan!


--
To view, visit http://gerrit.cloudera.org:8080/17834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib3da696b939204d23c5285dcf1bf6ee3a3738415
Gerrit-Change-Number: 17834
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Fri, 10 Sep 2021 15:27:27 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-10905: test iceberg.py test time travel fails in exhaustive builds

2021-09-10 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17835 )

Change subject: IMPALA-10905: test_iceberg.py test_time_travel fails in 
exhaustive builds
..


Patch Set 3: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/17835
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5c172df865bc35dd1792146f4045fbc16e04d116
Gerrit-Change-Number: 17835
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Fri, 10 Sep 2021 14:18:33 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-10905: test iceberg.py test time travel fails in exhaustive builds

2021-09-10 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/17835 )

Change subject: IMPALA-10905: test_iceberg.py test_time_travel fails in 
exhaustive builds
..

IMPALA-10905: test_iceberg.py test_time_travel fails in exhaustive builds

test_iceberg.py::test_time_travel sets the TIMEZONE query option
during execution. In the end it sets the TIMEZONE to 'Asia/Tokyo'.

The problem is that it sets the TIMEZONE of the client of the
ImpalaTestSuite. This means that a subsequent test will use
timezone 'Asia/Tokyo' instead of the local one.

This patch fixes this issue by creating a local Impala client for
the test.

Change-Id: I5c172df865bc35dd1792146f4045fbc16e04d116
Reviewed-on: http://gerrit.cloudera.org:8080/17835
Reviewed-by: Impala Public Jenkins 
Tested-by: Impala Public Jenkins 
---
M tests/common/impala_connection.py
M tests/query_test/test_iceberg.py
2 files changed, 90 insertions(+), 80 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

--
To view, visit http://gerrit.cloudera.org:8080/17835
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I5c172df865bc35dd1792146f4045fbc16e04d116
Gerrit-Change-Number: 17835
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Zoltan Borok-Nagy 


[Impala-ASF-CR] [WIP]: Initial commit: Add logic to sync to latest event id

2021-09-10 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17756 )

Change subject: [WIP]: Initial commit: Add logic to sync to latest event id
..


Patch Set 28: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/17756
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia822d15725d0a9a9ad1398e10ed4ae3288d0e9ad
Gerrit-Change-Number: 17756
Gerrit-PatchSet: 28
Gerrit-Owner: Sourabh Goyal 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Sourabh Goyal 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Fri, 10 Sep 2021 14:14:56 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-2581: LIMIT can be propagated down into some aggregations

2021-09-10 Thread Qifan Chen (Code Review)
Qifan Chen has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17821 )

Change subject: IMPALA-2581: LIMIT can be propagated down into some aggregations
..


Patch Set 12:

(6 comments)

Looks great.

The comment on subquery rewrite test is orthogonal to this patch.

http://gerrit.cloudera.org:8080/#/c/17821/12//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/17821/12//COMMIT_MSG@10
PS12, Line 10: pre-aggregation
nit. exceed the max line threshold.


http://gerrit.cloudera.org:8080/#/c/17821/12/be/src/exec/aggregation-node.cc
File be/src/exec/aggregation-node.cc:

http://gerrit.cloudera.org:8080/#/c/17821/12/be/src/exec/aggregation-node.cc@76
PS12, Line 76:  VLOG_QUERY << "the number of rows (" << aggs_[0]->GetNumKeys() 
<< ") returned"
 :   " from the aggregation node has exceeded the limit 
of " << limit();
nit. May use Substitute() which is faster.

Example:
orc-column-readers.cc:  return Substitute("$0 column (ORC id=$1)", 
node->toString(), node->getColumnId());

sort-node.cc:  VLOG(3) << Substitute("Sort estimation: 
estimated_full_input_size=$0 "


http://gerrit.cloudera.org:8080/#/c/17821/12/common/thrift/PlanNodes.thrift
File common/thrift/PlanNodes.thrift:

http://gerrit.cloudera.org:8080/#/c/17821/12/common/thrift/PlanNodes.thrift@479
PS12, Line 479: halt
nit. complete


http://gerrit.cloudera.org:8080/#/c/17821/12/fe/src/main/java/org/apache/impala/planner/AggregationNode.java
File fe/src/main/java/org/apache/impala/planner/AggregationNode.java:

http://gerrit.cloudera.org:8080/#/c/17821/12/fe/src/main/java/org/apache/impala/planner/AggregationNode.java@646
PS12, Line 646: For SELECT DISTINCT f1,f2,...fn FROM t LIMIT n
nit. I think we should mention the two conditions in the commit message here. 
The above SQL is an example.


http://gerrit.cloudera.org:8080/#/c/17821/12/fe/src/main/java/org/apache/impala/planner/AggregationNode.java@648
PS12, Line 648: Halt
nit. May use Complete, as Halt implies stop for some reason.


http://gerrit.cloudera.org:8080/#/c/17821/11/testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test
File 
testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test:

http://gerrit.cloudera.org:8080/#/c/17821/11/testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test@2934
PS11, Line 2934: limit: 2
> This is an SQL optimization. Look at SubqueryRewriter# mergeExpr. This is a
Since we do not push down id from the outer side to the inner, I would think we 
need to have ALL distinct values from the union side.

If limit 2 is for the # of groups (i.e., taking only 2 groups, or 2 distinct 
values), then is it right?



--
To view, visit http://gerrit.cloudera.org:8080/17821
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I930a6cb203615acfc03f23118d1bc1f0ea360995
Gerrit-Change-Number: 17821
Gerrit-PatchSet: 12
Gerrit-Owner: liuyao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: liuyao 
Gerrit-Comment-Date: Fri, 10 Sep 2021 13:40:16 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-9495: Support struct in select list for ORC tables

2021-09-10 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17638 )

Change subject: IMPALA-9495: Support struct in select list for ORC tables
..


Patch Set 16:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/9449/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/17638
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0fbe56bdcd372b72e99c0195d87a818e7fa4bc3a
Gerrit-Change-Number: 17638
Gerrit-PatchSet: 16
Gerrit-Owner: Gabor Kaszab 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Daniel Becker 
Gerrit-Reviewer: Gabor Kaszab 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Comment-Date: Fri, 10 Sep 2021 12:15:26 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-9495: Support struct in select list for ORC tables

2021-09-10 Thread Gabor Kaszab (Code Review)
Hello Quanlong Huang, Qifan Chen, Daniel Becker, Csaba Ringhofer, Impala Public 
Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/17638

to look at the new patch set (#16).

Change subject: IMPALA-9495: Support struct in select list for ORC tables
..

IMPALA-9495: Support struct in select list for ORC tables

This patch implements the functionality to allow structs in the select
list of inline views, topmost blocks. When displaying the value of a
struct it is formatted into a JSON value and returned as a string. An
example of such a value:

SELECT struct_col FROM some_table;
'{"int_struct_member":12,"string_struct_member":"string value"}'

Another example where we query a nested struct:
SELECT outer_struct_col FROM some_table;
'{"inner_struct":{"string_member":"string value","int_member":12}}'

Note, the conversion from struct to JSON happens on the server side
before sending out the value in HS2 to the client. However, HS2 is
capable of handling struct values as well so in a later change we might
want to add a functionality to send the struct in thrift to the client
so that the client can use the struct directly.

-- Internal representation of a struct:
When scanning a struct the rowbatch will hold the values of the
struct's children as if they were queried one by one directly in the
select list.

E.g. Taking the following table:
CREATE TABLE tbl (id int, s struct) STORED AS ORC

And running the following query:
SELECT id, s FROM tbl;

After scanning a row in a row batch will hold the following values:
(note the biggest size comes first)
 1: The pointer for the string in s.b
 2: The length for the string in s.b
 3: The int value for s.a
 4: The int value of id
 5: A single null byte for all the slots: id, s, s.a, s.b

The size of a struct has an effect on the order of the memory layout of
a row batch. The struct size is calculated by summing the size of its
fields and then the struct gets a place in the row batch to precede all
smaller slots by size. Note, all the fields of a struct are consecutive
to each other in the row batch. Inside a struct the order of the fields
is also based on their size as it does in a regular case for primitives.

When evaluating a struct as a SlotRef a newly introduced StructVal will
be used to refer to the actual values of a struct in the row batch.
This StructVal holds a vector of pointers where each pointer represents
a member of the struct. Following the above example the StructVal would
keep two pointers, one to point to an IntVal and one to point to a
StringVal.

-- Changes related to tuple and slot descriptors:
When providing a struct in the select list there is going to be a
SlotDescriptor for the struct slot in the topmost TupleDescriptor.
Additionally, another TupleDesriptor is created to hold SlotDescriptors
for each of the struct's children. The struct SlotDescriptor points to
the newly introduced TupleDescriptor using 'itemTupleId'.
The offsets for the children of the struct is calculated from the
beginning of the topmost TupleDescriptor and not from the
TupleDescriptor that directly holds the struct's children. The null
indicator bytes as well are stored on the level of the topmost
TupleDescriptor.

-- Changes related to scalar expressions:
A struct in the select list is translated into an expression tree where
the top of this tree is a SlotRef for the struct itself and its
children in the tree are SlotRefs for the members of the struct. When
evaluating a struct SlotRef after the null checks the evaluation is
delegated to the children SlotRefs.

-- Restrictions:
  - Codegen support is not included in this patch.
  - Only ORC file format is supported by this patch.
  - Only HS2 client supports returning structs. Beeswax support is not
implemented as it is going to be deprecated anyway. Currently we
receive an error when trying to query a struct through Beeswax.

-- Tests added:
  - The ORC and Parquet functional databases are extended with 3 new
tables:
1: A small table with one level structs, holding different
kind of primitive types as members.
2: A small table with 2 and 3 level nested structs.
3: A bigger, partitioned table constructed from alltypes where all
the columns except the 'id' column are put into a struct.
  - struct-in-select-list.test and nested-struct-in-select-list.test
uses these new tables to query structs directly or through an
inline view.

Change-Id: I0fbe56bdcd372b72e99c0195d87a818e7fa4bc3a
---
M be/src/exec/hdfs-orc-scanner.cc
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scanner.cc
M be/src/exec/orc-column-readers.cc
M be/src/exec/orc-column-readers.h
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/parquet/parquet-collection-column-reader.cc
M be/src/exprs/anyval-util.cc
M be/src/exprs/expr-value.h
M be/src/exprs/scalar-expr-evaluator.cc
M 

[Impala-ASF-CR] IMPALA-2581: LIMIT can be propagated down into some aggregations

2021-09-10 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17821 )

Change subject: IMPALA-2581: LIMIT can be propagated down into some aggregations
..


Patch Set 12:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/9448/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/17821
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I930a6cb203615acfc03f23118d1bc1f0ea360995
Gerrit-Change-Number: 17821
Gerrit-PatchSet: 12
Gerrit-Owner: liuyao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: liuyao 
Gerrit-Comment-Date: Fri, 10 Sep 2021 10:01:22 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-2581: LIMIT can be propagated down into some aggregations

2021-09-10 Thread liuyao (Code Review)
liuyao has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17821 )

Change subject: IMPALA-2581: LIMIT can be propagated down into some aggregations
..


Patch Set 12:

(8 comments)

http://gerrit.cloudera.org:8080/#/c/17821/11//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/17821/11//COMMIT_MSG@10
PS11, Line 10: When both con
> nit. When both conditions below are true,
Done


http://gerrit.cloudera.org:8080/#/c/17821/11/be/src/exec/aggregation-node-base.h
File be/src/exec/aggregation-node-base.h:

http://gerrit.cloudera.org:8080/#/c/17821/11/be/src/exec/aggregation-node-base.h@71
PS11, Line 71:   bool fast_limit_check_ = false;
> nit. May init to false here.
Done


http://gerrit.cloudera.org:8080/#/c/17821/11/be/src/exec/aggregation-node-base.cc
File be/src/exec/aggregation-node-base.cc:

http://gerrit.cloudera.org:8080/#/c/17821/11/be/src/exec/aggregation-node-base.cc@93
PS11, Line 93:
 : Status AggregationNodeBase::Prepare(RuntimeState* state) {
 :   SCOPED_TIMER(runtime_profile_->total_time_counter());
 :   RETURN_IF_ERROR(ExecNode::Prepare(state));
 :   for (auto& agg : aggs_) {
 : agg->SetDebugOptions
> I also wonder if fast_limit_check_ can be moved to the plan node for Aggreg
Done Only GroupingAggregator has limit, maybe limit should not be added to 
TAggregator


http://gerrit.cloudera.org:8080/#/c/17821/11/be/src/exec/aggregation-node.cc
File be/src/exec/aggregation-node.cc:

http://gerrit.cloudera.org:8080/#/c/17821/11/be/src/exec/aggregation-node.cc@75
PS11, Line 75: os = true;
 :   VLOG_QUERY << "the
> nit. May make the message more specific by plugging in the numbers. For exa
Done


http://gerrit.cloudera.org:8080/#/c/17821/11/be/src/exec/streaming-aggregation-node.cc
File be/src/exec/streaming-aggregation-node.cc:

http://gerrit.cloudera.org:8080/#/c/17821/11/be/src/exec/streaming-aggregation-node.cc@129
PS11, Line 129: DCHECK(limit() > -1);
> nit. We probably should DCHECK(limit() > -1) here before the IF.
Done


http://gerrit.cloudera.org:8080/#/c/17821/11/be/src/exec/streaming-aggregation-node.cc@133
PS11, Line 133: child_batch_->Reset();
  :   VLOG_QUERY << "the
> nit. same as the other one to make the message more specific.
Done


http://gerrit.cloudera.org:8080/#/c/17821/11/testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test
File 
testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test:

http://gerrit.cloudera.org:8080/#/c/17821/11/testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test@2934
PS11, Line 2934: limit: 2
> nit. I wonder if this correct. On paper, we need to look at all distinct va
This is an SQL optimization. Look at SubqueryRewriter# mergeExpr. This is a 
scalar subquery, the number of rows returned by this subquery cannot be greater 
than 1, so set limit = 2, and then check cardinality of result of subquery


http://gerrit.cloudera.org:8080/#/c/17821/11/testdata/workloads/targeted-perf/queries/aggregation.test
File testdata/workloads/targeted-perf/queries/aggregation.test:

http://gerrit.cloudera.org:8080/#/c/17821/11/testdata/workloads/targeted-perf/queries/aggregation.test@2731
PS11, Line 2731:
> This violates condition 1?
Done



--
To view, visit http://gerrit.cloudera.org:8080/17821
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I930a6cb203615acfc03f23118d1bc1f0ea360995
Gerrit-Change-Number: 17821
Gerrit-PatchSet: 12
Gerrit-Owner: liuyao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: liuyao 
Gerrit-Comment-Date: Fri, 10 Sep 2021 09:39:12 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-2581: LIMIT can be propagated down into some aggregations

2021-09-10 Thread liuyao (Code Review)
Hello Qifan Chen, Tim Armstrong, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/17821

to look at the new patch set (#12).

Change subject: IMPALA-2581: LIMIT can be propagated down into some aggregations
..

IMPALA-2581: LIMIT can be propagated down into some aggregations

This patch contains 2 parts:
1. When both conditions below are true, push down limit to pre-aggregation
 a) aggregation node has no aggregate function
 b) aggregation node has no predicate
2. finish aggregation when number of unique keys of hash table has
exceeded the limit.

Sample queries:
SELECT DISTINCT f FROM t LIMIT n
Can pass the LIMIT all the way down to the pre-aggregation, which
leads to a nearly unbounded speedup on these queries in large tables
when n is low.

Testing:
Add test targeted-perf/queries/aggregation.test
Pass core test

Change-Id: I930a6cb203615acfc03f23118d1bc1f0ea360995
---
M be/src/exec/aggregation-node-base.cc
M be/src/exec/aggregation-node-base.h
M be/src/exec/aggregation-node.cc
M be/src/exec/aggregator.h
M be/src/exec/grouping-aggregator.cc
M be/src/exec/grouping-aggregator.h
M be/src/exec/non-grouping-aggregator.h
M be/src/exec/streaming-aggregation-node.cc
M common/thrift/PlanNodes.thrift
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java
M 
testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test
M 
testdata/workloads/functional-planner/queries/PlannerTest/setoperation-rewrite.test
M 
testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q06.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q54.test
M testdata/workloads/functional-query/queries/QueryTest/spilling.test
M testdata/workloads/targeted-perf/queries/aggregation.test
18 files changed, 140 insertions(+), 28 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/17821/12
--
To view, visit http://gerrit.cloudera.org:8080/17821
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I930a6cb203615acfc03f23118d1bc1f0ea360995
Gerrit-Change-Number: 17821
Gerrit-PatchSet: 12
Gerrit-Owner: liuyao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Tim Armstrong 


[Impala-ASF-CR] IMPALA-10904: Add some perf tools to the repository

2021-09-10 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17834 )

Change subject: IMPALA-10904: Add some perf tools to the repository
..


Patch Set 3:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/9447/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/17834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib3da696b939204d23c5285dcf1bf6ee3a3738415
Gerrit-Change-Number: 17834
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Fri, 10 Sep 2021 09:17:10 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-10904: Add some perf tools to the repository

2021-09-10 Thread Zoltan Borok-Nagy (Code Review)
Zoltan Borok-Nagy has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17834 )

Change subject: IMPALA-10904: Add some perf tools to the repository
..


Patch Set 3:

(1 comment)

Thanks for trying out the scripts!

http://gerrit.cloudera.org:8080/#/c/17834/2/bin/perf_tools/perf-record.sh
File bin/perf_tools/perf-record.sh:

http://gerrit.cloudera.org:8080/#/c/17834/2/bin/perf_tools/perf-record.sh@79
PS2, Line 79: perf_record.
> Since the script name changed, should the output file name change as well?
Thanks for catching this! Done.



--
To view, visit http://gerrit.cloudera.org:8080/17834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib3da696b939204d23c5285dcf1bf6ee3a3738415
Gerrit-Change-Number: 17834
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Fri, 10 Sep 2021 08:54:27 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-10904: Add some perf tools to the repository

2021-09-10 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17834 )

Change subject: IMPALA-10904: Add some perf tools to the repository
..


Patch Set 3:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/17834/3/bin/perf_tools/perf-query.sh
File bin/perf_tools/perf-query.sh:

http://gerrit.cloudera.org:8080/#/c/17834/3/bin/perf_tools/perf-query.sh@47
PS3, Line 47:   echo "Please use 'git clone 
https://github.com/brendangregg/FlameGraph' in ${IMPALA_TOOLCHAIN}"
line too long (97 > 90)


http://gerrit.cloudera.org:8080/#/c/17834/3/bin/perf_tools/perf-record.sh
File bin/perf_tools/perf-record.sh:

http://gerrit.cloudera.org:8080/#/c/17834/3/bin/perf_tools/perf-record.sh@41
PS3, Line 41:   echo "Please use 'git clone 
https://github.com/brendangregg/FlameGraph' in ${IMPALA_TOOLCHAIN}"
line too long (97 > 90)



--
To view, visit http://gerrit.cloudera.org:8080/17834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib3da696b939204d23c5285dcf1bf6ee3a3738415
Gerrit-Change-Number: 17834
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Fri, 10 Sep 2021 08:53:31 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-10904: Add some perf tools to the repository

2021-09-10 Thread Zoltan Borok-Nagy (Code Review)
Hello Riza Suminto, Joe McDonnell, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/17834

to look at the new patch set (#3).

Change subject: IMPALA-10904: Add some perf tools to the repository
..

IMPALA-10904: Add some perf tools to the repository

This patch adds two perf tools to the repository. Both can be used to
generate flame graphs (https://www.brendangregg.com/flamegraphs.html).

perf-record.sh:
It samples the CPU stack traces for the entire system, or for a
specific PID until the user hits Ctrl+C. It can be useful if the
developer wants to take a look at what Impala is doing.
The resulting flame graph is written to an SVG file.

perf-query.sh:
It takes a query string as a parameter and passes it to the impala
shell to execute. While the query is executing the script samples
the CPU stack traces for the entire system. The resulting flame
graph is written to an SVG file.
E.g.:
perf-query.sh "select count(*) from tpch.lineitem group by l_returnflag"

Change-Id: Ib3da696b939204d23c5285dcf1bf6ee3a3738415
---
A bin/perf_tools/perf-query.sh
A bin/perf_tools/perf-record.sh
2 files changed, 171 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/17834/3
--
To view, visit http://gerrit.cloudera.org:8080/17834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib3da696b939204d23c5285dcf1bf6ee3a3738415
Gerrit-Change-Number: 17834
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Reviewer: Zoltan Borok-Nagy 


[Impala-ASF-CR] [WIP]: Initial commit: Add logic to sync to latest event id

2021-09-10 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17756 )

Change subject: [WIP]: Initial commit: Add logic to sync to latest event id
..


Patch Set 28:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7466/ 
DRY_RUN=true


--
To view, visit http://gerrit.cloudera.org:8080/17756
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia822d15725d0a9a9ad1398e10ed4ae3288d0e9ad
Gerrit-Change-Number: 17756
Gerrit-PatchSet: 28
Gerrit-Owner: Sourabh Goyal 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Sourabh Goyal 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Fri, 10 Sep 2021 08:08:22 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-10905: test iceberg.py test time travel fails in exhaustive builds

2021-09-10 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17835 )

Change subject: IMPALA-10905: test_iceberg.py test_time_travel fails in 
exhaustive builds
..


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7465/ 
DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/17835
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5c172df865bc35dd1792146f4045fbc16e04d116
Gerrit-Change-Number: 17835
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Fri, 10 Sep 2021 08:07:08 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-10905: test iceberg.py test time travel fails in exhaustive builds

2021-09-10 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17835 )

Change subject: IMPALA-10905: test_iceberg.py test_time_travel fails in 
exhaustive builds
..


Patch Set 3: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/17835
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5c172df865bc35dd1792146f4045fbc16e04d116
Gerrit-Change-Number: 17835
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Fri, 10 Sep 2021 08:07:07 +
Gerrit-HasComments: No