[Impala-ASF-CR] IMPALA-13477: Set request pool in QueryStateRecord for CTAS query
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21975 ) Change subject: IMPALA-13477: Set request_pool in QueryStateRecord for CTAS query .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21975 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6885192f1c2d563e58670f142b3c0df528032a6e Gerrit-Change-Number: 21975 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 25 Oct 2024 16:58:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13477: Set request pool in QueryStateRecord for CTAS query
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21975 ) Change subject: IMPALA-13477: Set request_pool in QueryStateRecord for CTAS query .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21975 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6885192f1c2d563e58670f142b3c0df528032a6e Gerrit-Change-Number: 21975 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 24 Oct 2024 18:39:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 ) Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes .. Patch Set 7: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21927 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc Gerrit-Change-Number: 21927 Gerrit-PatchSet: 7 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 18 Oct 2024 20:29:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 ) Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes .. Patch Set 6: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21927 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc Gerrit-Change-Number: 21927 Gerrit-PatchSet: 6 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 18 Oct 2024 17:49:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/21927 )
Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
..
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21927/5/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
File fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java:
http://gerrit.cloudera.org:8080/#/c/21927/5/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java@456
PS5, Line 456: minWriters = numInputNodes;
: maxWriters = numInputNodes;
In this case, byteBasedNumWriters = numInputNodes.
You can re-organize code as:
if (totalNumPartitions == 1) {
byteBasedNumWriters = numInputNodes;
} else {
long minWriters = Math.min(numInputNodes, totalNumPartitions);
long maxWriters = totalNumPartitions;
byteBasedNumWriters = Math.max(minWriters, byteBasedNumWriters);
byteBasedNumWriters = Math.min(maxWriters, byteBasedNumWriters);
}
--
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 5
Gerrit-Owner: Riza Suminto
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: David Rorke
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Fri, 18 Oct 2024 17:22:55 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13463: Impala should ignore case of Iceberg schema elements
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21950 ) Change subject: IMPALA-13463: Impala should ignore case of Iceberg schema elements .. Patch Set 4: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21950 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iedaf152d8a0c02a124c3dcf8acb59b4ba4e81cf4 Gerrit-Change-Number: 21950 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 17 Oct 2024 23:00:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 ) Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes .. Patch Set 4: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21927 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc Gerrit-Change-Number: 21927 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 17 Oct 2024 21:01:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 ) Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes .. Patch Set 3: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21927 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc Gerrit-Change-Number: 21927 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 17 Oct 2024 20:50:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 ) Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/21927/2/fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java File fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java: http://gerrit.cloudera.org:8080/#/c/21927/2/fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java@249 PS2, Line 249: int maxInstances = IntMath.saturatedMultiply( : inputFragment.getNumNodes(), analyzer.getMaxParallelismPerNode()); it's already calculated in line 240 as costBasedMaxWriter -- To view, visit http://gerrit.cloudera.org:8080/21927 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc Gerrit-Change-Number: 21927 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 17 Oct 2024 19:19:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 ) Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/21927/2/fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java File fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java: http://gerrit.cloudera.org:8080/#/c/21927/2/fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java@249 PS2, Line 249: int maxInstances = IntMath.saturatedMultiply( : inputFragment.getNumNodes(), analyzer.getMaxParallelismPerNode()); > it's already calculated in line 240 as costBasedMaxWriter Ignore it -- To view, visit http://gerrit.cloudera.org:8080/21927 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc Gerrit-Change-Number: 21927 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 17 Oct 2024 19:20:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13463: Impala should ignore case of Iceberg schema elements
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21950 ) Change subject: IMPALA-13463: Impala should ignore case of Iceberg schema elements .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21950 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iedaf152d8a0c02a124c3dcf8acb59b4ba4e81cf4 Gerrit-Change-Number: 21950 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 17 Oct 2024 16:02:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13445: Ignore num partition for unpartitioned writes
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/21927 )
Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21927/1/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
File fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java:
http://gerrit.cloudera.org:8080/#/c/21927/1/fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java@449
PS1, Line 449: if (totalNumPartitions >= maxNumWriters) {
: byteBasedNumWriters = maxNumWriters;
: } else {
: byteBasedNumWriters = (int) totalNumPartitions;
: }
byteBasedNumWriters value calculated in line 443 is not used for partitioned
insert. Are you going to change the behavior for partitioned insert? Or you can
move line 441 - 444 in else block.
--
To view, visit http://gerrit.cloudera.org:8080/21927
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc
Gerrit-Change-Number: 21927
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto
Gerrit-Reviewer: David Rorke
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Wed, 16 Oct 2024 23:53:46 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13132: Stop Kudu in kill-all.sh
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21915 ) Change subject: IMPALA-13132: Stop Kudu in kill-all.sh .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21915 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c40d881140658e8213f742d52e4eb73261e1b30 Gerrit-Change-Number: 21915 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 09 Oct 2024 20:06:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13427: Make connect timeout of statestore HA RPC tunable
Wenzhe Zhou has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21900 ) Change subject: IMPALA-13427: Make connect timeout of statestore HA RPC tunable .. IMPALA-13427: Make connect timeout of statestore HA RPC tunable In some deployment environment, unsuccessful socket connection attempts take long time to return error if connect timeout of the socket is set as 0. This causes statestored to tie up in unsuccessful connection attempts during initialization if its peer is not ready when statestore HA is enabled. Currently socket connect timeout of Thrift RPC is always set as 0. This patch makes socket connect timeout of Thrift RPC tunable, and adds a flag variable statestore_ha_client_rpc_conn_timeout_ms for the underlying socket connect timeout for Statestore HA RPC. Its default value equals 0, which equals the default value of TSocket.connTimeout_. Testing: - Added new test cases for statestore HA with non zero value for statestore_ha_client_rpc_conn_timeout_ms. - Passed core tests. Change-Id: Ie5840a76b4a34b4714c47b86f6366328b5ceed3a Reviewed-on: http://gerrit.cloudera.org:8080/21900 Reviewed-by: Norbert Luksa Tested-by: Impala Public Jenkins --- M be/src/rpc/thrift-client.h M be/src/runtime/client-cache.cc M be/src/runtime/client-cache.h M be/src/statestore/statestore.cc M tests/custom_cluster/test_statestored_ha.py 5 files changed, 68 insertions(+), 20 deletions(-) Approvals: Norbert Luksa: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/21900 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie5840a76b4a34b4714c47b86f6366328b5ceed3a Gerrit-Change-Number: 21900 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-13427: Make connect timeout of statestore HA RPC tunable
Wenzhe Zhou has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/21900 ) Change subject: IMPALA-13427: Make connect timeout of statestore HA RPC tunable .. IMPALA-13427: Make connect timeout of statestore HA RPC tunable In some deployment environment, unsuccessful socket connection attempts take long time to return error if connect timeout of the socket is set as 0. This causes statestored to tie up in unsuccessful connection attempts during initialization if its peer is not ready when statestore HA is enabled. Currently socket connect timeout of Thrift RPC is always set as 0. This patch makes socket connect timeout of Thrift RPC tunable, and adds a flag variable statestore_ha_client_rpc_conn_timeout_ms for the underlying socket connect timeout for Statestore HA RPC. Its default value equals 0, which equals the default value of TSocket.connTimeout_. Testing: - Added new test cases for statestore HA with non zero value for statestore_ha_client_rpc_conn_timeout_ms. - Passed core tests. Change-Id: Ie5840a76b4a34b4714c47b86f6366328b5ceed3a --- M be/src/rpc/thrift-client.h M be/src/runtime/client-cache.cc M be/src/runtime/client-cache.h M be/src/statestore/statestore.cc M tests/custom_cluster/test_statestored_ha.py 5 files changed, 68 insertions(+), 20 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/21900/2 -- To view, visit http://gerrit.cloudera.org:8080/21900 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie5840a76b4a34b4714c47b86f6366328b5ceed3a Gerrit-Change-Number: 21900 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] WIP: Add flag variable for Statestore HA Thrift RPC
Wenzhe Zhou has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21900 Change subject: WIP: Add flag variable for Statestore HA Thrift RPC .. WIP: Add flag variable for Statestore HA Thrift RPC Add flag variable statestore_ha_client_rpc_conn_timeout_ms for the underlying TSocket conn timeout for client RPC of Statestore HA service. The default value equals 0, which is same as the default value of connTimeout in TSocket. We could use this flag variable to prevent statestored from being tied up in unsuccessful connection attempts during initialization. Testing: - TODO add new test case - Passed test_statestored_ha.py Change-Id: Ie5840a76b4a34b4714c47b86f6366328b5ceed3a --- M be/src/rpc/thrift-client.h M be/src/runtime/client-cache.cc M be/src/runtime/client-cache.h M be/src/statestore/statestore.cc 4 files changed, 16 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/21900/1 -- To view, visit http://gerrit.cloudera.org:8080/21900 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ie5840a76b4a34b4714c47b86f6366328b5ceed3a Gerrit-Change-Number: 21900 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat
[Impala-ASF-CR] [WIP] IMPALA-12789: Fix unit-test code JdbcDataSourceTest.java
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21805 ) Change subject: [WIP] IMPALA-12789: Fix unit-test code JdbcDataSourceTest.java .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/21805/1/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java File fe/src/main/java/org/apache/impala/common/FileSystemUtil.java: http://gerrit.cloudera.org:8080/#/c/21805/1/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@798 PS1, Line 798: BackendConfig.INSTANCE != null Did this happen for Java based frontend unit-test? -- To view, visit http://gerrit.cloudera.org:8080/21805 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie07173d256d73c88f5a6c041f087db16b6ff3127 Gerrit-Change-Number: 21805 Gerrit-PatchSet: 1 Gerrit-Owner: Pranav Lodha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pranav Lodha Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 03 Oct 2024 22:02:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-12789: Fix unit-test code JdbcDataSourceTest.java
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/21805 )
Change subject: [WIP] IMPALA-12789: Fix unit-test code JdbcDataSourceTest.java
..
Patch Set 1:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/21805/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21805/1//COMMIT_MSG@18
PS1, Line 18: data source APIs.
mention how do you run the unit-test after your fixing
http://gerrit.cloudera.org:8080/#/c/21805/1/fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java
File
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java:
http://gerrit.cloudera.org:8080/#/c/21805/1/fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java@346
PS1, Line 346: props.put("maxTotal", "3");
: props.put("maxIdle", "3");
: props.put("minIdle", "0");
: props.put("maxWaitMillis", "1");
dbcp_max_conn_pool_size, dbcp_max_wait_millis_for_conn and
dbcp_data_source_idle_timeout_s are defined in be/src/service/frontend.cc with
default values. Use same default values here if those values could not be
passed from backend.
http://gerrit.cloudera.org:8080/#/c/21805/1/java/jamm-prefix/src/jamm
File java/jamm-prefix/src/jamm:
http://gerrit.cloudera.org:8080/#/c/21805/1/java/jamm-prefix/src/jamm@1
PS1, Line 1: Subproject commit 49c3e691c43a39ddaccaf19839ef316807009377
why do you need to add this file?
http://gerrit.cloudera.org:8080/#/c/21805/1/testdata/bin/create-ext-data-source-table.sql
File testdata/bin/create-ext-data-source-table.sql:
http://gerrit.cloudera.org:8080/#/c/21805/1/testdata/bin/create-ext-data-source-table.sql@31
PS1, Line 31: id INT,
unnecessary change
--
To view, visit http://gerrit.cloudera.org:8080/21805
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie07173d256d73c88f5a6c041f087db16b6ff3127
Gerrit-Change-Number: 21805
Gerrit-PatchSet: 1
Gerrit-Owner: Pranav Lodha
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Pranav Lodha
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Thu, 03 Oct 2024 21:59:59 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13388: fix unit-tests of Statestore HA for UBSAN builds
Wenzhe Zhou has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21820 ) Change subject: IMPALA-13388: fix unit-tests of Statestore HA for UBSAN builds .. IMPALA-13388: fix unit-tests of Statestore HA for UBSAN builds Sometimes in UBSAN builds, unit-tests of Statestore HA failed due to Thrift RPC receiving timeout. Standby statestored failed to send heartbeats to its subscribers so that failover was not triggered. The Thrift RPC failures still happened after increasing TCP timeout for Thrift RPCs between statestored and its subscribers. This patch adds a metric for number of subscribers which recevied heartbeats from statestored in a monitoring period. Unit-tests of Statestored HA for UBSAN build will be skipped if statestored failed to send heartbeats to more than half of subscribers. For other builds, throw exception with error message which complain Thrift RPC failure if statestored failed to send heartbeats to more than half of subscribers. Also fixed a bug which calls SecondsSinceHeartbeat() but compares the retutned value with time value in milli-seconds. Filed following up JIRA IMPALA-13399 to track the very root cause. Testing: - Looped to run test_statestored_ha.py for 100 times in UBSAN build without failed case, but 4 iterations out of 100 have skipped test cases. - Verified that the issue did not happen for ASAN build by running test_statestored_ha.py for 100 times in ASAN build. - Passed core test. Change-Id: Ie59d1e93c635411723f7044da52e4ab19c7d2fac Reviewed-on: http://gerrit.cloudera.org:8080/21820 Reviewed-by: Riza Suminto Tested-by: Impala Public Jenkins --- M be/src/statestore/statestore.cc M be/src/statestore/statestore.h M common/thrift/metrics.json M tests/custom_cluster/test_statestored_ha.py 4 files changed, 89 insertions(+), 22 deletions(-) Approvals: Riza Suminto: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/21820 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie59d1e93c635411723f7044da52e4ab19c7d2fac Gerrit-Change-Number: 21820 Gerrit-PatchSet: 6 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-13388: fix unit-tests of Statestore HA for UBSAN builds
Wenzhe Zhou has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/21820 ) Change subject: IMPALA-13388: fix unit-tests of Statestore HA for UBSAN builds .. IMPALA-13388: fix unit-tests of Statestore HA for UBSAN builds Sometimes in UBSAN builds, unit-tests of Statestore HA failed due to Thrift RPC receiving timeout. Standby statestored failed to send heartbeats to its subscribers so that failover was not triggered. The Thrift RPC failures still happened after increasing TCP timeout for Thrift RPCs between statestored and its subscribers. This patch adds a metric for number of subscribers which recevied heartbeats from statestored in a monitoring period. Unit-tests of Statestored HA for UBSAN build will be skipped if statestored failed to send heartbeats to more than half of subscribers. For other builds, throw exception with error message which complain Thrift RPC failure if statestored failed to send heartbeats to more than half of subscribers. Also fixed a bug which calls SecondsSinceHeartbeat() but compares the retutned value with time value in milli-seconds. Filed following up JIRA IMPALA-13399 to track the very root cause. Testing: - Looped to run test_statestored_ha.py for 100 times in UBSAN build without failed case, but 4 iterations out of 100 have skipped test cases. - Verified that the issue did not happen for ASAN build by running test_statestored_ha.py for 100 times in ASAN build. - Passed core test. Change-Id: Ie59d1e93c635411723f7044da52e4ab19c7d2fac --- M be/src/statestore/statestore.cc M be/src/statestore/statestore.h M common/thrift/metrics.json M tests/custom_cluster/test_statestored_ha.py 4 files changed, 89 insertions(+), 22 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/21820/5 -- To view, visit http://gerrit.cloudera.org:8080/21820 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie59d1e93c635411723f7044da52e4ab19c7d2fac Gerrit-Change-Number: 21820 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-13388: fix unit-tests of Statestore HA for UBSAN builds
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21820 ) Change subject: IMPALA-13388: fix unit-tests of Statestore HA for UBSAN builds .. Patch Set 5: (2 comments) http://gerrit.cloudera.org:8080/#/c/21820/4/be/src/statestore/statestore.cc File be/src/statestore/statestore.cc: http://gerrit.cloudera.org:8080/#/c/21820/4/be/src/statestore/statestore.cc@1468 PS4, Line 1468: subscriber.second->MilliSecondsSinceHeartbeat() : > FLAGS_heartbeat_monitoring_frequency_ms > nit: mention this bug fix in commit message? updated commit message http://gerrit.cloudera.org:8080/#/c/21820/4/tests/custom_cluster/test_statestored_ha.py File tests/custom_cluster/test_statestored_ha.py: http://gerrit.cloudera.org:8080/#/c/21820/4/tests/custom_cluster/test_statestored_ha.py@159 PS4, Line 159: # IMPALA-13388: Skip this tests for UBSAN build due to Thrift RPC failures. > nit: Please file a follow up JIRA so we can track this issue, find the very Filed following up JIRA IMPALA-13399 to track the very root cause. -- To view, visit http://gerrit.cloudera.org:8080/21820 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie59d1e93c635411723f7044da52e4ab19c7d2fac Gerrit-Change-Number: 21820 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 23 Sep 2024 22:31:35 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13388: fix unit-tests of Statestore HA for UBSAN builds
Wenzhe Zhou has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/21820 ) Change subject: IMPALA-13388: fix unit-tests of Statestore HA for UBSAN builds .. IMPALA-13388: fix unit-tests of Statestore HA for UBSAN builds Sometimes in UBSAN builds, unit-tests of Statestore HA failed due to Thrift RPC receiving timeout. Standby statestored failed to send heartbeats to its subscribers so that failover was not triggered. The Thrift RPC failures still happened after increasing TCP timeout for Thrift RPCs between statestored and its subscribers. This patch adds a metric for number of subscribers which recevied heartbeats from statestored in a monitoring period. Unit-tests of Statestored HA for UBSAN build will be skipped if statestored failed to send heartbeats to more than half of subscribers. For other builds, throw exception with error message which complain Thrift RPC failure if statestored failed to send heartbeats to more than half of subscribers. Testing: - Looped to run test_statestored_ha.py for 100 times in UBSAN build without failed case, but 4 iterations out of 100 have skipped test cases. - Verified that the issue did not happen for ASAN build by running test_statestored_ha.py for 100 times in ASAN build. - Passed core test. Change-Id: Ie59d1e93c635411723f7044da52e4ab19c7d2fac --- M be/src/statestore/statestore.cc M be/src/statestore/statestore.h M common/thrift/metrics.json M tests/custom_cluster/test_statestored_ha.py 4 files changed, 87 insertions(+), 22 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/21820/4 -- To view, visit http://gerrit.cloudera.org:8080/21820 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie59d1e93c635411723f7044da52e4ab19c7d2fac Gerrit-Change-Number: 21820 Gerrit-PatchSet: 4 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-13388: fix unit-tests of Statestore HA for UBSAN builds
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/21820 )
Change subject: IMPALA-13388: fix unit-tests of Statestore HA for UBSAN builds
..
Patch Set 4:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/21820/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21820/3//COMMIT_MSG@24
PS3, Line 24: Looped to run test_statestored_ha.py for 100 times in UBSAN
:build without failed case, but 4 iterations out of 100 h
> Question: How many iteration has skipped test out of 100?
4 out of 100
http://gerrit.cloudera.org:8080/#/c/21820/3/be/src/statestore/statestore.h
File be/src/statestore/statestore.h:
http://gerrit.cloudera.org:8080/#/c/21820/3/be/src/statestore/statestore.h@531
PS3, Line 531: return received_heartbeat_.Load();
> Can this be derived from existing last_heartbeat_ts_?
No, last_heartbeat_ts_ is set with timestamp when the subscriber is registered.
SecondsSinceHeartbeat() is used for output of Web page.
http://gerrit.cloudera.org:8080/#/c/21820/3/tests/custom_cluster/test_statestored_ha.py
File tests/custom_cluster/test_statestored_ha.py:
http://gerrit.cloudera.org:8080/#/c/21820/3/tests/custom_cluster/test_statestored_ha.py@156
PS3, Line 156: {0}
> Please mention total expected subscriber as well.
fixed
http://gerrit.cloudera.org:8080/#/c/21820/3/tests/custom_cluster/test_statestored_ha.py@157
PS3, Line 157: num_failed, num_subscribers)
: if ImpalaTestClusterProperties.get_instance().is_ubsan():
: # IMPALA-13388: Skip this tests for UBSAN build due to
Thrift RPC fail
> The JIRA report only test_statestored_auto_failover is flaky. I expect only
Ran UBSAN build multiple times for test_statestored_ha.py. The issue happened
randomly in different test cases. For IMPALA-13348, it happened in
test_statestored_manual_failover_with_failed_rpc.
http://gerrit.cloudera.org:8080/#/c/21820/3/tests/custom_cluster/test_statestored_ha.py@160
PS3, Line 160: if statestored_index ==
> Please copy the assert inside both if branch for more precise stacktrace if
fixed
--
To view, visit http://gerrit.cloudera.org:8080/21820
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie59d1e93c635411723f7044da52e4ab19c7d2fac
Gerrit-Change-Number: 21820
Gerrit-PatchSet: 4
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Mon, 23 Sep 2024 20:54:00 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13388: fix unit-tests of Statestore HA for UBSAN builds
Wenzhe Zhou has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21820 Change subject: IMPALA-13388: fix unit-tests of Statestore HA for UBSAN builds .. IMPALA-13388: fix unit-tests of Statestore HA for UBSAN builds Sometimes in UBSAN builds, unit-tests of Statestore HA failed due to Thrift RPC receiving timeout. Standby statestored failed to send heartbeats to its subscribers so that failover was not triggered. The Thrift RPC failures still happened after increasing TCP timeout for Thrift RPCs between statestored and its subscribers. This patch adds a metric for number of subscribers which recevied heartbeats from statestored in a monitoring period. Unit-tests of Statestored HA for UBSAN build will be skipped if statestored failed to send heartbeats to more than half of subscribers. For other builds, throw exception with error message which complain Thrift RPC failure if statestored failed to send heartbeats to more than half of subscribers. Testing: - Looped to run test_statestored_ha.py for 100 times in UBSAN build without failed case, but with a few skipped cases. - Passed core test. Change-Id: Ie59d1e93c635411723f7044da52e4ab19c7d2fac --- M be/src/statestore/statestore.cc M be/src/statestore/statestore.h M common/thrift/metrics.json M tests/custom_cluster/test_statestored_ha.py 4 files changed, 77 insertions(+), 22 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/21820/3 -- To view, visit http://gerrit.cloudera.org:8080/21820 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ie59d1e93c635411723f7044da52e4ab19c7d2fac Gerrit-Change-Number: 21820 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-13101: test data source tables fails with Data source does not exist
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21815 ) Change subject: IMPALA-13101: test_data_source_tables fails with Data source does not exist .. Patch Set 6: Code-Review+1 Tried the patch set 5 on my local machine. Both custom_cluster/test_ext_data_sources.py and query_test/test_ext_data_sources.py are working. Restarted GVO. -- To view, visit http://gerrit.cloudera.org:8080/21815 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29822855da8136e013c8a62bb0489a181bf131ae Gerrit-Change-Number: 21815 Gerrit-PatchSet: 6 Gerrit-Owner: Pranav Lodha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pranav Lodha Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 18 Sep 2024 16:05:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13101: test data source tables fails with Data source does not exist
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21815 ) Change subject: IMPALA-13101: test_data_source_tables fails with Data source does not exist .. Patch Set 4: (2 comments) http://gerrit.cloudera.org:8080/#/c/21815/4/tests/custom_cluster/test_ext_data_sources.py File tests/custom_cluster/test_ext_data_sources.py: http://gerrit.cloudera.org:8080/#/c/21815/4/tests/custom_cluster/test_ext_data_sources.py@66 PS4, Line 66: data-source-tables set $UNIQUE_DATASOURCE http://gerrit.cloudera.org:8080/#/c/21815/4/tests/custom_cluster/test_ext_data_sources.py@73 PS4, Line 73: data-source-tables set $UNIQUE_DATASOURCE -- To view, visit http://gerrit.cloudera.org:8080/21815 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29822855da8136e013c8a62bb0489a181bf131ae Gerrit-Change-Number: 21815 Gerrit-PatchSet: 4 Gerrit-Owner: Pranav Lodha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pranav Lodha Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 18 Sep 2024 06:34:33 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13101: test data source tables fails with Data source does not exist
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21815 ) Change subject: IMPALA-13101: test_data_source_tables fails with Data source does not exist .. Patch Set 3: QueryTest/data-source-tables is also called from custom_cluster/test_ext_data_sources.py -- To view, visit http://gerrit.cloudera.org:8080/21815 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29822855da8136e013c8a62bb0489a181bf131ae Gerrit-Change-Number: 21815 Gerrit-PatchSet: 3 Gerrit-Owner: Pranav Lodha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pranav Lodha Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 18 Sep 2024 00:19:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13390: Lower MEM LIMIT in test exchange mem usage scaling
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21817 ) Change subject: IMPALA-13390: Lower MEM_LIMIT in test_exchange_mem_usage_scaling .. Patch Set 2: Code-Review+2 carry +1 from Fang-Yu -- To view, visit http://gerrit.cloudera.org:8080/21817 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0a4bbfbec01ae63299d3de049f7766ba2ad0e913 Gerrit-Change-Number: 21817 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 17 Sep 2024 23:00:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13101: test data source tables fails with Data source does not exist
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21815 ) Change subject: IMPALA-13101: test_data_source_tables fails with Data source does not exist .. Patch Set 3: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21815 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29822855da8136e013c8a62bb0489a181bf131ae Gerrit-Change-Number: 21815 Gerrit-PatchSet: 3 Gerrit-Owner: Pranav Lodha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pranav Lodha Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 17 Sep 2024 18:46:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13101: test data source tables fails with Data source does not exist
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21815 ) Change subject: IMPALA-13101: test_data_source_tables fails with Data source does not exist .. Patch Set 1: Thanks to take care this issue. Looks good to me. Please update commit message as Riza suggested. -- To view, visit http://gerrit.cloudera.org:8080/21815 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29822855da8136e013c8a62bb0489a181bf131ae Gerrit-Change-Number: 21815 Gerrit-PatchSet: 1 Gerrit-Owner: Pranav Lodha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pranav Lodha Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 17 Sep 2024 18:16:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13378: Verify tuple ids in descriptor table received in executor side
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21794 ) Change subject: IMPALA-13378: Verify tuple ids in descriptor table received in executor side .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21794 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0c489d3bff7ae08813271c65086ea6b238420e47 Gerrit-Change-Number: 21794 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 13 Sep 2024 14:50:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13182: Support uploading additional jars
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21556 ) Change subject: IMPALA-13182: Support uploading additional jars .. Patch Set 5: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21556 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ica5fa4c0cd1a5c938f331f3a4bba85d4910db90e Gerrit-Change-Number: 21556 Gerrit-PatchSet: 5 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Thu, 12 Sep 2024 18:04:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13378: Verify tuple ids in descriptor table received in executor side
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21794 ) Change subject: IMPALA-13378: Verify tuple ids in descriptor table received in executor side .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21794 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0c489d3bff7ae08813271c65086ea6b238420e47 Gerrit-Change-Number: 21794 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 12 Sep 2024 17:25:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13182: Support uploading additional jars
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21556 ) Change subject: IMPALA-13182: Support uploading additional jars .. Patch Set 4: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21556 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ica5fa4c0cd1a5c938f331f3a4bba85d4910db90e Gerrit-Change-Number: 21556 Gerrit-PatchSet: 4 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Tue, 13 Aug 2024 20:45:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13182: Support uploading additional jars
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21556 ) Change subject: IMPALA-13182: Support uploading additional jars .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/21556/3/docker/daemon_entrypoint.sh File docker/daemon_entrypoint.sh: http://gerrit.cloudera.org:8080/#/c/21556/3/docker/daemon_entrypoint.sh@118 PS3, Line 118: CLASSPATH+=:/opt/impala/aux-jars > I will add the validation in the file download script. Prefer to append all of jars in the directory /opt/impala/aux-jars to classpath as line 110 - 114. -- To view, visit http://gerrit.cloudera.org:8080/21556 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ica5fa4c0cd1a5c938f331f3a4bba85d4910db90e Gerrit-Change-Number: 21556 Gerrit-PatchSet: 3 Gerrit-Owner: gaurav singh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Tue, 13 Aug 2024 16:27:02 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13250: [DOCS] Document ENABLED RUNTIME FILTER TYPES query option
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21645 ) Change subject: IMPALA-13250: [DOCS] Document ENABLED_RUNTIME_FILTER_TYPES query option .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21645 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib7a34782bed6f812fedf717d8a076e2706f0bba9 Gerrit-Change-Number: 21645 Gerrit-PatchSet: 2 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 07 Aug 2024 21:44:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13250: [DOCS] Document ENABLED RUNTIME FILTER TYPES query option
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21645 ) Change subject: IMPALA-13250: [DOCS] Document ENABLED_RUNTIME_FILTER_TYPES query option .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/21645/1/docs/topics/impala_enabled_runtime_filter_types.xml File docs/topics/impala_enabled_runtime_filter_types.xml: http://gerrit.cloudera.org:8080/#/c/21645/1/docs/topics/impala_enabled_runtime_filter_types.xml@67 PS1, Line 67: Impala backend expects comma separated values to be in quotes when executing the > Could this be a shared blurb? I assume we have other comma-separated query runtime_filter_ids_to_skip is also set with comma-separated values. -- To view, visit http://gerrit.cloudera.org:8080/21645 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib7a34782bed6f812fedf717d8a076e2706f0bba9 Gerrit-Change-Number: 21645 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 06 Aug 2024 16:44:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13142: [DOCS] Documentation for Impala StateStore & Catalogd HA
Wenzhe Zhou has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21615 ) Change subject: IMPALA-13142: [DOCS] Documentation for Impala StateStore & Catalogd HA .. IMPALA-13142: [DOCS] Documentation for Impala StateStore & Catalogd HA Change-Id: I8927c9cd61f0274ad9d6ac4a079f7a563197 Reviewed-on: http://gerrit.cloudera.org:8080/21615 Tested-by: Impala Public Jenkins Reviewed-by: Yida Wu Reviewed-by: Wenzhe Zhou --- M docs/impala.ditamap A docs/topics/impala_ha.xml A docs/topics/impala_ha_catalog.xml A docs/topics/impala_ha_statestore.xml 4 files changed, 252 insertions(+), 2 deletions(-) Approvals: Impala Public Jenkins: Verified Yida Wu: Looks good to me, but someone else must approve Wenzhe Zhou: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/21615 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8927c9cd61f0274ad9d6ac4a079f7a563197 Gerrit-Change-Number: 21615 Gerrit-PatchSet: 5 Gerrit-Owner: Sanjana Malhotra Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-13142: [DOCS] Documentation for Impala StateStore & Catalogd HA
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21615 ) Change subject: IMPALA-13142: [DOCS] Documentation for Impala StateStore & Catalogd HA .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21615 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8927c9cd61f0274ad9d6ac4a079f7a563197 Gerrit-Change-Number: 21615 Gerrit-PatchSet: 4 Gerrit-Owner: Sanjana Malhotra Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Thu, 01 Aug 2024 02:36:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21617 ) Change subject: IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table .. Patch Set 5: Thanks Michael, Abhishek, Yifan and Pranav. -- To view, visit http://gerrit.cloudera.org:8080/21617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334 Gerrit-Change-Number: 21617 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Pranav Lodha Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yifan Zhang Gerrit-Comment-Date: Thu, 01 Aug 2024 02:29:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
Wenzhe Zhou has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/21617 ) Change subject: IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table .. IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table Function DatabaseAccessor.getTotalNumberOfRecords() is defined with return data type as int. This caused Impala to throw an exception when executing COUNT(*) query for jdbc tables with more than 2G rows. This patch fixed the issue by changing the function return type as long. It also made number of rows in each TRowBatch fetched from jdbc data source not to exceed 2G. Testing: - Passed core-test and tpcds test for jdbc tables. - Manually created a jdbc table jdbc_table with more than 2G rows, verified that query 'select count(*) from jdbc_table' returned correct number of rows. Detailed steps were added in the comments of IMPALA-13256. Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334 --- M be/src/exec/data-source-scan-node.cc M fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java M fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java M fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java 4 files changed, 19 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/17/21617/5 -- To view, visit http://gerrit.cloudera.org:8080/21617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334 Gerrit-Change-Number: 21617 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Pranav Lodha Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yifan Zhang
[Impala-ASF-CR] IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
Wenzhe Zhou has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/21617 ) Change subject: IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table .. IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table Function DatabaseAccessor.getTotalNumberOfRecords() is defined with return data type as int. This caused Impala to throw an exception when executing COUNT(*) query for jdbc tables with more than 2G rows. This patch fixed the issue by changing the function return type as long. It also made number of rows in each TRowBatch fetched from jdbc data source not to exceed 2G. Testing: - Passed core-test and tpcds test for jdbc tables. - Manually created a jdbc table jdbc_table with more than 2G rows, verified that query 'select count(*) from jdbc_table' returned correct number of rows. Detailed steps were added in the comments of IMPALA-13256. Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334 --- M be/src/exec/data-source-scan-node.cc M fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java M fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java M fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java 4 files changed, 19 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/17/21617/4 -- To view, visit http://gerrit.cloudera.org:8080/21617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334 Gerrit-Change-Number: 21617 Gerrit-PatchSet: 4 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pranav Lodha Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yifan Zhang
[Impala-ASF-CR] IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21617 ) Change subject: IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table .. Patch Set 3: (5 comments) http://gerrit.cloudera.org:8080/#/c/21617/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21617/2//COMMIT_MSG@7 PS2, Line 7: COUNT(*) o > nit: Shouldn't it be COUNT(*) fixed http://gerrit.cloudera.org:8080/#/c/21617/2//COMMIT_MSG@10 PS2, Line 10: caused Impala to throw > nit: caused Impala to throw an fixed http://gerrit.cloudera.org:8080/#/c/21617/2//COMMIT_MSG@11 PS2, Line 11: COUNT(*) q > nit: Shouldn't it be COUNT(*) fixed http://gerrit.cloudera.org:8080/#/c/21617/2//COMMIT_MSG@18 PS2, Line 18: Manually created a jdbc table jdbc_table with more than 2G rows, :verified that query 'select count(*) from jdbc_table' returned :correct number of rows. > I think it'd be good if we can add some tests or mention steps/tables for w The table is too big and may hit memory issue when creating such big table in runtime. It takes additional disk space in Jenkins and developers' machine if add unit-test. I added detail steps in the comments of IMPALA-13256. http://gerrit.cloudera.org:8080/#/c/21617/2/be/src/exec/data-source-scan-node.cc File be/src/exec/data-source-scan-node.cc: http://gerrit.cloudera.org:8080/#/c/21617/2/be/src/exec/data-source-scan-node.cc@146 PS2, Line 146: // The capacity of output RowBatch is defined as 4 > nit: Can we add a comment about this check, something like this: Added a comment. -- To view, visit http://gerrit.cloudera.org:8080/21617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334 Gerrit-Change-Number: 21617 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pranav Lodha Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yifan Zhang Gerrit-Comment-Date: Sun, 28 Jul 2024 07:03:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table
Wenzhe Zhou has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/21617 ) Change subject: IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table .. IMPALA-13256: Support more than 2G rows for COUNT(*) on jdbc table Function DatabaseAccessor.getTotalNumberOfRecords() is defined with return data type as int. This caused Impala to throw an exception when executing COUNT(*) query for jdbc tables with more than 2G rows. This patch fixed the issue by changing the function return type as long. It also made number of rows in each TRowBatch fetched from jdbc data source not to exceed 2G. Testing: - Passed core-test, and tpcds test for jdbc tables. - Manually created a jdbc table jdbc_table with more than 2G rows, verified that query 'select count(*) from jdbc_table' returned correct number of rows. Detail steps were added in the comments of IMPALA-13256. Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334 --- M be/src/exec/data-source-scan-node.cc M fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java M fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java M fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java 4 files changed, 18 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/17/21617/3 -- To view, visit http://gerrit.cloudera.org:8080/21617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334 Gerrit-Change-Number: 21617 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pranav Lodha Gerrit-Reviewer: Yifan Zhang
[Impala-ASF-CR] IMPALA-13256: Support more than 2G rows for count star on jdbc table
Wenzhe Zhou has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21617 Change subject: IMPALA-13256: Support more than 2G rows for count star on jdbc table .. IMPALA-13256: Support more than 2G rows for count star on jdbc table Function DatabaseAccessor.getTotalNumberOfRecords() is defined with return data type as int. This causes Impala to thrown exception when executing count star query for jdbc tables with more than 2G rows. This patch fixed the issue by changing the function return type as long. The number of rows in each RowBatch fetched from jdbc data source cannot exceed 2G. Testing: - Passed core-test. - Manually created a jdbc table jdbc_table with more than 2G rows, verified that query 'select count(*) from jdbc_table' returned correct number of rows. Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334 --- M be/src/exec/data-source-scan-node.cc M fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java M fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java M fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java 4 files changed, 15 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/17/21617/2 -- To view, visit http://gerrit.cloudera.org:8080/21617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I47db58300cbe3270bab07da02c3fcde6d7072334 Gerrit-Change-Number: 21617 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Pranav Lodha Gerrit-Reviewer: Yifan Zhang
[Impala-ASF-CR] IMPALA-13142: [DOCS] Documentation for Impala StateStore & Catalogd HA
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21615 ) Change subject: IMPALA-13142: [DOCS] Documentation for Impala StateStore & Catalogd HA .. Patch Set 1: (5 comments) A few comments for minor changes. Thanks http://gerrit.cloudera.org:8080/#/c/21615/1/docs/topics/impala_HA.xml File docs/topics/impala_HA.xml: http://gerrit.cloudera.org:8080/#/c/21615/1/docs/topics/impala_HA.xml@44 PS1, Line 44: two Impala catalogd, two StateStore instances on two different nodes. Add one sentence: "Note that CatalogD HA and Statestore HA are independent. User could enable CatalogD HA, or enable Statestore HA, or enable both CatalogD HA and Statestore HA." http://gerrit.cloudera.org:8080/#/c/21615/1/docs/topics/impala_HA_catalog.xml File docs/topics/impala_HA_catalog.xml: http://gerrit.cloudera.org:8080/#/c/21615/1/docs/topics/impala_HA_catalog.xml@40 PS1, Line 40: hive nit: upper case 'H' http://gerrit.cloudera.org:8080/#/c/21615/1/docs/topics/impala_HA_statestore.xml File docs/topics/impala_HA_statestore.xml: http://gerrit.cloudera.org:8080/#/c/21615/1/docs/topics/impala_HA_statestore.xml@41 PS1, Line 41: Catalog nit: CatalogD http://gerrit.cloudera.org:8080/#/c/21615/1/docs/topics/impala_HA_statestore.xml@42 PS1, Line 42: Catalog nit: CatalogD http://gerrit.cloudera.org:8080/#/c/21615/1/docs/topics/impala_HA_statestore.xml@100 PS1, Line 100: Catalog nit: CatalogD -- To view, visit http://gerrit.cloudera.org:8080/21615 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8927c9cd61f0274ad9d6ac4a079f7a563197 Gerrit-Change-Number: 21615 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Fri, 26 Jul 2024 22:25:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13208: Add cluster id to the membership and request-queue topic names
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21573 ) Change subject: IMPALA-13208: Add cluster id to the membership and request-queue topic names .. Patch Set 5: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21573 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2ff41539f568ef03c0ee2284762b4116b313d90f Gerrit-Change-Number: 21573 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Wed, 17 Jul 2024 14:15:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13208: Add cluster id to the membership and request-queue topic names
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21573 ) Change subject: IMPALA-13208: Add cluster id to the membership and request-queue topic names .. Patch Set 4: Code-Review+1 (2 comments) http://gerrit.cloudera.org:8080/#/c/21573/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21573/3//COMMIT_MSG@10 PS3, Line 10: cluster id to the membership > Yeah, I think hostname+port should be unique. But I'm ok with adding the cl Ack http://gerrit.cloudera.org:8080/#/c/21573/4/tests/common/impala_test_suite.py File tests/common/impala_test_suite.py: http://gerrit.cloudera.org:8080/#/c/21573/4/tests/common/impala_test_suite.py@323 PS4, Line 323: host_port = host_ports[0] nit: add a comment here -- To view, visit http://gerrit.cloudera.org:8080/21573 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2ff41539f568ef03c0ee2284762b4116b313d90f Gerrit-Change-Number: 21573 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Wed, 17 Jul 2024 05:48:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13208: Add cluster id to the membership and request-queue topic names
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21573 ) Change subject: IMPALA-13208: Add cluster id to the membership and request-queue topic names .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/21573/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21573/3//COMMIT_MSG@10 PS3, Line 10: cluster id to the membership One question about subscriber-id. Each statestore subscriber (catalogd/coordinator/executor) send registration info to statestore including subscriber-id, and statestore use it to identify subscribers. subscriber-id is generated as hostname + krpc_port (https://github.com/apache/impala/blob/master/be/src/runtime/exec-env.cc#L286-L289). Can we guarantee subscriber-id unique across multi clusters? Add cluster id as prefix for subscriber id? -- To view, visit http://gerrit.cloudera.org:8080/21573 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2ff41539f568ef03c0ee2284762b4116b313d90f Gerrit-Change-Number: 21573 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 16 Jul 2024 23:17:04 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12093: impala-shell to preserve all cookies
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19827 ) Change subject: IMPALA-12093: impala-shell to preserve all cookies .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19827 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic81f790288460b086ab218e6701e8115a996dfa7 Gerrit-Change-Number: 19827 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 27 Jun 2024 17:01:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13180: Upgrade postgresql to 42.5.6
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21553 ) Change subject: IMPALA-13180: Upgrade postgresql to 42.5.6 .. Patch Set 3: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21553 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ida9fabcf217c1e566a45d9f2d48705f1d990b0d3 Gerrit-Change-Number: 21553 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 25 Jun 2024 18:55:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13175: Upgrade Spring Framework to 5.3.37
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21551 ) Change subject: IMPALA-13175: Upgrade Spring Framework to 5.3.37 .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21551 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic99eadb89f457b45b721b18054a06e497c3a82f5 Gerrit-Change-Number: 21551 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 25 Jun 2024 15:29:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12093: impala-shell to preserve all cookies
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19827 ) Change subject: IMPALA-12093: impala-shell to preserve all cookies .. Patch Set 5: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/19827 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic81f790288460b086ab218e6701e8115a996dfa7 Gerrit-Change-Number: 19827 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 24 Jun 2024 22:33:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12093: impala-shell to preserve all cookies
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19827 ) Change subject: IMPALA-12093: impala-shell to preserve all cookies .. Patch Set 3: (2 comments) Thanks to working on this. LGTM. http://gerrit.cloudera.org:8080/#/c/19827/3/be/src/rpc/authentication-util.cc File be/src/rpc/authentication-util.cc: http://gerrit.cloudera.org:8080/#/c/19827/3/be/src/rpc/authentication-util.cc@40 PS3, Line 40: DEFINE_string_hidden(cookie_name, "impala.auth", > This was the least invasive way I came up with to test wildcard cookie matc Use debug action to make Impala return an additional Set-Cookie http://gerrit.cloudera.org:8080/#/c/19827/3/be/src/rpc/authentication-util.cc@42 PS3, Line 42: > extra '>' ? -- To view, visit http://gerrit.cloudera.org:8080/19827 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic81f790288460b086ab218e6701e8115a996dfa7 Gerrit-Change-Number: 19827 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 21 Jun 2024 05:09:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12754: [DOCS] External JDBC table support
Wenzhe Zhou has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21539 ) Change subject: IMPALA-12754: [DOCS] External JDBC table support .. IMPALA-12754: [DOCS] External JDBC table support Created the docs for Impala external JDBC table support Change-Id: I5360389037ae9ee675ab406d87617d55d476bf8f Reviewed-on: http://gerrit.cloudera.org:8080/21539 Tested-by: Impala Public Jenkins Reviewed-by: gaurav singh Reviewed-by: Wenzhe Zhou --- M docs/impala.ditamap A docs/topics/impala_jdbc_external_table.xml 2 files changed, 231 insertions(+), 0 deletions(-) Approvals: Impala Public Jenkins: Verified gaurav singh: Looks good to me, but someone else must approve Wenzhe Zhou: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/21539 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I5360389037ae9ee675ab406d87617d55d476bf8f Gerrit-Change-Number: 21539 Gerrit-PatchSet: 4 Gerrit-Owner: Jankiram Balakrishnan Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jankiram Balakrishnan Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12754: [DOCS] External JDBC table support
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21539 ) Change subject: IMPALA-12754: [DOCS] External JDBC table support .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21539 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5360389037ae9ee675ab406d87617d55d476bf8f Gerrit-Change-Number: 21539 Gerrit-PatchSet: 3 Gerrit-Owner: Jankiram Balakrishnan Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jankiram Balakrishnan Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Thu, 20 Jun 2024 18:05:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12754: [DOCS] External JDBC table support
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21539 ) Change subject: IMPALA-12754: [DOCS] External JDBC table support .. Patch Set 3: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21539 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5360389037ae9ee675ab406d87617d55d476bf8f Gerrit-Change-Number: 21539 Gerrit-PatchSet: 3 Gerrit-Owner: Jankiram Balakrishnan Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jankiram Balakrishnan Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Thu, 20 Jun 2024 15:23:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12754: [DOCS] External JDBC table support
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21539 ) Change subject: IMPALA-12754: [DOCS] External JDBC table support .. Patch Set 2: (2 comments) looks good to me. http://gerrit.cloudera.org:8080/#/c/21539/2/docs/topics/impala_jdbc_external_table.xml File docs/topics/impala_jdbc_external_table.xml: http://gerrit.cloudera.org:8080/#/c/21539/2/docs/topics/impala_jdbc_external_table.xml@157 PS2, Line 157: add a line break before "complex data types". http://gerrit.cloudera.org:8080/#/c/21539/2/docs/topics/impala_jdbc_external_table.xml@157 PS2, Line 157: types data types -- To view, visit http://gerrit.cloudera.org:8080/21539 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5360389037ae9ee675ab406d87617d55d476bf8f Gerrit-Change-Number: 21539 Gerrit-PatchSet: 2 Gerrit-Owner: Jankiram Balakrishnan Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jankiram Balakrishnan Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Thu, 20 Jun 2024 04:32:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12754: [DOCS] External JDBC table support
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21539 ) Change subject: IMPALA-12754: [DOCS] External JDBC table support .. Patch Set 1: (14 comments) http://gerrit.cloudera.org:8080/#/c/21539/1/docs/topics/impala_jdbc_external_table.xml File docs/topics/impala_jdbc_external_table.xml: http://gerrit.cloudera.org:8080/#/c/21539/1/docs/topics/impala_jdbc_external_table.xml@22 PS1, Line 22: Support for Impala external JDBC data sources Change title as "Using Impala to Query External JDBC Data Sources" http://gerrit.cloudera.org:8080/#/c/21539/1/docs/topics/impala_jdbc_external_table.xml@40 PS1, Line 40: external table external JDBC table http://gerrit.cloudera.org:8080/#/c/21539/1/docs/topics/impala_jdbc_external_table.xml@41 PS1, Line 41: [EXTERNAL] key word 'EXTERNAL' is mandatory for creating JDBC table, remove square brackets http://gerrit.cloudera.org:8080/#/c/21539/1/docs/topics/impala_jdbc_external_table.xml@70 PS1, Line 70: port change to default Postgres port: 5432 http://gerrit.cloudera.org:8080/#/c/21539/1/docs/topics/impala_jdbc_external_table.xml@93 PS1, Line 93: port change to Impala default HS2 port: 21050 http://gerrit.cloudera.org:8080/#/c/21539/1/docs/topics/impala_jdbc_external_table.xml@98 PS1, Line 98: add a line break http://gerrit.cloudera.org:8080/#/c/21539/1/docs/topics/impala_jdbc_external_table.xml@107 PS1, Line 107: IMPALA or IMPALA http://gerrit.cloudera.org:8080/#/c/21539/1/docs/topics/impala_jdbc_external_table.xml@109 PS1, Line 109: hostname hostname/IP address http://gerrit.cloudera.org:8080/#/c/21539/1/docs/topics/impala_jdbc_external_table.xml@129 PS1, Line 129: Location change to URI http://gerrit.cloudera.org:8080/#/c/21539/1/docs/topics/impala_jdbc_external_table.xml@142 PS1, Line 142: an external an Impala external http://gerrit.cloudera.org:8080/#/c/21539/1/docs/topics/impala_jdbc_external_table.xml@154 PS1, Line 154: external Impala external http://gerrit.cloudera.org:8080/#/c/21539/1/docs/topics/impala_jdbc_external_table.xml@156 PS1, Line 156: Unsupported column data types include char, varchar, binary, complex : data types - struct, map, array, and nested type Following column data types are not supported: char, varchar, binary, complex data types - struct, map, array, and nested type http://gerrit.cloudera.org:8080/#/c/21539/1/docs/topics/impala_jdbc_external_table.xml@194 PS1, Line 194: replace it with one space http://gerrit.cloudera.org:8080/#/c/21539/1/docs/topics/impala_jdbc_external_table.xml@211 PS1, Line 211: JDBC external external JDBC -- To view, visit http://gerrit.cloudera.org:8080/21539 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5360389037ae9ee675ab406d87617d55d476bf8f Gerrit-Change-Number: 21539 Gerrit-PatchSet: 1 Gerrit-Owner: Jankiram Balakrishnan Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Thu, 20 Jun 2024 00:42:59 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13137: Add additional client fetch metrics columns to the queries page
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21482 ) Change subject: IMPALA-13137: Add additional client fetch metrics columns to the queries page .. Patch Set 8: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21482 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74a9393a7b38750de0c3f6230b6e5e048048c4b5 Gerrit-Change-Number: 21482 Gerrit-PatchSet: 8 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 18 Jun 2024 16:13:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13159: Fix query cancellation caused by statestore failover
Wenzhe Zhou has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/21520 ) Change subject: IMPALA-13159: Fix query cancellation caused by statestore failover .. IMPALA-13159: Fix query cancellation caused by statestore failover A momentary inconsistent cluster membership state after statestore failover results in query cancellation. We already have code to handle inconsistent cluster membership after statestore restarting by defining a post-recovery grace period. During the grace period, don't update the current cluster membership so that the inconsistent membership will not be used to cancel queries on coordinators and executors. This patch handles inconsistent cluster membership state after statestore failover in the same way. Testing: - Added a new test case to verify that inconsistent cluster membership after statestore failover will not result in query cancellation. - Fixed closing client issue for Catalogd HA test case test_catalogd_failover_with_sync_ddl when the test fails. - Passed core test. Change-Id: I720bec5199df46475b954558abb0637ca7e6298b --- M be/src/scheduling/cluster-membership-mgr.cc M be/src/statestore/statestore-subscriber.cc M be/src/statestore/statestore-subscriber.h M tests/custom_cluster/test_catalogd_ha.py M tests/custom_cluster/test_statestored_ha.py 5 files changed, 160 insertions(+), 31 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/21520/12 -- To view, visit http://gerrit.cloudera.org:8080/21520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I720bec5199df46475b954558abb0637ca7e6298b Gerrit-Change-Number: 21520 Gerrit-PatchSet: 12 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-13159: Fix query cancellation caused by statestore failover
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21520 ) Change subject: IMPALA-13159: Fix query cancellation caused by statestore failover .. Patch Set 10: (1 comment) http://gerrit.cloudera.org:8080/#/c/21520/9/be/src/statestore/statestore-subscriber.cc File be/src/statestore/statestore-subscriber.cc: http://gerrit.cloudera.org:8080/#/c/21520/9/be/src/statestore/statestore-subscriber.cc@1098 PS9, Line 1098: bool in_failover_grace_period = MilliSecondsSinceLastFailover() > At startup last_failover_time_ is zero, so for the first statestore_subscri At startup, last_failover_time_ is zero. If last_failover_time_ is not updated, MilliSecondsSinceLastFailover() return a very big number and it must be greater than FLAGS_statestore_subscriber_recovery_grace_period_ms so we don't need to check if last_failover_time_ is not zero. -- To view, visit http://gerrit.cloudera.org:8080/21520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I720bec5199df46475b954558abb0637ca7e6298b Gerrit-Change-Number: 21520 Gerrit-PatchSet: 10 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 17 Jun 2024 18:32:39 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-12093: impala-shell should preserver all cookies by default
Wenzhe Zhou has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19827 Change subject: WIP IMPALA-12093: impala-shell should preserver all cookies by default .. WIP IMPALA-12093: impala-shell should preserver all cookies by default Change-Id: Ic81f790288460b086ab218e6701e8115a996dfa7 --- M shell/ImpalaHttpClient.py M shell/cookie_util.py M shell/option_parser.py 3 files changed, 29 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/19827/2 -- To view, visit http://gerrit.cloudera.org:8080/19827 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ic81f790288460b086ab218e6701e8115a996dfa7 Gerrit-Change-Number: 19827 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-13159: Fix query cancellation caused by statestore failover
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21520 ) Change subject: IMPALA-13159: Fix query cancellation caused by statestore failover .. Patch Set 10: (2 comments) http://gerrit.cloudera.org:8080/#/c/21520/9/tests/custom_cluster/test_statestored_ha.py File tests/custom_cluster/test_statestored_ha.py: http://gerrit.cloudera.org:8080/#/c/21520/9/tests/custom_cluster/test_statestored_ha.py@704 PS9, Line 704: > nit: why sleep(1) when wait_until_ready=True above? add a buffer time for metrics to be updated on statestored. http://gerrit.cloudera.org:8080/#/c/21520/9/tests/custom_cluster/test_statestored_ha.py@705 PS9, Line 705: # Restart original active statestored. Verify that the statestored does not resume : # its active role. > nit: These assertion and others about active-status are not really straight added error messages for assertions. -- To view, visit http://gerrit.cloudera.org:8080/21520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I720bec5199df46475b954558abb0637ca7e6298b Gerrit-Change-Number: 21520 Gerrit-PatchSet: 10 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 17 Jun 2024 17:49:12 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13159: Fix query cancellation caused by statestore failover
Wenzhe Zhou has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/21520 ) Change subject: IMPALA-13159: Fix query cancellation caused by statestore failover .. IMPALA-13159: Fix query cancellation caused by statestore failover A momentary inconsistent cluster membership state after statestore failover results in query cancellation. We already have code to handle inconsistent cluster membership after statestore restarting by defining a post-recovery grace period. During the grace period, don't update the current cluster membership so that the inconsistent membership will not be used to cancel queries on coordinators and executors. This patch handles inconsistent cluster membership state after statestore failover in the same way. Testing: - Added a new test case to verify that inconsistent cluster membership after statestore failover will not result in query cancellation. - Fixed closing client issue for Catalogd HA test case test_catalogd_failover_with_sync_ddl when the test fails. - Passed core test. Change-Id: I720bec5199df46475b954558abb0637ca7e6298b --- M be/src/scheduling/cluster-membership-mgr.cc M be/src/statestore/statestore-subscriber.cc M be/src/statestore/statestore-subscriber.h M tests/custom_cluster/test_catalogd_ha.py M tests/custom_cluster/test_statestored_ha.py 5 files changed, 165 insertions(+), 31 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/21520/10 -- To view, visit http://gerrit.cloudera.org:8080/21520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I720bec5199df46475b954558abb0637ca7e6298b Gerrit-Change-Number: 21520 Gerrit-PatchSet: 10 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-13159: Fix query cancellation caused by statestore failover
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21520 ) Change subject: IMPALA-13159: Fix query cancellation caused by statestore failover .. Patch Set 9: (1 comment) http://gerrit.cloudera.org:8080/#/c/21520/8/tests/custom_cluster/test_statestored_ha.py File tests/custom_cluster/test_statestored_ha.py: http://gerrit.cloudera.org:8080/#/c/21520/8/tests/custom_cluster/test_statestored_ha.py@727 PS8, Line 727: P > flake8: W504 line break after binary operator Done -- To view, visit http://gerrit.cloudera.org:8080/21520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I720bec5199df46475b954558abb0637ca7e6298b Gerrit-Change-Number: 21520 Gerrit-PatchSet: 9 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 17 Jun 2024 16:52:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13159: Fix query cancellation caused by statestore failover
Wenzhe Zhou has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/21520 ) Change subject: IMPALA-13159: Fix query cancellation caused by statestore failover .. IMPALA-13159: Fix query cancellation caused by statestore failover A momentary inconsistent cluster membership state after statestore failover results in query cancellation. We already have code to handle inconsistent cluster membership after statestore restarting by defining a post-recovery grace period. During the grace period, don't update the current cluster membership so that the inconsistent membership will not be used to cancel queries on coordinators and executors. This patch handles inconsistent cluster membership state after statestore failover in the same way. Testing: - Added a new test case to verify that inconsistent cluster membership after statestore failover will not result in query cancellation. - Fixed closing client issue for Catalogd HA test case test_catalogd_failover_with_sync_ddl when the test fails. - Passed core test. Change-Id: I720bec5199df46475b954558abb0637ca7e6298b --- M be/src/scheduling/cluster-membership-mgr.cc M be/src/statestore/statestore-subscriber.cc M be/src/statestore/statestore-subscriber.h M tests/custom_cluster/test_catalogd_ha.py M tests/custom_cluster/test_statestored_ha.py 5 files changed, 157 insertions(+), 31 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/21520/9 -- To view, visit http://gerrit.cloudera.org:8080/21520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I720bec5199df46475b954558abb0637ca7e6298b Gerrit-Change-Number: 21520 Gerrit-PatchSet: 9 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-13159: Fix query cancellation caused by statestore failover
Wenzhe Zhou has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/21520 ) Change subject: IMPALA-13159: Fix query cancellation caused by statestore failover .. IMPALA-13159: Fix query cancellation caused by statestore failover A momentary inconsistent cluster membership state after statestore failover results in query cancellation. We already have code to handle inconsistent cluster membership after statestore restarting by defining a post-recovery grace period. During the grace period, don't update the current cluster membership so that the inconsistent membership will not be used to cancel queries on coordinators and executors. This patch handles inconsistent cluster membership state after statestore failover in the same way. Testing: - Added a new test case to verify that inconsistent cluster membership after statestore failover will not result in query cancellation. - Fixed closing client issue for Catalogd HA test case test_catalogd_failover_with_sync_ddl when the test fails. - Passed core test. Change-Id: I720bec5199df46475b954558abb0637ca7e6298b --- M be/src/scheduling/cluster-membership-mgr.cc M be/src/statestore/statestore-subscriber.cc M be/src/statestore/statestore-subscriber.h M tests/custom_cluster/test_catalogd_ha.py M tests/custom_cluster/test_statestored_ha.py 5 files changed, 157 insertions(+), 31 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/21520/8 -- To view, visit http://gerrit.cloudera.org:8080/21520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I720bec5199df46475b954558abb0637ca7e6298b Gerrit-Change-Number: 21520 Gerrit-PatchSet: 8 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-13159: Fix query cancellation caused by statestore failover
Wenzhe Zhou has removed a vote on this change. Change subject: IMPALA-13159: Fix query cancellation caused by statestore failover .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/21520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I720bec5199df46475b954558abb0637ca7e6298b Gerrit-Change-Number: 21520 Gerrit-PatchSet: 7 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-13159: Fix query cancellation caused by statestore failover
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21520 ) Change subject: IMPALA-13159: Fix query cancellation caused by statestore failover .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/21520/5/tests/custom_cluster/test_statestored_ha.py File tests/custom_cluster/test_statestored_ha.py: http://gerrit.cloudera.org:8080/#/c/21520/5/tests/custom_cluster/test_statestored_ha.py@727 PS5, Line 727: > flake8: E502 the backslash is redundant between brackets Done -- To view, visit http://gerrit.cloudera.org:8080/21520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I720bec5199df46475b954558abb0637ca7e6298b Gerrit-Change-Number: 21520 Gerrit-PatchSet: 6 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Sun, 16 Jun 2024 02:58:15 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13159: Fix query cancellation caused by statestore failover
Wenzhe Zhou has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/21520 ) Change subject: IMPALA-13159: Fix query cancellation caused by statestore failover .. IMPALA-13159: Fix query cancellation caused by statestore failover A momentary inconsistent cluster membership state after statestore failover results in query cancellation. We already have code to handle inconsistent cluster membership after statestore restarting by defining a post-recovery grace period. During the grace period, don't update the current cluster membership so that the inconsistent membership will not be used to cancel queries on coordinators and executors. This patch handles inconsistent cluster membership state after statestore failover in the same way. Testing: - Added a new test case to verify that inconsistent cluster membership after statestore failover will not result in query cancellation. - Fixed closing client issue for Catalogd HA test case test_catalogd_failover_with_sync_ddl when the test fails. - Passed core test. Change-Id: I720bec5199df46475b954558abb0637ca7e6298b --- M be/src/scheduling/cluster-membership-mgr.cc M be/src/statestore/statestore-subscriber.cc M be/src/statestore/statestore-subscriber.h M tests/custom_cluster/test_catalogd_ha.py M tests/custom_cluster/test_statestored_ha.py 5 files changed, 156 insertions(+), 31 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/21520/6 -- To view, visit http://gerrit.cloudera.org:8080/21520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I720bec5199df46475b954558abb0637ca7e6298b Gerrit-Change-Number: 21520 Gerrit-PatchSet: 6 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-13159: Fix query cancellation caused by statestore failover
Wenzhe Zhou has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/21520 ) Change subject: IMPALA-13159: Fix query cancellation caused by statestore failover .. IMPALA-13159: Fix query cancellation caused by statestore failover A momentary inconsistent cluster membership state after statestore failover results in query cancellation. We already have code to handle inconsistent cluster membership after statestore restarting by defining a post-recovery grace period. During the grace period, don't update the current cluster membership so that the inconsistent membership will not be used to cancel queries on coordinators and executors. This patch handles inconsistent cluster membership state after statestore failover in the same way. Testing: - Added a new test case to verify that inconsistent cluster membership after statestore failover will not result in query cancellation. - Fixed closing client issue for Catalogd HA test case test_catalogd_failover_with_sync_ddl when the test fails. - Passed core test. Change-Id: I720bec5199df46475b954558abb0637ca7e6298b --- M be/src/scheduling/cluster-membership-mgr.cc M be/src/statestore/statestore-subscriber.cc M be/src/statestore/statestore-subscriber.h M tests/custom_cluster/test_catalogd_ha.py M tests/custom_cluster/test_statestored_ha.py 5 files changed, 156 insertions(+), 31 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/21520/5 -- To view, visit http://gerrit.cloudera.org:8080/21520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I720bec5199df46475b954558abb0637ca7e6298b Gerrit-Change-Number: 21520 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-13159: Fix query cancellation caused by statestore failover
Wenzhe Zhou has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/21520 ) Change subject: IMPALA-13159: Fix query cancellation caused by statestore failover .. IMPALA-13159: Fix query cancellation caused by statestore failover A momentary inconsistent cluster membership state after statestore failover results in query cancellation. We already have code to handle inconsistent cluster membership after statestore restarting by defining a post-recovery grace period. During the grace period, don't update the current cluster membership so that the inconsistent membership will not be used to cancel queries on coordinators and executors. This patch handles inconsistent cluster membership state after statestore failover in the same way. Testing: - Added a new test case to verify that inconsistent cluster membership after statestore failover will not result in query cancellation. - Fixed closing client issue for Catalogd HA test case test_catalogd_failover_with_sync_ddl when the test fails. - Passed core test. Change-Id: I720bec5199df46475b954558abb0637ca7e6298b --- M be/src/scheduling/cluster-membership-mgr.cc M be/src/statestore/statestore-subscriber.cc M be/src/statestore/statestore-subscriber.h M tests/custom_cluster/test_catalogd_ha.py M tests/custom_cluster/test_statestored_ha.py 5 files changed, 156 insertions(+), 31 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/21520/4 -- To view, visit http://gerrit.cloudera.org:8080/21520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I720bec5199df46475b954558abb0637ca7e6298b Gerrit-Change-Number: 21520 Gerrit-PatchSet: 4 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-13159: Fix query cancellation caused by statestore failover
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21520 ) Change subject: IMPALA-13159: Fix query cancellation caused by statestore failover .. Patch Set 3: (7 comments) http://gerrit.cloudera.org:8080/#/c/21520/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21520/1//COMMIT_MSG@12 PS1, Line 12: y defining a post-recovery grace period. During : the grace period, don't update the current cluster membership so th > Thanks! Active statestored keeps sending cluster membership periodically to all subscribers, including coordinators/executors. Sending intervals are 100 ms by default. After grace period, the collected cluster membership should be consistent and it's safe to use the current cluster membership to detect if a node is unhealthy and cancel running queries which are assigned to unhealthy nodes. http://gerrit.cloudera.org:8080/#/c/21520/2/be/src/statestore/statestore-subscriber.h File be/src/statestore/statestore-subscriber.h: http://gerrit.cloudera.org:8080/#/c/21520/2/be/src/statestore/statestore-subscriber.h@345 PS2, Line 345: int64_t time_ms = MonotonicMillis() - last_failover_time_.Load(); > nit: Might be nice to DCHECK that the result is not negative. Added DCHECK http://gerrit.cloudera.org:8080/#/c/21520/2/be/src/statestore/statestore-subscriber.cc File be/src/statestore/statestore-subscriber.cc: http://gerrit.cloudera.org:8080/#/c/21520/2/be/src/statestore/statestore-subscriber.cc@1098 PS2, Line 1098: bool in_failover_grace_period = MilliSecondsSinceLastFailover() > Do we really need to check this? If it's 0, then MilliSecondsSinceLastFailo Right, removed has_failover_before http://gerrit.cloudera.org:8080/#/c/21520/1/tests/custom_cluster/test_statestored_ha.py File tests/custom_cluster/test_statestored_ha.py: http://gerrit.cloudera.org:8080/#/c/21520/1/tests/custom_cluster/test_statestored_ha.py@653 PS1, Line 653: """Test that a momentary inconsistent cluster membership state after statestore : service fail-over will not result in query cancellation. Also make sure that query : get cancelled if a backend actually went down aft > Is there an existing opposite test that shows query cancelled after grace p Second part of this test case shows query cancelled after grace period due to cluster membership change. Updated comments. http://gerrit.cloudera.org:8080/#/c/21520/1/tests/custom_cluster/test_statestored_ha.py@688 PS1, Line 688: # Now kill a backend, and make sure the query fails. > In that case, is it better to cancel through query handle instead? Cancellation reason is cluster membership change after grace period. http://gerrit.cloudera.org:8080/#/c/21520/1/tests/custom_cluster/test_statestored_ha.py@688 PS1, Line 688: # Now kill a backend, and make sure the query fails. > What will happen if, after client.execute_async(slow_query), you start 4th The query will not be affected if 4th impalad has no scheduled task. http://gerrit.cloudera.org:8080/#/c/21520/2/tests/custom_cluster/test_statestored_ha.py File tests/custom_cluster/test_statestored_ha.py: http://gerrit.cloudera.org:8080/#/c/21520/2/tests/custom_cluster/test_statestored_ha.py@692 PS2, Line 692: assert False, "Query expected to fail" > Can we verify that it had to wait longer than the CANCELLATION_GRACE_PERIOD Yes, need to add another test case. -- To view, visit http://gerrit.cloudera.org:8080/21520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I720bec5199df46475b954558abb0637ca7e6298b Gerrit-Change-Number: 21520 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 14 Jun 2024 23:51:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13159: Fix query cancellation caused by statestore failover
Wenzhe Zhou has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/21520 ) Change subject: IMPALA-13159: Fix query cancellation caused by statestore failover .. IMPALA-13159: Fix query cancellation caused by statestore failover A momentary inconsistent cluster membership state after statestore failover results in query cancellation. We already have code to handle inconsistent cluster membership after statestore restarting by defining a post-recovery grace period. During the grace period, don't update the current cluster membership so that the inconsistent membership will not be used to cancel queries on coordinators and executors. This patch handles inconsistent cluster membership state after statestore failover in the same way. Testing: - Added a new test case to verify that inconsistent cluster membership after statestore failover will not result in query cancellation. - Fixed closing client issue for Catalogd HA test case test_catalogd_failover_with_sync_ddl when the test fails. - Passed core test. Change-Id: I720bec5199df46475b954558abb0637ca7e6298b --- M be/src/scheduling/cluster-membership-mgr.cc M be/src/statestore/statestore-subscriber.cc M be/src/statestore/statestore-subscriber.h M tests/custom_cluster/test_catalogd_ha.py M tests/custom_cluster/test_statestored_ha.py 5 files changed, 113 insertions(+), 31 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/21520/3 -- To view, visit http://gerrit.cloudera.org:8080/21520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I720bec5199df46475b954558abb0637ca7e6298b Gerrit-Change-Number: 21520 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-13159: Fix query cancellation caused by statestore failover
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21520 ) Change subject: IMPALA-13159: Fix query cancellation caused by statestore failover .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/21520/1/tests/custom_cluster/test_statestored_ha.py File tests/custom_cluster/test_statestored_ha.py: http://gerrit.cloudera.org:8080/#/c/21520/1/tests/custom_cluster/test_statestored_ha.py@688 PS1, Line 688: # Now kill a backend, and make sure the query fails. > If the query is cancelled, its state will be changed to QueryState.CANCELLE Finishing this query takes about 120 seconds on my local machine. It seems too long for one test case. Without waiting to finish, it takes about 19 seconds. -- To view, visit http://gerrit.cloudera.org:8080/21520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I720bec5199df46475b954558abb0637ca7e6298b Gerrit-Change-Number: 21520 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 14 Jun 2024 20:25:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13159: Fix query cancellation caused by statestore failover
Wenzhe Zhou has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/21520 ) Change subject: IMPALA-13159: Fix query cancellation caused by statestore failover .. IMPALA-13159: Fix query cancellation caused by statestore failover A momentary inconsistent cluster membership state after statestore failover results in query cancellation. We already have code to handle inconsistent cluster membership after statestore restarting by defining a post-recovery grace period. During the grace period, don't update the current cluster membership so that the inconsistent membership will not be used to cancel queries on coordinators and executors. This patch handles inconsistent cluster membership state after statestore failover in the same way. Testing: - Added a new test case to verify that inconsistent cluster membership after statestore failover will not result in query cancellation. - Fixed closing client issue for Catalogd HA test case test_catalogd_failover_with_sync_ddl when the test fails. - Passed core test. Change-Id: I720bec5199df46475b954558abb0637ca7e6298b --- M be/src/scheduling/cluster-membership-mgr.cc M be/src/statestore/statestore-subscriber.cc M be/src/statestore/statestore-subscriber.h M tests/custom_cluster/test_catalogd_ha.py M tests/custom_cluster/test_statestored_ha.py 5 files changed, 112 insertions(+), 31 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/21520/2 -- To view, visit http://gerrit.cloudera.org:8080/21520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I720bec5199df46475b954558abb0637ca7e6298b Gerrit-Change-Number: 21520 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-13159: Fix query cancellation caused by statestore failover
Wenzhe Zhou has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21520 Change subject: IMPALA-13159: Fix query cancellation caused by statestore failover .. IMPALA-13159: Fix query cancellation caused by statestore failover A momentary inconsistent cluster membership state after statestore failover results in query cancellation. We already have code to handle inconsistent cluster membership after statestore restarting. This patch adds code to handle inconsistent cluster membership state after statestore failover in the same way. Testing: - Added a new test case to verify that inconsistent cluster membership after statestore failover will not result in query cancellation. - Fixed closing client issue for Catalogd HA test case test_catalogd_failover_with_sync_ddl when the test fails. - Passed core test. Change-Id: I720bec5199df46475b954558abb0637ca7e6298b --- M be/src/scheduling/cluster-membership-mgr.cc M be/src/statestore/statestore-subscriber.cc M be/src/statestore/statestore-subscriber.h M tests/custom_cluster/test_catalogd_ha.py M tests/custom_cluster/test_statestored_ha.py 5 files changed, 111 insertions(+), 30 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/21520/1 -- To view, visit http://gerrit.cloudera.org:8080/21520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I720bec5199df46475b954558abb0637ca7e6298b Gerrit-Change-Number: 21520 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-13131: Azure OpenAI API expects 'api-key' instead of 'Authorization' in the request header
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21493 ) Change subject: IMPALA-13131: Azure OpenAI API expects 'api-key' instead of 'Authorization' in the request header .. Patch Set 7: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21493 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If9cc07940ce355d511bcf0ee615ff31042d13eb5 Gerrit-Change-Number: 21493 Gerrit-PatchSet: 7 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Wed, 12 Jun 2024 17:41:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13131: Azure OpenAI API expects 'api-key' instead of 'Authorization' in the request header
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21493 ) Change subject: IMPALA-13131: Azure OpenAI API expects 'api-key' instead of 'Authorization' in the request header .. Patch Set 4: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21493 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If9cc07940ce355d511bcf0ee615ff31042d13eb5 Gerrit-Change-Number: 21493 Gerrit-PatchSet: 4 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Mon, 10 Jun 2024 17:08:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13137: Add additional client fetch metrics columns to the queries page
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21482 ) Change subject: IMPALA-13137: Add additional client fetch metrics columns to the queries page .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21482 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74a9393a7b38750de0c3f6230b6e5e048048c4b5 Gerrit-Change-Number: 21482 Gerrit-PatchSet: 2 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 10 Jun 2024 17:02:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13146: Download NodeJS from native toolchain
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21494 ) Change subject: IMPALA-13146: Download NodeJS from native toolchain .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21494 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1def801469cb68633e89b4a0f3c07a771febe599 Gerrit-Change-Number: 21494 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Sun, 09 Jun 2024 16:21:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13131: Azure OpenAI API expects 'api-key' instead of 'Authorization' in the request header
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/21493 )
Change subject: IMPALA-13131: Azure OpenAI API expects 'api-key' instead of
'Authorization' in the request header
..
Patch Set 3: Code-Review+1
(2 comments)
http://gerrit.cloudera.org:8080/#/c/21493/2/be/src/exprs/ai-functions-ir.cc
File be/src/exprs/ai-functions-ir.cc:
http://gerrit.cloudera.org:8080/#/c/21493/2/be/src/exprs/ai-functions-ir.cc@27
PS2, Line 27: ai_endpoint
> Renaming will be tricky since this is part of 4.4.0. We can clarify this in
ack
http://gerrit.cloudera.org:8080/#/c/21493/2/be/src/exprs/ai-functions-ir.cc@154
PS2, Line 154: switch(platform) {
: case AI_PLATFORM::OPEN_AI:
: return AiGenerateTextInternal(
: ctx, endpoint_sv, prompt, model, api_key_jceks_secret,
params, false);
: case AI_PLATFORM::AZURE_OPEN_AI:
: return AiGenerateTextInternal(
: ctx, endpoint_sv, prompt, model, api_key_jceks_secret,
params, false);
: default:
> This won't work because the template parameter 'platform' must be known at
ack
--
To view, visit http://gerrit.cloudera.org:8080/21493
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If9cc07940ce355d511bcf0ee615ff31042d13eb5
Gerrit-Change-Number: 21493
Gerrit-PatchSet: 3
Gerrit-Owner: Abhishek Rawat
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: Yida Wu
Gerrit-Comment-Date: Sat, 08 Jun 2024 23:56:59 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13143: Fix flaky test catalogd failover with sync ddl
Wenzhe Zhou has removed a vote on this change. Change subject: IMPALA-13143: Fix flaky test_catalogd_failover_with_sync_ddl .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/21491 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I15bb6aae23a2f544067f993533e322969372ebd5 Gerrit-Change-Number: 21491 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-13143: Fix flaky test catalogd failover with sync ddl
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21491 ) Change subject: IMPALA-13143: Fix flaky test_catalogd_failover_with_sync_ddl .. Patch Set 5: hit a flaky test -- To view, visit http://gerrit.cloudera.org:8080/21491 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I15bb6aae23a2f544067f993533e322969372ebd5 Gerrit-Change-Number: 21491 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Sat, 08 Jun 2024 00:34:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13131: Azure OpenAI API expects 'api-key' instead of 'Authorization' in the request header
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/21493 )
Change subject: IMPALA-13131: Azure OpenAI API expects 'api-key' instead of
'Authorization' in the request header
..
Patch Set 2:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/21493/2/be/src/exprs/ai-functions-ir.cc
File be/src/exprs/ai-functions-ir.cc:
http://gerrit.cloudera.org:8080/#/c/21493/2/be/src/exprs/ai-functions-ir.cc@27
PS2, Line 27: ai_endpoint
rename as default_ai_endpoint
http://gerrit.cloudera.org:8080/#/c/21493/2/be/src/exprs/ai-functions-ir.cc@154
PS2, Line 154: switch(platform) {
: case AI_PLATFORM::OPEN_AI:
: return AiGenerateTextInternal(
: ctx, endpoint_sv, prompt, model, api_key_jceks_secret,
params, false);
: case AI_PLATFORM::AZURE_OPEN_AI:
: return AiGenerateTextInternal(
: ctx, endpoint_sv, prompt, model, api_key_jceks_secret,
params, false);
: default:
simplify as:
if (platform != AI_PLATFORM::UNSUPPORTED) {
return AiGenerateTextInternal(
ctx, endpoint_sv, prompt, model, api_key_jceks_secret, params, false);
} else {
http://gerrit.cloudera.org:8080/#/c/21493/2/be/src/exprs/ai-functions-ir.cc@170
PS2, Line 170: switch(platform) {
same comment as last one
http://gerrit.cloudera.org:8080/#/c/21493/2/be/src/exprs/ai-functions-ir.cc@178
PS2, Line 178: default:
add DCHECK since the default endpoint must be supported.
--
To view, visit http://gerrit.cloudera.org:8080/21493
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If9cc07940ce355d511bcf0ee615ff31042d13eb5
Gerrit-Change-Number: 21493
Gerrit-PatchSet: 2
Gerrit-Owner: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: Yida Wu
Gerrit-Comment-Date: Fri, 07 Jun 2024 21:03:33 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13143: Fix flaky test catalogd failover with sync ddl
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21491 ) Change subject: IMPALA-13143: Fix flaky test_catalogd_failover_with_sync_ddl .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/21491/5/tests/custom_cluster/test_catalogd_ha.py File tests/custom_cluster/test_catalogd_ha.py: http://gerrit.cloudera.org:8080/#/c/21491/5/tests/custom_cluster/test_catalogd_ha.py@487 PS5, Line 487: QueryState.EXCEPTION, SYNC_DDL_DELAY_S * 2 + 10 > nit: Still not sure if query suppose to retry and talk with the newly activ DDL is not retried. Waiting longer for safe, it will not affect passed case. -- To view, visit http://gerrit.cloudera.org:8080/21491 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I15bb6aae23a2f544067f993533e322969372ebd5 Gerrit-Change-Number: 21491 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 07 Jun 2024 19:06:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13137: Add additional client fetch metrics columns to the queries page
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21482 ) Change subject: IMPALA-13137: Add additional client fetch metrics columns to the queries page .. Patch Set 1: (5 comments) http://gerrit.cloudera.org:8080/#/c/21482/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21482/1//COMMIT_MSG@9 PS1, Line 9: . nit: , http://gerrit.cloudera.org:8080/#/c/21482/1//COMMIT_MSG@10 PS1, Line 10: The nit: the http://gerrit.cloudera.org:8080/#/c/21482/1/be/src/service/impala-http-handler.cc File be/src/service/impala-http-handler.cc: http://gerrit.cloudera.org:8080/#/c/21482/1/be/src/service/impala-http-handler.cc@460 PS1, Line 460: , nit: move ',' to previous line http://gerrit.cloudera.org:8080/#/c/21482/1/be/src/service/impala-http-handler.cc@551 PS1, Line 551: first_row_fetched_ns = 0; nit: it's already inited as 0, not necessary to set http://gerrit.cloudera.org:8080/#/c/21482/1/be/src/service/impala-http-handler.cc@567 PS1, Line 567: , nit: move '' to previous line -- To view, visit http://gerrit.cloudera.org:8080/21482 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74a9393a7b38750de0c3f6230b6e5e048048c4b5 Gerrit-Change-Number: 21482 Gerrit-PatchSet: 1 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 07 Jun 2024 18:58:23 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13143: Fix flaky test catalogd failover with sync ddl
Wenzhe Zhou has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/21491 ) Change subject: IMPALA-13143: Fix flaky test_catalogd_failover_with_sync_ddl .. IMPALA-13143: Fix flaky test_catalogd_failover_with_sync_ddl The test_catalogd_failover_with_sync_ddl test which was added to custom_cluster/test_catalogd_ha.py in IMPALA-13134 failed on s3. The test relies on specific timing with a sleep injected via a debug action so that the DDL query is still running when catalogd failover is triggered. The failures were caused by slowly restarting for catalogd on s3 so that the query finished before catalogd failover was triggered. This patch fixed the issue by increasing the sleep time for s3 builds and other slow builds. Testing: - Ran the test 100 times in a loop on s3. Change-Id: I15bb6aae23a2f544067f993533e322969372ebd5 --- M tests/custom_cluster/test_catalogd_ha.py 1 file changed, 18 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/21491/5 -- To view, visit http://gerrit.cloudera.org:8080/21491 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I15bb6aae23a2f544067f993533e322969372ebd5 Gerrit-Change-Number: 21491 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-13143: Fix flaky test catalogd failover with sync ddl
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21491 ) Change subject: IMPALA-13143: Fix flaky test_catalogd_failover_with_sync_ddl .. Patch Set 4: Riza, I made small change. Could you review it again? -- To view, visit http://gerrit.cloudera.org:8080/21491 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I15bb6aae23a2f544067f993533e322969372ebd5 Gerrit-Change-Number: 21491 Gerrit-PatchSet: 4 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 07 Jun 2024 17:54:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13143: Fix flaky test catalogd failover with sync ddl
Wenzhe Zhou has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/21491 ) Change subject: IMPALA-13143: Fix flaky test_catalogd_failover_with_sync_ddl .. IMPALA-13143: Fix flaky test_catalogd_failover_with_sync_ddl The test_catalogd_failover_with_sync_ddl test which was added to custom_cluster/test_catalogd_ha.py in IMPALA-13134 failed on s3. The test relies on specific timing with a sleep injected via a debug action so that the DDL query is still running when catalogd failover is triggered. The failures were caused by slowly restarting for catalogd on s3 so that the query finished before catalogd failover was triggered. This patch fixed the issue by increasing the sleep time for s3 builds and other slow builds. Testing: - Ran the test 100 times in a loop on s3. Change-Id: I15bb6aae23a2f544067f993533e322969372ebd5 --- M tests/custom_cluster/test_catalogd_ha.py 1 file changed, 17 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/21491/4 -- To view, visit http://gerrit.cloudera.org:8080/21491 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I15bb6aae23a2f544067f993533e322969372ebd5 Gerrit-Change-Number: 21491 Gerrit-PatchSet: 4 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-13143: Fix flaky test catalogd failover with sync ddl
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21491 ) Change subject: IMPALA-13143: Fix flaky test_catalogd_failover_with_sync_ddl .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/21491/3/tests/custom_cluster/test_catalogd_ha.py File tests/custom_cluster/test_catalogd_ha.py: http://gerrit.cloudera.org:8080/#/c/21491/3/tests/custom_cluster/test_catalogd_ha.py@486 PS3, Line 486: 30 need to adjust the timeout after increase SYNC_DDL_DELAY_S since the query take longer time to fail. -- To view, visit http://gerrit.cloudera.org:8080/21491 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I15bb6aae23a2f544067f993533e322969372ebd5 Gerrit-Change-Number: 21491 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 07 Jun 2024 17:49:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13143: Fix flaky test catalogd failover with sync ddl
Wenzhe Zhou has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/21491 ) Change subject: IMPALA-13143: Fix flaky test_catalogd_failover_with_sync_ddl .. IMPALA-13143: Fix flaky test_catalogd_failover_with_sync_ddl The test_catalogd_failover_with_sync_ddl test which was added to custom_cluster/test_catalogd_ha.py in IMPALA-13134 failed on s3. The test relies on specific timing with a sleep injected via a debug action so that the DDL query is still running when catalogd failover is triggered. The failures were caused by slowly restarting for catalogd on s3 so that the query finished before catalogd failover was triggered. This patch fixed the issue by increasing the sleep time for s3 builds and other slow builds. Testing: - Ran the test 100 times in a loop on s3. Change-Id: I15bb6aae23a2f544067f993533e322969372ebd5 --- M tests/custom_cluster/test_catalogd_ha.py 1 file changed, 16 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/21491/3 -- To view, visit http://gerrit.cloudera.org:8080/21491 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I15bb6aae23a2f544067f993533e322969372ebd5 Gerrit-Change-Number: 21491 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-13134 (part 2): Fix flaky test catalogd failover with sync ddl
Wenzhe Zhou has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/21491 ) Change subject: IMPALA-13134 (part 2): Fix flaky test_catalogd_failover_with_sync_ddl .. IMPALA-13134 (part 2): Fix flaky test_catalogd_failover_with_sync_ddl The test_catalogd_failover_with_sync_ddl test which was added to custom_cluster/test_catalogd_ha.py in previous patch failed on s3. The test relies on specific timing with a sleep injected via a debug action so that the DDL query is still running when catalogd failover is triggered. The failures were caused by slowly restarting for catalogd on s3 so that the query finished before catalogd failover was triggered. This patch fixed the issue by increasing the sleep time for s3 builds and other slow builds. Testing: - Ran the test 100 times in a loop on s3. Change-Id: I15bb6aae23a2f544067f993533e322969372ebd5 --- M tests/custom_cluster/test_catalogd_ha.py 1 file changed, 16 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/21491/2 -- To view, visit http://gerrit.cloudera.org:8080/21491 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I15bb6aae23a2f544067f993533e322969372ebd5 Gerrit-Change-Number: 21491 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-13134 (part 2): Fix flaky test catalogd failover with sync ddl
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21491 ) Change subject: IMPALA-13134 (part 2): Fix flaky test_catalogd_failover_with_sync_ddl .. Patch Set 2: (4 comments) http://gerrit.cloudera.org:8080/#/c/21491/1/tests/custom_cluster/test_catalogd_ha.py File tests/custom_cluster/test_catalogd_ha.py: http://gerrit.cloudera.org:8080/#/c/21491/1/tests/custom_cluster/test_catalogd_ha.py@35 PS1, Line 35: _BUI > Is catalogd restart tend to slow? What factor in long catalogd restart time Doubled the base delay to 10s and 20s for slow build flavor. This happened for s3 nightly builds. Saw Joe fixed another flaky test on s3 due to long catalogd restart time during test. http://gerrit.cloudera.org:8080/#/c/21491/1/tests/custom_cluster/test_catalogd_ha.py@448 PS1, Line 448: e > flake8: E131 continuation line unaligned for hanging indent fixed http://gerrit.cloudera.org:8080/#/c/21491/1/tests/custom_cluster/test_catalogd_ha.py@470 PS1, Line 470: # Restart standby catalogd with force_catalogd_active as true. : start_s = time.time() > Can we log/assert the time until second catalogd active? Added log and assert http://gerrit.cloudera.org:8080/#/c/21491/1/tests/custom_cluster/test_catalogd_ha.py@475 PS1, Line 475: > Should this be adjusted too? No, since the restarted catalogd is already ready. -- To view, visit http://gerrit.cloudera.org:8080/21491 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I15bb6aae23a2f544067f993533e322969372ebd5 Gerrit-Change-Number: 21491 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 07 Jun 2024 17:30:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13134 (part 2): Fix flaky test catalogd failover with sync ddl
Wenzhe Zhou has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21491 Change subject: IMPALA-13134 (part 2): Fix flaky test_catalogd_failover_with_sync_ddl .. IMPALA-13134 (part 2): Fix flaky test_catalogd_failover_with_sync_ddl The test_catalogd_failover_with_sync_ddl test which was added to custom_cluster/test_catalogd_ha.py in previous patch failed on s3. The test relies on specific timing with a sleep injected via a debug action so that the DDL query is still running when catalogd failover is triggered. The failures were caused by slowly restarting for catalogd on s3 so that the query finished before catalogd failover was triggered. This patch fixed the issue by increasing the sleep time for s3 builds and other slow builds. Testing: - Ran the test 100 times in a loop on s3. Change-Id: I15bb6aae23a2f544067f993533e322969372ebd5 --- M tests/custom_cluster/test_catalogd_ha.py 1 file changed, 9 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/21491/1 -- To view, visit http://gerrit.cloudera.org:8080/21491 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I15bb6aae23a2f544067f993533e322969372ebd5 Gerrit-Change-Number: 21491 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-13130: Prioritize EndDataStream messages
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21476 ) Change subject: IMPALA-13130: Prioritize EndDataStream messages .. Patch Set 7: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21476 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2829e1ab5bcde36107e10bff5fe629c5ee60f3e8 Gerrit-Change-Number: 21476 Gerrit-PatchSet: 7 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 06 Jun 2024 20:03:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13119: Fix cost initialization at CostingSegment.java
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21468 ) Change subject: IMPALA-13119: Fix cost_ initialization at CostingSegment.java .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21468 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5b3c99c87a1d0a08edc8d276cf33d709bd39fe14 Gerrit-Change-Number: 21468 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 05 Jun 2024 21:31:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13134: DDL hang with SYNC DDL enabled when Catalogd is changed to standby status
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/21480 )
Change subject: IMPALA-13134: DDL hang with SYNC_DDL enabled when Catalogd is
changed to standby status
..
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21480/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/21480/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@3519
PS2, Line 3519: if (!serviceId.equals(JniCatalog.getServiceId())) {
> Ack.
Yes, it will wait one more attempt.
--
To view, visit http://gerrit.cloudera.org:8080/21480
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2dcd628cff3c10d2e7566ba2d9de0b5886a18fc1
Gerrit-Change-Number: 21480
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Wed, 05 Jun 2024 19:52:15 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13130: Prioritize EndDataStream messages
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21476 ) Change subject: IMPALA-13130: Prioritize EndDataStream messages .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/21476/4/be/src/runtime/krpc-data-stream-sender.cc File be/src/runtime/krpc-data-stream-sender.cc: http://gerrit.cloudera.org:8080/#/c/21476/4/be/src/runtime/krpc-data-stream-sender.cc@65 PS4, Line 65: Set to -1 > Yeah, I was omitting that. Maybe I should enforce a minimum timeout? I thin We should not allow to set it as 0. Maybe just update description. -- To view, visit http://gerrit.cloudera.org:8080/21476 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2829e1ab5bcde36107e10bff5fe629c5ee60f3e8 Gerrit-Change-Number: 21476 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 05 Jun 2024 19:49:57 + Gerrit-HasComments: Yes
