[Impala-ASF-CR] IMPALA-12950:Improve error message in case of out-of-range numeric conversions
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/21331 ) Change subject: IMPALA-12950:Improve error message in case of out-of-range numeric conversions .. Patch Set 2: Code-Review+1 lgtm (besides Peter's comments) -- To view, visit http://gerrit.cloudera.org:8080/21331 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieeed52e25f155818c35c11a8a6821708476ffb32 Gerrit-Change-Number: 21331 Gerrit-PatchSet: 2 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Tue, 23 Apr 2024 06:20:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13009: Fix catalogd not sending deletion updates for some dropped partitions
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/21326 ) Change subject: IMPALA-13009: Fix catalogd not sending deletion updates for some dropped partitions .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/21326/5/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/21326/5/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1125 PS5, Line 1125: collected from a new version How could the catalog collect the new version of the partition before collecting the deletion of the partition? If the partition was readded, shouldn't that operation also remove it from dropped_partitions? -- To view, visit http://gerrit.cloudera.org:8080/21326 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I12a68158dca18ee48c9564ea16b7484c9f5b5d21 Gerrit-Change-Number: 21326 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 23 Apr 2024 06:14:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13024: Ignore slots if using default pool and empty group
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/21340 ) Change subject: IMPALA-13024: Ignore slots if using default pool and empty group .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/21340/4/tests/custom_cluster/test_executor_groups.py File tests/custom_cluster/test_executor_groups.py: http://gerrit.cloudera.org:8080/#/c/21340/4/tests/custom_cluster/test_executor_groups.py@399 PS4, Line 399: QUERY = "select sleep(1000)" Isn't this a trivial query? By default enable_trivial_query_for_admission = true When I reproduced the issue, I used select * from functional.alltypestiny, which has very low cardinality but still has to scan a table. -- To view, visit http://gerrit.cloudera.org:8080/21340 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0b08dea7ba0c78ac6b98c7a0b148df8fb036c4d0 Gerrit-Change-Number: 21340 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 23 Apr 2024 06:05:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13015: Dataload fails due to concurrency issue with test.jceks
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/21346 ) Change subject: IMPALA-13015: Dataload fails due to concurrency issue with test.jceks .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7fbeffc19f2b78c19fee9acf7f96466c8f4f9bcd Gerrit-Change-Number: 21346 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 23 Apr 2024 05:51:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13015: Dataload fails due to concurrency issue with test.jceks
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21346 ) Change subject: IMPALA-13015: Dataload fails due to concurrency issue with test.jceks .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10574/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/21346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7fbeffc19f2b78c19fee9acf7f96466c8f4f9bcd Gerrit-Change-Number: 21346 Gerrit-PatchSet: 4 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 23 Apr 2024 05:53:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13015: Dataload fails due to concurrency issue with test.jceks
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21346 ) Change subject: IMPALA-13015: Dataload fails due to concurrency issue with test.jceks .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7fbeffc19f2b78c19fee9acf7f96466c8f4f9bcd Gerrit-Change-Number: 21346 Gerrit-PatchSet: 4 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 23 Apr 2024 05:53:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12777: Fix tpcds/tpcds-q66.test
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21344 ) Change subject: IMPALA-12777: Fix tpcds/tpcds-q66.test .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21344 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5b886f5dc1da213d25f33bd7b01dacca53eaef1b Gerrit-Change-Number: 21344 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 23 Apr 2024 04:50:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12777: Fix tpcds/tpcds-q66.test
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21344 ) Change subject: IMPALA-12777: Fix tpcds/tpcds-q66.test .. IMPALA-12777: Fix tpcds/tpcds-q66.test PlannerTest/tpcds/tpcds-q66.test was mistakenly a copy of PlannerTest/tpcds/tpcds-q61.test with different predicate values. This patch replace the wrong test file with correct TPC-DS Q66 query. Testing: - Pass FE test TpcdsPlannerTest#testQ66. Change-Id: I5b886f5dc1da213d25f33bd7b01dacca53eaef1b Reviewed-on: http://gerrit.cloudera.org:8080/21344 Reviewed-by: Wenzhe Zhou Tested-by: Impala Public Jenkins --- M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q66.test 1 file changed, 962 insertions(+), 1,020 deletions(-) Approvals: Wenzhe Zhou: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/21344 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I5b886f5dc1da213d25f33bd7b01dacca53eaef1b Gerrit-Change-Number: 21344 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12872: Use Calcite for optimization - part 1: simple queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21109 ) Change subject: IMPALA-12872: Use Calcite for optimization - part 1: simple queries .. Patch Set 25: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10572/ -- To view, visit http://gerrit.cloudera.org:8080/21109 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I453fd75b7b705f4d7de1ed73c3e24cafad0b8c98 Gerrit-Change-Number: 21109 Gerrit-PatchSet: 25 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Tue, 23 Apr 2024 04:20:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13015: Dataload fails due to concurrency issue with test.jceks
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21346 ) Change subject: IMPALA-13015: Dataload fails due to concurrency issue with test.jceks .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7fbeffc19f2b78c19fee9acf7f96466c8f4f9bcd Gerrit-Change-Number: 21346 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 23 Apr 2024 03:10:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13015: Dataload fails due to concurrency issue with test.jceks
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21346 ) Change subject: IMPALA-13015: Dataload fails due to concurrency issue with test.jceks .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15994/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7fbeffc19f2b78c19fee9acf7f96466c8f4f9bcd Gerrit-Change-Number: 21346 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 23 Apr 2024 02:32:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13015: Dataload fails due to concurrency issue with test.jceks
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/21346 ) Change subject: IMPALA-13015: Dataload fails due to concurrency issue with test.jceks .. Patch Set 3: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7fbeffc19f2b78c19fee9acf7f96466c8f4f9bcd Gerrit-Change-Number: 21346 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 23 Apr 2024 02:28:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13015: Dataload fails due to concurrency issue with test.jceks
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/21346 ) Change subject: IMPALA-13015: Dataload fails due to concurrency issue with test.jceks .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/21346/2/testdata/bin/create-load-data.sh File testdata/bin/create-load-data.sh: http://gerrit.cloudera.org:8080/#/c/21346/2/testdata/bin/create-load-data.sh@555 PS2, Line 555: run-step "Creating hadoop credential" create-hadoop-credential.log \ > How about adding a comment here to explain why we place the command here an Done -- To view, visit http://gerrit.cloudera.org:8080/21346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7fbeffc19f2b78c19fee9acf7f96466c8f4f9bcd Gerrit-Change-Number: 21346 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 23 Apr 2024 02:08:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13015: Dataload fails due to concurrency issue with test.jceks
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21346 ) Change subject: IMPALA-13015: Dataload fails due to concurrency issue with test.jceks .. Patch Set 3: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7fbeffc19f2b78c19fee9acf7f96466c8f4f9bcd Gerrit-Change-Number: 21346 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 23 Apr 2024 02:08:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13015: Dataload fails due to concurrency issue with test.jceks
Abhishek Rawat has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/21346 ) Change subject: IMPALA-13015: Dataload fails due to concurrency issue with test.jceks .. IMPALA-13015: Dataload fails due to concurrency issue with test.jceks Move 'hadoop credential' command used for creating test.jceks to testdata/bin/create-load-data.sh. Earlier it was in bin/load-data.py which is called in parallel and was causing failures due to race conditions. Testing: - Ran JniFrontendTest#testGetSecretFromKeyStore after data loading and test ran clean. Change-Id: I7fbeffc19f2b78c19fee9acf7f96466c8f4f9bcd --- M bin/load-data.py M testdata/bin/create-load-data.sh 2 files changed, 10 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/46/21346/3 -- To view, visit http://gerrit.cloudera.org:8080/21346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7fbeffc19f2b78c19fee9acf7f96466c8f4f9bcd Gerrit-Change-Number: 21346 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-13015: Dataload fails due to concurrency issue with test.jceks
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/21346 ) Change subject: IMPALA-13015: Dataload fails due to concurrency issue with test.jceks .. Patch Set 2: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/21346/2/testdata/bin/create-load-data.sh File testdata/bin/create-load-data.sh: http://gerrit.cloudera.org:8080/#/c/21346/2/testdata/bin/create-load-data.sh@555 PS2, Line 555: run-step "Creating hadoop credential" create-hadoop-credential.log \ How about adding a comment here to explain why we place the command here and the potential concurrency issue with the Jira number? -- To view, visit http://gerrit.cloudera.org:8080/21346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7fbeffc19f2b78c19fee9acf7f96466c8f4f9bcd Gerrit-Change-Number: 21346 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 23 Apr 2024 00:28:23 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13015: Dataload fails due to concurrency issue with test.jceks
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21346 ) Change subject: IMPALA-13015: Dataload fails due to concurrency issue with test.jceks .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15993/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7fbeffc19f2b78c19fee9acf7f96466c8f4f9bcd Gerrit-Change-Number: 21346 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 23 Apr 2024 00:16:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13015: Dataload fails due to concurrency issue with test.jceks
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21346 ) Change subject: IMPALA-13015: Dataload fails due to concurrency issue with test.jceks .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7fbeffc19f2b78c19fee9acf7f96466c8f4f9bcd Gerrit-Change-Number: 21346 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 23 Apr 2024 00:14:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13015: Dataload fails due to concurrency issue with test.jceks
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21346 ) Change subject: IMPALA-13015: Dataload fails due to concurrency issue with test.jceks .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15992/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7fbeffc19f2b78c19fee9acf7f96466c8f4f9bcd Gerrit-Change-Number: 21346 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 23 Apr 2024 00:03:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12997: Use graceful shutdown for query log tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21345 ) Change subject: IMPALA-12997: Use graceful shutdown for query log tests .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15991/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21345 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia123c53a952a77ff4a9c02736b5717ccaa3566dc Gerrit-Change-Number: 21345 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 23 Apr 2024 00:03:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12997: Use graceful shutdown for query log tests
Riza Suminto has posted comments on this change. (
http://gerrit.cloudera.org:8080/21345 )
Change subject: IMPALA-12997: Use graceful shutdown for query log tests
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21345/1/tests/custom_cluster/test_query_log.py
File tests/custom_cluster/test_query_log.py:
http://gerrit.cloudera.org:8080/#/c/21345/1/tests/custom_cluster/test_query_log.py@720
PS1, Line 720: client2 = self.create_client_for_nth_impalad(1,
vector.get_value('protocol'))
impalad_graceful_shutdown stays False for this test.
Even if first impalad is gracefully shutting down through L720, what will
ensure second impalad is also gracefully shutting down and flush query at L724?
--
To view, visit http://gerrit.cloudera.org:8080/21345
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia123c53a952a77ff4a9c02736b5717ccaa3566dc
Gerrit-Change-Number: 21345
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Smith
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jason Fehr
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Mon, 22 Apr 2024 23:58:22 +
Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-13020: Use 64-bit integer for Thrift max message size on C++
Hello Quanlong Huang, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21343 to look at the new patch set (#2). Change subject: IMPALA-13020: Use 64-bit integer for Thrift max message size on C++ .. IMPALA-13020: Use 64-bit integer for Thrift max message size on C++ Currently, Thrift's max message size is specified with a 32-bit signed integer, so it maxes out at 2GB. Impala has use cases that can produce messages larger than 2GB, so this patches Thrift to change the max message size to be an int64_t. This will allow Impala to specify a limit larger than 2GB. This only applies to Thrift's C++ code and it does not change Java. Change-Id: I94fdd7b07fcc8dca0639839b40a9eff815090835 --- M buildall.sh A source/thrift/thrift-0.16.0-patches/0007-IMPALA-13020-Use-64-bit-integer-for-max-message-size.patch 2 files changed, 125 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/43/21343/2 -- To view, visit http://gerrit.cloudera.org:8080/21343 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I94fdd7b07fcc8dca0639839b40a9eff815090835 Gerrit-Change-Number: 21343 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-13015: Dataload fails due to concurrency issue with test.jceks
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/21346 ) Change subject: IMPALA-13015: Dataload fails due to concurrency issue with test.jceks .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/21346/1/testdata/bin/create-load-data.sh File testdata/bin/create-load-data.sh: http://gerrit.cloudera.org:8080/#/c/21346/1/testdata/bin/create-load-data.sh@555 PS1, Line 555: run-step "Creating hadoop credential" create-hadoop-credential.log create-hadoop-credential > line too long (91 > 90) Done -- To view, visit http://gerrit.cloudera.org:8080/21346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7fbeffc19f2b78c19fee9acf7f96466c8f4f9bcd Gerrit-Change-Number: 21346 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Mon, 22 Apr 2024 23:55:22 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13015: Dataload fails due to concurrency issue with test.jceks
Abhishek Rawat has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/21346 ) Change subject: IMPALA-13015: Dataload fails due to concurrency issue with test.jceks .. IMPALA-13015: Dataload fails due to concurrency issue with test.jceks Move 'hadoop credential' command used for creating test.jceks to testdata/bin/create-load-data.sh. Earlier it was in bin/load-data.py which is called in parallel and was causing failures due to race conditions. Testing: - Ran JniFrontendTest#testGetSecretFromKeyStore after data loading and test ran clean. Change-Id: I7fbeffc19f2b78c19fee9acf7f96466c8f4f9bcd --- M bin/load-data.py M testdata/bin/create-load-data.sh 2 files changed, 9 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/46/21346/2 -- To view, visit http://gerrit.cloudera.org:8080/21346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7fbeffc19f2b78c19fee9acf7f96466c8f4f9bcd Gerrit-Change-Number: 21346 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21304 ) Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables .. Patch Set 14: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15990/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21304 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a Gerrit-Change-Number: 21304 Gerrit-PatchSet: 14 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 22 Apr 2024 23:47:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13015: Dataload fails due to concurrency issue with test.jceks
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21346 ) Change subject: IMPALA-13015: Dataload fails due to concurrency issue with test.jceks .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/21346/1/testdata/bin/create-load-data.sh File testdata/bin/create-load-data.sh: http://gerrit.cloudera.org:8080/#/c/21346/1/testdata/bin/create-load-data.sh@555 PS1, Line 555: run-step "Creating hadoop credential" create-hadoop-credential.log create-hadoop-credential line too long (91 > 90) -- To view, visit http://gerrit.cloudera.org:8080/21346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7fbeffc19f2b78c19fee9acf7f96466c8f4f9bcd Gerrit-Change-Number: 21346 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Mon, 22 Apr 2024 23:41:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12777: Fix tpcds/tpcds-q66.test
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21344 ) Change subject: IMPALA-12777: Fix tpcds/tpcds-q66.test .. Patch Set 1: > Patch Set 1: Code-Review+2 Thank you, Wenzhe! -- To view, visit http://gerrit.cloudera.org:8080/21344 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5b886f5dc1da213d25f33bd7b01dacca53eaef1b Gerrit-Change-Number: 21344 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 22 Apr 2024 23:41:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13015: Dataload fails due to concurrency issue with test.jceks
Abhishek Rawat has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21346 Change subject: IMPALA-13015: Dataload fails due to concurrency issue with test.jceks .. IMPALA-13015: Dataload fails due to concurrency issue with test.jceks Move 'hadoop credential' command used for creating test.jceks to testdata/bin/create-load-data.sh. Earlier it was in bin/load-data.py which is called in parallel and was causing failures due to race conditions. Testing: - Ran JniFrontendTest#testGetSecretFromKeyStore after data loading and test ran clean. Change-Id: I7fbeffc19f2b78c19fee9acf7f96466c8f4f9bcd --- M bin/load-data.py M testdata/bin/create-load-data.sh 2 files changed, 8 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/46/21346/1 -- To view, visit http://gerrit.cloudera.org:8080/21346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I7fbeffc19f2b78c19fee9acf7f96466c8f4f9bcd Gerrit-Change-Number: 21346 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-12777: Fix tpcds/tpcds-q66.test
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21344 ) Change subject: IMPALA-12777: Fix tpcds/tpcds-q66.test .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10573/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/21344 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5b886f5dc1da213d25f33bd7b01dacca53eaef1b Gerrit-Change-Number: 21344 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 22 Apr 2024 23:40:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12997: Use graceful shutdown for query log tests
Michael Smith has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21345 Change subject: IMPALA-12997: Use graceful shutdown for query log tests .. IMPALA-12997: Use graceful shutdown for query log tests Uses graceful shutdown for all tests that might insert into 'sys.impala_query_log' to avoid leaving the table locked in HMS by a SIGTERM. That's primarily any test that sets 'query_log_write_interval_s' or 'query_log_max_queued'. 'test_query_log_table_flush_on_shutdown' is unchanged because it explicitly triggers graceful shutdown during the test. Change-Id: Ia123c53a952a77ff4a9c02736b5717ccaa3566dc --- M tests/custom_cluster/test_query_live.py M tests/custom_cluster/test_query_log.py 2 files changed, 3 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/21345/1 -- To view, visit http://gerrit.cloudera.org:8080/21345 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ia123c53a952a77ff4a9c02736b5717ccaa3566dc Gerrit-Change-Number: 21345 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith
[Impala-ASF-CR] IMPALA-12777: Fix tpcds/tpcds-q66.test
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21344 ) Change subject: IMPALA-12777: Fix tpcds/tpcds-q66.test .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21344 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5b886f5dc1da213d25f33bd7b01dacca53eaef1b Gerrit-Change-Number: 21344 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 22 Apr 2024 23:37:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12872: Use Calcite for optimization - part 1: simple queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21109 ) Change subject: IMPALA-12872: Use Calcite for optimization - part 1: simple queries .. Patch Set 25: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15989/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21109 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I453fd75b7b705f4d7de1ed73c3e24cafad0b8c98 Gerrit-Change-Number: 21109 Gerrit-PatchSet: 25 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Mon, 22 Apr 2024 23:33:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/21304 )
Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
..
Patch Set 14:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/21304/6/be/src/service/frontend.cc
File be/src/service/frontend.cc:
http://gerrit.cloudera.org:8080/#/c/21304/6/be/src/service/frontend.cc@99
PS6, Line 99: DEFINE_int32(dbcp_max_conn_pool_size, 8,
> Ah, that makes it trickier. I assumed the point of this setting is to limit
This parameter limit each connection pool size, e.g. number of concurrent TCP
connections in a pool. We create DBCP connection pool for each unique
combination of database and username, which is implemented by setting cache key
as dataSourceCacheKey = String.format("%s.%s", jdbcUrl, username) in
GenericJdbcDatabaseAccessor.initializeDatabaseSource(). Each DataSource object
corresponds to one connection pool.
http://gerrit.cloudera.org:8080/#/c/21304/12/fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DataSourceObjectCache.java
File
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DataSourceObjectCache.java:
http://gerrit.cloudera.org:8080/#/c/21304/12/fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DataSourceObjectCache.java@93
PS12, Line 93: synchronized (this) {
> Ah, synchronized methods would have only prevented other calls to the same
'get', 'remove' and 'cleanup' use same synchronized object 'this'. Multiple
threads will not able to run multiple methods on the same synchronized object
simultaneously.
http://gerrit.cloudera.org:8080/#/c/21304/12/fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DataSourceObjectCache.java@148
PS12, Line 148: } catch (InterruptedException e) {
> I think the only exception we'd see here is InterruptedException, which doe
Lower the log level to info
--
To view, visit http://gerrit.cloudera.org:8080/21304
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
Gerrit-Change-Number: 21304
Gerrit-PatchSet: 14
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
Gerrit-Comment-Date: Mon, 22 Apr 2024 23:26:28 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has uploaded a new patch set (#14). (
http://gerrit.cloudera.org:8080/21304 )
Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
..
IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
This patch adds script to create external JDBC tables for the dataset of
TPCH and TPCDS, and adds unit-tests to run TPCH and TPCDS queries for
external JDBC tables with Impala-Impala federation. Notes that JDBC
tables are mapping tables, they don't take additional disk spaces.
It fixes the race condition when caching of SQL DataSource objects by
using a new DataSourceObjectCache class, which checks reference count
before closing SQL DataSource.
Adds a new query-option 'clean_dbcp_ds_cache' with default value as
true. When it's set as false, SQL DataSource object will not be closed
when its reference count equals 0 and will be kept in cache until
the SQL DataSource is idle for more than 5 minutes.
java.sql.Connection.close() fails to remove a closed connection from
connection pool sometimes, which causes JDBC working threads to wait
for available connections from the connection pool for a long time.
The work around is to call BasicDataSource.invalidateConnection() API
to close a connection.
Two flag variables are added for DBCP configuration properties
'maxTotal' and 'maxWaitMillis'. Notes that 'maxActive' and 'maxWait'
properties are renamed to 'maxTotal' and 'maxWaitMillis' respectively
in apache.commons.dbcp v2.
Fixes a bug for database type comparison since the type strings
specified by user could be lower case or mix of upper/lower cases, but
the code compares the types with upper case string.
Fixes issue to close SQL DataSource object in JdbcDataSource.open()
and JdbcDataSource.getNext() when some errors returned from DBCP APIs
or JDBC drivers.
testdata/bin/create-tpc-jdbc-tables.py supports to create JDBC tables
for Impala-Impala, Postgres and MySQL.
Following sample commands creates TPCDS JDBC tables for Impala-Impala
federation with remote coordinator running at 10.19.10.86, and Postgres
server running at 10.19.10.86:
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=IMPALA --database_host=10.19.10.86 --clean
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=POSTGRES --database_host=10.19.10.86 \
--database_name=tpcds --clean
Remaining Issues:
- tpcds-decimal_v2-q80a failed with returned rows not matching expected
results for some decimal values. This will be fixed in IMPALA-13018.
Testing:
- Passed core-test.
- Manually verified that only one SQL DataSource object was created for
test_tpcds_queries.py::TestTpcdsQueryForJdbcTables since query option
'clean_dbcp_ds_cache' was set as false, and the SQL DataSource object
was closed by cleanup thread.
Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
---
M be/src/exec/data-source-scan-node.cc
M be/src/service/frontend.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M common/thrift/ExternalDataSource.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java
A
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DataSourceObjectCache.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/JdbcRecordIterator.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M testdata/bin/create-load-data.sh
A testdata/bin/create-tpc-jdbc-tables.py
A testdata/datasets/tpcds/tpcds_jdbc_schema_template.sql
A testdata/datasets/tpch/tpch_jdbc_schema_template.sql
M tests/query_test/test_tpcds_queries.py
M tests/query_test/test_tpch_queries.py
22 files changed, 1,899 insertions(+), 98 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/21304/14
--
To view, visit http://gerrit.cloudera.org:8080/21304
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
Gerrit-Change-Number: 21304
Gerrit-PatchSet: 14
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12777: Fix tpcds/tpcds-q66.test
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21344 ) Change subject: IMPALA-12777: Fix tpcds/tpcds-q66.test .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15988/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21344 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5b886f5dc1da213d25f33bd7b01dacca53eaef1b Gerrit-Change-Number: 21344 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 22 Apr 2024 23:24:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12872: Use Calcite for optimization - part 1: simple queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21109 ) Change subject: IMPALA-12872: Use Calcite for optimization - part 1: simple queries .. Patch Set 25: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10572/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/21109 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I453fd75b7b705f4d7de1ed73c3e24cafad0b8c98 Gerrit-Change-Number: 21109 Gerrit-PatchSet: 25 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Mon, 22 Apr 2024 23:14:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12872: Use Calcite for optimization - part 1: simple queries
Hello Aman Sinha, Quanlong Huang, Joe McDonnell, Csaba Ringhofer, Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21109 to look at the new patch set (#25). Change subject: IMPALA-12872: Use Calcite for optimization - part 1: simple queries .. IMPALA-12872: Use Calcite for optimization - part 1: simple queries This is the first commit to use the Calcite library to parse, analyze, and optimize queries. The hook for the planner is through an override of the JniFrontend. The CalciteJniFrontend class is the driver that walks through each of the Calcite steps which are as follows: CalciteQueryParser: Takes the string query and outputs an AST in the form of Calcite's SqlNode object. CalciteMetadataHandler: Iterate through the SqlNode from the previous step and make sure all essential table metadata is retrieved from catalogd. CalciteValidator: Validate the SqlNode tree, akin to the Impala Analyzer. CalciteRelNodeConverter: Change the AST into a logical plan. In this first commit, the only logical nodes used are LogicalTableScan and LogicalProject. The LogicalTableScan will serve as the node that reads from an Hdfs Table and the LogicalProject will only project out the used columns in the query. In later versions, the LogicalProject will also handle function changes. CalciteOptimizer: This step is to optimize the query. In this cut, it will be a nop, but in later versions, it will perform logical optimizations via Calcite's rule mechanism. CalcitePhysPlanCreator: Converts the Calcite RelNode logical tree into Impala's PlanNode physical tree ExecRequestCreator: Implement the existing Impala steps that turn a Single Node Plan into a Distributed Plan. It will also create the TExecRequest object needed by the runtime server. Only some very basic queries will work with this commit. These include: select * from tbl <-- only needs the LogicalTableScan select c1 from tbl <-- Also uses the LogicalProject In the CalciteJniFrontend, there is some basic checks to make sure only select statements will get processed. Any non-query statement will revert back to the current Impala planner. In this iteration, any queries besides the minimal ones listed above will result in a caught exception which will then be run through the current Impala planner. The tests that do work can be found in calcite.test and run through the custom cluster test test_experimental_planner.py This iteration should support all types with the exception of complex types. Calcite does not have a STRING type, so the string type is represented as VARCHAR(MAXINT) similar to how Hive represents their STRING type. The ImpalaTypeConverter file is used to convert the Impala Type object to corresponding Calcite objects. Authorization is not yet working with this current commit. A Jira has been filed (IMPALA-13011) to deal with this. Change-Id: I453fd75b7b705f4d7de1ed73c3e24cafad0b8c98 --- M bin/set-classpath.sh M bin/start-impala-cluster.py M fe/src/main/java/org/apache/impala/analysis/TableName.java M fe/src/main/java/org/apache/impala/planner/PlannerContext.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A java/calcite-planner/pom.xml A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ConvertToImpalaRelRules.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaHdfsScanRel.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaPlanRel.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaProjectRel.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/NodeWithExprs.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ParentPlanRelContext.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/phys/ImpalaHdfsScanNode.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/util/CreateExprVisitor.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteDb.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaCalciteCatalogReader.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteJniFrontend.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteMetadataHandler.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalcitePhysPlanCreator.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteQueryParser.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteRelNodeCon
[Impala-ASF-CR] IMPALA-12872: Use Calcite for optimization - part 1: simple queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21109 ) Change subject: IMPALA-12872: Use Calcite for optimization - part 1: simple queries .. Patch Set 25: (1 comment) http://gerrit.cloudera.org:8080/#/c/21109/25/java/calcite-planner/src/main/java/org/apache/impala/calcite/validate/ImpalaConformance.java File java/calcite-planner/src/main/java/org/apache/impala/calcite/validate/ImpalaConformance.java: http://gerrit.cloudera.org:8080/#/c/21109/25/java/calcite-planner/src/main/java/org/apache/impala/calcite/validate/ImpalaConformance.java@26 PS25, Line 26: * https://calcite.apache.org/javadocAggregate/org/apache/calcite/sql/validate/SqlConformance.html line too long (98 > 90) -- To view, visit http://gerrit.cloudera.org:8080/21109 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I453fd75b7b705f4d7de1ed73c3e24cafad0b8c98 Gerrit-Change-Number: 21109 Gerrit-PatchSet: 25 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Mon, 22 Apr 2024 23:09:43 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12777: Fix tpcds/tpcds-q66.test
Riza Suminto has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21344 Change subject: IMPALA-12777: Fix tpcds/tpcds-q66.test .. IMPALA-12777: Fix tpcds/tpcds-q66.test PlannerTest/tpcds/tpcds-q66.test was mistakenly a copy of PlannerTest/tpcds/tpcds-q61.test with different predicate values. This patch replace the wrong test file with correct TPC-DS Q66 query. Testing: - Pass FE test TpcdsPlannerTest#testQ66. Change-Id: I5b886f5dc1da213d25f33bd7b01dacca53eaef1b --- M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q66.test 1 file changed, 962 insertions(+), 1,020 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/21344/1 -- To view, visit http://gerrit.cloudera.org:8080/21344 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I5b886f5dc1da213d25f33bd7b01dacca53eaef1b Gerrit-Change-Number: 21344 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21304 ) Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables .. Patch Set 13: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15987/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21304 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a Gerrit-Change-Number: 21304 Gerrit-PatchSet: 13 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 22 Apr 2024 22:30:17 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-13020: Use 64-bit integer for Thrift max message size on C++
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/21343 ) Change subject: IMPALA-13020: Use 64-bit integer for Thrift max message size on C++ .. Patch Set 1: Here is a better view of the new Thrift patch: https://github.com/joemcdonnell/thrift/commit/4cc1f0100f60c86cbf1068f8785e9d0cc8c6334f -- To view, visit http://gerrit.cloudera.org:8080/21343 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I94fdd7b07fcc8dca0639839b40a9eff815090835 Gerrit-Change-Number: 21343 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 22 Apr 2024 22:22:45 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-13020: Use 64-bit integer for Thrift max message size on C++
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21343 Change subject: IMPALA-13020: Use 64-bit integer for Thrift max message size on C++ .. IMPALA-13020: Use 64-bit integer for Thrift max message size on C++ Currently, Thrift's max message size is specified with a 32-bit signed integer, so it maxes out at 2GB. Impala has use cases that can produce messages larger than 2GB, so this patches Thrift to change the max message size to be an int64_t. This will allow Impala to specify a limit larger than 2GB. Change-Id: I94fdd7b07fcc8dca0639839b40a9eff815090835 --- M buildall.sh A source/thrift/thrift-0.16.0-patches/0007-IMPALA-13020-Use-64-bit-integer-for-max-message-size.patch 2 files changed, 125 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/43/21343/1 -- To view, visit http://gerrit.cloudera.org:8080/21343 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I94fdd7b07fcc8dca0639839b40a9eff815090835 Gerrit-Change-Number: 21343 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has uploaded a new patch set (#13). (
http://gerrit.cloudera.org:8080/21304 )
Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
..
IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
This patch adds script to create external JDBC tables for the dataset of
TPCH and TPCDS, and adds unit-tests to run TPCH and TPCDS queries for
external JDBC tables with Impala-Impala federation. Notes that JDBC
tables are mapping tables, they don't take additional disk spaces.
It fixes the race condition when caching of SQL DataSource objects by
using a new DataSourceObjectCache class, which checks reference count
before closing SQL DataSource.
Adds a new query-option 'clean_dbcp_ds_cache' with default value as
true. When it's set as false, SQL DataSource object will not be closed
when its reference count equals 0 and will be kept in cache until
the SQL DataSource is idle for more than 5 minutes.
java.sql.Connection.close() fails to remove a closed connection from
connection pool sometimes, which causes JDBC working threads to wait
for available connections from the connection pool for a long time.
The work around is to call BasicDataSource.invalidateConnection() API
to close a connection.
Two flag variables are added for DBCP configuration properties
'maxTotal' and 'maxWaitMillis'. Notes that 'maxActive' and 'maxWait'
properties are renamed to 'maxTotal' and 'maxWaitMillis' respectively
in apache.commons.dbcp v2.
Fixes a bug for database type comparison since the type strings
specified by user could be lower case or mix of upper/lower cases, but
the code compares the types with upper case string.
Fixes issue to close SQL DataSource object in JdbcDataSource.open()
and JdbcDataSource.getNext() when some errors returned from DBCP APIs
or JDBC drivers.
testdata/bin/create-tpc-jdbc-tables.py supports to create JDBC tables
for Impala-Impala, Postgres and MySQL.
Following sample commands creates TPCDS JDBC tables for Impala-Impala
federation with remote coordinator running at 10.19.10.86, and Postgres
server running at 10.19.10.86:
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=IMPALA --database_host=10.19.10.86 --clean
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=POSTGRES --database_host=10.19.10.86 \
--database_name=tpcds --clean
Remaining Issues:
- tpcds-decimal_v2-q80a failed with returned rows not matching expected
results for some decimal values. This will be fixed in IMPALA-13018.
Testing:
- Passed core-test.
- Manually verified that only one SQL DataSource object was created for
test_tpcds_queries.py::TestTpcdsQueryForJdbcTables since query option
'clean_dbcp_ds_cache' was set as false, and the SQL DataSource object
was closed by cleanup thread.
Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
---
M be/src/exec/data-source-scan-node.cc
M be/src/service/frontend.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M common/thrift/ExternalDataSource.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java
A
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DataSourceObjectCache.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/JdbcRecordIterator.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M testdata/bin/create-load-data.sh
A testdata/bin/create-tpc-jdbc-tables.py
A testdata/datasets/tpcds/tpcds_jdbc_schema_template.sql
A testdata/datasets/tpch/tpch_jdbc_schema_template.sql
M tests/query_test/test_tpcds_queries.py
M tests/query_test/test_tpch_queries.py
22 files changed, 1,899 insertions(+), 98 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/21304/13
--
To view, visit http://gerrit.cloudera.org:8080/21304
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
Gerrit-Change-Number: 21304
Gerrit-PatchSet: 13
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21304 ) Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables .. Patch Set 12: (1 comment) http://gerrit.cloudera.org:8080/#/c/21304/12/tests/query_test/test_tpcds_queries.py File tests/query_test/test_tpcds_queries.py: http://gerrit.cloudera.org:8080/#/c/21304/12/tests/query_test/test_tpcds_queries.py@791 PS12, Line 791: a > It's complaining that the 'and' is at the end of this line and not the star Will move 'and' to the start of next line. Thanks -- To view, visit http://gerrit.cloudera.org:8080/21304 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a Gerrit-Change-Number: 21304 Gerrit-PatchSet: 12 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 22 Apr 2024 21:08:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21304 ) Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables .. Patch Set 12: (1 comment) http://gerrit.cloudera.org:8080/#/c/21304/12/tests/query_test/test_tpcds_queries.py File tests/query_test/test_tpcds_queries.py: http://gerrit.cloudera.org:8080/#/c/21304/12/tests/query_test/test_tpcds_queries.py@791 PS12, Line 791: a > If adding '\' after 'and', it complains 'the backslash is redundant between It's complaining that the 'and' is at the end of this line and not the start of the next line. This is a warning I've seen recently that doesn't seem very consistent with how most of Impala's python is written. -- To view, visit http://gerrit.cloudera.org:8080/21304 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a Gerrit-Change-Number: 21304 Gerrit-PatchSet: 12 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 22 Apr 2024 20:33:10 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21304 ) Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15986/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21304 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a Gerrit-Change-Number: 21304 Gerrit-PatchSet: 12 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 22 Apr 2024 20:02:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21304 ) Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables .. Patch Set 12: (1 comment) http://gerrit.cloudera.org:8080/#/c/21304/12/tests/query_test/test_tpcds_queries.py File tests/query_test/test_tpcds_queries.py: http://gerrit.cloudera.org:8080/#/c/21304/12/tests/query_test/test_tpcds_queries.py@791 PS12, Line 791: a > flake8: W504 line break after binary operator If adding '\' after 'and', it complains 'the backslash is redundant between brackets'. Ignore this warning. -- To view, visit http://gerrit.cloudera.org:8080/21304 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a Gerrit-Change-Number: 21304 Gerrit-PatchSet: 12 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 22 Apr 2024 19:50:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21304 ) Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables .. Patch Set 12: (1 comment) http://gerrit.cloudera.org:8080/#/c/21304/12/tests/query_test/test_tpcds_queries.py File tests/query_test/test_tpcds_queries.py: http://gerrit.cloudera.org:8080/#/c/21304/12/tests/query_test/test_tpcds_queries.py@791 PS12, Line 791: a flake8: W504 line break after binary operator -- To view, visit http://gerrit.cloudera.org:8080/21304 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a Gerrit-Change-Number: 21304 Gerrit-PatchSet: 12 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 22 Apr 2024 19:39:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has uploaded a new patch set (#12). (
http://gerrit.cloudera.org:8080/21304 )
Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
..
IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
This patch adds script to create external JDBC tables for the dataset of
TPCH and TPCDS, and adds unit-tests to run TPCH and TPCDS queries for
external JDBC tables with Impala-Impala federation. Notes that JDBC
tables are mapping tables, they don't take additional disk spaces.
It fixes the race condition when caching of SQL DataSource objects by
using a new DataSourceObjectCache class, which checks reference count
before closing SQL DataSource.
Adds a new query-option 'clean_dbcp_ds_cache' with default value as
true. When it's set as false, SQL DataSource object will not be closed
when its reference count equals 0 and will be kept in cache until
the SQL DataSource is idle for more than 5 minutes.
java.sql.Connection.close() fails to remove a closed connection from
connection pool sometimes, which causes JDBC working threads to wait
for available connections from the connection pool for a long time.
The work around is to call BasicDataSource.invalidateConnection() API
to close a connection.
Two flag variables are added for DBCP configuration properties
'maxTotal' and 'maxWaitMillis'. Notes that 'maxActive' and 'maxWait'
properties are renamed to 'maxTotal' and 'maxWaitMillis' respectively
in apache.commons.dbcp v2.
Fixes a bug for database type comparison since the type strings
specified by user could be lower case or mix of upper/lower cases, but
the code compares the types with upper case string.
Fixes issue to close SQL DataSource object in JdbcDataSource.open()
and JdbcDataSource.getNext() when some errors returned from DBCP APIs
or JDBC drivers.
testdata/bin/create-tpc-jdbc-tables.py supports to create JDBC tables
for Impala-Impala, Postgres and MySQL.
Following sample commands creates TPCDS JDBC tables for Impala-Impala
federation with remote coordinator running at 10.19.10.86, and Postgres
server running at 10.19.10.86:
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=IMPALA --database_host=10.19.10.86 --clean
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=POSTGRES --database_host=10.19.10.86 \
--database_name=tpcds --clean
Remaining Issues:
- tpcds-decimal_v2-q80a failed with returned rows not matching expected
results for some decimal values. This will be fixed in IMPALA-13018.
Testing:
- Passed core-test.
- Manually verified that only one SQL DataSource object was created for
test_tpcds_queries.py::TestTpcdsQueryForJdbcTables since query option
'clean_dbcp_ds_cache' was set as false, and the SQL DataSource object
was closed by cleanup thread.
Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
---
M be/src/exec/data-source-scan-node.cc
M be/src/service/frontend.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M common/thrift/ExternalDataSource.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java
A
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DataSourceObjectCache.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/JdbcRecordIterator.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M testdata/bin/create-load-data.sh
A testdata/bin/create-tpc-jdbc-tables.py
A testdata/datasets/tpcds/tpcds_jdbc_schema_template.sql
A testdata/datasets/tpch/tpch_jdbc_schema_template.sql
M tests/query_test/test_tpcds_queries.py
M tests/query_test/test_tpch_queries.py
22 files changed, 1,899 insertions(+), 98 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/21304/12
--
To view, visit http://gerrit.cloudera.org:8080/21304
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
Gerrit-Change-Number: 21304
Gerrit-PatchSet: 12
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21304 ) Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15985/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21304 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a Gerrit-Change-Number: 21304 Gerrit-PatchSet: 11 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 22 Apr 2024 19:21:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21304 ) Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15984/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21304 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a Gerrit-Change-Number: 21304 Gerrit-PatchSet: 10 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 22 Apr 2024 19:10:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13024: Ignore slots if using default pool and empty group
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21340 ) Change subject: IMPALA-13024: Ignore slots if using default pool and empty group .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15983/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21340 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0b08dea7ba0c78ac6b98c7a0b148df8fb036c4d0 Gerrit-Change-Number: 21340 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 22 Apr 2024 19:00:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21304 ) Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables .. Patch Set 11: (2 comments) http://gerrit.cloudera.org:8080/#/c/21304/11/tests/query_test/test_tpcds_queries.py File tests/query_test/test_tpcds_queries.py: http://gerrit.cloudera.org:8080/#/c/21304/11/tests/query_test/test_tpcds_queries.py@790 PS11, Line 790: \ flake8: E502 the backslash is redundant between brackets http://gerrit.cloudera.org:8080/#/c/21304/11/tests/query_test/test_tpcds_queries.py@791 PS11, Line 791: \ flake8: E502 the backslash is redundant between brackets -- To view, visit http://gerrit.cloudera.org:8080/21304 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a Gerrit-Change-Number: 21304 Gerrit-PatchSet: 11 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 22 Apr 2024 18:58:10 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21304 ) Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables .. Patch Set 11: (1 comment) http://gerrit.cloudera.org:8080/#/c/21304/10/tests/query_test/test_tpcds_queries.py File tests/query_test/test_tpcds_queries.py: http://gerrit.cloudera.org:8080/#/c/21304/10/tests/query_test/test_tpcds_queries.py@791 PS10, Line 791: a > flake8: W504 line break after binary operator done -- To view, visit http://gerrit.cloudera.org:8080/21304 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a Gerrit-Change-Number: 21304 Gerrit-PatchSet: 11 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 22 Apr 2024 18:57:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has uploaded a new patch set (#11). (
http://gerrit.cloudera.org:8080/21304 )
Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
..
IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
This patch adds script to create external JDBC tables for the dataset of
TPCH and TPCDS, and adds unit-tests to run TPCH and TPCDS queries for
external JDBC tables with Impala-Impala federation. Notes that JDBC
tables are mapping tables, they don't take additional disk spaces.
It fixes the race condition when caching of SQL DataSource objects by
using a new DataSourceObjectCache class, which checks reference count
before closing SQL DataSource.
Adds a new query-option 'clean_dbcp_ds_cache' with default value as
true. When it's set as false, SQL DataSource object will not be closed
when its reference count equals 0 and will be kept in cache until
the SQL DataSource is idle for more than 5 minutes.
java.sql.Connection.close() fails to remove a closed connection from
connection pool sometimes, which causes JDBC working threads to wait
for available connections from the connection pool for a long time.
The work around is to call BasicDataSource.invalidateConnection() API
to close a connection.
Two flag variables are added for DBCP configuration properties
'maxTotal' and 'maxWaitMillis'. Notes that 'maxActive' and 'maxWait'
properties are renamed to 'maxTotal' and 'maxWaitMillis' respectively
in apache.commons.dbcp v2.
Fixes a bug for database type comparison since the type strings
specified by user could be lower case or mix of upper/lower cases, but
the code compares the types with upper case string.
Fixes issue to close SQL DataSource object in JdbcDataSource.open()
and JdbcDataSource.getNext() when some errors returned from DBCP APIs
or JDBC drivers.
testdata/bin/create-tpc-jdbc-tables.py supports to create JDBC tables
for Impala-Impala, Postgres and MySQL.
Following sample commands creates TPCDS JDBC tables for Impala-Impala
federation with remote coordinator running at 10.19.10.86, and Postgres
server running at 10.19.10.86:
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=IMPALA --database_host=10.19.10.86 --clean
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=POSTGRES --database_host=10.19.10.86 \
--database_name=tpcds --clean
Remaining Issues:
- tpcds-decimal_v2-q80a failed with returned rows not matching expected
results for some decimal values. This will be fixed in IMPALA-13018.
Testing:
- Passed core-test.
- Manually verified that only one SQL DataSource object was created for
test_tpcds_queries.py::TestTpcdsQueryForJdbcTables since query option
'clean_dbcp_ds_cache' was set as false, and the SQL DataSource object
was closed by cleanup thread.
Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
---
M be/src/exec/data-source-scan-node.cc
M be/src/service/frontend.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M common/thrift/ExternalDataSource.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java
A
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DataSourceObjectCache.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/JdbcRecordIterator.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M testdata/bin/create-load-data.sh
A testdata/bin/create-tpc-jdbc-tables.py
A testdata/datasets/tpcds/tpcds_jdbc_schema_template.sql
A testdata/datasets/tpch/tpch_jdbc_schema_template.sql
M tests/query_test/test_tpcds_queries.py
M tests/query_test/test_tpch_queries.py
22 files changed, 1,899 insertions(+), 98 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/21304/11
--
To view, visit http://gerrit.cloudera.org:8080/21304
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
Gerrit-Change-Number: 21304
Gerrit-PatchSet: 11
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Wenzhe Zhou has uploaded a new patch set (#10). (
http://gerrit.cloudera.org:8080/21304 )
Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
..
IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
This patch adds script to create external JDBC tables for the dataset of
TPCH and TPCDS, and adds unit-tests to run TPCH and TPCDS queries for
external JDBC tables with Impala-Impala federation. Notes that JDBC
tables are mapping tables, they don't take additional disk spaces.
It fixes the race condition when caching of SQL DataSource objects by
using a new DataSourceObjectCache class, which checks reference count
before closing SQL DataSource.
Adds a new query-option 'clean_dbcp_ds_cache' with default value as
true. When it's set as false, SQL DataSource object will not be closed
when its reference count equals 0 and will be kept in cache until
the SQL DataSource is idle for more than 5 minutes.
java.sql.Connection.close() fails to remove a closed connection from
connection pool sometimes, which causes JDBC working threads to wait
for available connections from the connection pool for a long time.
The work around is to call BasicDataSource.invalidateConnection() API
to close a connection.
Two flag variables are added for DBCP configuration properties
'maxTotal' and 'maxWaitMillis'. Notes that 'maxActive' and 'maxWait'
properties are renamed to 'maxTotal' and 'maxWaitMillis' respectively
in apache.commons.dbcp v2.
Fixes a bug for database type comparison since the type strings
specified by user could be lower case or mix of upper/lower cases, but
the code compares the types with upper case string.
Fixes issue to close SQL DataSource object in JdbcDataSource.open()
and JdbcDataSource.getNext() when some errors returned from DBCP APIs
or JDBC drivers.
testdata/bin/create-tpc-jdbc-tables.py supports to create JDBC tables
for Impala-Impala, Postgres and MySQL.
Following sample commands creates TPCDS JDBC tables for Impala-Impala
federation with remote coordinator running at 10.19.10.86, and Postgres
server running at 10.19.10.86:
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=IMPALA --database_host=10.19.10.86 --clean
${IMPALA_HOME}/testdata/bin/create-tpc-jdbc-tables.py \
--jdbc_db_name=tpcds_jdbc --workload=tpcds \
--database_type=POSTGRES --database_host=10.19.10.86 \
--database_name=tpcds --clean
Remaining Issues:
- tpcds-decimal_v2-q80a failed with returned rows not matching expected
results for some decimal values. This will be fixed in IMPALA-13018.
Testing:
- Passed core-test.
- Manually verified that only one SQL DataSource object was created for
test_tpcds_queries.py::TestTpcdsQueryForJdbcTables since query option
'clean_dbcp_ds_cache' was set as false, and the SQL DataSource object
was closed by cleanup thread.
Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
---
M be/src/exec/data-source-scan-node.cc
M be/src/service/frontend.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M common/thrift/ExternalDataSource.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/extdatasource/jdbc/JdbcDataSource.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/conf/JdbcStorageConfigManager.java
A
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DataSourceObjectCache.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/DatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java
M
fe/src/main/java/org/apache/impala/extdatasource/jdbc/dao/JdbcRecordIterator.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M testdata/bin/create-load-data.sh
A testdata/bin/create-tpc-jdbc-tables.py
A testdata/datasets/tpcds/tpcds_jdbc_schema_template.sql
A testdata/datasets/tpch/tpch_jdbc_schema_template.sql
M tests/query_test/test_tpcds_queries.py
M tests/query_test/test_tpch_queries.py
22 files changed, 1,899 insertions(+), 98 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/21304/10
--
To view, visit http://gerrit.cloudera.org:8080/21304
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a
Gerrit-Change-Number: 21304
Gerrit-PatchSet: 10
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: gaurav singh
[Impala-ASF-CR] IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21304 ) Change subject: IMPALA-12910: Support running TPCH/TPCDS queries for JDBC tables .. Patch Set 10: (1 comment) http://gerrit.cloudera.org:8080/#/c/21304/10/tests/query_test/test_tpcds_queries.py File tests/query_test/test_tpcds_queries.py: http://gerrit.cloudera.org:8080/#/c/21304/10/tests/query_test/test_tpcds_queries.py@791 PS10, Line 791: a flake8: W504 line break after binary operator -- To view, visit http://gerrit.cloudera.org:8080/21304 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I44e8c1bb020e90559c7f22483a7ab7a151b8f48a Gerrit-Change-Number: 21304 Gerrit-PatchSet: 10 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: gaurav singh Gerrit-Comment-Date: Mon, 22 Apr 2024 18:49:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13024: Ignore slots if using default pool and empty group
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21340 ) Change subject: IMPALA-13024: Ignore slots if using default pool and empty group .. Patch Set 4: (2 comments) http://gerrit.cloudera.org:8080/#/c/21340/3/be/src/scheduling/admission-controller-test.cc File be/src/scheduling/admission-controller-test.cc: http://gerrit.cloudera.org:8080/#/c/21340/3/be/src/scheduling/admission-controller-test.cc@605 PS3, Line 605: // Create ScheduleStates to run on "default-pool" on 12 hosts. : // Running both distributed and coordinator-only schedule. > Drop this comment. Leftover from copy-paste. Done http://gerrit.cloudera.org:8080/#/c/21340/3/be/src/scheduling/admission-controller-test.cc@629 PS3, Line 629: ERT_TRUE(admission_controller->C > "All schedules should be admitted". Done -- To view, visit http://gerrit.cloudera.org:8080/21340 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0b08dea7ba0c78ac6b98c7a0b148df8fb036c4d0 Gerrit-Change-Number: 21340 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 22 Apr 2024 18:36:39 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13024: Ignore slots if using default pool and empty group
Hello Abhishek Rawat, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21340
to look at the new patch set (#4).
Change subject: IMPALA-13024: Ignore slots if using default pool and empty group
..
IMPALA-13024: Ignore slots if using default pool and empty group
Slot based admission should not be enabled when using default pool.
There is a bug where coordinator-only query still does slot based
admission because executor group name set to
ClusterMembershipMgr::EMPTY_GROUP_NAME ("empty group (using coordinator
only)"). This patch add check to recognize coordinator-only query at
default pool and skip it from slot checking.
Testing:
- Add BE test AdmissionControllerTest.CanAdmitRequestSlotsDefault.
- In test_executor_groups.py, split test_coordinator_concurrency to
test_coordinator_concurrency_default and
test_coordinator_concurrency_two_exec_group_cluster to show the
behavior change.
- Pass core tests in ASAN build.
Change-Id: I0b08dea7ba0c78ac6b98c7a0b148df8fb036c4d0
---
M be/src/scheduling/admission-controller-test.cc
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admission-controller.h
M be/src/scheduling/cluster-membership-mgr.cc
M be/src/scheduling/cluster-membership-mgr.h
M be/src/scheduling/request-pool-service.cc
M be/src/scheduling/request-pool-service.h
M tests/custom_cluster/test_executor_groups.py
8 files changed, 101 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/21340/4
--
To view, visit http://gerrit.cloudera.org:8080/21340
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0b08dea7ba0c78ac6b98c7a0b148df8fb036c4d0
Gerrit-Change-Number: 21340
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-13024: Ignore slots if using default pool and empty group
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21340 ) Change subject: IMPALA-13024: Ignore slots if using default pool and empty group .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15982/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21340 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0b08dea7ba0c78ac6b98c7a0b148df8fb036c4d0 Gerrit-Change-Number: 21340 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 22 Apr 2024 17:09:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11499: Refactor UrlEncode function to handle special characters
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21131 ) Change subject: IMPALA-11499: Refactor UrlEncode function to handle special characters .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15981/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88c4aba5d811dfcec809583d0c16fcbc0ca730fb Gerrit-Change-Number: 21131 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zihao Ye Gerrit-Comment-Date: Mon, 22 Apr 2024 16:59:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13024: Ignore slots if using default pool and empty group
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21340 ) Change subject: IMPALA-13024: Ignore slots if using default pool and empty group .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/21340/3/be/src/scheduling/admission-controller-test.cc File be/src/scheduling/admission-controller-test.cc: http://gerrit.cloudera.org:8080/#/c/21340/3/be/src/scheduling/admission-controller-test.cc@605 PS3, Line 605: // Create ScheduleStates to run on QUEUE_D on 12 hosts. : // Running in both default and non-default executor groups is simulated. Drop this comment. Leftover from copy-paste. http://gerrit.cloudera.org:8080/#/c/21340/3/be/src/scheduling/admission-controller-test.cc@629 PS3, Line 629: Schedule should all be admitted. "All schedules should be admitted". -- To view, visit http://gerrit.cloudera.org:8080/21340 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0b08dea7ba0c78ac6b98c7a0b148df8fb036c4d0 Gerrit-Change-Number: 21340 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 22 Apr 2024 16:52:34 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13024: Ignore slots if using default pool and empty group
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21340 ) Change subject: IMPALA-13024: Ignore slots if using default pool and empty group .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/21340/2/tests/custom_cluster/test_executor_groups.py File tests/custom_cluster/test_executor_groups.py: http://gerrit.cloudera.org:8080/#/c/21340/2/tests/custom_cluster/test_executor_groups.py@402 PS2, Line 402: # Try to run two queries and observe that the second one > This comment is stale. Done -- To view, visit http://gerrit.cloudera.org:8080/21340 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0b08dea7ba0c78ac6b98c7a0b148df8fb036c4d0 Gerrit-Change-Number: 21340 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 22 Apr 2024 16:50:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13024: Ignore slots if using default pool and empty group
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21340 ) Change subject: IMPALA-13024: Ignore slots if using default pool and empty group .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15980/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21340 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0b08dea7ba0c78ac6b98c7a0b148df8fb036c4d0 Gerrit-Change-Number: 21340 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 22 Apr 2024 16:50:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13024: Ignore slots if using default pool and empty group
Hello Abhishek Rawat, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21340
to look at the new patch set (#3).
Change subject: IMPALA-13024: Ignore slots if using default pool and empty group
..
IMPALA-13024: Ignore slots if using default pool and empty group
Slot based admission should not be enabled when using default pool.
There is a bug where coordinator-only query still does slot based
admission because executor group name set to
ClusterMembershipMgr::EMPTY_GROUP_NAME ("empty group (using coordinator
only)"). This patch add check to recognize coordinator-only query at
default pool and skip it from slot checking.
Testing:
- Add BE test AdmissionControllerTest.CanAdmitRequestSlotsDefault.
- In test_executor_groups.py, split test_coordinator_concurrency to
test_coordinator_concurrency_default and
test_coordinator_concurrency_two_exec_group_cluster to show the
behavior change.
- Pass core tests in ASAN build.
Change-Id: I0b08dea7ba0c78ac6b98c7a0b148df8fb036c4d0
---
M be/src/scheduling/admission-controller-test.cc
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admission-controller.h
M be/src/scheduling/cluster-membership-mgr.cc
M be/src/scheduling/cluster-membership-mgr.h
M be/src/scheduling/request-pool-service.cc
M be/src/scheduling/request-pool-service.h
M tests/custom_cluster/test_executor_groups.py
8 files changed, 103 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/21340/3
--
To view, visit http://gerrit.cloudera.org:8080/21340
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0b08dea7ba0c78ac6b98c7a0b148df8fb036c4d0
Gerrit-Change-Number: 21340
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-13024: Ignore slots if using default pool and empty group
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21340 ) Change subject: IMPALA-13024: Ignore slots if using default pool and empty group .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/21340/2/tests/custom_cluster/test_executor_groups.py File tests/custom_cluster/test_executor_groups.py: http://gerrit.cloudera.org:8080/#/c/21340/2/tests/custom_cluster/test_executor_groups.py@402 PS2, Line 402: # Try to run two queries and observe that one gets queued This comment is stale. -- To view, visit http://gerrit.cloudera.org:8080/21340 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0b08dea7ba0c78ac6b98c7a0b148df8fb036c4d0 Gerrit-Change-Number: 21340 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 22 Apr 2024 16:42:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11499: Refactor UrlEncode function to handle special characters
pranav.lo...@cloudera.com has posted comments on this change. (
http://gerrit.cloudera.org:8080/21131 )
Change subject: IMPALA-11499: Refactor UrlEncode function to handle special
characters
..
Patch Set 5:
(3 comments)
> Uploaded patch set 5.
http://gerrit.cloudera.org:8080/#/c/21131/3/be/src/util/coding-util.cc
File be/src/util/coding-util.cc:
http://gerrit.cloudera.org:8080/#/c/21131/3/be/src/util/coding-util.cc@83
PS3, Line 83: } else {
> The exception is thrown here in the catalogd:
I've tried adding all the cases in the map now.
http://gerrit.cloudera.org:8080/#/c/21131/4/be/src/util/coding-util.cc
File be/src/util/coding-util.cc:
http://gerrit.cloudera.org:8080/#/c/21131/4/be/src/util/coding-util.cc@83
PS4, Line 83: } else {
> Repeated lookup on specialCharacterMap. You could do
Done
http://gerrit.cloudera.org:8080/#/c/21131/4/be/src/util/coding-util.cc@117
PS4, Line 117: int value = 0;
> Avoid unnecessary whitespace change.
Done
--
To view, visit http://gerrit.cloudera.org:8080/21131
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I88c4aba5d811dfcec809583d0c16fcbc0ca730fb
Gerrit-Change-Number: 21131
Gerrit-PatchSet: 5
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Daniel Becker
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Zihao Ye
Gerrit-Comment-Date: Mon, 22 Apr 2024 16:33:12 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11499: Refactor UrlEncode function to handle special characters
pranav.lo...@cloudera.com has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/21131 ) Change subject: IMPALA-11499: Refactor UrlEncode function to handle special characters .. IMPALA-11499: Refactor UrlEncode function to handle special characters The previous code suffered from URL encoding errors due to incomplete character whitelisting and inadequate handling of Unicode characters. This commit addresses these issues by introducing a specialCharacterMap that accurately maps special characters to their URL-encoded forms. The UrlEncode function now utilizes this map to determine whether a character should be escaped. If present in the map, the character's encoded form is used; otherwise, it falls back to hexadecimal encoding. Additionally, the function clears the output string before encoding to ensure a clean start and efficiently accumulates encoded characters using stringstream. These improvements ensure accurate encoding of all characters, including special and Unicode characters, resolving the errors encountered during URL encoding. Testing: Some basic tests are provided in unicode-column-name.test. Change-Id: I88c4aba5d811dfcec809583d0c16fcbc0ca730fb --- M be/src/util/coding-util.cc M testdata/workloads/functional-query/queries/QueryTest/unicode-column-name.test 2 files changed, 78 insertions(+), 19 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/21131/5 -- To view, visit http://gerrit.cloudera.org:8080/21131 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I88c4aba5d811dfcec809583d0c16fcbc0ca730fb Gerrit-Change-Number: 21131 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zihao Ye
[Impala-ASF-CR] IMPALA-13024: Ignore slots if using default pool and empty group
Hello Abhishek Rawat, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21340
to look at the new patch set (#2).
Change subject: IMPALA-13024: Ignore slots if using default pool and empty group
..
IMPALA-13024: Ignore slots if using default pool and empty group
Slot based admission should not be enabled when using default pool.
There is a bug where coordinator-only query still does slot based
admission because executor group name set to
ClusterMembershipMgr::EMPTY_GROUP_NAME ("empty group (using coordinator
only)"). This patch add check to recognize coordinator-only query at
default pool and skip it from slot checking.
Testing:
- Add BE test AdmissionControllerTest.CanAdmitRequestSlotsDefault.
- In test_executor_groups.py, split test_coordinator_concurrency to
test_coordinator_concurrency_default and
test_coordinator_concurrency_two_exec_group_cluster to show the
behavior change.
- Pass core tests in ASAN build.
Change-Id: I0b08dea7ba0c78ac6b98c7a0b148df8fb036c4d0
---
M be/src/scheduling/admission-controller-test.cc
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admission-controller.h
M be/src/scheduling/cluster-membership-mgr.cc
M be/src/scheduling/cluster-membership-mgr.h
M be/src/scheduling/request-pool-service.cc
M be/src/scheduling/request-pool-service.h
M tests/custom_cluster/test_executor_groups.py
8 files changed, 103 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/21340/2
--
To view, visit http://gerrit.cloudera.org:8080/21340
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0b08dea7ba0c78ac6b98c7a0b148df8fb036c4d0
Gerrit-Change-Number: 21340
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-13002: Iceberg V2 tables with Avro delete files aren't read properly
Zoltan Borok-Nagy has posted comments on this change. (
http://gerrit.cloudera.org:8080/21301 )
Change subject: IMPALA-13002: Iceberg V2 tables with Avro delete files aren't
read properly
..
Patch Set 1:
(2 comments)
Thanks for the comments!
http://gerrit.cloudera.org:8080/#/c/21301/1/fe/src/main/java/org/apache/impala/catalog/IcebergDeleteTable.java
File fe/src/main/java/org/apache/impala/catalog/IcebergDeleteTable.java:
http://gerrit.cloudera.org:8080/#/c/21301/1/fe/src/main/java/org/apache/impala/catalog/IcebergDeleteTable.java@87
PS1, Line 87: if (desc.hdfsTable.isSetAvroSchema()) {
> I guess the issue is also true for AVRO equality delete files. Should we al
Yes, it would definitely be useful to have such tests. Probably in a separate
CR, as adding such tables is cumbersome.
http://gerrit.cloudera.org:8080/#/c/21301/1/testdata/workloads/functional-query/queries/QueryTest/iceberg-mixed-format-position-deletes.test
File
testdata/workloads/functional-query/queries/QueryTest/iceberg-mixed-format-position-deletes.test:
http://gerrit.cloudera.org:8080/#/c/21301/1/testdata/workloads/functional-query/queries/QueryTest/iceberg-mixed-format-position-deletes.test@92
PS1, Line 92:
row_regex:'$NAMENODE/test-warehouse/$DATABASE.db/ice_mixed_formats_partitioned/data/j_trunc=2/.*-data-.*.orc','.*B','','.*'
> there should be 2 ORC data files in the j_trunc=2, right? One for (2,2) and
With VERIFY_IS_SUBSET we only check that each line is present in the result
set. I.e. adding more lines with the same content wouldn't have an effect:
https://github.com/apache/impala/blob/9b05a205fec397fa1e19ae467b1cc406ca43d948/tests/common/test_result_verifier.py#L258-L259
--
To view, visit http://gerrit.cloudera.org:8080/21301
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff13198991caf32c51cd9e0ace4454fd00216cf6
Gerrit-Change-Number: 21301
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy
Gerrit-Reviewer: Daniel Becker
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Mon, 22 Apr 2024 15:30:48 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13002: Iceberg V2 tables with Avro delete files aren't read properly
Gabor Kaszab has posted comments on this change. (
http://gerrit.cloudera.org:8080/21301 )
Change subject: IMPALA-13002: Iceberg V2 tables with Avro delete files aren't
read properly
..
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/21301/1/fe/src/main/java/org/apache/impala/catalog/IcebergDeleteTable.java
File fe/src/main/java/org/apache/impala/catalog/IcebergDeleteTable.java:
http://gerrit.cloudera.org:8080/#/c/21301/1/fe/src/main/java/org/apache/impala/catalog/IcebergDeleteTable.java@87
PS1, Line 87: if (desc.hdfsTable.isSetAvroSchema()) {
I guess the issue is also true for AVRO equality delete files. Should we also
add test coverage for that? (could be separate patch)
http://gerrit.cloudera.org:8080/#/c/21301/1/testdata/workloads/functional-query/queries/QueryTest/iceberg-mixed-format-position-deletes.test
File
testdata/workloads/functional-query/queries/QueryTest/iceberg-mixed-format-position-deletes.test:
http://gerrit.cloudera.org:8080/#/c/21301/1/testdata/workloads/functional-query/queries/QueryTest/iceberg-mixed-format-position-deletes.test@92
PS1, Line 92:
row_regex:'$NAMENODE/test-warehouse/$DATABASE.db/ice_mixed_formats_partitioned/data/j_trunc=2/.*-data-.*.orc','.*B','','.*'
there should be 2 ORC data files in the j_trunc=2, right? One for (2,2) and one
for (3,3). You only check for 1 of such line.
--
To view, visit http://gerrit.cloudera.org:8080/21301
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iff13198991caf32c51cd9e0ace4454fd00216cf6
Gerrit-Change-Number: 21301
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy
Gerrit-Reviewer: Daniel Becker
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Mon, 22 Apr 2024 15:20:10 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12973,IMPALA-11491,IMPALA-12651: Support BINARY nested in complex types in select list
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21269 ) Change subject: IMPALA-12973,IMPALA-11491,IMPALA-12651: Support BINARY nested in complex types in select list .. Patch Set 8: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/15979/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/21269 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b1d7fa332a901f05a46e0199e13fb841d2687c2 Gerrit-Change-Number: 21269 Gerrit-PatchSet: 8 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Comment-Date: Mon, 22 Apr 2024 15:01:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12973,IMPALA-11491,IMPALA-12651: Support BINARY nested in complex types in select list
Daniel Becker has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/21269 ) Change subject: IMPALA-12973,IMPALA-11491,IMPALA-12651: Support BINARY nested in complex types in select list .. IMPALA-12973,IMPALA-11491,IMPALA-12651: Support BINARY nested in complex types in select list Binary fields in complex types are currently not supported at all for regular tables (an error is returned). For Iceberg metadata tables, IMPALA-12899 added a temporary workaround to allow queries that contain these fields to succeed by NULLing them out. This change adds support for displaying them with base64 encoding for both regular and Iceberg metadata tables. Complex types are displayed in JSON format, so simply inserting the bytes of the binary fields is not acceptable as it would produce invalid JSON. Base64 is a widely used encoding that allows representing arbitrary binary information using only a limited set of ASCII characters. This change also adds support for top level binary columns in Iceberg metadata tables. However, these are not base64 encoded but are returned in raw byte format - this is consistent with how top level binary columns from regular (non-metadata) tables are handled. Testing: - added test queries in iceberg-metadata-tables.test referencing both nested and top level binary fields; also updated existing queries - moved relevant tests (queries extracting binary fields from within complex types) from nested-types-scanner-basic.test to a new binary-in-complex-type.test file and also added a query that selects the containing complex types; this new test file is run from test_scanners.py::TestBinaryInComplexType::\ test_binary_in_complex_type - moved negative tests in AnalyzerTest.TestUnsupportedTypes() to AnalyzeStmtsTest.TestComplexTypesInSelectList() and converted them to positive tests (expecting success); a negative test already in AnalyzeStmtsTest.TestComplexTypesInSelectList() was also converted Change-Id: I7b1d7fa332a901f05a46e0199e13fb841d2687c2 --- M be/src/exec/iceberg-metadata/iceberg-metadata-scanner.cc M be/src/exec/iceberg-metadata/iceberg-metadata-scanner.h M be/src/exec/iceberg-metadata/iceberg-row-reader.cc M be/src/exec/iceberg-metadata/iceberg-row-reader.h M be/src/rpc/jni-thrift-util.h M be/src/runtime/complex-value-writer.inline.h M be/src/util/jni-util.cc M be/src/util/jni-util.h M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/util/IcebergMetadataScanner.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java M testdata/data/README A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_with_key_metadata/data/0-0-data-danielbecker_20240408174043_c3737eaf-db30-4b88-aafb-f23c0f3c1dd3-job_17125053806420_0002-1-1.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_with_key_metadata/metadata/64da0e56-efa3-4025-bef1-1047fdd9a2b0-m0.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_with_key_metadata/metadata/snap-3079551887386250470-1-64da0e56-efa3-4025-bef1-1047fdd9a2b0.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_with_key_metadata/metadata/v1.metadata.json A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_with_key_metadata/metadata/v2.metadata.json A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_with_key_metadata/metadata/version-hint.txt M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv A testdata/workloads/functional-query/queries/QueryTest/binary-in-complex-type.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-metadata-tables.test M testdata/workloads/functional-query/queries/QueryTest/nested-types-scanner-basic.test M tests/query_test/test_scanners.py 26 files changed, 439 insertions(+), 155 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/21269/8 -- To view, visit http://gerrit.cloudera.org:8080/21269 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7b1d7fa332a901f05a46e0199e13fb841d2687c2 Gerrit-Change-Number: 21269 Gerrit-PatchSet: 8 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs
[Impala-ASF-CR] IMPALA-12973,IMPALA-11491,IMPALA-12651: Support BINARY nested in complex types in select list
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21269 ) Change subject: IMPALA-12973,IMPALA-11491,IMPALA-12651: Support BINARY nested in complex types in select list .. Patch Set 8: (1 comment) http://gerrit.cloudera.org:8080/#/c/21269/8/be/src/util/jni-util.h File be/src/util/jni-util.h: http://gerrit.cloudera.org:8080/#/c/21269/8/be/src/util/jni-util.h@115 PS8, Line 115: /// is more restricted, see https://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/functions.html#GetPrimitiveArrayCritical_ReleasePrimitiveArrayCritical line too long (162 > 90) -- To view, visit http://gerrit.cloudera.org:8080/21269 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b1d7fa332a901f05a46e0199e13fb841d2687c2 Gerrit-Change-Number: 21269 Gerrit-PatchSet: 8 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Comment-Date: Mon, 22 Apr 2024 14:38:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12973,IMPALA-11491,IMPALA-12651: Support BINARY nested in complex types in select list
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/21269 ) Change subject: IMPALA-12973,IMPALA-11491,IMPALA-12651: Support BINARY nested in complex types in select list .. Patch Set 7: (2 comments) Thanks for the review, Gábor! http://gerrit.cloudera.org:8080/#/c/21269/7/be/src/exec/iceberg-metadata/iceberg-row-reader.cc File be/src/exec/iceberg-metadata/iceberg-row-reader.cc: http://gerrit.cloudera.org:8080/#/c/21269/7/be/src/exec/iceberg-metadata/iceberg-row-reader.cc@227 PS7, Line 227: RETURN_IF_ERROR(GuardType::create(env, jbuffer, &jbuffer_guard)); > if create() returned an error here we'd leak memory because of 'jbuffer', r No, the type of 'jbuffer' is jbyteArray or jstring, and they are local JNI object references which will be freed by the JVM. Only the buffer returned by env->GetByteArrayElements() or env->GetStringUTFChars() (called within create()) has to be freed manually, that's what 'jbuffer_guard' takes care of. http://gerrit.cloudera.org:8080/#/c/21269/7/testdata/workloads/functional-query/queries/QueryTest/iceberg-metadata-tables.test File testdata/workloads/functional-query/queries/QueryTest/iceberg-metadata-tables.test: http://gerrit.cloudera.org:8080/#/c/21269/7/testdata/workloads/functional-query/queries/QueryTest/iceberg-metadata-tables.test@804 PS7, Line 804: select data_file from functional_parquet.iceberg_query_metadata.entries; > The query should go into the QUERY section Thanks for catching it. -- To view, visit http://gerrit.cloudera.org:8080/21269 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b1d7fa332a901f05a46e0199e13fb841d2687c2 Gerrit-Change-Number: 21269 Gerrit-PatchSet: 7 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Comment-Date: Mon, 22 Apr 2024 14:26:29 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12950:Improve error message in case of out-of-range numeric conversions
Peter Rozsa has posted comments on this change. (
http://gerrit.cloudera.org:8080/21331 )
Change subject: IMPALA-12950:Improve error message in case of out-of-range
numeric conversions
..
Patch Set 2:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/21331/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21331/2//COMMIT_MSG@7
PS2, Line 7: :
nit: missing whitespace
http://gerrit.cloudera.org:8080/#/c/21331/2//COMMIT_MSG@12
PS2, Line 12: floating point
nit: floating-point
http://gerrit.cloudera.org:8080/#/c/21331/2/be/src/exprs/cast-functions-ir.cc
File be/src/exprs/cast-functions-ir.cc:
http://gerrit.cloudera.org:8080/#/c/21331/2/be/src/exprs/cast-functions-ir.cc@76
PS2, Line 76: }
The default case is missing, it could be added as "UNKNOWN TYPE" or something
similar.
http://gerrit.cloudera.org:8080/#/c/21331/2/be/src/exprs/cast-functions-ir.cc@182
PS2, Line 182: err = Substitute("NaN value of type $0 cannot be converted
to $1.",
Could you please add test cases to cover each condition?
--
To view, visit http://gerrit.cloudera.org:8080/21331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ieeed52e25f155818c35c11a8a6821708476ffb32
Gerrit-Change-Number: 21331
Gerrit-PatchSet: 2
Gerrit-Owner: Daniel Becker
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Peter Rozsa
Gerrit-Comment-Date: Mon, 22 Apr 2024 12:25:05 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10408: Support build using Apache components
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/18977 )
Change subject: IMPALA-10408: Support build using Apache components
..
Patch Set 15:
(4 comments)
Good to know the build succeeds!
> I have one question. Why does impala use toolchains from cloudera internal
> repository when building? It is called "Apache Impala" and open source. It
> should be built with apache sources.
I think we used to support building on Apache components. But due to the
official releases are usually slow (e.g. Hive-4.0), we switch to build on
CDH/CDP components which have the bug fixes or improvements to unblock our work.
> It will be easier to build apache impala if there is a way to download or
> upload binary packages for apache components like
> https://github.com/zer0beat/apache-ranger-compiled/releases for Apache Ranger.
We can consider uploading this to the S3 bucket of our native-toolchain. An
alternative is building Apache Ranger in bin/bootstrap_system.sh if it doesn't
take long. We used to build hadoop-lzo there in 3.x (it's removed later since
we deprecated the support on LZO).
http://gerrit.cloudera.org:8080/#/c/18977/15//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/18977/15//COMMIT_MSG@8
PS15, Line 8:
Could you add some notes as the commit message? E.g. overview of this patch,
what problems you have solved, what tests you have done, etc. Just like other
commits do.
http://gerrit.cloudera.org:8080/#/c/18977/15/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/18977/15/bin/impala-config.sh@249
PS15, Line 249: \export APACHE_HADOOP_VERSION=3.3.6
Redundant backslash "\"
http://gerrit.cloudera.org:8080/#/c/18977/15/bin/impala-config.sh@358
PS15, Line 358: USE_APACHE
nit: Can we use a more concrete name, e.g. USE_APACHE_BUILDS,
USE_APACHE_BINARIES, or USE_APACHE_COMPONENTS ?
http://gerrit.cloudera.org:8080/#/c/18977/15/java/TableFlattener/pom.xml
File java/TableFlattener/pom.xml:
http://gerrit.cloudera.org:8080/#/c/18977/15/java/TableFlattener/pom.xml@67
PS15, Line 67:
: commons-lang
: commons-lang
: ${commons-io.version}
:
Is this added for some reason? Could you mention the compilation error without
it in the commit message?
--
To view, visit http://gerrit.cloudera.org:8080/18977
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8730dd182b367c9daa94303937ad249db72b1399
Gerrit-Change-Number: 18977
Gerrit-PatchSet: 15
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Comment-Date: Mon, 22 Apr 2024 11:02:37 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12973,IMPALA-11491,IMPALA-12651: Support BINARY nested in complex types in select list
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/21269 ) Change subject: IMPALA-12973,IMPALA-11491,IMPALA-12651: Support BINARY nested in complex types in select list .. Patch Set 7: (2 comments) http://gerrit.cloudera.org:8080/#/c/21269/7/be/src/exec/iceberg-metadata/iceberg-row-reader.cc File be/src/exec/iceberg-metadata/iceberg-row-reader.cc: http://gerrit.cloudera.org:8080/#/c/21269/7/be/src/exec/iceberg-metadata/iceberg-row-reader.cc@227 PS7, Line 227: RETURN_IF_ERROR(GuardType::create(env, jbuffer, &jbuffer_guard)); if create() returned an error here we'd leak memory because of 'jbuffer', right? http://gerrit.cloudera.org:8080/#/c/21269/7/testdata/workloads/functional-query/queries/QueryTest/iceberg-metadata-tables.test File testdata/workloads/functional-query/queries/QueryTest/iceberg-metadata-tables.test: http://gerrit.cloudera.org:8080/#/c/21269/7/testdata/workloads/functional-query/queries/QueryTest/iceberg-metadata-tables.test@804 PS7, Line 804: select data_file from functional_parquet.iceberg_query_metadata.entries; The query should go into the QUERY section -- To view, visit http://gerrit.cloudera.org:8080/21269 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b1d7fa332a901f05a46e0199e13fb841d2687c2 Gerrit-Change-Number: 21269 Gerrit-PatchSet: 7 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Comment-Date: Mon, 22 Apr 2024 10:46:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13000: Document OPTIMIZE TABLE
Zoltan Borok-Nagy has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21320 ) Change subject: IMPALA-13000: Document OPTIMIZE TABLE .. IMPALA-13000: Document OPTIMIZE TABLE Document OPTIMIZE TABLE syntax and behaviour. Testing: - built docs locally Change-Id: I851669686ed4da610dcac97c9b88ff23b0a4a647 Reviewed-on: http://gerrit.cloudera.org:8080/21320 Tested-by: Impala Public Jenkins Reviewed-by: Zoltan Borok-Nagy Reviewed-by: Daniel Becker --- M docs/topics/impala_iceberg.xml 1 file changed, 47 insertions(+), 0 deletions(-) Approvals: Impala Public Jenkins: Verified Zoltan Borok-Nagy: Looks good to me, but someone else must approve Daniel Becker: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/21320 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I851669686ed4da610dcac97c9b88ff23b0a4a647 Gerrit-Change-Number: 21320 Gerrit-PatchSet: 4 Gerrit-Owner: Noemi Pap-Takacs Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-13000: Document OPTIMIZE TABLE
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/21320 ) Change subject: IMPALA-13000: Document OPTIMIZE TABLE .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21320 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I851669686ed4da610dcac97c9b88ff23b0a4a647 Gerrit-Change-Number: 21320 Gerrit-PatchSet: 3 Gerrit-Owner: Noemi Pap-Takacs Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 22 Apr 2024 10:40:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13000: Document OPTIMIZE TABLE
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/21320 ) Change subject: IMPALA-13000: Document OPTIMIZE TABLE .. Patch Set 3: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21320 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I851669686ed4da610dcac97c9b88ff23b0a4a647 Gerrit-Change-Number: 21320 Gerrit-PatchSet: 3 Gerrit-Owner: Noemi Pap-Takacs Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 22 Apr 2024 10:39:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13000: Document OPTIMIZE TABLE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21320 ) Change subject: IMPALA-13000: Document OPTIMIZE TABLE .. Patch Set 3: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/762/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/21320 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I851669686ed4da610dcac97c9b88ff23b0a4a647 Gerrit-Change-Number: 21320 Gerrit-PatchSet: 3 Gerrit-Owner: Noemi Pap-Takacs Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 22 Apr 2024 09:42:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13000: Document OPTIMIZE TABLE
Noemi Pap-Takacs has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/21320 ) Change subject: IMPALA-13000: Document OPTIMIZE TABLE .. IMPALA-13000: Document OPTIMIZE TABLE Document OPTIMIZE TABLE syntax and behaviour. Testing: - built docs locally Change-Id: I851669686ed4da610dcac97c9b88ff23b0a4a647 --- M docs/topics/impala_iceberg.xml 1 file changed, 47 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/21320/3 -- To view, visit http://gerrit.cloudera.org:8080/21320 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I851669686ed4da610dcac97c9b88ff23b0a4a647 Gerrit-Change-Number: 21320 Gerrit-PatchSet: 3 Gerrit-Owner: Noemi Pap-Takacs Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-13000: Document OPTIMIZE TABLE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21320 ) Change subject: IMPALA-13000: Document OPTIMIZE TABLE .. Patch Set 3: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/762/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/21320 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I851669686ed4da610dcac97c9b88ff23b0a4a647 Gerrit-Change-Number: 21320 Gerrit-PatchSet: 3 Gerrit-Owner: Noemi Pap-Takacs Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 22 Apr 2024 09:35:29 + Gerrit-HasComments: No
