[Impala-ASF-CR] IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19052 ) Change subject: IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8653/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/19052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f1b470f40e0aaf891c9f3f327af393b2f9c74bc Gerrit-Change-Number: 19052 Gerrit-PatchSet: 6 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Tue, 04 Oct 2022 01:47:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19052 ) Change subject: IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8652/ -- To view, visit http://gerrit.cloudera.org:8080/19052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f1b470f40e0aaf891c9f3f327af393b2f9c74bc Gerrit-Change-Number: 19052 Gerrit-PatchSet: 4 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Tue, 04 Oct 2022 01:21:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19052 ) Change subject: IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11523/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f1b470f40e0aaf891c9f3f327af393b2f9c74bc Gerrit-Change-Number: 19052 Gerrit-PatchSet: 6 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Tue, 04 Oct 2022 01:03:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19052 ) Change subject: IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11522/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f1b470f40e0aaf891c9f3f327af393b2f9c74bc Gerrit-Change-Number: 19052 Gerrit-PatchSet: 5 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Tue, 04 Oct 2022 00:51:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala
Yu-Wen Lai has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/19052 ) Change subject: IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala .. IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala In this patch, we use TUpdateCatalogRequest to refresh metadata after 'LOAD DATA' instead of TResetMetadataRequest so that we can reuse the code for 'INSERT' statements. It will fire an insert event just same as what we did for 'INSERT' statements. We also fix the inconsistent indentation in event_processor_utils.py. Testing: - Run existing test_load.py - Added test_load_data_from_impala() in test_event_processing.py Change-Id: I7f1b470f40e0aaf891c9f3f327af393b2f9c74bc --- M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M common/thrift/Frontend.thrift M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/metadata/test_event_processing.py M tests/util/event_processor_utils.py 7 files changed, 194 insertions(+), 84 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/19052/6 -- To view, visit http://gerrit.cloudera.org:8080/19052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7f1b470f40e0aaf891c9f3f327af393b2f9c74bc Gerrit-Change-Number: 19052 Gerrit-PatchSet: 6 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yu-Wen Lai
[Impala-ASF-CR] IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19052 ) Change subject: IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/19052/5/tests/metadata/test_event_processing.py File tests/metadata/test_event_processing.py: http://gerrit.cloudera.org:8080/#/c/19052/5/tests/metadata/test_event_processing.py@380 PS5, Line 380: p flake8: F821 undefined name 'pytest' -- To view, visit http://gerrit.cloudera.org:8080/19052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f1b470f40e0aaf891c9f3f327af393b2f9c74bc Gerrit-Change-Number: 19052 Gerrit-PatchSet: 5 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Tue, 04 Oct 2022 00:32:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala
Yu-Wen Lai has posted comments on this change. (
http://gerrit.cloudera.org:8080/19052 )
Change subject: IMPALA-8592: Add support for insert events for 'LOAD DATA'
statements from Impala
..
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19052/4/tests/metadata/test_event_processing.py
File tests/metadata/test_event_processing.py:
http://gerrit.cloudera.org:8080/#/c/19052/4/tests/metadata/test_event_processing.py@408
PS4, Line 408: into table {1}.{2}".format(staging_dir, unique_database,
tbl_nopart))
> I think we need to mark this test using @pytest.mark.execute_serially. Othe
Thanks for pointing out this mark!
--
To view, visit http://gerrit.cloudera.org:8080/19052
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7f1b470f40e0aaf891c9f3f327af393b2f9c74bc
Gerrit-Change-Number: 19052
Gerrit-PatchSet: 5
Gerrit-Owner: Yu-Wen Lai
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Yu-Wen Lai
Gerrit-Comment-Date: Tue, 04 Oct 2022 00:31:13 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala
Yu-Wen Lai has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/19052 ) Change subject: IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala .. IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala In this patch, we use TUpdateCatalogRequest to refresh metadata after 'LOAD DATA' instead of TResetMetadataRequest so that we can reuse the code for 'INSERT' statements. It will fire an insert event just same as what we did for 'INSERT' statements. We also fix the inconsistent indentation in event_processor_utils.py. Testing: - Run existing test_load.py - Added test_load_data_from_impala() in test_event_processing.py Change-Id: I7f1b470f40e0aaf891c9f3f327af393b2f9c74bc --- M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M common/thrift/Frontend.thrift M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/metadata/test_event_processing.py M tests/util/event_processor_utils.py 7 files changed, 195 insertions(+), 84 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/19052/5 -- To view, visit http://gerrit.cloudera.org:8080/19052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7f1b470f40e0aaf891c9f3f327af393b2f9c74bc Gerrit-Change-Number: 19052 Gerrit-PatchSet: 5 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yu-Wen Lai
[Impala-ASF-CR] IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19052 ) Change subject: IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala .. Patch Set 4: (2 comments) http://gerrit.cloudera.org:8080/#/c/19052/4/tests/metadata/test_event_processing.py File tests/metadata/test_event_processing.py: http://gerrit.cloudera.org:8080/#/c/19052/4/tests/metadata/test_event_processing.py@396 PS4, Line 396: parquet".format(unique_database, tbl_part)) Can we keep the wait of "EventProcessorUtils.wait_for_event_processing(self)"? I'm just concerning the test become flaky if the CREATE_TABLE events come late. http://gerrit.cloudera.org:8080/#/c/19052/4/tests/metadata/test_event_processing.py@408 PS4, Line 408: assert len(events) == 1 I think we need to mark this test using @pytest.mark.execute_serially. Otherwise, it runs concurrently with other metadata tests, which can be flaky as well. -- To view, visit http://gerrit.cloudera.org:8080/19052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f1b470f40e0aaf891c9f3f327af393b2f9c74bc Gerrit-Change-Number: 19052 Gerrit-PatchSet: 4 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Mon, 03 Oct 2022 23:17:17 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11526: Install en US.UTF-8 locale into docker images
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19080 ) Change subject: IMPALA-11526: Install en_US.UTF-8 locale into docker images .. Patch Set 4: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/19080 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I353f257b3cb6d45f7d0a28f7d5319fdb457e6e3d Gerrit-Change-Number: 19080 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 03 Oct 2022 22:15:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11634: Provide an option to use Java 11 for docker images
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19031 ) Change subject: IMPALA-11634: Provide an option to use Java 11 for docker images .. Patch Set 13: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/19031 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icc1dbd3f6a2279840218dc1da2b60077e211a328 Gerrit-Change-Number: 19031 Gerrit-PatchSet: 13 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 03 Oct 2022 21:56:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11632: Exclude log4j-1.2-api in some Ranger artifacts
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19085 ) Change subject: IMPALA-11632: Exclude log4j-1.2-api in some Ranger artifacts .. IMPALA-11632: Exclude log4j-1.2-api in some Ranger artifacts After RANGER-3498, Ranger's ranger-plugins-audit, ranger-plugins-common start pulling in log4j-1.2-api, which is currently banned by Impala's frontend. To be able to compile Impala after RANGER-3498, this patch excludes log4j-1.2-api when adding those Ranger dependencies mentioned above. Change-Id: I72bf989a4fa04b23daf9093df8764670cc5e974d Reviewed-on: http://gerrit.cloudera.org:8080/19085 Reviewed-by: Joe McDonnell Tested-by: Impala Public Jenkins --- M fe/pom.xml 1 file changed, 14 insertions(+), 0 deletions(-) Approvals: Joe McDonnell: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/19085 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I72bf989a4fa04b23daf9093df8764670cc5e974d Gerrit-Change-Number: 19085 Gerrit-PatchSet: 2 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Yida Wu Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-11632: Exclude log4j-1.2-api in some Ranger artifacts
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19085 ) Change subject: IMPALA-11632: Exclude log4j-1.2-api in some Ranger artifacts .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19085 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I72bf989a4fa04b23daf9093df8764670cc5e974d Gerrit-Change-Number: 19085 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Yida Wu Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 03 Oct 2022 21:55:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11526: Install en US.UTF-8 locale into docker images
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19080 ) Change subject: IMPALA-11526: Install en_US.UTF-8 locale into docker images .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19080 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I353f257b3cb6d45f7d0a28f7d5319fdb457e6e3d Gerrit-Change-Number: 19080 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 03 Oct 2022 21:52:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11610: Pass environment variables into dockerized-impala-run-tests.sh
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19088 ) Change subject: IMPALA-11610: Pass environment variables into dockerized-impala-run-tests.sh .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19088 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I325217c731883c087c724194b45d50b790c7c280 Gerrit-Change-Number: 19088 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 03 Oct 2022 21:38:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11604 Planner changes for CPU usage
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/19033 )
Change subject: IMPALA-11604 Planner changes for CPU usage
..
Patch Set 16:
(9 comments)
http://gerrit.cloudera.org:8080/#/c/19033/16/common/thrift/Frontend.thrift
File common/thrift/Frontend.thrift:
http://gerrit.cloudera.org:8080/#/c/19033/16/common/thrift/Frontend.thrift@740
PS16, Line 740: select
nit: selecting
http://gerrit.cloudera.org:8080/#/c/19033/16/common/thrift/Frontend.thrift@750
PS16, Line 750: amount of data
What's the unit? in bytes?
http://gerrit.cloudera.org:8080/#/c/19033/16/fe/src/main/java/org/apache/impala/planner/PlanFragment.java
File fe/src/main/java/org/apache/impala/planner/PlanFragment.java:
http://gerrit.cloudera.org:8080/#/c/19033/16/fe/src/main/java/org/apache/impala/planner/PlanFragment.java@311
PS16, Line 311: CPU
processing
http://gerrit.cloudera.org:8080/#/c/19033/16/fe/src/main/java/org/apache/impala/planner/ResourceProfile.java
File fe/src/main/java/org/apache/impala/planner/ResourceProfile.java:
http://gerrit.cloudera.org:8080/#/c/19033/16/fe/src/main/java/org/apache/impala/planner/ResourceProfile.java@137
PS16, Line 137: TODO:
Processing cost is already reported. What's the TODO?
http://gerrit.cloudera.org:8080/#/c/19033/16/fe/src/main/java/org/apache/impala/planner/SubplanNode.java
File fe/src/main/java/org/apache/impala/planner/SubplanNode.java:
http://gerrit.cloudera.org:8080/#/c/19033/16/fe/src/main/java/org/apache/impala/planner/SubplanNode.java@106
PS16, Line 106: if (LOG.isTraceEnabled()) {
:
LOG.trace(AutoScaleUtil.getProcessingCostComputationDetails(getDisplayLabel(),
: getChild(0).getCardinality(), 0, getAvgRowSize(),
numInstances_));
: }
:
: long processingCost = Math.max(cardinality_, 0) * ((long)
(getAvgRowSize()))
: / Math.max(numInstances_, 1);
Call AutoScaleUtil.computeAndLogProcessingCost()
http://gerrit.cloudera.org:8080/#/c/19033/16/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/19033/16/fe/src/main/java/org/apache/impala/service/Frontend.java@1728
PS16, Line 1728: cpu usage
processing cost
http://gerrit.cloudera.org:8080/#/c/19033/16/fe/src/main/java/org/apache/impala/service/Frontend.java@1799
PS16, Line 1799: 64*MEGABYTE
Why set this value?
http://gerrit.cloudera.org:8080/#/c/19033/16/fe/src/main/java/org/apache/impala/service/Frontend.java@1846
PS16, Line 1846: poolConfig.getMax_query_mem_limit()
Why set the same value as query memory limit?
http://gerrit.cloudera.org:8080/#/c/19033/16/fe/src/main/java/org/apache/impala/util/AutoScaleUtil.java
File fe/src/main/java/org/apache/impala/util/AutoScaleUtil.java:
http://gerrit.cloudera.org:8080/#/c/19033/16/fe/src/main/java/org/apache/impala/util/AutoScaleUtil.java@39
PS16, Line 39: if (LOG.isTraceEnabled()) {
: LOG.trace(getProcessingCostComputationDetails(
: label, cardinality, exprsCost, avgRowSize,
numInstances));
: }
: return computeProcessingCost(cardinality, exprsCost,
avgRowSize, numInstances);
This function call computeProcessingCost() twice. Could we reorganize the code
to call computeProcessingCost() once?
--
To view, visit http://gerrit.cloudera.org:8080/19033
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If32dc770dfffcdd0be2ba789a7720952c68a
Gerrit-Change-Number: 19033
Gerrit-PatchSet: 16
Gerrit-Owner: Qifan Chen
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Qifan Chen
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Mon, 03 Oct 2022 21:25:50 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19052 ) Change subject: IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11521/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f1b470f40e0aaf891c9f3f327af393b2f9c74bc Gerrit-Change-Number: 19052 Gerrit-PatchSet: 4 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Mon, 03 Oct 2022 20:33:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala
Yu-Wen Lai has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/19052 ) Change subject: IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala .. IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala In this patch, we use TUpdateCatalogRequest to refresh metadata after 'LOAD DATA' instead of TResetMetadataRequest so that we can reuse the code for 'INSERT' statements. It will fire an insert event just same as what we did for 'INSERT' statements. We also fix the inconsistent indentation in event_processor_utils.py. Testing: - Run existing test_load.py - Added test_load_data_from_impala() in test_event_processing.py Change-Id: I7f1b470f40e0aaf891c9f3f327af393b2f9c74bc --- M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M common/thrift/Frontend.thrift M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/metadata/test_event_processing.py M tests/util/event_processor_utils.py 7 files changed, 193 insertions(+), 84 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/19052/4 -- To view, visit http://gerrit.cloudera.org:8080/19052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7f1b470f40e0aaf891c9f3f327af393b2f9c74bc Gerrit-Change-Number: 19052 Gerrit-PatchSet: 4 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yu-Wen Lai
[Impala-ASF-CR] IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19052 ) Change subject: IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8652/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/19052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f1b470f40e0aaf891c9f3f327af393b2f9c74bc Gerrit-Change-Number: 19052 Gerrit-PatchSet: 4 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Mon, 03 Oct 2022 20:13:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10851: Codegen for structs
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/18526 ) Change subject: IMPALA-10851: Codegen for structs .. Patch Set 13: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/18526/12/be/src/runtime/descriptors.cc File be/src/runtime/descriptors.cc: http://gerrit.cloudera.org:8080/#/c/18526/12/be/src/runtime/descriptors.cc@972 PS12, Line 972: case TYPE_ARRAY: // CollectionVal has same memory layout as StringVal. > You're right, I forgot to add it. Done in the new patch. Done -- To view, visit http://gerrit.cloudera.org:8080/18526 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5272c3f095fd9f07877104ee03c8e43d0c4ec0b6 Gerrit-Change-Number: 18526 Gerrit-PatchSet: 13 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 03 Oct 2022 17:08:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11610: Pass environment variables into dockerized-impala-run-tests.sh
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19088 ) Change subject: IMPALA-11610: Pass environment variables into dockerized-impala-run-tests.sh .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11520/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19088 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I325217c731883c087c724194b45d50b790c7c280 Gerrit-Change-Number: 19088 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Mon, 03 Oct 2022 16:57:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala
Yu-Wen Lai has posted comments on this change. ( http://gerrit.cloudera.org:8080/19052 ) Change subject: IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/19052/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19052/1//COMMIT_MSG@16 PS1, Line 16: - Run existing test_load.py > I see. Can we use the hive_client to fetch and verify the INSERT events dir Cool. Let me try that. http://gerrit.cloudera.org:8080/#/c/19052/3/be/src/service/client-request-state.cc File be/src/service/client-request-state.cc: http://gerrit.cloudera.org:8080/#/c/19052/3/be/src/service/client-request-state.cc@2047 PS3, Line 2047: > nit: 2 spaces indent here Ack -- To view, visit http://gerrit.cloudera.org:8080/19052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f1b470f40e0aaf891c9f3f327af393b2f9c74bc Gerrit-Change-Number: 19052 Gerrit-PatchSet: 3 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Mon, 03 Oct 2022 16:57:25 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11632: Exclude log4j-1.2-api in some Ranger artifacts
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19085 ) Change subject: IMPALA-11632: Exclude log4j-1.2-api in some Ranger artifacts .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8651/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19085 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I72bf989a4fa04b23daf9093df8764670cc5e974d Gerrit-Change-Number: 19085 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Yida Wu Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 03 Oct 2022 16:51:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11632: Exclude log4j-1.2-api in some Ranger artifacts
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/19085 ) Change subject: IMPALA-11632: Exclude log4j-1.2-api in some Ranger artifacts .. Patch Set 1: Code-Review+2 This makes sense to me -- To view, visit http://gerrit.cloudera.org:8080/19085 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I72bf989a4fa04b23daf9093df8764670cc5e974d Gerrit-Change-Number: 19085 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Yida Wu Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 03 Oct 2022 16:50:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11632: Exclude log4j-1.2-api in some Ranger artifacts
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/19085 ) Change subject: IMPALA-11632: Exclude log4j-1.2-api in some Ranger artifacts .. Patch Set 1: According to https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/17736/, this patch passed Impala's core tests. -- To view, visit http://gerrit.cloudera.org:8080/19085 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I72bf989a4fa04b23daf9093df8764670cc5e974d Gerrit-Change-Number: 19085 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Yida Wu Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 03 Oct 2022 16:47:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11526: Install en US.UTF-8 locale into docker images
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19080 ) Change subject: IMPALA-11526: Install en_US.UTF-8 locale into docker images .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8650/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/19080 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I353f257b3cb6d45f7d0a28f7d5319fdb457e6e3d Gerrit-Change-Number: 19080 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 03 Oct 2022 16:47:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11610: Pass environment variables into dockerized-impala-run-tests.sh
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19088 ) Change subject: IMPALA-11610: Pass environment variables into dockerized-impala-run-tests.sh .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11519/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19088 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I325217c731883c087c724194b45d50b790c7c280 Gerrit-Change-Number: 19088 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Mon, 03 Oct 2022 16:43:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11526: Install en US.UTF-8 locale into docker images
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19080 ) Change subject: IMPALA-11526: Install en_US.UTF-8 locale into docker images .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11518/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19080 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I353f257b3cb6d45f7d0a28f7d5319fdb457e6e3d Gerrit-Change-Number: 19080 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 03 Oct 2022 16:43:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11634: Provide an option to use Java 11 for docker images
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19031 ) Change subject: IMPALA-11634: Provide an option to use Java 11 for docker images .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11517/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19031 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icc1dbd3f6a2279840218dc1da2b60077e211a328 Gerrit-Change-Number: 19031 Gerrit-PatchSet: 12 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Mon, 03 Oct 2022 16:42:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8770: Support building Docker images on Redhat-based distributions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19006 ) Change subject: IMPALA-8770: Support building Docker images on Redhat-based distributions .. Patch Set 13: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11516/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19006 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibaff2560ef971ac2c2231a8e43921164ea1d2f4d Gerrit-Change-Number: 19006 Gerrit-PatchSet: 13 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 03 Oct 2022 16:42:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11610: Pass environment variables into dockerized-impala-run-tests.sh
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/19088 ) Change subject: IMPALA-11610: Pass environment variables into dockerized-impala-run-tests.sh .. Patch Set 1: (3 comments) http://gerrit.cloudera.org:8080/#/c/19088/1/bin/jenkins/dockerized-impala-preserve-vars.py File bin/jenkins/dockerized-impala-preserve-vars.py: http://gerrit.cloudera.org:8080/#/c/19088/1/bin/jenkins/dockerized-impala-preserve-vars.py@34 PS1, Line 34: def main(): > flake8: E302 expected 2 blank lines, found 1 Done http://gerrit.cloudera.org:8080/#/c/19088/1/bin/jenkins/dockerized-impala-preserve-vars.py@48 PS1, Line 48: ) > flake8: E501 line too long (91 > 90 characters) Done http://gerrit.cloudera.org:8080/#/c/19088/1/bin/jenkins/dockerized-impala-preserve-vars.py@54 PS1, Line 54: if __name__ == "__main__": main() > flake8: E305 expected 2 blank lines after class or function definition, fou Done -- To view, visit http://gerrit.cloudera.org:8080/19088 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I325217c731883c087c724194b45d50b790c7c280 Gerrit-Change-Number: 19088 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Mon, 03 Oct 2022 16:28:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11610: Pass environment variables into dockerized-impala-run-tests.sh
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19088 to look at the new patch set (#2). Change subject: IMPALA-11610: Pass environment variables into dockerized-impala-run-tests.sh .. IMPALA-11610: Pass environment variables into dockerized-impala-run-tests.sh Because dockerized-impala-bootstrap-test.sh does a relogin while calling dockerized-impala-run-tests.sh, the environment is not preserved. This adds a script dockerized-impala-preserve-vars.py that takes a list of environment variables to preserve and appends export statements to bin/impala-config-local.sh. Since dockerized-impala-run-tests.sh sourcs bin/impala-config.sh, these variables will be carried into the test execution. This starts by adding environment variables used by upstream Jenkin's ubuntu-16.04-dockerized-tests. Jenkins jobs can also call dockerized-impala-preserve-vars.py directly. Testing: - Hand tested the preservation script - Verified ubuntu-16.04-dockerized-tests now respected EE_TEST argument. Change-Id: I325217c731883c087c724194b45d50b790c7c280 --- M bin/jenkins/dockerized-impala-bootstrap-and-test.sh A bin/jenkins/dockerized-impala-preserve-vars.py 2 files changed, 63 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/19088/2 -- To view, visit http://gerrit.cloudera.org:8080/19088 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I325217c731883c087c724194b45d50b790c7c280 Gerrit-Change-Number: 19088 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11526: Install en US.UTF-8 locale into docker images
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19080 to look at the new patch set (#3). Change subject: IMPALA-11526: Install en_US.UTF-8 locale into docker images .. IMPALA-11526: Install en_US.UTF-8 locale into docker images In IMPALA-11492, ExprTest.Utf8MaskTest was failing on some configurations because the en_US.UTF-8 was missing. Since the Docker images don't contain en_US.UTF-8, they are subject to the same bug. This was confirmed by adding tests cases to the test_utf8_strings.py end-to-end test and running it in the dockerized tests. This add the appropriate language pack to the list of packages installed for the Docker build. Testing: - This adds end-to-end tests to test_utf8_strings.py covering the same cases that were failing in ExprTest.Utf8MaskTest. They failed without the added languages packs, and now succeed. Change-Id: I353f257b3cb6d45f7d0a28f7d5319fdb457e6e3d --- M docker/install_os_packages.sh M testdata/workloads/functional-query/queries/QueryTest/utf8-string-functions.test 2 files changed, 82 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/19080/3 -- To view, visit http://gerrit.cloudera.org:8080/19080 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I353f257b3cb6d45f7d0a28f7d5319fdb457e6e3d Gerrit-Change-Number: 19080 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8770: Support building Docker images on Redhat-based distributions
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19006 to look at the new patch set (#13). Change subject: IMPALA-8770: Support building Docker images on Redhat-based distributions .. IMPALA-8770: Support building Docker images on Redhat-based distributions Currently, Impala supports building and testing Docker images on Ubuntu. This extends that same support to Redhat-based distributions: 1. This splits out the Docker build's OS package installation into a separate install_os_packages.sh script. This script detects the OS and calls apt or yum as appropriate. The script takes the argument --install-debug-tools, which installs extra tools like iproute2 and ping. This defaults to true for debug images and false for release images. 2. This modifies daemon_entrypoint.sh to detect the OS and set LD_LIBRARY_PATH appropriate to account for different locations of Java. 3. This modifies docker/setup_build_context.py to handle different locations of libkudu_client.so and add extra sanity checks on various libraries found via globs. 4. This modifies bin/jenkins/dockerized-*.sh test infrastructure to be able to install docker on either Ubuntu or Redhat. It also changes the exit logic to collect the container logs. Developers can override the base image for Redhat 7 and Redhat 8 builds via the IMPALA_REDHAT7_DOCKER_BASE and IMPALA_REDHAT8_DOCKER_BASE environment variables. These default to open source Redhat equivalents (Centos 7.9 and Rocky 8.5 respectively), but they are also known to work with Redhat UBI images. Testing: - Ran dockerised testing on Rocky 8.5 via the rocky-8.5-dockerised-tests job. - Ran GVO - Ran a Docker build on Centos7 with UBI7 as the base image Change-Id: Ibaff2560ef971ac2c2231a8e43921164ea1d2f4d --- M bin/impala-config.sh M bin/jenkins/dockerized-impala-bootstrap-and-test.sh M bin/jenkins/dockerized-impala-run-tests.sh A bin/jenkins/install_docker.sh M docker/CMakeLists.txt M docker/admissiond/Dockerfile M docker/catalogd/Dockerfile M docker/daemon_entrypoint.sh M docker/impala_base/Dockerfile M docker/impala_profile_tool/Dockerfile M docker/impalad_coord_exec/Dockerfile M docker/impalad_coordinator/Dockerfile M docker/impalad_executor/Dockerfile A docker/install_os_packages.sh M docker/setup_build_context.py M docker/statestored/Dockerfile 16 files changed, 471 insertions(+), 92 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/19006/13 -- To view, visit http://gerrit.cloudera.org:8080/19006 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibaff2560ef971ac2c2231a8e43921164ea1d2f4d Gerrit-Change-Number: 19006 Gerrit-PatchSet: 13 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11610: Pass environment variables into dockerized-impala-run-tests.sh
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19088 Change subject: IMPALA-11610: Pass environment variables into dockerized-impala-run-tests.sh .. IMPALA-11610: Pass environment variables into dockerized-impala-run-tests.sh Because dockerized-impala-bootstrap-test.sh does a relogin while calling dockerized-impala-run-tests.sh, the environment is not preserved. This adds a script dockerized-impala-preserve-vars.py that takes a list of environment variables to preserve and appends export statements to bin/impala-config-local.sh. Since dockerized-impala-run-tests.sh sourcs bin/impala-config.sh, these variables will be carried into the test execution. This starts by adding environment variables used by upstream Jenkin's ubuntu-16.04-dockerized-tests. Jenkins jobs can also call dockerized-impala-preserve-vars.py directly. Testing: - Hand tested the preservation script - Verified ubuntu-16.04-dockerized-tests now respected EE_TEST argument. Change-Id: I325217c731883c087c724194b45d50b790c7c280 --- M bin/jenkins/dockerized-impala-bootstrap-and-test.sh A bin/jenkins/dockerized-impala-preserve-vars.py 2 files changed, 61 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/19088/1 -- To view, visit http://gerrit.cloudera.org:8080/19088 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I325217c731883c087c724194b45d50b790c7c280 Gerrit-Change-Number: 19088 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[Impala-ASF-CR] IMPALA-11634: Provide an option to use Java 11 for docker images
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19031 to look at the new patch set (#12). Change subject: IMPALA-11634: Provide an option to use Java 11 for docker images .. IMPALA-11634: Provide an option to use Java 11 for docker images Currently, Docker images install Java 8 for Impala's use. This adds the IMPALA_DOCKER_USE_JAVA11 environment variable. When set to true, this installs Java 11 rather than Java 8. It defaults to false. The daemon_entrypoint.sh script is modified to detect Java 11 correctly. As a workaround for IMPALA-11260, this appends a list of "--add-opens" statements to JAVA_TOOL_OPTIONS when running with Java 11. Testing: - Ran a set of dockerized tests wi Change-Id: Icc1dbd3f6a2279840218dc1da2b60077e211a328 --- M bin/impala-config.sh M bin/jenkins/dockerized-impala-bootstrap-and-test.sh M docker/CMakeLists.txt M docker/daemon_entrypoint.sh M docker/install_os_packages.sh 5 files changed, 92 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/19031/12 -- To view, visit http://gerrit.cloudera.org:8080/19031 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icc1dbd3f6a2279840218dc1da2b60077e211a328 Gerrit-Change-Number: 19031 Gerrit-PatchSet: 12 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-11610: Pass environment variables into dockerized-impala-run-tests.sh
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19088 ) Change subject: IMPALA-11610: Pass environment variables into dockerized-impala-run-tests.sh .. Patch Set 1: (3 comments) http://gerrit.cloudera.org:8080/#/c/19088/1/bin/jenkins/dockerized-impala-preserve-vars.py File bin/jenkins/dockerized-impala-preserve-vars.py: http://gerrit.cloudera.org:8080/#/c/19088/1/bin/jenkins/dockerized-impala-preserve-vars.py@34 PS1, Line 34: def main(): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/19088/1/bin/jenkins/dockerized-impala-preserve-vars.py@48 PS1, Line 48: ) flake8: E501 line too long (91 > 90 characters) http://gerrit.cloudera.org:8080/#/c/19088/1/bin/jenkins/dockerized-impala-preserve-vars.py@54 PS1, Line 54: if __name__ == "__main__": main() flake8: E305 expected 2 blank lines after class or function definition, found 1 -- To view, visit http://gerrit.cloudera.org:8080/19088 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I325217c731883c087c724194b45d50b790c7c280 Gerrit-Change-Number: 19088 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 03 Oct 2022 16:24:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11604 Planner changes for CPU usage
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19033 ) Change subject: IMPALA-11604 Planner changes for CPU usage .. Patch Set 16: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11515/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19033 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If32dc770dfffcdd0be2ba789a7720952c68a Gerrit-Change-Number: 19033 Gerrit-PatchSet: 16 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 03 Oct 2022 15:56:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11604 Planner changes for CPU usage
Qifan Chen has uploaded a new patch set (#16). ( http://gerrit.cloudera.org:8080/19033 ) Change subject: IMPALA-11604 Planner changes for CPU usage .. IMPALA-11604 Planner changes for CPU usage This patch augments IMPALA-10992 by establishing an infrastructure to allow the weighted total amount of data to process per instance to be used as a new factor in the definition and selection of an executor group. In this patch the weight component is set to 1. The weighted amount of data processed is the sum of that in every fragment in the query to reflect the worse case of minimal parallel execution among fragments. A fragment's total is the sum of that of every node in the fragment to reflect these nodes are executed in a single thread. For each node, the weighted amount of data processed is computed with a general formula as follows. D = (I * C * W) / N where D is the weighted amount of data processed I is input cardinality C is expression evaluation cost per row, set to 1 W is average row size N is number of instances A description of the computation for each kind of plan node is given below. 1. Aggregation node: C and W are the sum of the costs and partial row widths for each AggregateInfo object. 2. AnalyticEval node: C is sum of the evaluation costs for analytic functions, partition by equal and order by equal predicate; 3. CardinalityCheck node: Both C and I are 1; 4. DataSource scan node: C is computed from a subset of the selection predicates excluding data source accepted predicates; 5. EmptySet node: I is 0; 6. Exchange node: A modification of the general formula when in broadcast mode: D = D * number of receivers; 7. Hash join node: probe side = (I0 * C(equi-join predicate) * W) / N + (output cardinality * C(other join predicate) * W) / N build side = (I1 * C(equi-join predicate) * W) / N 8. Hbase scan node: N is 1 9. Hdfs and Kudu scan node: N is mt_dop when query option mt_dop >= 1, otherwise N is number of nodes * max scan threads; 10. Nested loop join node: When the right child is not a SingularRowSrc node: probe side = (I0 * C(equi-join predicate) * W) / N + (output cardinality * C(other join predicate) * W) / N build side = (I1 * C(equi-join predicate) * W) / N When the right child is a SingularRowSrc node: probe side = (I0 * W) / N build side = I0 * ((I1 * W) / N) 11. Select node: Use the general formula; 12. SingularRowSrc node: Since the node is involved once per input in nested loop join, the contribution of this node is computed in nested loop join; 13. Sort node: C is the evalation cost for the sort expression and W is the width of the intermediate tuple being sorted; 14. Subplan node: C is 1. I is the multiplication of the cardinality of the left and the right child; 15. Union node: C is the cost of materializing rows from all non pass-through children. W is the width of all non pass-through children; 16. Unnest node: I is the cardinality of the containing subplan node and C is 1. Testing: 1. Unit test by examining processing cost computed for all plan nodes in all fragments for a small set of queries; 2. Core tests. Change-Id: If32dc770dfffcdd0be2ba789a7720952c68a --- M common/thrift/Frontend.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/analysis/SortInfo.java M fe/src/main/java/org/apache/impala/planner/AggregationNode.java M fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java M fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M fe/src/main/java/org/apache/impala/planner/EmptySetNode.java M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java M fe/src/main/java/org/apache/impala/planner/PlanFragment.java M fe/src/main/java/org/apache/impala/planner/PlanNode.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/planner/ResourceProfile.java M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java M fe/src/main/java/org/apache/impala/planner/ScanNode.java M fe/src/main/java/org/apache/impala/planner/SelectNode.java M fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java M fe/src/main/java/org/apache/impala/planner/SortNode.java M
[Impala-ASF-CR] IMPALA-11604 Planner changes for CPU usage
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/19033 ) Change subject: IMPALA-11604 Planner changes for CPU usage .. Patch Set 16: Improved formula for hash and nested joins. -- To view, visit http://gerrit.cloudera.org:8080/19033 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If32dc770dfffcdd0be2ba789a7720952c68a Gerrit-Change-Number: 19033 Gerrit-PatchSet: 16 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 03 Oct 2022 15:35:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11604 Planner changes for CPU usage
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19033 ) Change subject: IMPALA-11604 Planner changes for CPU usage .. Patch Set 15: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11514/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19033 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If32dc770dfffcdd0be2ba789a7720952c68a Gerrit-Change-Number: 19033 Gerrit-PatchSet: 15 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 03 Oct 2022 15:08:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11604 Planner changes for CPU usage
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/19033 ) Change subject: IMPALA-11604 Planner changes for CPU usage .. Patch Set 15: (1 comment) Refactoring on processing cost by moving the actual computation into AutoScaleUtil. http://gerrit.cloudera.org:8080/#/c/19033/14/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java File fe/src/main/java/org/apache/impala/planner/HashJoinNode.java: http://gerrit.cloudera.org:8080/#/c/19033/14/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@329 PS14, Line 329: // Almost all resource consumption is in the build, or shared between the build and > line too long (91 > 90) Done -- To view, visit http://gerrit.cloudera.org:8080/19033 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If32dc770dfffcdd0be2ba789a7720952c68a Gerrit-Change-Number: 19033 Gerrit-PatchSet: 15 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 03 Oct 2022 14:48:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11604 Planner changes for CPU usage
Qifan Chen has uploaded a new patch set (#15). ( http://gerrit.cloudera.org:8080/19033 ) Change subject: IMPALA-11604 Planner changes for CPU usage .. IMPALA-11604 Planner changes for CPU usage This patch augments IMPALA-10992 by establishing an infrastructure to allow the weighted total amount of data to process per instance to be used as a new factor in the definition and selection of an executor group. In this patch the weight component is set to 1. The weighted amount of data processed is the sum of that in every fragment in the query to reflect the worse case of minimal parallel execution among fragments. A fragment's total is the sum of that of every node in the fragment to reflect these nodes are executed in a single thread. For each node, the weighted amount of data processed is computed with a general formula as follows. D = (I * C * W) / N where D is the weighted amount of data processed I is input cardinality C is expression evaluation cost per row, set to 1 W is average row size N is number of instances A description of the computation for each kind of plan node is given below. 1. Aggregation node: C and W are the sum of the costs and partial row widths for each AggregateInfo object. 2. AnalyticEval node: C is sum of the evaluation costs for analytic functions, partition by equal and order by equal predicate; 3. CardinalityCheck node: Both C and I are 1; 4. DataSource scan node: C is computed from a subset of the selection predicates excluding data source accepted predicates; 5. EmptySet node: I is 0; 6. Exchange node: A modification of the general formula when in broadcast mode: D = (I * C * W / N) * number of receivers; 7. Hash join node: C is sum of the evaluation cost for equi-join predicate and for other join predicate, for both probe and build side; 8. Hbase scan node: N is 1 9. Hdfs and Kudu scan node: N is mt_dop when query option mt_dop >= 1, otherwise N is number of nodes * max scan threads; 10. Nested loop join node: When the right child is not a SingularRowSrc node, C is sum of the evaluation cost for equi-join predicate and for other join predicate, for both probe and build side. When the right child is a SingularRowSrc node, the cost for build side is multiplied by the number of probes; 11. Select node: Use the general formula; 12. SingularRowSrc node: Since the node is involved once per input in nested loop join, the contribution of this node is computed in nested loop join; 13. Sort node: C is the evalation cost for the sort expression and W is the width of the intermediate tuple being sorted; 14. Subplan node: C is 1. I is the multiplication of the cardinality of the left and the right child; 15. Union node: C is the cost of materializing rows from all non pass-through children. W is the width of all non pass-through children; 16. Unnest node: I is the cardinality of the containing subplan node and C is 1. Testing: [TBD] Change-Id: If32dc770dfffcdd0be2ba789a7720952c68a --- M common/thrift/Frontend.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/analysis/SortInfo.java M fe/src/main/java/org/apache/impala/planner/AggregationNode.java M fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java M fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M fe/src/main/java/org/apache/impala/planner/EmptySetNode.java M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java M fe/src/main/java/org/apache/impala/planner/PlanFragment.java M fe/src/main/java/org/apache/impala/planner/PlanNode.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/planner/ResourceProfile.java M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java M fe/src/main/java/org/apache/impala/planner/ScanNode.java M fe/src/main/java/org/apache/impala/planner/SelectNode.java M fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java M fe/src/main/java/org/apache/impala/planner/SortNode.java M fe/src/main/java/org/apache/impala/planner/SubplanNode.java M fe/src/main/java/org/apache/impala/planner/UnionNode.java M fe/src/main/java/org/apache/impala/planner/UnnestNode.java M fe/src/main/java/org/apache/impala/service/Frontend.java A
[Impala-ASF-CR] IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19048 ) Change subject: IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value .. Patch Set 8: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8649/ -- To view, visit http://gerrit.cloudera.org:8080/19048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I952377ed14eba26e3774e7776eb81a95d1d8e76f Gerrit-Change-Number: 19048 Gerrit-PatchSet: 8 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Mon, 03 Oct 2022 14:39:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11482: Alter Table Execute Rollback for Iceberg tables.
Gergely Fürnstáhl has posted comments on this change. (
http://gerrit.cloudera.org:8080/19002 )
Change subject: IMPALA-11482: Alter Table Execute Rollback for Iceberg tables.
..
Patch Set 7:
(7 comments)
Generally looks good to me, added a few small comments
http://gerrit.cloudera.org:8080/#/c/19002/7/common/thrift/JniCatalog.thrift
File common/thrift/JniCatalog.thrift:
http://gerrit.cloudera.org:8080/#/c/19002/7/common/thrift/JniCatalog.thrift@416
PS7, Line 416: TExecuteExpireSnapshotsParams
I think this should be
TAlterTableExecuteExpireSnapshotsParams
http://gerrit.cloudera.org:8080/#/c/19002/7/common/thrift/JniCatalog.thrift@505
PS7, Line 505: // Parameters for ALTER TABLE EXECUTE EXPIRE_SNAPSHOTS
nit: // Parameters for ALTER TABLE EXECUTE ... operations.
http://gerrit.cloudera.org:8080/#/c/19002/7/fe/src/main/java/org/apache/impala/analysis/AlterTableExecuteRollbackStmt.java
File
fe/src/main/java/org/apache/impala/analysis/AlterTableExecuteRollbackStmt.java:
http://gerrit.cloudera.org:8080/#/c/19002/7/fe/src/main/java/org/apache/impala/analysis/AlterTableExecuteRollbackStmt.java@74
PS7, Line 74: if (fnParamValue_.getType().isIntegerType()) {
: // Parameter is a snapshot id
: kind_ = TRollbackType.VERSION_ID;
: snapshotVersion_ = fnParamValue_.evalToInteger(analyzer,
USAGE);
: if (snapshotVersion_ < 0) {
: throw new AnalysisException("Invalid version number has
been given to " + USAGE
: + ": " + snapshotVersion_);
: }
: LOG.debug(USAGE + " version: " + snapshotVersion_);
: return;
: } else if (fnParamValue_.getType().isDecimal()) {
: throw new AnalysisException(USAGE
: + " must be an integer type or a
timestamp, but is '"
: + fnParamValue_.getType() + "': EXECUTE " + toSql());
: }
: }
:
: // Parameter is a timestamp
: kind_ = TRollbackType.TIME_ID;
: analyzeOlderThan(analyzer, USAGE);
The control flow is kinda messy here. Is being decimal important at all?
Falling back to parsing it as timestamp could produce hard to understand error
messages too.
Maybe something like this would be easier to follow/maintain/extend it later:
if(literal && integer) { ... }
else if(covertibleToTimestamp()) { ... } // this could be extracted from
analyzeOlderThan
else { ... } // not supported, must be integer or timestamp
http://gerrit.cloudera.org:8080/#/c/19002/7/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/19002/7/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1338
PS7, Line 1338: alterTableExecute
Shouldn't be this method be modified to alterTableExecuteExpireSnapshots too?
http://gerrit.cloudera.org:8080/#/c/19002/7/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1343
PS7, Line 1343: Alter table execute has new format.
nit: Maybe this is a bit more telling for debugging purposes.
"Alter table execute statement is not implemented."
http://gerrit.cloudera.org:8080/#/c/19002/7/testdata/workloads/functional-query/queries/QueryTest/iceberg-rollback.test
File
testdata/workloads/functional-query/queries/QueryTest/iceberg-rollback.test:
http://gerrit.cloudera.org:8080/#/c/19002/7/testdata/workloads/functional-query/queries/QueryTest/iceberg-rollback.test@18
PS7, Line 18: ROLLBACK('')
: CATCH
: Invalid TIMESTAMP expression
This is what I meant with weird error message, if I want to rollback to my
id= snapshot, but Impala is telling its not a valid timestamp, hard to find
that the mistake is the added ' '.
http://gerrit.cloudera.org:8080/#/c/19002/7/tests/common/iceberg_test_suite.py
File tests/common/iceberg_test_suite.py:
http://gerrit.cloudera.org:8080/#/c/19002/7/tests/common/iceberg_test_suite.py@105
PS7, Line 105: parts = raw_string.split("\t")
Could you add here the expected format of the raw string in a comment?
--
To view, visit http://gerrit.cloudera.org:8080/19002
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic74913d3b81103949ffb5eef7cc936303494f8b9
Gerrit-Change-Number: 19002
Gerrit-PatchSet: 7
Gerrit-Owner: Andrew Sherman
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Gergely Fürnstáhl
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tamas Mate
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Mon, 03 Oct 2022 13:53:50 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10851: Codegen for structs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18526 ) Change subject: IMPALA-10851: Codegen for structs .. Patch Set 13: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11513/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18526 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5272c3f095fd9f07877104ee03c8e43d0c4ec0b6 Gerrit-Change-Number: 18526 Gerrit-PatchSet: 13 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 03 Oct 2022 09:47:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19048 ) Change subject: IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I952377ed14eba26e3774e7776eb81a95d1d8e76f Gerrit-Change-Number: 19048 Gerrit-PatchSet: 8 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Mon, 03 Oct 2022 09:39:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19048 ) Change subject: IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8649/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I952377ed14eba26e3774e7776eb81a95d1d8e76f Gerrit-Change-Number: 19048 Gerrit-PatchSet: 8 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Mon, 03 Oct 2022 09:39:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10851: Codegen for structs
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/18526 ) Change subject: IMPALA-10851: Codegen for structs .. Patch Set 13: (2 comments) http://gerrit.cloudera.org:8080/#/c/18526/12/be/src/runtime/descriptors.cc File be/src/runtime/descriptors.cc: http://gerrit.cloudera.org:8080/#/c/18526/12/be/src/runtime/descriptors.cc@972 PS12, Line 972: case TYPE_ARRAY: // CollectionVal has same memory layout as StringVal. > My point in an earlier comment was that https://gerrit.cloudera.org/c/18736 You're right, I forgot to add it. Done in the new patch. Also added it in codegen-anyval.cc. http://gerrit.cloudera.org:8080/#/c/18526/10/tests/query_test/test_nested_types.py File tests/query_test/test_nested_types.py: http://gerrit.cloudera.org:8080/#/c/18526/10/tests/query_test/test_nested_types.py@143 PS10, Line 143: def test_struct_in_select_list(self, vector): > Reversing order seems fine, but I'd add a comment why. I'd also add the oth Done -- To view, visit http://gerrit.cloudera.org:8080/18526 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5272c3f095fd9f07877104ee03c8e43d0c4ec0b6 Gerrit-Change-Number: 18526 Gerrit-PatchSet: 13 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 03 Oct 2022 09:27:13 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10851: Codegen for structs
Daniel Becker has uploaded a new patch set (#13). ( http://gerrit.cloudera.org:8080/18526 ) Change subject: IMPALA-10851: Codegen for structs .. IMPALA-10851: Codegen for structs IMPALA-9495 added support for struct types in SELECT lists but only with codegen turned off. This commit implements codegen for struct types. To facilitate this, code generation for reading and writing 'AnyVal's has been refactored. A new class, 'CodegenAnyValReadWriteInfo' is introduced. This class is an interface between sources and destinations, one of which is an 'AnyVal' object: sources generate an instance of this class and destinations take that instance and use it to write the value. The other side can for example be tuples from which we read (in the case of 'SlotRef') or tuples we write into (in case of materialisation, see Tuple::CodegenMaterializeExprs()). The main advantage is that sources do not have to know how to write their destinations, only how to read the values (and vice versa). Testing: - enabling tests for structs in select list in codegen mode in tests/query_test/test_nested_types.py Change-Id: I5272c3f095fd9f07877104ee03c8e43d0c4ec0b6 --- M be/src/codegen/CMakeLists.txt A be/src/codegen/codegen-anyval-read-write-info.cc A be/src/codegen/codegen-anyval-read-write-info.h M be/src/codegen/codegen-anyval.cc M be/src/codegen/codegen-anyval.h M be/src/codegen/gen_ir_descriptions.py M be/src/codegen/impala-ir.cc M be/src/codegen/llvm-codegen.cc M be/src/codegen/llvm-codegen.h M be/src/exec/aggregator.cc M be/src/exec/filter-context.cc M be/src/exec/hash-table.cc M be/src/exec/hash-table.h M be/src/exprs/CMakeLists.txt M be/src/exprs/hive-udf-call-ir.cc M be/src/exprs/hive-udf-call.cc M be/src/exprs/hive-udf-call.h M be/src/exprs/kudu-partition-expr.cc A be/src/exprs/scalar-expr-evaluator-ir.cc M be/src/exprs/scalar-expr-evaluator.h M be/src/exprs/slot-ref.cc M be/src/exprs/slot-ref.h M be/src/runtime/descriptors.cc M be/src/runtime/descriptors.h M be/src/runtime/fragment-state.cc M be/src/runtime/fragment-state.h M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/tuple.cc M be/src/udf/udf-internal.h M be/src/udf/udf-ir.cc M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M testdata/workloads/functional-query/queries/QueryTest/nested-struct-in-select-list.test M tests/authorization/test_ranger.py M tests/query_test/test_nested_types.py 35 files changed, 2,201 insertions(+), 808 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/18526/13 -- To view, visit http://gerrit.cloudera.org:8080/18526 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5272c3f095fd9f07877104ee03c8e43d0c4ec0b6 Gerrit-Change-Number: 18526 Gerrit-PatchSet: 13 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19052 ) Change subject: IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/19052/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19052/1//COMMIT_MSG@16 PS1, Line 16: - Run existing test_load.py > I realized that replication cannot be used as a verification of insert even I see. Can we use the hive_client to fetch and verify the INSERT events directly? We use it to fetch the latest event id here: https://github.com/apache/impala/blob/68650057a163ac23e1ca85b7d9d8dbfd975a69ff/tests/util/event_processor_utils.py#L125 Probably we can use get_next_notification() to fetch the INSERT events. http://gerrit.cloudera.org:8080/#/c/19052/3/be/src/service/client-request-state.cc File be/src/service/client-request-state.cc: http://gerrit.cloudera.org:8080/#/c/19052/3/be/src/service/client-request-state.cc@2047 PS3, Line 2047: nit: 2 spaces indent here -- To view, visit http://gerrit.cloudera.org:8080/19052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f1b470f40e0aaf891c9f3f327af393b2f9c74bc Gerrit-Change-Number: 19052 Gerrit-PatchSet: 3 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Mon, 03 Oct 2022 08:27:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11504: Specializing DecimalUtil::GetScaleMultiplier().
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/18861 ) Change subject: IMPALA-11504: Specializing DecimalUtil::GetScaleMultiplier(). .. Patch Set 6: (4 comments) http://gerrit.cloudera.org:8080/#/c/18861/6//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/18861/6//COMMIT_MSG@12 PS6, Line 12: Indicate also here how much the speedup is resulting from this template specialisation. http://gerrit.cloudera.org:8080/#/c/18861/6/be/src/benchmarks/decimal-util-benchmark.cc File be/src/benchmarks/decimal-util-benchmark.cc: http://gerrit.cloudera.org:8080/#/c/18861/6/be/src/benchmarks/decimal-util-benchmark.cc@26 PS6, Line 26: // Machine Info: AMD Ryzen 9 5950X 16-Core Processor I agree with Lipenglin, we should mention that it was done in a Docker container. http://gerrit.cloudera.org:8080/#/c/18861/6/be/src/benchmarks/decimal-util-benchmark.cc@53 PS6, Line 53: static volatile int64_t black_hole = 0; I think in this case 'volatile_var' or something similar would be more descriptive - when assigning to this variable, the reader could have an idea that we have this variable to block the compiler from removing the code. Also, a comment before this variable should describe why we have it. http://gerrit.cloudera.org:8080/#/c/18861/4/be/src/util/decimal-util.h File be/src/util/decimal-util.h: http://gerrit.cloudera.org:8080/#/c/18861/4/be/src/util/decimal-util.h@37 PS4, Line 37: // The scale upper bound for GetScaleMultiplier() : static constexpr int INT32_SCALE_UPPER_BOUND = ColumnType::MAX_DECIMAL4_PRECISION + 1; : // The scale upper bound for GetScaleMultiplier() : static constexpr int INT64_SCALE_UPPER_BOUND = ColumnType::MAX_DECIMAL8_PRECISION + 1; : // The scale upper bound for GetScaleMultiplier() : static constexpr int INT128_SCALE_UPPER_BOUND = ColumnType::MAX_PRECISION + 1; : // The scale upper bound for GetScaleMultiplier() : static constexpr int INT256_SCALE_UPPER_BOUND = 77; > These constants are referenced in decimal-test.cc and decimal-util-benchmar Ok, we can let them stay. -- To view, visit http://gerrit.cloudera.org:8080/18861 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I969e2977d51313e738f72c8246db003ae43a3782 Gerrit-Change-Number: 18861 Gerrit-PatchSet: 6 Gerrit-Owner: Xiang Yang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Xiang Yang Gerrit-Comment-Date: Mon, 03 Oct 2022 07:42:41 + Gerrit-HasComments: Yes
