[Impala-ASF-CR] Prototype to provide an option to install Java 11
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19031 to look at the new patch set (#10). Change subject: Prototype to provide an option to install Java 11 .. Prototype to provide an option to install Java 11 Change-Id: Icc1dbd3f6a2279840218dc1da2b60077e211a328 --- M bin/impala-config.sh M docker/CMakeLists.txt M docker/daemon_entrypoint.sh M docker/install_os_packages.sh 4 files changed, 111 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/19031/10 -- To view, visit http://gerrit.cloudera.org:8080/19031 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icc1dbd3f6a2279840218dc1da2b60077e211a328 Gerrit-Change-Number: 19031 Gerrit-PatchSet: 10 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-11612: fix ORDER BY expression rewrite bug
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19049 ) Change subject: IMPALA-11612: fix ORDER BY expression rewrite bug .. Patch Set 2: (1 comment) Thanks for your contribution! Let's start with adding the commit message. http://gerrit.cloudera.org:8080/#/c/19049/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19049/2//COMMIT_MSG@8 PS2, Line 8: Please write a good, clear commit message, with a short, descriptive title and a message that is exactly long enough to explain what the problem was, and how it was fixed. Each should have 72 or fewer characters if possible. https://cwiki.apache.org/confluence/display/IMPALA/Contributing+to+Impala -- To view, visit http://gerrit.cloudera.org:8080/19049 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I438c61c647e3f99665c94ea15e3a9d98b9ff9912 Gerrit-Change-Number: 19049 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 29 Sep 2022 05:47:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19052 ) Change subject: IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f1b470f40e0aaf891c9f3f327af393b2f9c74bc Gerrit-Change-Number: 19052 Gerrit-PatchSet: 1 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Thu, 29 Sep 2022 05:19:19 + Gerrit-HasComments: No
[Impala-ASF-CR] Prototype to provide an option to install Java 11
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19031 ) Change subject: Prototype to provide an option to install Java 11 .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11482/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19031 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icc1dbd3f6a2279840218dc1da2b60077e211a328 Gerrit-Change-Number: 19031 Gerrit-PatchSet: 9 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Thu, 29 Sep 2022 05:04:28 + Gerrit-HasComments: No
[Impala-ASF-CR] Prototype to provide an option to install Java 11
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19031 to look at the new patch set (#9). Change subject: Prototype to provide an option to install Java 11 .. Prototype to provide an option to install Java 11 Change-Id: Icc1dbd3f6a2279840218dc1da2b60077e211a328 --- M bin/impala-config.sh M docker/CMakeLists.txt M docker/daemon_entrypoint.sh M docker/install_os_packages.sh 4 files changed, 111 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/19031/9 -- To view, visit http://gerrit.cloudera.org:8080/19031 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icc1dbd3f6a2279840218dc1da2b60077e211a328 Gerrit-Change-Number: 19031 Gerrit-PatchSet: 9 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-6684: Fix untracked memory in KRPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18798 ) Change subject: IMPALA-6684: Fix untracked memory in KRPC .. Patch Set 15: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11481/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2ba2b907ce4f275a7a1fb8cf75453c7003eb4b82 Gerrit-Change-Number: 18798 Gerrit-PatchSet: 15 Gerrit-Owner: Omid Shahidi Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Omid Shahidi Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 29 Sep 2022 04:41:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6684: Fix untracked memory in KRPC
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/18798 ) Change subject: IMPALA-6684: Fix untracked memory in KRPC .. Patch Set 15: Added KRPC memory estimation, DataStreamSink.estimateOutboundRowBatchBuffers(), at patch set 15. -- To view, visit http://gerrit.cloudera.org:8080/18798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2ba2b907ce4f275a7a1fb8cf75453c7003eb4b82 Gerrit-Change-Number: 18798 Gerrit-PatchSet: 15 Gerrit-Owner: Omid Shahidi Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Omid Shahidi Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 29 Sep 2022 04:23:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6684: Fix untracked memory in KRPC
Riza Suminto has uploaded a new patch set (#15) to the change originally created by Omid Shahidi. ( http://gerrit.cloudera.org:8080/18798 ) Change subject: IMPALA-6684: Fix untracked memory in KRPC .. IMPALA-6684: Fix untracked memory in KRPC During serialization of an row batch header, a tuple_data_ is created which will hold the compressed tuple data for an outbound row batch. We would like this tuple data to be trackable as it is responsible for a significant portion of untrackable memory from the krpc data stream sender. By using free pool, we are able to allocate tuple data and compression scratch and account for it in the memory tracker of the KrpcDataStreamSender. This solution creates a RAII class responsible for memory allocation and changes the existing code to use a char buffer pointed by a char* tuple_data_ instead of the previously used std::string tuple_data_. The thrift implementation is left unchanged and the protobuf implementation is seperated. Testing: - Passed core tests. - Ran a single node benchmark which shows no regression. - Updated row-batch-serialize-test and row-batch-serialize-benchmark to test the row-batch serialization used by KRPC. - Manually collected query-profile, heap growth, and memory usage log showing untracked memory decreased by 1/2. - Add end-end unit-test to verify the new counters in runtime profile New row-batch serialization benchmark: serialize: Func10% 50% 90% 10% 50% 90% ile (rel) (rel) (rel) --- ser_no_dups_baseline8.36 8.6 8.7 1X 1X 1X ser_no_dups 6.73 6.85 6.93 0.804X 0.796X 0.796X ser_no_dups_full5.28 5.38 5.55 0.631X 0.625X 0.637X ser_adjacent_dups_baseline 12.9 13.2 13.4 1X 1X 1X ser_adjacent_dups 23.2 23.7 24.1 1.8X 1.8X 1.8X ser_adjacent_dups_full 19.9 20.3 20.7 1.54X 1.54X 1.55X ser_dups_baseline 9.17 9.54 9.72 1X 1X 1X ser_dups7.45 7.69 7.86 0.812X 0.806X 0.809X ser_dups_full 14.6 15 15.3 1.6X 1.57X 1.57X deserialize: Func10% 50% 90% 10% 50% 90% ile (rel) (rel) (rel) --- deser_no_dups_baseline 32.6 33.5 34 1X 1X1X deser_no_dups 32.5 33.1 33.7 0.999X 0.99X 0.992X deser_adjacent_dups_baseline 53.1 54 54.7 1X 1X 1X deser_adjacent_dups 80.3 81.6 82.5 1.51X 1.51X 1.51X deser_dups_baseline 52.4 54 54.7 1X 1X 1X deser_dups 86.8 88.4 89.7 1.66X 1.64X 1.64X Change-Id: I2ba2b907ce4f275a7a1fb8cf75453c7003eb4b82 --- M be/src/benchmarks/row-batch-serialize-benchmark.cc M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/krpc-data-stream-sender.h M be/src/runtime/row-batch-serialize-test.cc M be/src/runtime/row-batch.cc M be/src/runtime/row-batch.h A be/src/runtime/row-batch.inline.h M fe/src/main/java/org/apache/impala/planner/DataStreamSink.java M testdata/workloads/functional-planner/queries/PlannerTest/max-row-size.test M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/result-spooling.test M testdata/workloads/functional-planner/queries/PlannerTest/spillable-buffer-sizing.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q01.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q02.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q03.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q04.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q05.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q06.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q07.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q08.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q09.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q10a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q11.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q12.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q13.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q15.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q16.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q17.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q18.test M
[Impala-ASF-CR] IMPALA-11606: add 'untracked memory' metric.
Xianqing He has posted comments on this change. ( http://gerrit.cloudera.org:8080/19029 ) Change subject: IMPALA-11606: add 'untracked memory' metric. .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/19029/2/be/src/util/memory-metrics.h File be/src/util/memory-metrics.h: http://gerrit.cloudera.org:8080/#/c/19029/2/be/src/util/memory-metrics.h@37 PS2, Line 37: class ExecEnv; nit: suggest ordering alphabetically -- To view, visit http://gerrit.cloudera.org:8080/19029 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib16e00109d732f759c96c7a967eb1cc32124a03f Gerrit-Change-Number: 19029 Gerrit-PatchSet: 2 Gerrit-Owner: Xiang Yang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Thu, 29 Sep 2022 03:50:16 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP: Build Impala with cdw dependencies
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18994 ) Change subject: WIP: Build Impala with cdw dependencies .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18994 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id379030f4b314e139c875584eee438b7416d89a4 Gerrit-Change-Number: 18994 Gerrit-PatchSet: 5 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 29 Sep 2022 03:40:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11621: Remove hiveserver2.pid when shutting down HiveServer2
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19051 ) Change subject: IMPALA-11621: Remove hiveserver2.pid when shutting down HiveServer2 .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19051 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I813626d06829a86854c6d2c1715f0c5f5109836d Gerrit-Change-Number: 19051 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 29 Sep 2022 03:06:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19048 ) Change subject: IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8635/ -- To view, visit http://gerrit.cloudera.org:8080/19048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I952377ed14eba26e3774e7776eb81a95d1d8e76f Gerrit-Change-Number: 19048 Gerrit-PatchSet: 5 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Thu, 29 Sep 2022 03:00:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19046 ) Change subject: IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11480/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19046 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1003556d3afc8e8216142cac4007a4c99046caeb Gerrit-Change-Number: 19046 Gerrit-PatchSet: 4 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Xiang Yang Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Thu, 29 Sep 2022 02:55:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses
Xianqing He has posted comments on this change. ( http://gerrit.cloudera.org:8080/19046 ) Change subject: IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/19046/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19046/3//COMMIT_MSG@10 PS3, Line 10: so we have to ensure only one : Kudu client created for a given Kudu master address. > nit: so we have to ensure only one Kudu client created for a given Kudu mas Done -- To view, visit http://gerrit.cloudera.org:8080/19046 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1003556d3afc8e8216142cac4007a4c99046caeb Gerrit-Change-Number: 19046 Gerrit-PatchSet: 4 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Xiang Yang Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Thu, 29 Sep 2022 02:35:04 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses
Xianqing He has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/19046 ) Change subject: IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses .. IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses Creating Kudu clients is very expensive as each will fetch metadata from the Kudu master, so we have to ensure only one Kudu client created for a given Kudu master address. The solution is to ensure KuduUtil.getKuduClient created only one KuduClient for the specified Kudu master addresses by using 'computeIfAbsent' of the ConcurrentHashMap. Testing: - Manually ran a stress test: scan of a Kudu table, 1000 concurrent queries and verified the untracked memory - Added concurrent tests for KuduUtil.getKuduClient - Ran the full set of verifications in Impala Public Jenkins Change-Id: I1003556d3afc8e8216142cac4007a4c99046caeb --- M fe/src/main/java/org/apache/impala/util/KuduUtil.java A fe/src/test/java/org/apache/impala/util/KuduUtilTest.java 2 files changed, 89 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/46/19046/4 -- To view, visit http://gerrit.cloudera.org:8080/19046 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1003556d3afc8e8216142cac4007a4c99046caeb Gerrit-Change-Number: 19046 Gerrit-PatchSet: 4 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Xiang Yang Gerrit-Reviewer: Xianqing He
[Impala-ASF-CR] Prototype to provide an option to install Java 11
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19031 ) Change subject: Prototype to provide an option to install Java 11 .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11479/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19031 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icc1dbd3f6a2279840218dc1da2b60077e211a328 Gerrit-Change-Number: 19031 Gerrit-PatchSet: 8 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Thu, 29 Sep 2022 02:29:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8770: Support building Docker images on Redhat-based distributions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19006 ) Change subject: IMPALA-8770: Support building Docker images on Redhat-based distributions .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11478/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19006 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibaff2560ef971ac2c2231a8e43921164ea1d2f4d Gerrit-Change-Number: 19006 Gerrit-PatchSet: 12 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 29 Sep 2022 02:21:54 + Gerrit-HasComments: No
[Impala-ASF-CR] Prototype to provide an option to install Java 11
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19031 to look at the new patch set (#8). Change subject: Prototype to provide an option to install Java 11 .. Prototype to provide an option to install Java 11 Change-Id: Icc1dbd3f6a2279840218dc1da2b60077e211a328 --- M bin/impala-config.sh M docker/CMakeLists.txt M docker/daemon_entrypoint.sh M docker/install_os_packages.sh 4 files changed, 102 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/19031/8 -- To view, visit http://gerrit.cloudera.org:8080/19031 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icc1dbd3f6a2279840218dc1da2b60077e211a328 Gerrit-Change-Number: 19031 Gerrit-PatchSet: 8 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-8770: Support building Docker images on Redhat-based distributions
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19006 to look at the new patch set (#12). Change subject: IMPALA-8770: Support building Docker images on Redhat-based distributions .. IMPALA-8770: Support building Docker images on Redhat-based distributions Currently, Impala supports building and testing Docker images on Ubuntu. This extends that same support to Redhat-based distributions: 1. This splits out the Docker build's OS package installation into a separate install_os_packages.sh script. This script detects the OS and calls apt or yum as appropriate. The script takes the argument --install-debug-tools, which installs extra tools like iproute2 and ping. This defaults to true for debug images and false for release images. 2. This modifies daemon_entrypoint.sh to detect the OS and set LD_LIBRARY_PATH appropriate to account for different locations of Java. 3. This modifies docker/setup_build_context.py to handle different locations of libkudu_client.so and add extra sanity checks on various libraries found via globs. 4. This modifies bin/jenkins/dockerized-*.sh test infrastructure to be able to install docker on either Ubuntu or Redhat. It also changes the exit logic to collect the container logs. Developers can override the base image for Redhat 7 and Redhat 8 builds via the IMPALA_REDHAT7_DOCKER_BASE and IMPALA_REDHAT8_DOCKER_BASE environment variables. These default to open source Redhat equivalents (Centos 7.9 and Rocky 8.5 respectively), but they are also known to work with Redhat UBI images. Testing: - Ran dockerised testing on Rocky 8.5 via the rocky-8.5-dockerised-tests job. - Ran GVO - Ran a Docker build on Centos7 with UBI7 as the base image Change-Id: Ibaff2560ef971ac2c2231a8e43921164ea1d2f4d --- M bin/impala-config.sh M bin/jenkins/dockerized-impala-bootstrap-and-test.sh M bin/jenkins/dockerized-impala-run-tests.sh A bin/jenkins/install_docker.sh M docker/CMakeLists.txt M docker/daemon_entrypoint.sh M docker/impala_base/Dockerfile M docker/impala_profile_tool/Dockerfile A docker/install_os_packages.sh M docker/setup_build_context.py 10 files changed, 417 insertions(+), 92 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/19006/12 -- To view, visit http://gerrit.cloudera.org:8080/19006 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibaff2560ef971ac2c2231a8e43921164ea1d2f4d Gerrit-Change-Number: 19006 Gerrit-PatchSet: 12 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19052 ) Change subject: IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11477/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f1b470f40e0aaf891c9f3f327af393b2f9c74bc Gerrit-Change-Number: 19052 Gerrit-PatchSet: 1 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 29 Sep 2022 00:00:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19052 ) Change subject: IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8638/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/19052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f1b470f40e0aaf891c9f3f327af393b2f9c74bc Gerrit-Change-Number: 19052 Gerrit-PatchSet: 1 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 28 Sep 2022 23:54:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala
Yu-Wen Lai has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19052 Change subject: IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala .. IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala In this patch, we use TUpdateCatalogRequest to refresh metadata after 'LOAD DATA' instead of TResetMetadataRequest so that we can reuse the code for 'INSERT' statements. It will fire an insert event just same as what we did for 'INSERT' statements. Testing: - Run existing test_load.py Change-Id: I7f1b470f40e0aaf891c9f3f327af393b2f9c74bc --- M be/src/service/client-request-state.cc M common/thrift/Frontend.thrift M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/main/java/org/apache/impala/service/Frontend.java 4 files changed, 67 insertions(+), 28 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/19052/1 -- To view, visit http://gerrit.cloudera.org:8080/19052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I7f1b470f40e0aaf891c9f3f327af393b2f9c74bc Gerrit-Change-Number: 19052 Gerrit-PatchSet: 1 Gerrit-Owner: Yu-Wen Lai
[Impala-ASF-CR] IMPALA-8770: Support building Docker images on Redhat-based distributions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19006 ) Change subject: IMPALA-8770: Support building Docker images on Redhat-based distributions .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11476/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19006 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibaff2560ef971ac2c2231a8e43921164ea1d2f4d Gerrit-Change-Number: 19006 Gerrit-PatchSet: 11 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 28 Sep 2022 23:34:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8770: Support building Docker images on Redhat-based distributions
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19006 to look at the new patch set (#11). Change subject: IMPALA-8770: Support building Docker images on Redhat-based distributions .. IMPALA-8770: Support building Docker images on Redhat-based distributions Currently, Impala supports building and testing Docker images on Ubuntu. This extends that same support to Redhat-based distributions: 1. This splits out the Docker build's OS package installation into a separate install_os_packages.sh script. This script detects the OS and calls apt or yum as appropriate. The script takes the argument --install-debug-tools, which installs extra tools like iproute2 and ping. This defaults to true for debug images and false for release images. 2. This modifies daemon_entrypoint.sh to detect the OS and set LD_LIBRARY_PATH appropriate to account for different locations of Java. 3. This modifies docker/setup_build_context.py to handle different locations of libkudu_client.so and add extra sanity checks on various libraries found via globs. 4. This modifies bin/jenkins/dockerized-*.sh test infrastructure to be able to install docker on either Ubuntu or Redhat. It also changes the exit logic to collect the container logs. Developers can override the base image for Redhat 7 and Redhat 8 builds via the IMPALA_REDHAT7_DOCKER_BASE and IMPALA_REDHAT8_DOCKER_BASE environment variables. These default to open source Redhat equivalents (Centos 7.9 and Rocky 8.5 respectively), but they are also known to work with Redhat UBI images. Testing: - Ran dockerised testing on Rocky 8.5 via the rocky-8.5-dockerised-tests job. - Ran GVO - Ran a Docker build on Centos7 with UBI7 as the base image Change-Id: Ibaff2560ef971ac2c2231a8e43921164ea1d2f4d --- M bin/impala-config.sh M bin/jenkins/dockerized-impala-bootstrap-and-test.sh M bin/jenkins/dockerized-impala-run-tests.sh A bin/jenkins/install_docker.sh M docker/CMakeLists.txt M docker/daemon_entrypoint.sh M docker/impala_base/Dockerfile M docker/impala_profile_tool/Dockerfile A docker/install_os_packages.sh M docker/setup_build_context.py 10 files changed, 417 insertions(+), 92 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/19006/11 -- To view, visit http://gerrit.cloudera.org:8080/19006 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibaff2560ef971ac2c2231a8e43921164ea1d2f4d Gerrit-Change-Number: 19006 Gerrit-PatchSet: 11 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] Prototype to provide an option to install Java 11
Joe McDonnell has posted comments on this change. (
http://gerrit.cloudera.org:8080/19031 )
Change subject: Prototype to provide an option to install Java 11
..
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19031/5/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/19031/5/bin/impala-config.sh@243
PS5, Line 243: export
IMPALA_DOCKER_USE_JAVA11=${IMPALA_DOCKER_USE_JAVA11:-"true"}
TODO: Needs to default to false.
--
To view, visit http://gerrit.cloudera.org:8080/19031
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icc1dbd3f6a2279840218dc1da2b60077e211a328
Gerrit-Change-Number: 19031
Gerrit-PatchSet: 5
Gerrit-Owner: Joe McDonnell
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Comment-Date: Wed, 28 Sep 2022 23:02:24 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8770: Support building Docker images on Redhat-based distributions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19006 ) Change subject: IMPALA-8770: Support building Docker images on Redhat-based distributions .. Patch Set 10: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/11474/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/19006 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibaff2560ef971ac2c2231a8e43921164ea1d2f4d Gerrit-Change-Number: 19006 Gerrit-PatchSet: 10 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 28 Sep 2022 23:03:50 + Gerrit-HasComments: No
[Impala-ASF-CR] Prototype to provide an option to install Java 11
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19031 ) Change subject: Prototype to provide an option to install Java 11 .. Patch Set 5: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/11475/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/19031 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icc1dbd3f6a2279840218dc1da2b60077e211a328 Gerrit-Change-Number: 19031 Gerrit-PatchSet: 5 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 28 Sep 2022 23:03:44 + Gerrit-HasComments: No
[Impala-ASF-CR] Prototype to provide an option to install Java 11
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19031 to look at the new patch set (#5). Change subject: Prototype to provide an option to install Java 11 .. Prototype to provide an option to install Java 11 Change-Id: Icc1dbd3f6a2279840218dc1da2b60077e211a328 --- M bin/impala-config.sh M docker/CMakeLists.txt M docker/daemon_entrypoint.sh M docker/install_os_packages.sh 4 files changed, 102 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/19031/5 -- To view, visit http://gerrit.cloudera.org:8080/19031 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icc1dbd3f6a2279840218dc1da2b60077e211a328 Gerrit-Change-Number: 19031 Gerrit-PatchSet: 5 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-8770: Support building Docker images on Redhat-based distributions
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19006 to look at the new patch set (#10). Change subject: IMPALA-8770: Support building Docker images on Redhat-based distributions .. IMPALA-8770: Support building Docker images on Redhat-based distributions Currently, Impala supports building and testing Docker images on Ubuntu. This extends that same support to Redhat-based distributions: 1. This splits out the Docker build's OS package installation into a separate install_os_packages.sh script. This script detects the OS and calls apt or yum as appropriate. The script takes the argument --install-debug-tools, which installs extra tools like iproute2 and ping. This defaults to true for debug images and false for release images. 2. This modifies daemon_entrypoint.sh to detect the OS and set LD_LIBRARY_PATH appropriate to account for different locations of Java. 3. This modifies docker/setup_build_context.py to handle different locations of libkudu_client.so and add extra sanity checks on various libraries found via globs. 4. This modifies bin/jenkins/dockerized-*.sh test infrastructure to be able to install docker on either Ubuntu or Redhat. It also changes the exit logic to collect the container logs. Developers can override the base image for Redhat 7 and Redhat 8 builds via the IMPALA_REDHAT7_DOCKER_BASE and IMPALA_REDHAT8_DOCKER_BASE environment variables. These default to open source Redhat equivalents (Centos 7.9 and Rocky 8.5 respectively), but they are also known to work with Redhat UBI images. Testing: - Ran dockerised testing on Rocky 8.5 via the rocky-8.5-dockerised-tests job. - Ran GVO - Ran a Docker build on Centos7 with UBI7 as the base image Change-Id: Ibaff2560ef971ac2c2231a8e43921164ea1d2f4d --- M bin/impala-config.sh M bin/jenkins/dockerized-impala-bootstrap-and-test.sh M bin/jenkins/dockerized-impala-run-tests.sh A bin/jenkins/install_docker.sh M docker/CMakeLists.txt M docker/daemon_entrypoint.sh M docker/impala_base/Dockerfile M docker/impala_profile_tool/Dockerfile A docker/install_os_packages.sh M docker/setup_build_context.py 10 files changed, 417 insertions(+), 92 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/19006/10 -- To view, visit http://gerrit.cloudera.org:8080/19006 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibaff2560ef971ac2c2231a8e43921164ea1d2f4d Gerrit-Change-Number: 19006 Gerrit-PatchSet: 10 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] WIP: Build Impala with cdw dependencies
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18994 ) Change subject: WIP: Build Impala with cdw dependencies .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11473/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18994 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id379030f4b314e139c875584eee438b7416d89a4 Gerrit-Change-Number: 18994 Gerrit-PatchSet: 5 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 28 Sep 2022 22:49:35 + Gerrit-HasComments: No
[Impala-ASF-CR] Speed up default configuration for Docker-based tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19038 ) Change subject: Speed up default configuration for Docker-based tests .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19038 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifb609bcfb10e9f9b281cc6b375c36c9638db168b Gerrit-Change-Number: 19038 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Wed, 28 Sep 2022 22:47:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11562: Revert support for o3fs as default filesystem
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19001 ) Change subject: IMPALA-11562: Revert support for o3fs as default filesystem .. IMPALA-11562: Revert support for o3fs as default filesystem Reverts support for o3fs as a default filesystem added in IMPALA-9442. Updates test setup to use ofs instead. Munges absolute paths in Iceberg metadata to match the new location required for ofs. Ozone has strict requirements on volume and bucket names, so all tables must be created within a bucket (e.g. inside /impala/test-warehouse/). Change-Id: I45e90d30b2e68876dec0db3c43ac15ee510b17bd Reviewed-on: http://gerrit.cloudera.org:8080/19001 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M bin/impala-config.sh M bin/run-all-tests.sh M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M testdata/bin/create-tpcds-testcase-files.sh M testdata/bin/load-metastore-snapshot.sh M testdata/bin/load-test-warehouse-snapshot.sh A testdata/bin/rewrite-iceberg-metadata.py M testdata/cluster/admin M testdata/workloads/functional-query/queries/QueryTest/iceberg-alter.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-create-table-like-table.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-create.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-ctas.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-insert.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-old-fileformat.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test M testdata/workloads/functional-query/queries/QueryTest/multiple-filesystems.test M testdata/workloads/functional-query/queries/QueryTest/partition-col-types.test M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test M tests/authorization/test_ranger.py M tests/common/file_utils.py M tests/common/impala_test_suite.py M tests/custom_cluster/test_startup_filesystem_checks.py M tests/metadata/test_ddl.py M tests/metadata/test_explain.py M tests/metadata/test_load.py M tests/metadata/test_recover_partitions.py M tests/metadata/test_recursive_listing.py M tests/metadata/test_show_create_table.py M tests/metadata/test_testcase_builder.py M tests/query_test/test_compressed_formats.py M tests/query_test/test_insert_behaviour.py M tests/query_test/test_insert_parquet.py M tests/query_test/test_observability.py M tests/query_test/test_runtime_filters.py M tests/util/filesystem_utils.py M tests/util/hdfs_util.py 37 files changed, 355 insertions(+), 241 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/19001 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I45e90d30b2e68876dec0db3c43ac15ee510b17bd Gerrit-Change-Number: 19001 Gerrit-PatchSet: 10 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-11562: Revert support for o3fs as default filesystem
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19001 ) Change subject: IMPALA-11562: Revert support for o3fs as default filesystem .. Patch Set 9: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19001 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45e90d30b2e68876dec0db3c43ac15ee510b17bd Gerrit-Change-Number: 19001 Gerrit-PatchSet: 9 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 28 Sep 2022 22:35:47 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: Build Impala with cdw dependencies
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18994 ) Change subject: WIP: Build Impala with cdw dependencies .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/18994/5/fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java File fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java: http://gerrit.cloudera.org:8080/#/c/18994/5/fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java@308 PS5, Line 308: dropDbIfExists(dropDatabaseRequest.getName(), dropDatabaseRequest.isIgnoreUnknownDb(), line too long (92 > 90) -- To view, visit http://gerrit.cloudera.org:8080/18994 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id379030f4b314e139c875584eee438b7416d89a4 Gerrit-Change-Number: 18994 Gerrit-PatchSet: 5 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 28 Sep 2022 22:28:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP: Build Impala with cdw dependencies
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18994 ) Change subject: WIP: Build Impala with cdw dependencies .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8637/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18994 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id379030f4b314e139c875584eee438b7416d89a4 Gerrit-Change-Number: 18994 Gerrit-PatchSet: 5 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 28 Sep 2022 22:28:05 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: Build Impala with cdw dependencies
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18994 to look at the new patch set (#5). Change subject: WIP: Build Impala with cdw dependencies .. WIP: Build Impala with cdw dependencies Other changes: - updated resource-requirements.test due to different number of files tpch_orc_def.lineitem - added redirections/implementations for new HMS APIs in MetastoreService - fixes the bug mentioned in IMPALA-11397 Change-Id: Id379030f4b314e139c875584eee438b7416d89a4 --- M bin/impala-config.sh M bin/jenkins/archive_m2_directory.sh M fe/pom.xml M fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/catalog/metastore/HmsApiNameEnum.java M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java M java/pom.xml M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test 8 files changed, 249 insertions(+), 39 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/94/18994/5 -- To view, visit http://gerrit.cloudera.org:8080/18994 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id379030f4b314e139c875584eee438b7416d89a4 Gerrit-Change-Number: 18994 Gerrit-PatchSet: 5 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19048 ) Change subject: IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11472/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I952377ed14eba26e3774e7776eb81a95d1d8e76f Gerrit-Change-Number: 19048 Gerrit-PatchSet: 4 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Wed, 28 Sep 2022 22:07:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10851: Codegen for structs
Michael Smith has posted comments on this change. (
http://gerrit.cloudera.org:8080/18526 )
Change subject: IMPALA-10851: Codegen for structs
..
Patch Set 10:
(7 comments)
I think I understand most of what's going on, but have a few questions and
suggestions.
http://gerrit.cloudera.org:8080/#/c/18526/10/be/src/codegen/codegen-anyval.h
File be/src/codegen/codegen-anyval.h:
http://gerrit.cloudera.org:8080/#/c/18526/10/be/src/codegen/codegen-anyval.h@21
PS10, Line 21: #include "codegen/codegen-anyval-read-write-info.h"
This could be a forward declaration of CodegenAnyValReadWriteInfo rather than
including the header.
http://gerrit.cloudera.org:8080/#/c/18526/10/be/src/codegen/codegen-anyval.cc
File be/src/codegen/codegen-anyval.cc:
http://gerrit.cloudera.org:8080/#/c/18526/10/be/src/codegen/codegen-anyval.cc@a669
PS10, Line 669:
Needs a rebase to resolve adding TYPE_MAP here.
http://gerrit.cloudera.org:8080/#/c/18526/10/be/src/codegen/codegen-anyval.cc@1025
PS10, Line 1025: // of i8.
This doesn't really explain why we handle booleans as i8 here, but store them
as i1.
http://gerrit.cloudera.org:8080/#/c/18526/10/be/src/exec/hash-table.cc
File be/src/exec/hash-table.cc:
http://gerrit.cloudera.org:8080/#/c/18526/10/be/src/exec/hash-table.cc@a786
PS10, Line 786:
Was this essentially NaN? I'm trying to figure out why this store op seems to
have disappeared, and what it was doing.
http://gerrit.cloudera.org:8080/#/c/18526/10/be/src/exprs/scalar-expr-evaluator-ir.cc
File be/src/exprs/scalar-expr-evaluator-ir.cc:
http://gerrit.cloudera.org:8080/#/c/18526/10/be/src/exprs/scalar-expr-evaluator-ir.cc@134
PS10, Line 134: if (v.is_null) return nullptr;
Could the 'v.is_null' check be done before the case statement? Looks like
AnyVal stores is_null, so the reinterpret_cast shouldn't change how it's
checked.
http://gerrit.cloudera.org:8080/#/c/18526/10/be/src/udf/udf-ir.cc
File be/src/udf/udf-ir.cc:
http://gerrit.cloudera.org:8080/#/c/18526/10/be/src/udf/udf-ir.cc@58
PS10, Line 58: uint8_t* FnCtxAllocateForResults(FunctionContext* ctx, int64_t
byte_size) {
Where is this used?
http://gerrit.cloudera.org:8080/#/c/18526/10/tests/query_test/test_nested_types.py
File tests/query_test/test_nested_types.py:
http://gerrit.cloudera.org:8080/#/c/18526/10/tests/query_test/test_nested_types.py@143
PS10, Line 143: 'disable_codegen': ['True', 'False']}))
Why reverse the order here? Looking at how create_exec_option_dimension works,
I think you could set disable_codegen_rows_threshold and
exec_single_node_rows_threshold as part of this dictionary.
Also I'm confused about these tests setting disable_codegen explicitly. These
are the defaults for create_exec_option_dimension.
--
To view, visit http://gerrit.cloudera.org:8080/18526
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5272c3f095fd9f07877104ee03c8e43d0c4ec0b6
Gerrit-Change-Number: 18526
Gerrit-PatchSet: 10
Gerrit-Owner: Daniel Becker
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Daniel Becker
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Comment-Date: Wed, 28 Sep 2022 22:06:02 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11621: Remove hiveserver2.pid when shutting down HiveServer2
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19051 ) Change subject: IMPALA-11621: Remove hiveserver2.pid when shutting down HiveServer2 .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8636/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/19051 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I813626d06829a86854c6d2c1715f0c5f5109836d Gerrit-Change-Number: 19051 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 28 Sep 2022 21:54:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/19048 ) Change subject: IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value .. Patch Set 4: Code-Review+2 Carry +2 -- To view, visit http://gerrit.cloudera.org:8080/19048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I952377ed14eba26e3774e7776eb81a95d1d8e76f Gerrit-Change-Number: 19048 Gerrit-PatchSet: 4 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Wed, 28 Sep 2022 21:50:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value
Daniel Becker has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/19048 ) Change subject: IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value .. IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value If the query contains a VALUES clause with a single value, the 'analyzed query' in the level 2 explain plan contains a UNION where both sides are the value from the VALUES clause. For INSERT INTO double_tbl VALUES (-0.43149576573887316); we get SELECT CAST(-0.43149576573887316 AS DECIMAL(17,17)) UNION SELECT CAST(-0.43149576573887316 AS DECIMAL(17,17)); This is because of a bug in 'SetOperationStmt::toSql()'; 'ValuesStmt's are implemented as special 'UnionStmt's, which derive from 'SetOperationStmt'. 'SetOperationStmt::toSql()' implicitly assumes that there are at least 2 operands and always prints the first one and the last one separately, even if they are the same. Note that 'ValuesStmt' is the only 'SetOperationStmt' that may contain a single operand - it is syntactically impossible in SQL to create other 'SetOperationStmt's with only one operand. This patch modifies 'SetOperationStmt::toSql()' so that if there is only one operand, only that operand is printed (so there is no reference to the set operation in the resulting string). This change does not affect how such queries are handled, only how the analyzed query is printed. Testing: - Extended AnalyzeStmtsTest.TestValuesStmt so it now also checks what VALUES clauses with one and two operands are rewritten to during analysis. Change-Id: I952377ed14eba26e3774e7776eb81a95d1d8e76f --- M fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test 4 files changed, 31 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/19048/4 -- To view, visit http://gerrit.cloudera.org:8080/19048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I952377ed14eba26e3774e7776eb81a95d1d8e76f Gerrit-Change-Number: 19048 Gerrit-PatchSet: 4 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa
[Impala-ASF-CR] IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19048 ) Change subject: IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I952377ed14eba26e3774e7776eb81a95d1d8e76f Gerrit-Change-Number: 19048 Gerrit-PatchSet: 5 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Wed, 28 Sep 2022 21:50:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19048 ) Change subject: IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8635/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I952377ed14eba26e3774e7776eb81a95d1d8e76f Gerrit-Change-Number: 19048 Gerrit-PatchSet: 5 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Wed, 28 Sep 2022 21:50:50 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: Build Impala with cdw dependencies
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18994 ) Change subject: WIP: Build Impala with cdw dependencies .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8632/ -- To view, visit http://gerrit.cloudera.org:8080/18994 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id379030f4b314e139c875584eee438b7416d89a4 Gerrit-Change-Number: 18994 Gerrit-PatchSet: 4 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 28 Sep 2022 19:54:08 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-11604 Planner changes for CPU usage
Kurt Deschler has posted comments on this change. (
http://gerrit.cloudera.org:8080/19033 )
Change subject: [WIP] IMPALA-11604 Planner changes for CPU usage
..
Patch Set 10:
(7 comments)
Good start. I think the following changes are necessary and worth spending
extra time on:
1) Fragment cost should conceptually be an amount of time required to process
the estimated data for the fragment. That way, we can divide by cores available
and factor consider SLA times through direct calculations. Any functions and
variables should probably naming like CPUCost or ProcessingCost.
2) It's probably worth spending time making a new set of Expr costing functions
instead of multiplying the EXPR constants by the row width. Comments in
Expr.java state that those constants are not suitable for absolute cost
computation. It should be reasonable to add a mechanism to do actual time
measurements for Expr evaluation during init and use measurements to compute
accurate time costs.
http://gerrit.cloudera.org:8080/#/c/19033/10/common/thrift/Frontend.thrift
File common/thrift/Frontend.thrift:
http://gerrit.cloudera.org:8080/#/c/19033/10/common/thrift/Frontend.thrift@752
PS10, Line 752: 5: optional i64 max_data_processed_limit
It's probably best to stick with vcores here as what the executor group
provides. The coordinator can work backwards from the query costing to
determine vcore sizing for a query.
http://gerrit.cloudera.org:8080/#/c/19033/10/common/thrift/Query.thrift
File common/thrift/Query.thrift:
http://gerrit.cloudera.org:8080/#/c/19033/10/common/thrift/Query.thrift@873
PS10, Line 873: 13: optional i64 data_processed_bytes;
Vcores here also.
http://gerrit.cloudera.org:8080/#/c/19033/10/fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java
File fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java:
http://gerrit.cloudera.org:8080/#/c/19033/10/fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java@727
PS10, Line 727: * getIntermediateTupleDesc().getByteSize()) /
Math.max(numInstances, 1);
Don't divide by numInstances in these functions. Instead accumulate the
fragment cost then divide by a (cost/core) number to arrive at a number of
cores required. SLA can also be factored in later for to fragment That way.
http://gerrit.cloudera.org:8080/#/c/19033/10/fe/src/main/java/org/apache/impala/planner/AggregationNode.java
File fe/src/main/java/org/apache/impala/planner/AggregationNode.java:
http://gerrit.cloudera.org:8080/#/c/19033/10/fe/src/main/java/org/apache/impala/planner/AggregationNode.java@627
PS10, Line 627: .setDataProcessedBytes(
Use (abstract) processingCost naming instead of processedBytes.
http://gerrit.cloudera.org:8080/#/c/19033/10/fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java
File fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java:
http://gerrit.cloudera.org:8080/#/c/19033/10/fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java@362
PS10, Line 362: public long computeDataProcessedBytes() {
rename all of these to computeProcessingCost or similar
http://gerrit.cloudera.org:8080/#/c/19033/10/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
File fe/src/main/java/org/apache/impala/planner/HashJoinNode.java:
http://gerrit.cloudera.org:8080/#/c/19033/10/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@308
PS10, Line 308: float eualJoinPredicateEvalCost =
eqJoinPredicateEvalCost
http://gerrit.cloudera.org:8080/#/c/19033/10/fe/src/main/java/org/apache/impala/util/ExprUtil.java
File fe/src/main/java/org/apache/impala/util/ExprUtil.java:
http://gerrit.cloudera.org:8080/#/c/19033/10/fe/src/main/java/org/apache/impala/util/ExprUtil.java@109
PS10, Line 109: public static float computeConjuctsTotalCost(List
conjuncts) {
Use List to share the same function.
--
To view, visit http://gerrit.cloudera.org:8080/19033
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If32dc770dfffcdd0be2ba789a7720952c68a
Gerrit-Change-Number: 19033
Gerrit-PatchSet: 10
Gerrit-Owner: Qifan Chen
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Qifan Chen
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Wed, 28 Sep 2022 19:17:43 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11621: Remove hiveserver2.pid when shutting down HiveServer2
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/19051 ) Change subject: IMPALA-11621: Remove hiveserver2.pid when shutting down HiveServer2 .. Patch Set 1: > This looks fine. How hard would it to be to use Hive's graceful > shutdown? I don't think it would be hard to switch to using the shutdown logic from the Hive script. At the moment, Hive's graceful shutdown also doesn't remove the pid file. I'm talking to them about it. -- To view, visit http://gerrit.cloudera.org:8080/19051 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I813626d06829a86854c6d2c1715f0c5f5109836d Gerrit-Change-Number: 19051 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 28 Sep 2022 18:55:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11621: Remove hiveserver2.pid when shutting down HiveServer2
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/19051 ) Change subject: IMPALA-11621: Remove hiveserver2.pid when shutting down HiveServer2 .. Patch Set 1: Code-Review+1 This looks fine. How hard would it to be to use Hive's graceful shutdown? -- To view, visit http://gerrit.cloudera.org:8080/19051 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I813626d06829a86854c6d2c1715f0c5f5109836d Gerrit-Change-Number: 19051 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 28 Sep 2022 18:46:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11621: Remove hiveserver2.pid when shutting down HiveServer2
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19051 ) Change subject: IMPALA-11621: Remove hiveserver2.pid when shutting down HiveServer2 .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11471/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19051 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I813626d06829a86854c6d2c1715f0c5f5109836d Gerrit-Change-Number: 19051 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 28 Sep 2022 18:46:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11621: Remove hiveserver2.pid when shutting down HiveServer2
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19051 Change subject: IMPALA-11621: Remove hiveserver2.pid when shutting down HiveServer2 .. IMPALA-11621: Remove hiveserver2.pid when shutting down HiveServer2 In HIVE-22193, Hive added graceful shutdown for HiveServer2. This modified some of the startup scripts to maintain a pid file. On startup, it verified that the pid is not already running. Impala uses the hive startup script, but it kills HiveServer2 with testdata/bin/kill-java-service.sh. If the pid file outlives the process, then the OS may reuse the pid and this can cause problems on startup. This modifies testdata/bin/kill-hive-server.sh to remove the pid file. testdata/bin/kill-java-service.sh would fail if it did not kill HiveServer2, so it should be safe to remove the pid file after the kill-java-service.sh call. Testing: - Verified the hiveserver2.pid file is remove after shutting down. Change-Id: I813626d06829a86854c6d2c1715f0c5f5109836d --- M testdata/bin/kill-hive-server.sh 1 file changed, 5 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/19051/1 -- To view, visit http://gerrit.cloudera.org:8080/19051 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I813626d06829a86854c6d2c1715f0c5f5109836d Gerrit-Change-Number: 19051 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[Impala-ASF-CR] Speed up default configuration for Docker-based tests
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/19038 ) Change subject: Speed up default configuration for Docker-based tests .. Patch Set 1: Code-Review+2 This makes sense to me -- To view, visit http://gerrit.cloudera.org:8080/19038 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifb609bcfb10e9f9b281cc6b375c36c9638db168b Gerrit-Change-Number: 19038 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Wed, 28 Sep 2022 17:30:37 + Gerrit-HasComments: No
[Impala-ASF-CR] Speed up default configuration for Docker-based tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19038 ) Change subject: Speed up default configuration for Docker-based tests .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8634/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/19038 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifb609bcfb10e9f9b281cc6b375c36c9638db168b Gerrit-Change-Number: 19038 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Wed, 28 Sep 2022 17:30:54 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-11604 Planner changes for CPU usage
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19033 ) Change subject: [WIP] IMPALA-11604 Planner changes for CPU usage .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11470/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19033 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If32dc770dfffcdd0be2ba789a7720952c68a Gerrit-Change-Number: 19033 Gerrit-PatchSet: 10 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 28 Sep 2022 17:30:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11562: Revert support for o3fs as default filesystem
Joe McDonnell has posted comments on this change. (
http://gerrit.cloudera.org:8080/19001 )
Change subject: IMPALA-11562: Revert support for o3fs as default filesystem
..
Patch Set 8: Code-Review+2
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19001/8/tests/authorization/test_ranger.py
File tests/authorization/test_ranger.py:
http://gerrit.cloudera.org:8080/#/c/19001/8/tests/authorization/test_ranger.py@309
PS8, Line 309: TestRanger._check_privileges(result, [
: [kw, id, "", "", "", "{0}{1}".format(NAMENODE,
'/tmp'), "", "all", "false"]])
> Originally this would grant with the uri '/tmp', then check privileges with
Ah, I see. I didn't realize NAMENODE included FILESYSTEM_PREFIX. Makes sense.
--
To view, visit http://gerrit.cloudera.org:8080/19001
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45e90d30b2e68876dec0db3c43ac15ee510b17bd
Gerrit-Change-Number: 19001
Gerrit-PatchSet: 8
Gerrit-Owner: Michael Smith
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Wed, 28 Sep 2022 17:22:42 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11562: Revert support for o3fs as default filesystem
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19001 ) Change subject: IMPALA-11562: Revert support for o3fs as default filesystem .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8633/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19001 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45e90d30b2e68876dec0db3c43ac15ee510b17bd Gerrit-Change-Number: 19001 Gerrit-PatchSet: 9 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 28 Sep 2022 17:24:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11562: Revert support for o3fs as default filesystem
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19001 ) Change subject: IMPALA-11562: Revert support for o3fs as default filesystem .. Patch Set 9: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19001 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45e90d30b2e68876dec0db3c43ac15ee510b17bd Gerrit-Change-Number: 19001 Gerrit-PatchSet: 9 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 28 Sep 2022 17:24:31 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-11604 Planner changes for CPU usage
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/19033 ) Change subject: [WIP] IMPALA-11604 Planner changes for CPU usage .. Patch Set 10: Added nodes in the commit message about each node (up to SingularRowSrc) and improved in the corresponding class. -- To view, visit http://gerrit.cloudera.org:8080/19033 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If32dc770dfffcdd0be2ba789a7720952c68a Gerrit-Change-Number: 19033 Gerrit-PatchSet: 10 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 28 Sep 2022 17:10:08 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump impyla dependency to 0.18.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18456 ) Change subject: Bump impyla dependency to 0.18.0 .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11469/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18456 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7265558b0e07959c606cba73cd251c3edfcb3ed5 Gerrit-Change-Number: 18456 Gerrit-PatchSet: 4 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 28 Sep 2022 17:10:19 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-11604 Planner changes for CPU usage
Qifan Chen has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/19033 ) Change subject: [WIP] IMPALA-11604 Planner changes for CPU usage .. [WIP] IMPALA-11604 Planner changes for CPU usage This patch augments IMPALA-10992 by allowing the weighted amount of data processed to be used as a new factor in the definition and selection of an executor group. The weighted amount of data processed is the sum of that in every fragment, which is the sum of that in every node in the fragment. For each node, the weighted amount of data processed is computed with a general formula as follows. D = I * C * W / N where D is the weighted amount of data processed I is input cardinality C is expression evaluation cost per row W is average row size N is number of instances A description of the computation for each kind of plan node is given below. 1. Aggregation node: C and W are the sum of the costs and partial row widths for each AggregateInfo object. 2. AnalyticEval node: C is sum of the evaluation costs for analytic functions, partition by equal and order by equal predicate; 3. CardinalityCheck node: Both C and I are 1; 4. DataSource scan node: C is computed from a subset of the selection predicates excluding data source accepted predicates; 5. EmptySet node: I is 0; 6. Exchange node: A modification of the general formula when in broadcast mode: D = (I * C * W / N) * number of receivers; 7. Hash join node: C is sum of the evaluation cost for equi-join predicate and for other join predicate, for both probe and build side; 8. Hbase scan node: N is 1 9. Hdfs and Kudu scan node: N is mt_dop when query option mt_dop >= 1, otherwise N is number of nodes * max scan threads; 10. Nested loop join node: When the right child is not a SingularRowSrc node, C is sum of the evaluation cost for equi-join predicate and for other join predicate, for both probe and build side. When the right child is a SingularRowSrc node, the cost for build side is multiplied by the cardinality from the probe side; 11. Select node: Use the general formula; 12. SingularRowSrc node: I is 1. Since the node is involved once per input in nested loop join, the total cost of this node is computed in nested loop join. 13. Sort node: 14. Subplan node: 15. Union node: 16. Unnest node: Change-Id: If32dc770dfffcdd0be2ba789a7720952c68a --- M common/thrift/Frontend.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/planner/AggregationNode.java M fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java M fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M fe/src/main/java/org/apache/impala/planner/EmptySetNode.java M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java M fe/src/main/java/org/apache/impala/planner/PlanFragment.java M fe/src/main/java/org/apache/impala/planner/PlanNode.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/planner/ResourceProfile.java M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java M fe/src/main/java/org/apache/impala/planner/ScanNode.java M fe/src/main/java/org/apache/impala/planner/SelectNode.java M fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java M fe/src/main/java/org/apache/impala/planner/SortNode.java M fe/src/main/java/org/apache/impala/planner/SubplanNode.java M fe/src/main/java/org/apache/impala/planner/UnionNode.java M fe/src/main/java/org/apache/impala/planner/UnnestNode.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/ExprUtil.java 29 files changed, 336 insertions(+), 56 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/19033/10 -- To view, visit http://gerrit.cloudera.org:8080/19033 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If32dc770dfffcdd0be2ba789a7720952c68a Gerrit-Change-Number: 19033 Gerrit-PatchSet: 10 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] Bump impyla dependency to 0.18.0
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18456 ) Change subject: Bump impyla dependency to 0.18.0 .. Patch Set 5: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/18456/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/18456/5//COMMIT_MSG@7 PS5, Line 7: Bump impyla dependency to 0.18.0 nit: Should you add Jira number? -- To view, visit http://gerrit.cloudera.org:8080/18456 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7265558b0e07959c606cba73cd251c3edfcb3ed5 Gerrit-Change-Number: 18456 Gerrit-PatchSet: 5 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 28 Sep 2022 16:57:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11562: Revert support for o3fs as default filesystem
Michael Smith has posted comments on this change. (
http://gerrit.cloudera.org:8080/19001 )
Change subject: IMPALA-11562: Revert support for o3fs as default filesystem
..
Patch Set 8:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19001/8/tests/authorization/test_ranger.py
File tests/authorization/test_ranger.py:
http://gerrit.cloudera.org:8080/#/c/19001/8/tests/authorization/test_ranger.py@309
PS8, Line 309: TestRanger._check_privileges(result, [
: [kw, id, "", "", "", "{0}{1}".format(NAMENODE,
'/tmp'), "", "all", "false"]])
> Just curious, why is /tmp special? We execute the grant with uri but then c
Originally this would grant with the uri '/tmp', then check privileges with
'$NAMENODE/tmp'.
hdfs_util.py sets NAMENODE = FILESYSTEM_PREFIX or CORE_CONF.get('fs.defaultFS')
For Ozone, FILESYSTEM_PREFIX includes the volume name -
ofs://localhost:9862/impala - so a bunch of warehouse paths go to the correct
location.
For this test, my interpretation for grant/check values is
- HDFS: /tmp, hdfs://localhost:20500
- Ozone: /impala/tmp, ofs://localhost:9862/impala/tmp
- S3: /tmp, s3a://impala-test-uswest2-2/tmp
There are probably other ways to get there, but this seemed like the smallest
change to adjust.
--
To view, visit http://gerrit.cloudera.org:8080/19001
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45e90d30b2e68876dec0db3c43ac15ee510b17bd
Gerrit-Change-Number: 19001
Gerrit-PatchSet: 8
Gerrit-Owner: Michael Smith
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Wed, 28 Sep 2022 16:53:37 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] Bump impyla dependency to 0.18.0
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18456 ) Change subject: Bump impyla dependency to 0.18.0 .. Patch Set 3: > (1 comment) > > What's the additional work to keep the patch in WIP state? Switched to the non-alpha 0.18.0 Impyla release and removed WIP state. -- To view, visit http://gerrit.cloudera.org:8080/18456 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7265558b0e07959c606cba73cd251c3edfcb3ed5 Gerrit-Change-Number: 18456 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 28 Sep 2022 16:51:10 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump impyla dependency to 0.18.0
Hello Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18456 to look at the new patch set (#5). Change subject: Bump impyla dependency to 0.18.0 .. Bump impyla dependency to 0.18.0 IMPALA_THRIFT_PY_VERSION is also bumped to 0.16.0p3. As 0.16.0p3 Thrift does not contain Python related patches and Impyla 0.18.0 depends on Thrift 0.16.0, now we are consistently using Thrift 0.16.0 in all Python code. Change-Id: I7265558b0e07959c606cba73cd251c3edfcb3ed5 --- M bin/impala-config.sh M infra/python/deps/requirements.txt M shell/packaging/requirements.txt 3 files changed, 3 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/18456/5 -- To view, visit http://gerrit.cloudera.org:8080/18456 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7265558b0e07959c606cba73cd251c3edfcb3ed5 Gerrit-Change-Number: 18456 Gerrit-PatchSet: 5 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] Bump impyla dependency to 0.18.0
Hello Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18456 to look at the new patch set (#4). Change subject: Bump impyla dependency to 0.18.0 .. Bump impyla dependency to 0.18.0 IMPALA_THRIFT_PY_VERSION is also bumped to 0.16.0p3. As 0.16.0p3 Thrift does not contain Python related patches and Impyla 0.18.0 depends on Thrift 0.16.0, now we are consistantly using Thrift 0.16.0 in all Python code. Change-Id: I7265558b0e07959c606cba73cd251c3edfcb3ed5 --- M bin/impala-config.sh M infra/python/deps/requirements.txt M shell/packaging/requirements.txt 3 files changed, 3 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/18456/4 -- To view, visit http://gerrit.cloudera.org:8080/18456 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7265558b0e07959c606cba73cd251c3edfcb3ed5 Gerrit-Change-Number: 18456 Gerrit-PatchSet: 4 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19048 ) Change subject: IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8631/ -- To view, visit http://gerrit.cloudera.org:8080/19048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I952377ed14eba26e3774e7776eb81a95d1d8e76f Gerrit-Change-Number: 19048 Gerrit-PatchSet: 3 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Wed, 28 Sep 2022 16:46:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11562: Revert support for o3fs as default filesystem
Joe McDonnell has posted comments on this change. (
http://gerrit.cloudera.org:8080/19001 )
Change subject: IMPALA-11562: Revert support for o3fs as default filesystem
..
Patch Set 8: Code-Review+1
(1 comment)
This makes sense to me.
One thing that can happen is that developers add new tests without the
appropriate prefix. One extension is that we could do is to have a way to run
our normal tests with a special prefix and make precommit use it. We don't need
to do that in this change, but it might be something to look at.
http://gerrit.cloudera.org:8080/#/c/19001/8/tests/authorization/test_ranger.py
File tests/authorization/test_ranger.py:
http://gerrit.cloudera.org:8080/#/c/19001/8/tests/authorization/test_ranger.py@309
PS8, Line 309: TestRanger._check_privileges(result, [
: [kw, id, "", "", "", "{0}{1}".format(NAMENODE,
'/tmp'), "", "all", "false"]])
Just curious, why is /tmp special? We execute the grant with uri but then check
/tmp here.
--
To view, visit http://gerrit.cloudera.org:8080/19001
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45e90d30b2e68876dec0db3c43ac15ee510b17bd
Gerrit-Change-Number: 19001
Gerrit-PatchSet: 8
Gerrit-Owner: Michael Smith
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Wed, 28 Sep 2022 16:32:14 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11064 Optimizing Temporary File Structure for Batch Reading
Michael Smith has posted comments on this change. (
http://gerrit.cloudera.org:8080/18219 )
Change subject: IMPALA-11064 Optimizing Temporary File Structure for Batch
Reading
..
Patch Set 6:
(1 comment)
This seems like it could use a rebase.
http://gerrit.cloudera.org:8080/#/c/18219/5/be/src/runtime/tmp-file-mgr.h
File be/src/runtime/tmp-file-mgr.h:
http://gerrit.cloudera.org:8080/#/c/18219/5/be/src/runtime/tmp-file-mgr.h@617
PS5, Line 617: if (it != tmp_files_remote_by_spill_id_.end()) {
> Moved the not thread-safe public functions to protected and added some comm
Done
--
To view, visit http://gerrit.cloudera.org:8080/18219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If913785cac9e2dafa20013b6600c87fcaf3e2018
Gerrit-Change-Number: 18219
Gerrit-PatchSet: 6
Gerrit-Owner: Yida Wu
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Yida Wu
Gerrit-Comment-Date: Wed, 28 Sep 2022 16:21:14 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11555: external Udf support decimal(*, *)
lipeng...@sensorsdata.cn has posted comments on this change. ( http://gerrit.cloudera.org:8080/19021 ) Change subject: IMPALA-11555: external Udf support decimal(*, *) .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/19021/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19021/3//COMMIT_MSG@9 PS3, Line 9: Increase syntax parsing and remove some restrictions. To help others to understand the sense of "decimal(*, *)", could you please give more description? -- To view, visit http://gerrit.cloudera.org:8080/19021 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I09f19da5b1660ed0b882497fc321e6cc1b9b55d0 Gerrit-Change-Number: 19021 Gerrit-PatchSet: 3 Gerrit-Owner: Minghui Zhu Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Minghui Zhu Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 28 Sep 2022 16:07:29 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP: Build Impala with cdw dependencies
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18994 ) Change subject: WIP: Build Impala with cdw dependencies .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11468/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18994 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id379030f4b314e139c875584eee438b7416d89a4 Gerrit-Change-Number: 18994 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 28 Sep 2022 15:48:48 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: Build Impala with cdw dependencies
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18994 ) Change subject: WIP: Build Impala with cdw dependencies .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8632/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18994 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id379030f4b314e139c875584eee438b7416d89a4 Gerrit-Change-Number: 18994 Gerrit-PatchSet: 4 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 28 Sep 2022 15:27:44 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: Build Impala with cdw dependencies
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18994 ) Change subject: WIP: Build Impala with cdw dependencies .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/18994/3/fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java File fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java: http://gerrit.cloudera.org:8080/#/c/18994/3/fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java@308 PS3, Line 308: dropDbIfExists(dropDatabaseRequest.getName(), dropDatabaseRequest.isIgnoreUnknownDb(), line too long (92 > 90) -- To view, visit http://gerrit.cloudera.org:8080/18994 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id379030f4b314e139c875584eee438b7416d89a4 Gerrit-Change-Number: 18994 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 28 Sep 2022 15:27:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP: Build Impala with cdw dependencies
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18994 to look at the new patch set (#3). Change subject: WIP: Build Impala with cdw dependencies .. WIP: Build Impala with cdw dependencies Other changes: - updated resource-requirements.test due to different number of files tpch_orc_def.lineitem - added redirections/implementations for new HMS APIs in MetastoreService - fixes the bug mentioned in IMPALA-11397 Change-Id: Id379030f4b314e139c875584eee438b7416d89a4 --- M bin/impala-config.sh M bin/jenkins/archive_m2_directory.sh M fe/pom.xml M fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/catalog/metastore/HmsApiNameEnum.java M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java M java/pom.xml M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test 8 files changed, 249 insertions(+), 39 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/94/18994/3 -- To view, visit http://gerrit.cloudera.org:8080/18994 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id379030f4b314e139c875584eee438b7416d89a4 Gerrit-Change-Number: 18994 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11504:Specializing DecimalUtil::GetScaleMultiplier().
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/18861 ) Change subject: IMPALA-11504:Specializing DecimalUtil::GetScaleMultiplier(). .. Patch Set 4: (2 comments) http://gerrit.cloudera.org:8080/#/c/18861/3/be/src/benchmarks/decimal-util-benchmark.cc File be/src/benchmarks/decimal-util-benchmark.cc: http://gerrit.cloudera.org:8080/#/c/18861/3/be/src/benchmarks/decimal-util-benchmark.cc@51 PS3, Line 51: } > I guess the difference is a total time comparison, incuding the target func I meant why it makes a huge difference if the volatile variable is static or not. http://gerrit.cloudera.org:8080/#/c/18861/4/be/src/runtime/decimal-test.cc File be/src/runtime/decimal-test.cc: http://gerrit.cloudera.org:8080/#/c/18861/4/be/src/runtime/decimal-test.cc@1022 PS4, Line 1022: TestGetScaleMultiplier(DecimalUtil::INT32_SCALE_UPPER_BOUND); > Since we didn't specialize the double edition, I think we needn't to test i I still think we should test it, it's good to add tests to old code that hasn't been tested so far. -- To view, visit http://gerrit.cloudera.org:8080/18861 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I969e2977d51313e738f72c8246db003ae43a3782 Gerrit-Change-Number: 18861 Gerrit-PatchSet: 4 Gerrit-Owner: Xiang Yang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Xiang Yang Gerrit-Comment-Date: Wed, 28 Sep 2022 15:09:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11576: Fix for test multiple storage locations on S3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19045 ) Change subject: IMPALA-11576: Fix for test_multiple_storage_locations on S3 .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19045 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I08badd96d060a2377c9a1eafa287a3adf8fa11db Gerrit-Change-Number: 19045 Gerrit-PatchSet: 4 Gerrit-Owner: Gergely Fürnstáhl Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 28 Sep 2022 13:19:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11576: Fix for test multiple storage locations on S3
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19045 ) Change subject: IMPALA-11576: Fix for test_multiple_storage_locations on S3 .. IMPALA-11576: Fix for test_multiple_storage_locations on S3 Fixed absolute/relative path handling in filesystems not supporting block location (e.g. S3). Testing: - test_multiple_storage_location passes on S3 Change-Id: I08badd96d060a2377c9a1eafa287a3adf8fa11db Reviewed-on: http://gerrit.cloudera.org:8080/19045 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java 3 files changed, 7 insertions(+), 8 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/19045 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I08badd96d060a2377c9a1eafa287a3adf8fa11db Gerrit-Change-Number: 19045 Gerrit-PatchSet: 5 Gerrit-Owner: Gergely Fürnstáhl Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-11504:Specializing DecimalUtil::GetScaleMultiplier().
Xiang Yang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18861 ) Change subject: IMPALA-11504:Specializing DecimalUtil::GetScaleMultiplier(). .. Patch Set 4: (2 comments) Hi Daniel, I still have a question to discuss, Looking forward to your reply, thanks. http://gerrit.cloudera.org:8080/#/c/18861/3/be/src/benchmarks/decimal-util-benchmark.cc File be/src/benchmarks/decimal-util-benchmark.cc: http://gerrit.cloudera.org:8080/#/c/18861/3/be/src/benchmarks/decimal-util-benchmark.cc@51 PS3, Line 51: } > I experimented a bit with it and a simple global static volatile variable m I guess the difference is a total time comparison, incuding the target function's cost, and the addition's cost, and from int256 to int64 conversion's cost. Because different test method's distribution ratio are different, the result is different. http://gerrit.cloudera.org:8080/#/c/18861/4/be/src/runtime/decimal-test.cc File be/src/runtime/decimal-test.cc: http://gerrit.cloudera.org:8080/#/c/18861/4/be/src/runtime/decimal-test.cc@1022 PS4, Line 1022: TestGetScaleMultiplier(DecimalUtil::INT32_SCALE_UPPER_BOUND); > We could also test with double, I saw this function used with double in the Since we didn't specialize the double edition, I think we needn't to test it. -- To view, visit http://gerrit.cloudera.org:8080/18861 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I969e2977d51313e738f72c8246db003ae43a3782 Gerrit-Change-Number: 18861 Gerrit-PatchSet: 4 Gerrit-Owner: Xiang Yang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Xiang Yang Gerrit-Comment-Date: Wed, 28 Sep 2022 12:05:28 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11417: Support outer join elimination optimization
Csaba Ringhofer has posted comments on this change. (
http://gerrit.cloudera.org:8080/18705 )
Change subject: IMPALA-11417: Support outer join elimination optimization
..
Patch Set 6:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18705/6/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/18705/6/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3178
PS6, Line 3178: public Set materializeSlots(List
exprs, boolean areJoinOnConds) {
I think that it would be better to keep a materializeSlots(List exprs)
overload that calls this function with false argument. This would reduce the
size of this change.
Another solution would be to create another function that only deals with the
join case.
http://gerrit.cloudera.org:8080/#/c/18705/6/testdata/workloads/functional-planner/queries/PlannerTest/outer-join-elimination.test
File
testdata/workloads/functional-planner/queries/PlannerTest/outer-join-elimination.test:
http://gerrit.cloudera.org:8080/#/c/18705/6/testdata/workloads/functional-planner/queries/PlannerTest/outer-join-elimination.test@1
PS6, Line 1: select catalog_sales.cs_item_sk, catalog_sales.cs_order_number
Can you add a few more tests? I was thinking about the following cases:
- the column in the join condition from the left table also occurs in the
select list
- a column from the right side is also used in the where clause (we should not
eliminate the join in this case)
- the join happens in an inline view or with clause
- see what happens in case of right join (even if it is unoptimized)
- see what happens in case of full join (even if it is unoptimized)
- see what happens if there are more than 2 tables joined
Also, these queries could be also tested in an EE test to ensure that that the
optimization does not change the results.
--
To view, visit http://gerrit.cloudera.org:8080/18705
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If2e68263a029ac84a4f35b0846b22aa42d7ceece
Gerrit-Change-Number: 18705
Gerrit-PatchSet: 6
Gerrit-Owner: Jian Zhang
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Xiang Yang
Gerrit-Comment-Date: Wed, 28 Sep 2022 11:41:35 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19046 ) Change subject: IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses .. Patch Set 3: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/19046/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19046/3//COMMIT_MSG@10 PS3, Line 10: so we can ensure one Kudu : client created for the specified Kudu master addresses. nit: so we have to ensure only one Kudu client created for a given Kudu master address. -- To view, visit http://gerrit.cloudera.org:8080/19046 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1003556d3afc8e8216142cac4007a4c99046caeb Gerrit-Change-Number: 19046 Gerrit-PatchSet: 3 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Xiang Yang Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Wed, 28 Sep 2022 11:35:08 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19048 ) Change subject: IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8631/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I952377ed14eba26e3774e7776eb81a95d1d8e76f Gerrit-Change-Number: 19048 Gerrit-PatchSet: 3 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Wed, 28 Sep 2022 11:29:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19048 ) Change subject: IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I952377ed14eba26e3774e7776eb81a95d1d8e76f Gerrit-Change-Number: 19048 Gerrit-PatchSet: 3 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Wed, 28 Sep 2022 11:29:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19048 ) Change subject: IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11467/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I952377ed14eba26e3774e7776eb81a95d1d8e76f Gerrit-Change-Number: 19048 Gerrit-PatchSet: 2 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Wed, 28 Sep 2022 11:19:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11417: Support outer join elimination optimization
Xiang Yang has posted comments on this change. (
http://gerrit.cloudera.org:8080/18705 )
Change subject: IMPALA-11417: Support outer join elimination optimization
..
Patch Set 6:
(2 comments)
Hi jian, thanks for your contribution!
http://gerrit.cloudera.org:8080/#/c/18705/6/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
File fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java:
http://gerrit.cloudera.org:8080/#/c/18705/6/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java@2117
PS6, Line 2117: while (conjunctIter.hasNext()) {
since you didn't remove eqJoinConjuncts's element, you can use 'for
(BinaryPredicate conjunct: eqJoinConjuncts) {...}' to iterate List.
http://gerrit.cloudera.org:8080/#/c/18705/6/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java@2123
PS6, Line 2123: SlotRef innerSlotRef = eq.getChild(1).unwrapSlotRef(true);
is the 2nd child must belong to the right table?
--
To view, visit http://gerrit.cloudera.org:8080/18705
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If2e68263a029ac84a4f35b0846b22aa42d7ceece
Gerrit-Change-Number: 18705
Gerrit-PatchSet: 6
Gerrit-Owner: Jian Zhang
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Xiang Yang
Gerrit-Comment-Date: Wed, 28 Sep 2022 11:02:31 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/19048 ) Change subject: IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I952377ed14eba26e3774e7776eb81a95d1d8e76f Gerrit-Change-Number: 19048 Gerrit-PatchSet: 2 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Wed, 28 Sep 2022 11:00:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/19048 ) Change subject: IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/19048/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19048/1//COMMIT_MSG@7 PS1, Line 7: Fix SetOperationStmt::toSql() for VALUES statement with : single value > This could include the info that the single value case was wrong, e.g. Done -- To view, visit http://gerrit.cloudera.org:8080/19048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I952377ed14eba26e3774e7776eb81a95d1d8e76f Gerrit-Change-Number: 19048 Gerrit-PatchSet: 2 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Wed, 28 Sep 2022 10:58:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value
Daniel Becker has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/19048 ) Change subject: IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value .. IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value If the query contains a VALUES clause with a single value, the 'analyzed query' in the level 2 explain plan contains a UNION where both sides are the value from the VALUES clause. For INSERT INTO double_tbl VALUES (-0.43149576573887316); we get SELECT CAST(-0.43149576573887316 AS DECIMAL(17,17)) UNION SELECT CAST(-0.43149576573887316 AS DECIMAL(17,17)); This is because of a bug in 'SetOperationStmt::toSql()'; 'ValuesStmt's are implemented as special 'UnionStmt's, which derive from 'SetOperationStmt'. 'SetOperationStmt::toSql()' implicitly assumes that there are at least 2 operands and always prints the first one and the last one separately, even if they are the same. Note that 'ValuesStmt' is the only 'SetOperationStmt' that may contain a single operand - it is syntactically impossible in SQL to create other 'SetOperationStmt's with only one operand. This patch modifies 'SetOperationStmt::toSql()' so that if there is only one operand, only that operand is printed (so there is no reference to the set operation in the resulting string). This change does not affect how such queries are handled, only how the analyzed query is printed. Testing: - Extended AnalyzeStmtsTest.TestValuesStmt so it now also checks what VALUES clauses with one and two operands are rewritten to during analysis. Change-Id: I952377ed14eba26e3774e7776eb81a95d1d8e76f --- M fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java 2 files changed, 29 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/19048/2 -- To view, visit http://gerrit.cloudera.org:8080/19048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I952377ed14eba26e3774e7776eb81a95d1d8e76f Gerrit-Change-Number: 19048 Gerrit-PatchSet: 2 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa
[Impala-ASF-CR] IMPALA-11504:Specializing DecimalUtil::GetScaleMultiplier().
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/18861 ) Change subject: IMPALA-11504:Specializing DecimalUtil::GetScaleMultiplier(). .. Patch Set 4: (8 comments) http://gerrit.cloudera.org:8080/#/c/18861/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/18861/3//COMMIT_MSG@15 PS3, Line 15: - Add decimal-util-benchmark. > done, but ran in a docker container, will that be a problem? It's ok if it is in a Docker container. I would mention in the commit msg how big the speedup is. http://gerrit.cloudera.org:8080/#/c/18861/3/be/src/benchmarks/decimal-util-benchmark.cc File be/src/benchmarks/decimal-util-benchmark.cc: http://gerrit.cloudera.org:8080/#/c/18861/3/be/src/benchmarks/decimal-util-benchmark.cc@51 PS3, Line 51: } > done. I experimented a bit with it and a simple global static volatile variable may be enough. I couldn't get it to work with int256_t but using an int64_t I got an even greater difference, around 6000x. This time I simply assigned the result of the function to the global volatile int64_t, didn't add anything. Interestingly if the global volatile variable is not static, the difference is only 1400x... I don't know the cause of the difference but I think we should try the global, static volatile int64_t variable that we assign the result to. http://gerrit.cloudera.org:8080/#/c/18861/4/be/src/benchmarks/decimal-util-benchmark.cc File be/src/benchmarks/decimal-util-benchmark.cc: http://gerrit.cloudera.org:8080/#/c/18861/4/be/src/benchmarks/decimal-util-benchmark.cc@56 PS4, Line 56: int256_t If we stay with summing the results, I'm not sure int256_t is OK, it may have undefined behaviour on overflow. http://gerrit.cloudera.org:8080/#/c/18861/4/be/src/runtime/decimal-test.cc File be/src/runtime/decimal-test.cc: http://gerrit.cloudera.org:8080/#/c/18861/4/be/src/runtime/decimal-test.cc@1012 PS4, Line 1012: DCHECK(expect * 10 >= expect); This is not really a good check because if a signed type (such as int32_t etc.) overflows it is undefined behaviour so the value could be anything. In this test I think we can tolerate not checking overflow. http://gerrit.cloudera.org:8080/#/c/18861/4/be/src/runtime/decimal-test.cc@1013 PS4, Line 1013: expect *= 10; We don't need to prepare a fresh 'expected' in each iteration, we can start with 1 outside the loop and multiply it with 10 at the end of each iteration. http://gerrit.cloudera.org:8080/#/c/18861/4/be/src/runtime/decimal-test.cc@1022 PS4, Line 1022: TestGetScaleMultiplier(DecimalUtil::INT32_SCALE_UPPER_BOUND); We could also test with double, I saw this function used with double in the code base. http://gerrit.cloudera.org:8080/#/c/18861/4/be/src/util/decimal-util.h File be/src/util/decimal-util.h: http://gerrit.cloudera.org:8080/#/c/18861/4/be/src/util/decimal-util.h@37 PS4, Line 37: // The scale upper bound for GetScaleMultiplier() : static constexpr int INT32_SCALE_UPPER_BOUND = ColumnType::MAX_DECIMAL4_PRECISION + 1; : // The scale upper bound for GetScaleMultiplier() : static constexpr int INT64_SCALE_UPPER_BOUND = ColumnType::MAX_DECIMAL8_PRECISION + 1; : // The scale upper bound for GetScaleMultiplier() : static constexpr int INT128_SCALE_UPPER_BOUND = ColumnType::MAX_PRECISION + 1; : // The scale upper bound for GetScaleMultiplier() : static constexpr int INT256_SCALE_UPPER_BOUND = 77; I don't think these should be public class-level constants, they should be in the respective functions. http://gerrit.cloudera.org:8080/#/c/18861/3/be/src/util/decimal-util.h File be/src/util/decimal-util.h: http://gerrit.cloudera.org:8080/#/c/18861/3/be/src/util/decimal-util.h@342 PS3, Line 342: * i10e18 * i10e18 * 10ll > use 'num_values' instead of "sizeof(values) / sizeof(int256_t)"? it seems t I agree now this assert statement is not needed, we can remove it. -- To view, visit http://gerrit.cloudera.org:8080/18861 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I969e2977d51313e738f72c8246db003ae43a3782 Gerrit-Change-Number: 18861 Gerrit-PatchSet: 4 Gerrit-Owner: Xiang Yang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Xiang Yang Gerrit-Comment-Date: Wed, 28 Sep 2022 10:53:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11562: Revert support for o3fs as default filesystem
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/19001 ) Change subject: IMPALA-11562: Revert support for o3fs as default filesystem .. Patch Set 8: (2 comments) LGTM! http://gerrit.cloudera.org:8080/#/c/19001/8/testdata/workloads/functional-query/queries/QueryTest/iceberg-alter.test File testdata/workloads/functional-query/queries/QueryTest/iceberg-alter.test: http://gerrit.cloudera.org:8080/#/c/19001/8/testdata/workloads/functional-query/queries/QueryTest/iceberg-alter.test@77 PS8, Line 77: other > As far as I can tell the test cleans up after itself. Format deletes everyt Thanks for the clarification. http://gerrit.cloudera.org:8080/#/c/19001/8/tests/custom_cluster/test_startup_filesystem_checks.py File tests/custom_cluster/test_startup_filesystem_checks.py: http://gerrit.cloudera.org:8080/#/c/19001/8/tests/custom_cluster/test_startup_filesystem_checks.py@50 PS8, Line 50: "/" > Actually it's just checking that they're valid directories, checking '/' is Thanks for the explanation. -- To view, visit http://gerrit.cloudera.org:8080/19001 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45e90d30b2e68876dec0db3c43ac15ee510b17bd Gerrit-Change-Number: 19001 Gerrit-PatchSet: 8 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 28 Sep 2022 09:18:33 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10791 Add batch reading for remote temporary files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17979 ) Change subject: IMPALA-10791 Add batch reading for remote temporary files .. Patch Set 23: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/17979 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1dcc5d0881ffaeff09c5c514306cd668373ad31b Gerrit-Change-Number: 17979 Gerrit-PatchSet: 23 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Wed, 28 Sep 2022 09:02:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10791 Add batch reading for remote temporary files
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17979 ) Change subject: IMPALA-10791 Add batch reading for remote temporary files .. IMPALA-10791 Add batch reading for remote temporary files The patch adds a feature to batch read from a remote temporary file in order to improve the reading performance for the spilled remote data. Originally, the design is to use the local disk file as the buffer for batch read from the remote file. But in practice, it doesn't help to improve the performance. Therefore, the design is changed to use the memory as the read buffer. Currently, each TmpFileRemote has two DiskFile, one is for the remote, and one is for the local buffer. The patch adds MemBlocks to the local buffer file. Each local buffer file is divided into several MemBlocks evenly. Moreover, in order to guarantee a single page not being cut into two parts in different blocks, the block size could be a little different to each other in practice. The default block size is the minimum value between the default file size and MAX_REMOTE_READ_MEM_BLOCK_THRESHOLD_BYTES, which is 16MB. When pinning a page, the system will detect if there is enough memory for the block that holds the page. If yes, the block will be stored in the memory until all of the pages in the block are read or the query ends. If not, we will go reading the page directly and disable this block, because it may be good to avoid duplicated reads from the remote fs for the same content. One challenge of the read buffer is where to get the extra memory for it, because when impala starts to spill data, it means the process lacks of memory to use. By default, impala process will reserve 20% of the total system memory as unused memory, and here we will use this unused memory for the read buffer because it is reasonable to use it for the emergency case like spilling and the memory of the read buffer will be returned immediately after the use. For system reliability consideration, we set a restriction that, the maximum bytes of the read buffer memory are no more than 10% of the total system memory and 50% of the unused memory. Also, if the unused memory is less than 5% of the total system memory, the read buffer will be disabled. Two start options have been added for the new feature. 1. remote_batch_read. Default is false. If set true, the batch read is enabled. 2. remote_read_memory_buffer_size. Default is 1G. The maximum memory that can be used by the read buffer. The number is also restricted by the process memory limit, which can not exceed 10% of the process memory limit. Added metrics ScratchReadsUseMem/ScratchBytesReadUseMem/ ScratchBytesReadUseLocalDisk to the query profile. The patch also increases the MAX_REMOTE_TMPFILE_SIZE_THRESHOLD_MB from 256 to 512. Tests: Ran core and exhaustive tests. Added and ran TmpFileMgrTest::TestBatchReadingFromRemote. Added e2e test test_scratch_dirs_batch_reading. Change-Id: I1dcc5d0881ffaeff09c5c514306cd668373ad31b Reviewed-on: http://gerrit.cloudera.org:8080/17979 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/runtime/io/CMakeLists.txt A be/src/runtime/io/disk-file-test.cc M be/src/runtime/io/disk-file.cc M be/src/runtime/io/disk-file.h M be/src/runtime/io/disk-io-mgr-test.cc M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/request-context.cc M be/src/runtime/io/request-context.h M be/src/runtime/io/request-ranges.h M be/src/runtime/io/scan-range.cc M be/src/runtime/tmp-file-mgr-internal.h M be/src/runtime/tmp-file-mgr-test.cc M be/src/runtime/tmp-file-mgr.cc M be/src/runtime/tmp-file-mgr.h M be/src/util/mem-info.cc M be/src/util/mem-info.h M be/src/util/metrics.h M common/thrift/metrics.json M tests/custom_cluster/test_scratch_disk.py 19 files changed, 1,666 insertions(+), 156 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/17979 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I1dcc5d0881ffaeff09c5c514306cd668373ad31b Gerrit-Change-Number: 17979 Gerrit-PatchSet: 24 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-4052: CREATE TABLE LIKE for Kudu tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18729 ) Change subject: IMPALA-4052: CREATE TABLE LIKE for Kudu tables .. Patch Set 22: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18729 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia3d276a6465301dbcfed17bb713aca06367d9a42 Gerrit-Change-Number: 18729 Gerrit-PatchSet: 22 Gerrit-Owner: Xiaoqing Gao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Xiaoqing Gao Gerrit-Comment-Date: Wed, 28 Sep 2022 08:31:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11576: Fix for test multiple storage locations on S3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19045 ) Change subject: IMPALA-11576: Fix for test_multiple_storage_locations on S3 .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19045 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I08badd96d060a2377c9a1eafa287a3adf8fa11db Gerrit-Change-Number: 19045 Gerrit-PatchSet: 4 Gerrit-Owner: Gergely Fürnstáhl Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 28 Sep 2022 08:08:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11576: Fix for test multiple storage locations on S3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19045 ) Change subject: IMPALA-11576: Fix for test_multiple_storage_locations on S3 .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8630/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19045 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I08badd96d060a2377c9a1eafa287a3adf8fa11db Gerrit-Change-Number: 19045 Gerrit-PatchSet: 4 Gerrit-Owner: Gergely Fürnstáhl Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 28 Sep 2022 08:08:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10356: Analyzed query in explain plan is not right for insert with values clause
Peter Rozsa has posted comments on this change. ( http://gerrit.cloudera.org:8080/19048 ) Change subject: IMPALA-10356: Analyzed query in explain plan is not right for insert with values clause .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/19048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I952377ed14eba26e3774e7776eb81a95d1d8e76f Gerrit-Change-Number: 19048 Gerrit-PatchSet: 1 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Wed, 28 Sep 2022 08:01:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses
Jian Zhang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19046 ) Change subject: IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses .. Patch Set 3: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/19046 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1003556d3afc8e8216142cac4007a4c99046caeb Gerrit-Change-Number: 19046 Gerrit-PatchSet: 3 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Xiang Yang Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Wed, 28 Sep 2022 07:56:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19046 ) Change subject: IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11466/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19046 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1003556d3afc8e8216142cac4007a4c99046caeb Gerrit-Change-Number: 19046 Gerrit-PatchSet: 3 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Xiang Yang Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Wed, 28 Sep 2022 08:00:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11555: external Udf support decimal(*, *)
Jian Zhang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19021 ) Change subject: IMPALA-11555: external Udf support decimal(*, *) .. Patch Set 3: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/19021 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I09f19da5b1660ed0b882497fc321e6cc1b9b55d0 Gerrit-Change-Number: 19021 Gerrit-PatchSet: 3 Gerrit-Owner: Minghui Zhu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Minghui Zhu Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 28 Sep 2022 07:51:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses
Xianqing He has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/19046 ) Change subject: IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses .. IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses Creating Kudu clients is very expensive as each will fetch metadata from the Kudu master, so we can ensure one Kudu client created for the specified Kudu master addresses. The solution is to ensure KuduUtil.getKuduClient created only one KuduClient for the specified Kudu master addresses by using 'computeIfAbsent' of the ConcurrentHashMap. Testing: - Manually ran a stress test: scan of a Kudu table, 1000 concurrent queries and verified the untracked memory - Added concurrent tests for KuduUtil.getKuduClient - Ran the full set of verifications in Impala Public Jenkins Change-Id: I1003556d3afc8e8216142cac4007a4c99046caeb --- M fe/src/main/java/org/apache/impala/util/KuduUtil.java A fe/src/test/java/org/apache/impala/util/KuduUtilTest.java 2 files changed, 89 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/46/19046/3 -- To view, visit http://gerrit.cloudera.org:8080/19046 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1003556d3afc8e8216142cac4007a4c99046caeb Gerrit-Change-Number: 19046 Gerrit-PatchSet: 3 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Xiang Yang Gerrit-Reviewer: Xianqing He
[Impala-ASF-CR] IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
Jian Zhang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17830 ) Change subject: IMPALA-10902: In the case of outer join connections, skip unnecessary BE check .. Patch Set 4: could you add some unit tests to ensure these expressions are evaluated in the frontend? -- To view, visit http://gerrit.cloudera.org:8080/17830 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6f4db9b5aa00ee5117118696c58c07ffaaf0b66 Gerrit-Change-Number: 17830 Gerrit-PatchSet: 4 Gerrit-Owner: pengdou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Comment-Date: Wed, 28 Sep 2022 07:25:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11612: fix ORDER BY expression rewrite bug
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19049 ) Change subject: IMPALA-11612: fix ORDER BY expression rewrite bug .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11465/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19049 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I438c61c647e3f99665c94ea15e3a9d98b9ff9912 Gerrit-Change-Number: 19049 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 28 Sep 2022 07:20:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11235: Support Pluggable Authentication for Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19042 ) Change subject: IMPALA-11235: Support Pluggable Authentication for Impala .. Patch Set 3: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/11464/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/19042 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieaaef6bffe641ece30e7fc3756688ef30722dfe1 Gerrit-Change-Number: 19042 Gerrit-PatchSet: 3 Gerrit-Owner: Minghui Zhu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Minghui Zhu Gerrit-Comment-Date: Wed, 28 Sep 2022 07:03:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19046 ) Change subject: IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11463/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19046 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1003556d3afc8e8216142cac4007a4c99046caeb Gerrit-Change-Number: 19046 Gerrit-PatchSet: 2 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Xiang Yang Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Wed, 28 Sep 2022 07:02:10 + Gerrit-HasComments: No
