[Impala-ASF-CR] IMPALA-11008: fix incorrect to propagate inferred predicates
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18234 ) Change subject: IMPALA-11008: fix incorrect to propagate inferred predicates .. Patch Set 9: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7989/ -- To view, visit http://gerrit.cloudera.org:8080/18234 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e64230f6d0c2b9ef1560186ceba349a5920ccdf Gerrit-Change-Number: 18234 Gerrit-PatchSet: 9 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Thu, 31 Mar 2022 06:32:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18324 ) Change subject: IMPALA-11181: Improving performance of compaction checking .. IMPALA-11181: Improving performance of compaction checking After HIVE-25753, we don't need to explicitly set all partitions' name to get the latest compaction id. Besides, we can also send the last compaction id over to HMS so that HMS will send back compaction info only if there are newer compactions. In this way, we can avoid unnecessary data transmitted between HMS and Catalogd. Testing: existing tests Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c Reviewed-on: http://gerrit.cloudera.org:8080/18324 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java 3 files changed, 31 insertions(+), 25 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/18324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c Gerrit-Change-Number: 18324 Gerrit-PatchSet: 6 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Yu-Wen Lai
[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 ) Change subject: IMPALA-11181: Improving performance of compaction checking .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c Gerrit-Change-Number: 18324 Gerrit-PatchSet: 5 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Thu, 31 Mar 2022 05:50:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11126: impala-shell: Support configurable socket timeout for http client
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18336 ) Change subject: IMPALA-11126: impala-shell: Support configurable socket timeout for http client .. Patch Set 12: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7990/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18336 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29fa4ff96cdcf154c3aac7e43340af60d7d61e94 Gerrit-Change-Number: 18336 Gerrit-PatchSet: 12 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 31 Mar 2022 05:05:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11126: impala-shell: Support configurable socket timeout for http client
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18336 ) Change subject: IMPALA-11126: impala-shell: Support configurable socket timeout for http client .. Patch Set 12: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7987/ -- To view, visit http://gerrit.cloudera.org:8080/18336 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29fa4ff96cdcf154c3aac7e43340af60d7d61e94 Gerrit-Change-Number: 18336 Gerrit-PatchSet: 12 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 31 Mar 2022 04:44:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 ) Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches .. Patch Set 1: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7986/ -- To view, visit http://gerrit.cloudera.org:8080/18370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 Gerrit-Change-Number: 18370 Gerrit-PatchSet: 1 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 31 Mar 2022 03:49:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11008: fix incorrect to propagate inferred predicates
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18234 ) Change subject: IMPALA-11008: fix incorrect to propagate inferred predicates .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7989/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18234 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e64230f6d0c2b9ef1560186ceba349a5920ccdf Gerrit-Change-Number: 18234 Gerrit-PatchSet: 9 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Thu, 31 Mar 2022 02:10:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 ) Change subject: IMPALA-11181: Improving performance of compaction checking .. Patch Set 4: Code-Review+2 LGTM. Thanks for working on this! -- To view, visit http://gerrit.cloudera.org:8080/18324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c Gerrit-Change-Number: 18324 Gerrit-PatchSet: 4 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Thu, 31 Mar 2022 01:22:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 ) Change subject: IMPALA-11181: Improving performance of compaction checking .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7988/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c Gerrit-Change-Number: 18324 Gerrit-PatchSet: 5 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Thu, 31 Mar 2022 01:23:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 ) Change subject: IMPALA-11181: Improving performance of compaction checking .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c Gerrit-Change-Number: 18324 Gerrit-PatchSet: 5 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Thu, 31 Mar 2022 01:23:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11204: Template implementation for OrcStringColumnReader::ReadValue
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18366 ) Change subject: IMPALA-11204: Template implementation for OrcStringColumnReader::ReadValue .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10368/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18366 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5e5f88c28059fb3d3ac1172e6d383d06ee3bedd5 Gerrit-Change-Number: 18366 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Thu, 31 Mar 2022 01:16:27 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-10854: Use ZSTD in toolchain to build ORC
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18367 ) Change subject: IMPALA-10854: Use ZSTD in toolchain to build ORC .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/18367 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifa8f0f55254e777b0132480a80737d9169410c73 Gerrit-Change-Number: 18367 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 31 Mar 2022 01:14:30 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-10854: Use ZSTD in toolchain to build ORC
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18367 ) Change subject: IMPALA-10854: Use ZSTD in toolchain to build ORC .. Patch Set 1: > Patch Set 1: > > We could remove the protobuf related patch when disable building ORC test. That will bring down the version from 1.7.0-p7 to 1.7.0-p6. I'm afraid it will introduce misunderstaning.. I think we can drop it next time when we bump the base version. -- To view, visit http://gerrit.cloudera.org:8080/18367 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifa8f0f55254e777b0132480a80737d9169410c73 Gerrit-Change-Number: 18367 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 31 Mar 2022 01:02:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11204: Template implementation for OrcStringColumnReader::ReadValue
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18366 ) Change subject: IMPALA-11204: Template implementation for OrcStringColumnReader::ReadValue .. Patch Set 5: (2 comments) http://gerrit.cloudera.org:8080/#/c/18366/4/be/src/exec/orc-column-readers.cc File be/src/exec/orc-column-readers.cc: http://gerrit.cloudera.org:8080/#/c/18366/4/be/src/exec/orc-column-readers.cc@184 PS4, Line 184: If it doesn't turn out to be too complicated, I would prefer to use a templ Done http://gerrit.cloudera.org:8080/#/c/18366/4/be/src/exec/orc-column-readers.cc@228 PS4, Line 228: > I don't know whether we have a rule for this, but I would prefer SLOT_TYPE I can't find a style guide about this in https://google.github.io/styleguide/cppguide.html I fount that we have both cases: https://github.com/apache/impala/blob/b1c1be12f3cceed48e93eddae8b9512737e3e0d2/be/src/exec/parquet/parquet-column-readers.cc#L56 https://github.com/apache/impala/blob/b1c1be12f3cceed48e93eddae8b9512737e3e0d2/be/src/exprs/date-functions.h#L132 I'm ok to use uppercase. -- To view, visit http://gerrit.cloudera.org:8080/18366 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5e5f88c28059fb3d3ac1172e6d383d06ee3bedd5 Gerrit-Change-Number: 18366 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Thu, 31 Mar 2022 00:56:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11204: Template implementation for OrcStringColumnReader::ReadValue
Hello Riza Suminto, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18366 to look at the new patch set (#5). Change subject: IMPALA-11204: Template implementation for OrcStringColumnReader::ReadValue .. IMPALA-11204: Template implementation for OrcStringColumnReader::ReadValue There are some checks in OrcStringColumnReader::ReadValue() that we can determine outside the scope of this method. They should be optimized since this is a critical method that will be executed for each row (and for each string column). With these checks, the method is too complex to be inlined in OrcBatchedReader::ReadValueBatch() by the compiler. This patch templates OrcStringColumnReader::ReadValue() with two parameters, one for the target slot type (i.e. STRING/CHAR/VARCHAR), ther other for whether the column is dictionary encoded. Also adds an ALWAYS_INLINE marker to force inlining it. OrcStringColumnReader::ReadValueBatch() will call a template version of ReadValue() based on the slot type and the orc batch encoded state. Ran a single node perf test on TPCH(30) on my dev box using 3 impalad instances. There are some improvements and no significant regressions: +--++-++ | Query| Avg(s) | Base Avg(s) | Delta(Avg) | +--++-++ | TPCH-Q19 | 5.62 | 6.07| I -7.41% | | TPCH-Q6 | 2.56 | 2.78| I -7.77% | | TPCH-Q4 | 3.85 | 4.25| I -9.42% | | TPCH-Q12 | 4.25 | 4.99| I -14.78% | +--++-++ Base commit: ff21728 File Format: orc/snap/block Iterations: 30 Change-Id: I5e5f88c28059fb3d3ac1172e6d383d06ee3bedd5 --- M be/src/exec/orc-column-readers.cc M be/src/exec/orc-column-readers.h 2 files changed, 77 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/18366/5 -- To view, visit http://gerrit.cloudera.org:8080/18366 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5e5f88c28059fb3d3ac1172e6d383d06ee3bedd5 Gerrit-Change-Number: 18366 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-10737: Optimize the number of Iceberg API Metadata requests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18353 ) Change subject: IMPALA-10737: Optimize the number of Iceberg API Metadata requests .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7985/ -- To view, visit http://gerrit.cloudera.org:8080/18353 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5492e0cdb31602f0276029c2645d14ff5cb2f672 Gerrit-Change-Number: 18353 Gerrit-PatchSet: 2 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 31 Mar 2022 00:53:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11210: Impala can only handle lowercase schema elements of Iceberg table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18368 ) Change subject: IMPALA-11210: Impala can only handle lowercase schema elements of Iceberg table .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18368 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iffd910f76844fbf34db805dda6c3053c5ad1cf79 Gerrit-Change-Number: 18368 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 31 Mar 2022 00:49:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11126: impala-shell: Support configurable socket timeout for http client
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18336 ) Change subject: IMPALA-11126: impala-shell: Support configurable socket timeout for http client .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10367/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18336 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29fa4ff96cdcf154c3aac7e43340af60d7d61e94 Gerrit-Change-Number: 18336 Gerrit-PatchSet: 11 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 31 Mar 2022 00:09:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11126: impala-shell: Support configurable socket timeout for http client
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18336 ) Change subject: IMPALA-11126: impala-shell: Support configurable socket timeout for http client .. Patch Set 12: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7987/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18336 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29fa4ff96cdcf154c3aac7e43340af60d7d61e94 Gerrit-Change-Number: 18336 Gerrit-PatchSet: 12 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 31 Mar 2022 00:00:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11126: impala-shell: Support configurable socket timeout for http client
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18336 ) Change subject: IMPALA-11126: impala-shell: Support configurable socket timeout for http client .. Patch Set 11: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/18336 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29fa4ff96cdcf154c3aac7e43340af60d7d61e94 Gerrit-Change-Number: 18336 Gerrit-PatchSet: 11 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 30 Mar 2022 23:59:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11126: impala-shell: Support configurable socket timeout for http client
Abhishek Rawat has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/18336 ) Change subject: IMPALA-11126: impala-shell: Support configurable socket timeout for http client .. IMPALA-11126: impala-shell: Support configurable socket timeout for http client In 'hs2-http' mode, the socket timeout is None, which could cause hang like symptoms in case of a problematic remote server. Added support for configurable socket timeout using the new impala-shell config option '--http_socket_timeout_s'. If a reasonable timeout is set, impala-shell client can retry in case of connection issues, when possible. The default value of '--http_socket_timeout_s' is set to None, to prevent behavior changes for existing clients. More details on socket timeout here: https://docs.python.org/3/library/socket.html#socket-timeouts Testing: - Added tests for various timeout values in test_shell_commandline.py - Ran e2e shell tests. Change-Id: I29fa4ff96cdcf154c3aac7e43340af60d7d61e94 --- M shell/ImpalaHttpClient.py M shell/impala_client.py M shell/impala_shell.py M shell/impala_shell_config_defaults.py M shell/option_parser.py M tests/shell/test_shell_commandline.py 6 files changed, 70 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/36/18336/11 -- To view, visit http://gerrit.cloudera.org:8080/18336 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I29fa4ff96cdcf154c3aac7e43340af60d7d61e94 Gerrit-Change-Number: 18336 Gerrit-PatchSet: 11 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 ) Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7986/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 Gerrit-Change-Number: 18370 Gerrit-PatchSet: 1 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 30 Mar 2022 23:21:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 ) Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10366/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 Gerrit-Change-Number: 18370 Gerrit-PatchSet: 1 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 30 Mar 2022 23:18:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Steve Carlin has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18370 Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches .. IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches The strict hs2 protocol mode is broken when fetching large results. The FetchResults.hasMoreRows field is always returned as false. When there are no more results, Hive returns an empty batch with no rows. Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 --- M shell/impala_client.py M tests/shell/test_shell_commandline.py 2 files changed, 13 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/18370/1 -- To view, visit http://gerrit.cloudera.org:8080/18370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24 Gerrit-Change-Number: 18370 Gerrit-PatchSet: 1 Gerrit-Owner: Steve Carlin
[Impala-ASF-CR] IMPALA-11196 Fix a ClientCacheTest flaky test
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18363 ) Change subject: IMPALA-11196 Fix a ClientCacheTest flaky test .. IMPALA-11196 Fix a ClientCacheTest flaky test The memory leak detection testcase is flaky, in local environment, it could fail over 10 times out of 100. The fix changes to use the ordinary ADD_BE_TEST to replace ADD_UNIFIED_BE_LSAN_TEST, there could be some elements in the unified test which affect the process vm size occasionally. After the fix, the testcase ran 5000 times without a failure, and can still catch the memory leak issue in IMPALA-11176. Tests: Ran Core Tests in DEBUG/RELEASE/ASAN build. Ran consecutive 5000 times of the testcase and all passed. Change-Id: I6ee273db4f9b0a518b6438df6982325f5ad1fa0c Reviewed-on: http://gerrit.cloudera.org:8080/18363 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/runtime/CMakeLists.txt M be/src/runtime/client-cache-test.cc 2 files changed, 17 insertions(+), 16 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/18363 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I6ee273db4f9b0a518b6438df6982325f5ad1fa0c Gerrit-Change-Number: 18363 Gerrit-PatchSet: 3 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11196 Fix a ClientCacheTest flaky test
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18363 ) Change subject: IMPALA-11196 Fix a ClientCacheTest flaky test .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18363 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6ee273db4f9b0a518b6438df6982325f5ad1fa0c Gerrit-Change-Number: 18363 Gerrit-PatchSet: 2 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 30 Mar 2022 22:07:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11210: Impala can only handle lowercase schema elements of Iceberg table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18368 ) Change subject: IMPALA-11210: Impala can only handle lowercase schema elements of Iceberg table .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10365/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18368 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iffd910f76844fbf34db805dda6c3053c5ad1cf79 Gerrit-Change-Number: 18368 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 30 Mar 2022 20:49:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11210: Impala can only handle lowercase schema elements of Iceberg table
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18368 ) Change subject: IMPALA-11210: Impala can only handle lowercase schema elements of Iceberg table .. Patch Set 2: (1 comment) Thanks for the comment! http://gerrit.cloudera.org:8080/#/c/18368/2/testdata/datasets/functional/functional_schema_template.sql File testdata/datasets/functional/functional_schema_template.sql: http://gerrit.cloudera.org:8080/#/c/18368/2/testdata/datasets/functional/functional_schema_template.sql@3309 PS2, Line 3309: iceberg_uppercase_col > The new files could be mentioned in https://github.com/apache/impala/blob/m Done -- To view, visit http://gerrit.cloudera.org:8080/18368 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iffd910f76844fbf34db805dda6c3053c5ad1cf79 Gerrit-Change-Number: 18368 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 30 Mar 2022 20:30:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11210: Impala can only handle lowercase schema elements of Iceberg table
Hello Tamas Mate, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18368 to look at the new patch set (#3). Change subject: IMPALA-11210: Impala can only handle lowercase schema elements of Iceberg table .. IMPALA-11210: Impala can only handle lowercase schema elements of Iceberg table When Impala/Hive creates a table they lowercase the schema elements. When Spark creates an Iceberg table it doesn't lowercase the names of the columns in the Iceberg metadata. This triggers a precondition check in Impala which makes such Iceberg tables unloadable. This patch converts column names to lowercase when converting Iceberg schemas to Hive/Impala schemas. Testing: * added e2e test Change-Id: Iffd910f76844fbf34db805dda6c3053c5ad1cf79 --- M fe/src/main/java/org/apache/impala/catalog/IcebergColumn.java M fe/src/main/java/org/apache/impala/util/IcebergSchemaConverter.java M testdata/data/README A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_uppercase_col/metadata/v1.metadata.json A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_uppercase_col/metadata/version-hint.txt M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test 8 files changed, 272 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/18368/3 -- To view, visit http://gerrit.cloudera.org:8080/18368 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iffd910f76844fbf34db805dda6c3053c5ad1cf79 Gerrit-Change-Number: 18368 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate
[Impala-ASF-CR] IMPALA-10737: Optimize the number of Iceberg API Metadata requests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18353 ) Change subject: IMPALA-10737: Optimize the number of Iceberg API Metadata requests .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7985/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18353 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5492e0cdb31602f0276029c2645d14ff5cb2f672 Gerrit-Change-Number: 18353 Gerrit-PatchSet: 2 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 30 Mar 2022 20:27:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11210: Impala can only handle lowercase schema elements of Iceberg table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18368 ) Change subject: IMPALA-11210: Impala can only handle lowercase schema elements of Iceberg table .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7984/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18368 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iffd910f76844fbf34db805dda6c3053c5ad1cf79 Gerrit-Change-Number: 18368 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Wed, 30 Mar 2022 20:21:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10737: Optimize the number of Iceberg API Metadata requests
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18353 ) Change subject: IMPALA-10737: Optimize the number of Iceberg API Metadata requests .. Patch Set 2: (4 comments) http://gerrit.cloudera.org:8080/#/c/18353/2/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java File fe/src/main/java/org/apache/impala/catalog/IcebergTable.java: http://gerrit.cloudera.org:8080/#/c/18353/2/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java@336 PS2, Line 336: FeIcebergTable.super Is supert.snapshotId() not enough? http://gerrit.cloudera.org:8080/#/c/18353/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java: http://gerrit.cloudera.org:8080/#/c/18353/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1062 PS2, Line 1062: iceberg api metadata iceberg api table? http://gerrit.cloudera.org:8080/#/c/18353/2/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java File fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java: http://gerrit.cloudera.org:8080/#/c/18353/2/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java@145 PS2, Line 145: // TODO: get rid of try-catch TableLoadingException once we have IMPALA-10737. Please remove the try-catch http://gerrit.cloudera.org:8080/#/c/18353/2/fe/src/main/java/org/apache/impala/util/IcebergUtil.java File fe/src/main/java/org/apache/impala/util/IcebergUtil.java: http://gerrit.cloudera.org:8080/#/c/18353/2/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@853 PS2, Line 853: //TODO: avoid loading the table once we have IMPALA-10737 again: : Table iceTbl = loadTable(feTbl); Please use getIcebergApiTable() -- To view, visit http://gerrit.cloudera.org:8080/18353 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5492e0cdb31602f0276029c2645d14ff5cb2f672 Gerrit-Change-Number: 18353 Gerrit-PatchSet: 2 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 30 Mar 2022 20:17:04 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11210: Impala can only handle lowercase schema elements of Iceberg table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18368 ) Change subject: IMPALA-11210: Impala can only handle lowercase schema elements of Iceberg table .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7982/ -- To view, visit http://gerrit.cloudera.org:8080/18368 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iffd910f76844fbf34db805dda6c3053c5ad1cf79 Gerrit-Change-Number: 18368 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Wed, 30 Mar 2022 19:53:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11210: Impala can only handle lowercase schema elements of Iceberg table
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18368 ) Change subject: IMPALA-11210: Impala can only handle lowercase schema elements of Iceberg table .. Patch Set 2: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/18368/2/testdata/datasets/functional/functional_schema_template.sql File testdata/datasets/functional/functional_schema_template.sql: http://gerrit.cloudera.org:8080/#/c/18368/2/testdata/datasets/functional/functional_schema_template.sql@3309 PS2, Line 3309: iceberg_uppercase_col The new files could be mentioned in https://github.com/apache/impala/blob/master/testdata/data/README -- To view, visit http://gerrit.cloudera.org:8080/18368 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iffd910f76844fbf34db805dda6c3053c5ad1cf79 Gerrit-Change-Number: 18368 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Wed, 30 Mar 2022 19:19:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11123: Optimize count(star) for ORC scans
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18327 ) Change subject: IMPALA-11123: Optimize count(star) for ORC scans .. Patch Set 8: Code-Review+1 (7 comments) Looks great! http://gerrit.cloudera.org:8080/#/c/18327/8//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/18327/8//COMMIT_MSG@9 PS8, Line 9: IMPALA-5036 added optimization for count(star) in Parquet scans that : avoid materializing dummy rows. nit. This patch provides count(star) optimization for ORC scans, similar to the work done in IMPALA-5036 for Parquet scans. http://gerrit.cloudera.org:8080/#/c/18327/8/testdata/workloads/functional-planner/queries/PlannerTest/orc-stats-agg.test File testdata/workloads/functional-planner/queries/PlannerTest/orc-stats-agg.test: http://gerrit.cloudera.org:8080/#/c/18327/8/testdata/workloads/functional-planner/queries/PlannerTest/orc-stats-agg.test@3 PS8, Line 3: # a text table, so the optimization is not applied. nit. May add a comment: The optimization is observed when the cardinality of the ORC scan (24) is the same as the # of files (24). http://gerrit.cloudera.org:8080/#/c/18327/8/testdata/workloads/functional-planner/queries/PlannerTest/orc-stats-agg.test@126 PS8, Line 126: should be nit. is http://gerrit.cloudera.org:8080/#/c/18327/8/testdata/workloads/functional-planner/queries/PlannerTest/orc-stats-agg.test@168 PS8, Line 168: there are two aggregate functions nit. it can not be applied to the 1st aggregate function. http://gerrit.cloudera.org:8080/#/c/18327/8/testdata/workloads/functional-planner/queries/PlannerTest/orc-stats-agg.test@181 PS8, Line 181: because the inner count(*) is not materialized. The outer : # count(*) does not reference a base table. nit. The optimization does apply to the inner count(*). http://gerrit.cloudera.org:8080/#/c/18327/8/testdata/workloads/functional-planner/queries/PlannerTest/orc-stats-agg.test@277 PS8, Line 277: there nit. all predicates are on partition columns only. http://gerrit.cloudera.org:8080/#/c/18327/8/testdata/workloads/functional-planner/queries/PlannerTest/orc-stats-agg.test@386 PS8, Line 386: # Optimization is not applied when there is a distinct agg. nit. in general. -- To view, visit http://gerrit.cloudera.org:8080/18327 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0fafa1182f97323aeb9ee39dd4e8ecd418fa6091 Gerrit-Change-Number: 18327 Gerrit-PatchSet: 8 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Wed, 30 Mar 2022 18:19:12 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-11129: Support running KRPC over Unix Domain Socket
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 ) Change subject: WIP IMPALA-11129: Support running KRPC over Unix Domain Socket .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10364/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 Gerrit-Change-Number: 18369 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 30 Mar 2022 17:47:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11196 Fix a ClientCacheTest flaky test
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18363 ) Change subject: IMPALA-11196 Fix a ClientCacheTest flaky test .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7983/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18363 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6ee273db4f9b0a518b6438df6982325f5ad1fa0c Gerrit-Change-Number: 18363 Gerrit-PatchSet: 2 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 30 Mar 2022 17:38:23 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-11129: Support running KRPC over Unix Domain Socket
Wenzhe Zhou has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/18369 ) Change subject: WIP IMPALA-11129: Support running KRPC over Unix Domain Socket .. WIP IMPALA-11129: Support running KRPC over Unix Domain Socket This patch make following changes to support running KRPC over UDS. - Add a FLAG_rpc_use_unix_domain_socket to enable running KRPC over UDS. - Add field uds_address in NetworkAddressPB. Replace TNetworkAddress with NetworkAddressPB for RPC related class variables and APIs. - Set UDS address for each daemon as @impala-kprc: during initialization. - When FLAG_rpc_use_unix_domain_socket is true, the socket of KRPC server will be binded to the UDS address of the daemon. KRPC Client will connect to KRPC server with the UDS address of the server when creating proxy service, which in turn call kudu::Socket::Connect() function to connect KRPC server. TODO: - Find a way to advertise BackendId of admissiond in global admission control mode. - Fix rpcz web page when FLAG_rpc_use_unix_domain_socket is true. webserver.test_web_pages.TestWebPage.test_krpc_rpcz is skiped now when FLAG_rpc_use_unix_domain_socket is set as true. There are two issues: - rpcz web page show TCP related stats which are not available for UDS. We may use a different tmpl file for UDS. - Cannot get remote address from Unix domain socket for inbound connection. - Add more unit-tests. Testing: - Passed core test with FLAG_rpc_use_unix_domain_socket as fault value false. - Passed core test with FLAG_rpc_use_unix_domain_socket as true. Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 --- M be/src/benchmarks/expr-benchmark.cc M be/src/rpc/impala-service-pool.cc M be/src/rpc/impala-service-pool.h M be/src/rpc/rpc-mgr-kerberized-test.cc M be/src/rpc/rpc-mgr-test.cc M be/src/rpc/rpc-mgr-test.h M be/src/rpc/rpc-mgr.cc M be/src/rpc/rpc-mgr.h M be/src/rpc/rpc-mgr.inline.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/data-stream-test.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/krpc-data-stream-mgr.cc M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/query-driver.cc M be/src/runtime/query-state.cc M be/src/runtime/runtime-filter-bank.cc M be/src/runtime/test-env.cc M be/src/runtime/tmp-file-mgr.cc M be/src/scheduling/admissiond-env.cc M be/src/scheduling/admissiond-env.h M be/src/service/client-request-state.cc M be/src/service/control-service.cc M be/src/service/control-service.h M be/src/service/data-stream-service.cc M be/src/service/data-stream-service.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/testutil/in-process-servers.cc M be/src/util/container-util.h M be/src/util/network-util.cc M be/src/util/network-util.h M common/protobuf/common.proto M tests/custom_cluster/test_restart_services.py M tests/webserver/test_web_pages.py 36 files changed, 348 insertions(+), 116 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/18369/2 -- To view, visit http://gerrit.cloudera.org:8080/18369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 Gerrit-Change-Number: 18369 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-11008: fix incorrect to propagate inferred predicates
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18234 ) Change subject: IMPALA-11008: fix incorrect to propagate inferred predicates .. Patch Set 9: (2 comments) http://gerrit.cloudera.org:8080/#/c/18234/7/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/18234/7/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@2150 PS7, Line 2150: final substitution. : ExprSubstitutionMap smap = new ExprSubstitutionMap(); : Exp > The 'ojmap' is not a subset of 'smap'. The slots in 'ojNullableEqualSlots' Done http://gerrit.cloudera.org:8080/#/c/18234/7/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@2729 PS7, Line 2729: // the joined table -> ignore this, we can't reason about when it'll : // actually be true > 1.We don't add value-transfer edges based on full outer joins, see #2697. Done -- To view, visit http://gerrit.cloudera.org:8080/18234 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e64230f6d0c2b9ef1560186ceba349a5920ccdf Gerrit-Change-Number: 18234 Gerrit-PatchSet: 9 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Wed, 30 Mar 2022 17:24:17 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-11129: Support running KRPC over Unix Domain Socket
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 ) Change subject: WIP IMPALA-11129: Support running KRPC over Unix Domain Socket .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/10363/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/18369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 Gerrit-Change-Number: 18369 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 30 Mar 2022 16:46:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 ) Change subject: IMPALA-11181: Improving performance of compaction checking .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10362/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c Gerrit-Change-Number: 18324 Gerrit-PatchSet: 4 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Wed, 30 Mar 2022 16:38:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11204: Template implementation for OrcStringColumnReader::ReadValue
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18366 ) Change subject: IMPALA-11204: Template implementation for OrcStringColumnReader::ReadValue .. Patch Set 4: Code-Review+1 (2 comments) http://gerrit.cloudera.org:8080/#/c/18366/4/be/src/exec/orc-column-readers.cc File be/src/exec/orc-column-readers.cc: http://gerrit.cloudera.org:8080/#/c/18366/4/be/src/exec/orc-column-readers.cc@184 PS4, Line 184: READ_STRING_BATCH If it doesn't turn out to be too complicated, I would prefer to use a templated function instead. http://gerrit.cloudera.org:8080/#/c/18366/4/be/src/exec/orc-column-readers.cc@228 PS4, Line 228: slot_type I don't know whether we have a rule for this, but I would prefer SLOT_TYPE to avoid looking like a simple variable. -- To view, visit http://gerrit.cloudera.org:8080/18366 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5e5f88c28059fb3d3ac1172e6d383d06ee3bedd5 Gerrit-Change-Number: 18366 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Wed, 30 Mar 2022 16:29:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10838: Error when struct returned from WITH()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17847 ) Change subject: IMPALA-10838: Error when struct returned from WITH() .. Patch Set 19: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10361/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17847 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iadb9233677355b85d424cc3f22b00b5a3bf61c57 Gerrit-Change-Number: 17847 Gerrit-PatchSet: 19 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Comment-Date: Wed, 30 Mar 2022 16:28:17 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-11129: Support running KRPC over Unix Domain Socket
Wenzhe Zhou has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18369 Change subject: WIP IMPALA-11129: Support running KRPC over Unix Domain Socket .. WIP IMPALA-11129: Support running KRPC over Unix Domain Socket This patch make following changes to support running KRPC over UDS. - Add a FLAG_rpc_use_unix_domain_socket to enable running KRPC over UDS. - Add field uds_address in NetworkAddressPB. Replace TNetworkAddress with NetworkAddressPB for RPC related class variables and APIs. - Set UDS address for each daemon as @impala-kprc: during initialization. - When FLAG_rpc_use_unix_domain_socket is true, the socket of KRPC server will be binded to the UDS address of the daemon. KRPC Client will connect to KRPC server with the UDS address of the server when creating proxy service, which in turn call kudu::Socket::Connect() function to connect KRPC server. TODO: - Find a way to advertise BackendId of admissiond in global admission control mode. - Fix rpcz web page when FLAG_rpc_use_unix_domain_socket is true. webserver.test_web_pages.TestWebPage.test_krpc_rpcz is skiped now when FLAG_rpc_use_unix_domain_socket is set as true. There are two issues: - rpcz web page show TCP related stats which are not available for UDS. We may use a different tmpl file for UDS. - Cannot get remote address from Unix domain socket for inbound connection. - Add more unit-tests. Testing: - Passed core test with FLAG_rpc_use_unix_domain_socket as fault value false. - Passed core test with FLAG_rpc_use_unix_domain_socket as true. Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 --- M be/src/benchmarks/expr-benchmark.cc M be/src/rpc/impala-service-pool.cc M be/src/rpc/impala-service-pool.h M be/src/rpc/rpc-mgr-kerberized-test.cc M be/src/rpc/rpc-mgr-test.cc M be/src/rpc/rpc-mgr-test.h M be/src/rpc/rpc-mgr.cc M be/src/rpc/rpc-mgr.h M be/src/rpc/rpc-mgr.inline.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/data-stream-test.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/krpc-data-stream-mgr.cc M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/query-driver.cc M be/src/runtime/query-state.cc M be/src/runtime/runtime-filter-bank.cc M be/src/runtime/test-env.cc M be/src/runtime/tmp-file-mgr.cc M be/src/scheduling/admissiond-env.cc M be/src/scheduling/admissiond-env.h M be/src/service/client-request-state.cc M be/src/service/control-service.cc M be/src/service/control-service.h M be/src/service/data-stream-service.cc M be/src/service/data-stream-service.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/testutil/in-process-servers.cc M be/src/util/container-util.h M be/src/util/network-util.cc M be/src/util/network-util.h M common/protobuf/common.proto M tests/custom_cluster/test_restart_services.py M tests/webserver/test_web_pages.py 36 files changed, 348 insertions(+), 116 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/18369/1 -- To view, visit http://gerrit.cloudera.org:8080/18369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 Gerrit-Change-Number: 18369 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking
Yu-Wen Lai has posted comments on this change. (
http://gerrit.cloudera.org:8080/18324 )
Change subject: IMPALA-11181: Improving performance of compaction checking
..
Patch Set 4:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18324/3/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/18324/3/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@705
PS3, Line 705: if
(partNameToCompactionId.containsKey(entry.getKey().getName())) {
: stalePartitions.add(entry.getKey());
: iter.remove();
> nit: Can we optimize this to the following case?
Done
http://gerrit.cloudera.org:8080/#/c/18324/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/18324/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@295
PS3, Line 295:
> nit: Could you add a blank line before this?
Done
--
To view, visit http://gerrit.cloudera.org:8080/18324
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
Gerrit-Change-Number: 18324
Gerrit-PatchSet: 4
Gerrit-Owner: Yu-Wen Lai
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Sourabh Goyal
Gerrit-Reviewer: Yu-Wen Lai
Gerrit-Comment-Date: Wed, 30 Mar 2022 16:20:28 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking
Yu-Wen Lai has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/18324 ) Change subject: IMPALA-11181: Improving performance of compaction checking .. IMPALA-11181: Improving performance of compaction checking After HIVE-25753, we don't need to explicitly set all partitions' name to get the latest compaction id. Besides, we can also send the last compaction id over to HMS so that HMS will send back compaction info only if there are newer compactions. In this way, we can avoid unnecessary data transmitted between HMS and Catalogd. Testing: existing tests Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c --- M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java 3 files changed, 31 insertions(+), 25 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/18324/4 -- To view, visit http://gerrit.cloudera.org:8080/18324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c Gerrit-Change-Number: 18324 Gerrit-PatchSet: 4 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Yu-Wen Lai
[Impala-ASF-CR] IMPALA-10838: Error when struct returned from WITH()
Daniel Becker has uploaded a new patch set (#19). ( http://gerrit.cloudera.org:8080/17847 ) Change subject: IMPALA-10838: Error when struct returned from WITH() .. IMPALA-10838: Error when struct returned from WITH() The following query fails: ''' with sub as ( select id, outer_struct from functional_orc_def.complextypes_nested_structs) select sub.id, sub.outer_struct.inner_struct2 from sub; ''' with the following error: ''' ERROR: IllegalStateException: Illegal reference to non-materialized tuple: debugname=InlineViewRef sub alias=sub tid=6 ''' while if 'outer_struct.inner_struct2' is added to the select list of the inline view, the query works as expected. This change fixes the problem by two modifications: - if a field of a struct needs to be materialised, also materialise all of its enclosing structs (ancestors) - in InlineViewRef, struct fields are inserted into the 'smap' and 'baseTableSmap' with the appropriate inline view prefix TODO: Is this just a hack with the labels? This change also changes the way struct fields are materialised: until now, if a member of a struct was needed to be materialised, the whole struct, including other members of the struct were materialised. This behaviour can lead to using significantly more memory than necessary if we for example query a single member of a large struct. This change modifies this behaviour so that we only materialise the struct members that are actually needed. Tests: - added queries that are fixed by this change (including the one above) in nested-struct-in-select-list.test - added a planner test in fe/src/test/java/org/apache/impala/planner/PlannerTest.java that asserts that only the required parts of structs are materialised Change-Id: Iadb9233677355b85d424cc3f22b00b5a3bf61c57 --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/CollectionTableRef.java M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java M fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java M fe/src/main/java/org/apache/impala/analysis/Path.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/SortInfo.java M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java M testdata/workloads/functional-query/queries/QueryTest/nested-struct-in-select-list.test 16 files changed, 814 insertions(+), 97 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/17847/19 -- To view, visit http://gerrit.cloudera.org:8080/17847 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iadb9233677355b85d424cc3f22b00b5a3bf61c57 Gerrit-Change-Number: 17847 Gerrit-PatchSet: 19 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 ) Change subject: IMPALA-11153: Make lock wait time configurable for the users .. Patch Set 7: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7981/ -- To view, visit http://gerrit.cloudera.org:8080/18289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc Gerrit-Change-Number: 18289 Gerrit-PatchSet: 7 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 30 Mar 2022 15:53:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11210: Impala can only handle lowercase schema elements of Iceberg table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18368 ) Change subject: IMPALA-11210: Impala can only handle lowercase schema elements of Iceberg table .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10360/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18368 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iffd910f76844fbf34db805dda6c3053c5ad1cf79 Gerrit-Change-Number: 18368 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Wed, 30 Mar 2022 15:48:40 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-10854: Use ZSTD in toolchain to build ORC
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18367 ) Change subject: IMPALA-10854: Use ZSTD in toolchain to build ORC .. Patch Set 1: We could remove the protobuf related patch when disable building ORC test. -- To view, visit http://gerrit.cloudera.org:8080/18367 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifa8f0f55254e777b0132480a80737d9169410c73 Gerrit-Change-Number: 18367 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 30 Mar 2022 15:43:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11210: Impala can only handle lowercase schema elements of Iceberg table
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18368 Change subject: IMPALA-11210: Impala can only handle lowercase schema elements of Iceberg table .. IMPALA-11210: Impala can only handle lowercase schema elements of Iceberg table When Impala/Hive creates a table they lowercase the schema elements. When Spark creates an Iceberg table it doesn't lowercase the names of the columns in the Iceberg metadata. This triggers a precondition check in Impala which makes such Iceberg tables unloadable. This patch converts column names to lowercase when converting Iceberg schemas to Hive/Impala schemas. Testing: * added e2e test Change-Id: Iffd910f76844fbf34db805dda6c3053c5ad1cf79 --- M fe/src/main/java/org/apache/impala/catalog/IcebergColumn.java M fe/src/main/java/org/apache/impala/util/IcebergSchemaConverter.java A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_uppercase_col/metadata/v1.metadata.json A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_uppercase_col/metadata/version-hint.txt M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test 7 files changed, 269 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/18368/1 -- To view, visit http://gerrit.cloudera.org:8080/18368 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iffd910f76844fbf34db805dda6c3053c5ad1cf79 Gerrit-Change-Number: 18368 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-11210: Impala can only handle lowercase schema elements of Iceberg table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18368 ) Change subject: IMPALA-11210: Impala can only handle lowercase schema elements of Iceberg table .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7982/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18368 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iffd910f76844fbf34db805dda6c3053c5ad1cf79 Gerrit-Change-Number: 18368 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Wed, 30 Mar 2022 15:30:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 ) Change subject: IMPALA-11153: Make lock wait time configurable for the users .. Patch Set 6: Code-Review+2 Carry +2 -- To view, visit http://gerrit.cloudera.org:8080/18289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc Gerrit-Change-Number: 18289 Gerrit-PatchSet: 6 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 30 Mar 2022 11:29:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 ) Change subject: IMPALA-11153: Make lock wait time configurable for the users .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7981/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc Gerrit-Change-Number: 18289 Gerrit-PatchSet: 7 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 30 Mar 2022 11:29:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11153: Make lock wait time configurable for the users
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18289 ) Change subject: IMPALA-11153: Make lock wait time configurable for the users .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18289 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I055b76138dd30b2c40eedb48905cb3bade1438fc Gerrit-Change-Number: 18289 Gerrit-PatchSet: 7 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 30 Mar 2022 11:29:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11038: Zipping unnest from view
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18094 ) Change subject: IMPALA-11038: Zipping unnest from view .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10359/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18094 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I68f649dda9e41f257e7f6596193d07b24049f92a Gerrit-Change-Number: 18094 Gerrit-PatchSet: 8 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 30 Mar 2022 09:30:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11038: Zipping unnest from view
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/18094 ) Change subject: IMPALA-11038: Zipping unnest from view .. Patch Set 8: (5 comments) http://gerrit.cloudera.org:8080/#/c/18094/7/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/18094/7/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@510 PS7, Line 510: If the table ref is originated from a view then also add the tuple IDs for the : // respective table refs from the view. > Is this still true in the last patch? I think this is still true as e.g. for a from clause unnest I still had to store the zipping unnest IDs both in the FromClause and in CollectionTableRef.analyze(). For the select list syntax again we store the zipping unnest IDs in the FromClause (in the re-analysis phase) and in UnnestExpr.analyze(). See the calsites for addZippingUnnestTupleId() http://gerrit.cloudera.org:8080/#/c/18094/7/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@515 PS7, Line 515: public int numZippingUnnests = 0; > Do we still need this? I think that zippingUnnestTupleIds's length should b see comment above. http://gerrit.cloudera.org:8080/#/c/18094/7/fe/src/main/java/org/apache/impala/analysis/CollectionTableRef.java File fe/src/main/java/org/apache/impala/analysis/CollectionTableRef.java: http://gerrit.cloudera.org:8080/#/c/18094/7/fe/src/main/java/org/apache/impala/analysis/CollectionTableRef.java@206 PS7, Line 206: isCollectionInSelectList > I couldn't find where we use this. I use it twice in UnnestExpr. http://gerrit.cloudera.org:8080/#/c/18094/7/fe/src/main/java/org/apache/impala/analysis/FromClause.java File fe/src/main/java/org/apache/impala/analysis/FromClause.java: http://gerrit.cloudera.org:8080/#/c/18094/7/fe/src/main/java/org/apache/impala/analysis/FromClause.java@150 PS7, Line 150: // Don't do any checks of the collection that came from a view as getTable() would : // return null in that case. : if (collRef.getTable() == null) return; > Shouldn't line 149 be enough to rule out removing this check would crash this query: select id, a1, a2 from ( select id, unnest(arr1) a1, unnest(arr2) a2 from complextypes_arrays where id = 3 or id = 4) x where a1 > 8 and a2 = 'ten'; http://gerrit.cloudera.org:8080/#/c/18094/7/fe/src/main/java/org/apache/impala/planner/PlanNode.java File fe/src/main/java/org/apache/impala/planner/PlanNode.java: http://gerrit.cloudera.org:8080/#/c/18094/7/fe/src/main/java/org/apache/impala/planner/PlanNode.java@506 PS7, Line 506: removeZippingUnnestConjuncts > Is this still needed? What I don't get is why is it needed now, but why it hmm, it was a long time ago when I wrote this ... :D I did some poking here and it seems that for some reason the SingularRowSrcNode picks up zipping unnest conjuncts (and fails on a Precondition as it shouldn't) instead of the UnnestNode. Also, if I remove this function the "select list unnest" tests also break. Currently, I don't recall why this is needed for unnesting from views but not for tables. I'll take a look to figure out. Update: Ok, I figured it out why this is needed. In case there are conjuncts on an unnested array where the array is coming from a view the UnnestNode originally (and none of the other nodes) didn't pick up these conjuncts and they weren't evaluated. I had to make the changes on isBoundByTupleIds() functions in UnnestExpr and SlotRef but then the ScanNode also wanted to pick up these, also the SingularRowSrcNode so this is what I came up with to prevent them to pick up these unnested array related conjuncts and let UnnestNode to take care of them. The above only applies if there are more than 1 of these unnest conjuncts. If there is one, than ScanNode handles it. -- To view, visit http://gerrit.cloudera.org:8080/18094 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I68f649dda9e41f257e7f6596193d07b24049f92a Gerrit-Change-Number: 18094 Gerrit-PatchSet: 8 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 30 Mar 2022 09:15:16 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11038: Zipping unnest from view
Hello Daniel Becker, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18094 to look at the new patch set (#8). Change subject: IMPALA-11038: Zipping unnest from view .. IMPALA-11038: Zipping unnest from view IMPALA-10920 introduced zipping unnest functionality for arrays that are in a table. This patch improves that support further by accepting inputs from views as well. Testing: - Added planner tests to verify which execution node handles the predicates on unnested items. - E2E tests for both unnesting syntaxes (select list and from clause) to cover when the source of the unnested arrays is not a table but a view. Also tested multi-level views and filtering the unnested items on different levels. Change-Id: I68f649dda9e41f257e7f6596193d07b24049f92a --- M be/src/exec/subplan-node.cc M be/src/exec/unnest-node.cc M be/src/exec/unnest-node.h M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/CollectionTableRef.java M fe/src/main/java/org/apache/impala/analysis/FromClause.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/UnnestExpr.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/PlanNode.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java M fe/src/main/java/org/apache/impala/planner/UnnestNode.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java A testdata/workloads/functional-planner/queries/PlannerTest/zipping-unnest.test M testdata/workloads/functional-query/queries/QueryTest/zipping-unnest-from-view.test M testdata/workloads/functional-query/queries/QueryTest/zipping-unnest-in-select-list.test 19 files changed, 942 insertions(+), 46 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/94/18094/8 -- To view, visit http://gerrit.cloudera.org:8080/18094 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I68f649dda9e41f257e7f6596193d07b24049f92a Gerrit-Change-Number: 18094 Gerrit-PatchSet: 8 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins
[native-toolchain-CR] IMPALA-10854: Use ZSTD in toolchain to build ORC
Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18367 Change subject: IMPALA-10854: Use ZSTD in toolchain to build ORC .. IMPALA-10854: Use ZSTD in toolchain to build ORC The ORC build will download and compile ZSTD if env var ZSTD_HOME is undefined. We already have ZSTD in the toolchain. This patch let ORC depends on the ZSTD in our toolchain. Also disables building ORC tests. Tests: - Built native-toolchain and verified check/orc-1.7.0-p7.log that zstd is not downloaded again. - Compiled Impala with it and ran TPCH tests locally. Change-Id: Ifa8f0f55254e777b0132480a80737d9169410c73 --- M buildall.sh M source/orc/build.sh 2 files changed, 3 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/67/18367/1 -- To view, visit http://gerrit.cloudera.org:8080/18367 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ifa8f0f55254e777b0132480a80737d9169410c73 Gerrit-Change-Number: 18367 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang
[Impala-ASF-CR] IMPALA-11204: Template implementation for OrcStringColumnReader::ReadValue
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18366 ) Change subject: IMPALA-11204: Template implementation for OrcStringColumnReader::ReadValue .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10358/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18366 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5e5f88c28059fb3d3ac1172e6d383d06ee3bedd5 Gerrit-Change-Number: 18366 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 30 Mar 2022 07:34:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11204: Template implementation for OrcStringColumnReader::ReadValue
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18366 ) Change subject: IMPALA-11204: Template implementation for OrcStringColumnReader::ReadValue .. Patch Set 3: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/10357/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/18366 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5e5f88c28059fb3d3ac1172e6d383d06ee3bedd5 Gerrit-Change-Number: 18366 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 30 Mar 2022 07:33:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11204: Template implementation for OrcStringColumnReader::ReadValue
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18366 to look at the new patch set (#4). Change subject: IMPALA-11204: Template implementation for OrcStringColumnReader::ReadValue .. IMPALA-11204: Template implementation for OrcStringColumnReader::ReadValue There are some checks in OrcStringColumnReader::ReadValue() that we can determine outside the scope of this method. They should be optimized since this is a critical method that will be executed for each row (and for each string column). With these checks, the method is too complex to be inlined in OrcBatchedReader::ReadValueBatch() by the compiler. This patch templates OrcStringColumnReader::ReadValue() with two parameters, one for the target slot type (i.e. STRING/CHAR/VARCHAR), ther other for whether the column is dictionary encoded. Also adds an ALWAYS_INLINE marker to force inlining it. OrcStringColumnReader::ReadValueBatch() will call a template version of ReadValue() based on the slot type and the orc batch encoded state. Ran a single node perf test on TPCH(30) on my dev box using 3 impalad instances. There are some improvements and no significant regressions: +--++-++ | Query| Avg(s) | Base Avg(s) | Delta(Avg) | +--++-++ | TPCH-Q19 | 5.62 | 6.07| I -7.41% | | TPCH-Q6 | 2.56 | 2.78| I -7.77% | | TPCH-Q4 | 3.85 | 4.25| I -9.42% | | TPCH-Q12 | 4.25 | 4.99| I -14.78% | +--++-++ Base commit: ff21728 File Format: orc/snap/block Iterations: 30 Change-Id: I5e5f88c28059fb3d3ac1172e6d383d06ee3bedd5 --- M be/src/exec/orc-column-readers.cc M be/src/exec/orc-column-readers.h 2 files changed, 66 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/18366/4 -- To view, visit http://gerrit.cloudera.org:8080/18366 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5e5f88c28059fb3d3ac1172e6d383d06ee3bedd5 Gerrit-Change-Number: 18366 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11204: Template implementation for OrcStringColumnReader::ReadValue
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18366 to look at the new patch set (#3). Change subject: IMPALA-11204: Template implementation for OrcStringColumnReader::ReadValue .. IMPALA-11204: Template implementation for OrcStringColumnReader::ReadValue There are some checks in OrcStringColumnReader::ReadValue() that we can determine outside the scope of this method. They should be optimized since this is a critical method that will be executed for each row (and for each string column). With these checks, the method is too complex to be inlined in OrcBatchedReader::ReadValueBatch() by the compiler. This patch templates OrcStringColumnReader::ReadValue() with two parameters, one for the target slot type (i.e. STRING/CHAR/VARCHAR), ther other for whether the column is dictionary encoded. Also adds an ALWAYS_INLINE marker to force inlining it. OrcStringColumnReader::ReadValueBatch() will call a template version of ReadValue() based on the slot type and the orc batch encoded state. Ran a single node perf test on TPCH(30) on my dev box using 3 impalad instances. There are some improvements and no significant regressions: +--++-++ | Query| Avg(s) | Base Avg(s) | Delta(Avg) | +--++-++ | TPCH-Q19 | 5.62 | 6.07| I -7.41% | | TPCH-Q6 | 2.56 | 2.78| I -7.77% | | TPCH-Q4 | 3.85 | 4.25| I -9.42% | | TPCH-Q12 | 4.25 | 4.99| I -14.78% | +--++-++ Base commit: ff21728 File Format: orc/snap/block Iterations: 30 Change-Id: I5e5f88c28059fb3d3ac1172e6d383d06ee3bedd5 --- M be/src/exec/orc-column-readers.cc M be/src/exec/orc-column-readers.h 2 files changed, 66 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/18366/3 -- To view, visit http://gerrit.cloudera.org:8080/18366 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5e5f88c28059fb3d3ac1172e6d383d06ee3bedd5 Gerrit-Change-Number: 18366 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins
