[Impala-ASF-CR] IMPALA-11237: Preserve the behavior of CONCAT WS() before IMPALA-8891
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18440 ) Change subject: IMPALA-11237: Preserve the behavior of CONCAT_WS() before IMPALA-8891 .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/10479/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/18440 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I295adf4d4eeec7af43bb05644462e44024b6a2f6 Gerrit-Change-Number: 18440 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vincent Tran Gerrit-Comment-Date: Fri, 22 Apr 2022 04:35:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11237: Preserve the behavior of CONCAT WS() before IMPALA-8891
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18440 ) Change subject: IMPALA-11237: Preserve the behavior of CONCAT_WS() before IMPALA-8891 .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/18440/1/tests/custom_cluster/test_concat_ws.py File tests/custom_cluster/test_concat_ws.py: http://gerrit.cloudera.org:8080/#/c/18440/1/tests/custom_cluster/test_concat_ws.py@3 PS1, Line 3: class TestConcatWs(CustomClusterTestSuite): flake8: E302 expected 2 blank lines, found 1 -- To view, visit http://gerrit.cloudera.org:8080/18440 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I295adf4d4eeec7af43bb05644462e44024b6a2f6 Gerrit-Change-Number: 18440 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vincent Tran Gerrit-Comment-Date: Fri, 22 Apr 2022 04:16:10 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11237: Preserve the behavior of concat ws() before IMPALA-8891
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/18423 ) Change subject: IMPALA-11237: Preserve the behavior of concat_ws() before IMPALA-8891 .. Patch Set 1: Thanks for the comments Csaba and Quanlong! I also prepared another patch at https://gerrit.cloudera.org/c/18440/ that uses the approach suggested by Csaba. It's not that easy to determine which approach is better. For a user that relies on Impala's old behavior, it may be easier for the user to start the Impala daemons with 'concat_ws_return_null_if_any_is_null' set to true. No change has to be made to the user's SQL scripts with respect to the behavior of CONCAT_WS(). However, it may be easier for such a user to start adapting to the new behavior of CONCAT_WS() in that no restart of Impala daemons is required. Let me know if I missed something. I think I am fine with both approaches. -- To view, visit http://gerrit.cloudera.org:8080/18423 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6fe11971e0c56b868bdf8bf1b071d40d9837acda Gerrit-Change-Number: 18423 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vincent Tran Gerrit-Comment-Date: Fri, 22 Apr 2022 04:15:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11237: Preserve the behavior of CONCAT WS() before IMPALA-8891
Fang-Yu Rao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18440 Change subject: IMPALA-11237: Preserve the behavior of CONCAT_WS() before IMPALA-8891 .. IMPALA-11237: Preserve the behavior of CONCAT_WS() before IMPALA-8891 This patch preserves the behavior of CONCAT_WS() before IMPALA-8891 by introducing a flag of 'concat_ws_return_null_if_any_is_null', which is set to false by default. Specifically, if 'concat_ws_return_null_if_any_is_null' is set to true when the Impala daemon is started, CONCAT_WS() will return NULL as long as any of the argument values is NULL, which corresponds to the behavior before IMPALA-8891. Testing: - Added an E2E test to verifiy the behavior before IMPALA-8891 could be preserved once the flag above is set to true when the Impala daemon is started. Change-Id: I295adf4d4eeec7af43bb05644462e44024b6a2f6 --- M be/src/exprs/string-functions-ir.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h A testdata/workloads/functional-query/queries/QueryTest/test_concat_ws.test A tests/custom_cluster/test_concat_ws.py 5 files changed, 100 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/18440/1 -- To view, visit http://gerrit.cloudera.org:8080/18440 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I295adf4d4eeec7af43bb05644462e44024b6a2f6 Gerrit-Change-Number: 18440 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao
[Impala-ASF-CR] IMPALA-11237: Preserve the behavior of concat ws() before IMPALA-8891
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18423 ) Change subject: IMPALA-11237: Preserve the behavior of concat_ws() before IMPALA-8891 .. Patch Set 1: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/18423/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/18423/1//COMMIT_MSG@10 PS1, Line 10: query option of 'CONCAT_WS_RETURN_NULL_IF_ANY_IS_NULL' > My understanding is that generally skipping nulls is the expected behavior I tend to the current query option approach since it can help users migrate queries to the new behavior. So this won't be an upgrade blocker. -- To view, visit http://gerrit.cloudera.org:8080/18423 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6fe11971e0c56b868bdf8bf1b071d40d9837acda Gerrit-Change-Number: 18423 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vincent Tran Gerrit-Comment-Date: Fri, 22 Apr 2022 01:27:01 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11263: Coordinator hang when cancelling a query
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18439 ) Change subject: IMPALA-11263: Coordinator hang when cancelling a query .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/18439/1/tests/custom_cluster/test_rpc_timeout.py File tests/custom_cluster/test_rpc_timeout.py: http://gerrit.cloudera.org:8080/#/c/18439/1/tests/custom_cluster/test_rpc_timeout.py@211 PS1, Line 211: _ > flake8: E131 continuation line unaligned for hanging indent fixed -- To view, visit http://gerrit.cloudera.org:8080/18439 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I915511afe2df3017cbbf37f6aff3c5ff7f5473be Gerrit-Change-Number: 18439 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 21 Apr 2022 23:11:02 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11263: Coordinator hang when cancelling a query
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18439 ) Change subject: IMPALA-11263: Coordinator hang when cancelling a query .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10478/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18439 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I915511afe2df3017cbbf37f6aff3c5ff7f5473be Gerrit-Change-Number: 18439 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 21 Apr 2022 23:08:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11263: Coordinator hang when cancelling a query
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18439 ) Change subject: IMPALA-11263: Coordinator hang when cancelling a query .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10477/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18439 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I915511afe2df3017cbbf37f6aff3c5ff7f5473be Gerrit-Change-Number: 18439 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 21 Apr 2022 22:58:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11263: Coordinator hang when cancelling a query
Wenzhe Zhou has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/18439 ) Change subject: IMPALA-11263: Coordinator hang when cancelling a query .. IMPALA-11263: Coordinator hang when cancelling a query In a rare case, callback Coordinator::BackendState::ExecCompleteCb() was not called for the corresponding ExecQueryFInstances RPC. This caused coordinator to wait indefinitely when calling Coordinator::BackendState::Cancel() to cancel one fragment instance. This patch added timeout for BackendState::WaitOnExecLocked() so that coordinator will not be blocked indefinitely when cancelling a query. Testing: - Added a test case to simulate the callback missing when a query is failed. Verified that the coordinator would hang without the fixing. - Passed core tests. Change-Id: I915511afe2df3017cbbf37f6aff3c5ff7f5473be --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M tests/custom_cluster/test_rpc_timeout.py 3 files changed, 154 insertions(+), 96 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/18439/2 -- To view, visit http://gerrit.cloudera.org:8080/18439 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I915511afe2df3017cbbf37f6aff3c5ff7f5473be Gerrit-Change-Number: 18439 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11263: Coordinator hang when cancelling a query
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18439 ) Change subject: IMPALA-11263: Coordinator hang when cancelling a query .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/18439/1/tests/custom_cluster/test_rpc_timeout.py File tests/custom_cluster/test_rpc_timeout.py: http://gerrit.cloudera.org:8080/#/c/18439/1/tests/custom_cluster/test_rpc_timeout.py@211 PS1, Line 211: _ flake8: E131 continuation line unaligned for hanging indent -- To view, visit http://gerrit.cloudera.org:8080/18439 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I915511afe2df3017cbbf37f6aff3c5ff7f5473be Gerrit-Change-Number: 18439 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 21 Apr 2022 22:39:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11263: Coordinator hang when cancelling a query
Wenzhe Zhou has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18439 Change subject: IMPALA-11263: Coordinator hang when cancelling a query .. IMPALA-11263: Coordinator hang when cancelling a query In a rare case, callback Coordinator::BackendState::ExecCompleteCb() was not called for the corresponding ExecQueryFInstances RPC. This caused coordinator to wait indefinitely when calling Coordinator::BackendState::Cancel() to cancel one fragment instance. This patch added timeout for BackendState::WaitOnExecLocked() so that coordinator will not be blocked indefinitely when cancelling a query. Testing: - Added a test case to simulate the callback missing when a query is failed. Verified that the coordinator would hang without the fixing. - Passed core tests. Change-Id: I915511afe2df3017cbbf37f6aff3c5ff7f5473be --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M tests/custom_cluster/test_rpc_timeout.py 3 files changed, 154 insertions(+), 96 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/18439/1 -- To view, visit http://gerrit.cloudera.org:8080/18439 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I915511afe2df3017cbbf37f6aff3c5ff7f5473be Gerrit-Change-Number: 18439 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou
[native-toolchain-CR] IMPALA-11259: Bump Kudu version to 1.16 release (5cd6779)
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18437 ) Change subject: IMPALA-11259: Bump Kudu version to 1.16 release (5cd6779) .. Patch Set 4: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/18437 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I044e24d7402080447188355780a5f1c846a87103 Gerrit-Change-Number: 18437 Gerrit-PatchSet: 4 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Laszlo Gaal (Cloudera) Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 21 Apr 2022 22:25:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11127: [DOCS] document the UTF8 MODE query option
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18424 ) Change subject: IMPALA-11127: [DOCS] document the UTF8_MODE query option .. Patch Set 3: Code-Review+2 LGTM. Carry Tamas's +1. Thank Shajini! -- To view, visit http://gerrit.cloudera.org:8080/18424 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifac5812a3f5e105a73ac87c1ae5fce69a776fb92 Gerrit-Change-Number: 18424 Gerrit-PatchSet: 3 Gerrit-Owner: Shajini Thayasingh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Thu, 21 Apr 2022 22:24:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11127: [DOCS] document the UTF8 MODE query option
Quanlong Huang has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18424 ) Change subject: IMPALA-11127: [DOCS] document the UTF8_MODE query option .. IMPALA-11127: [DOCS] document the UTF8_MODE query option add the new query option UTF8_MODE topic update impala_string topic as requested in the first review create a new topic for UTF_8 mode under SQL ref discuss the new query option Change-Id: Ifac5812a3f5e105a73ac87c1ae5fce69a776fb92 Reviewed-on: http://gerrit.cloudera.org:8080/18424 Tested-by: Impala Public Jenkins Reviewed-by: Quanlong Huang --- M docs/impala.ditamap M docs/topics/impala_string.xml A docs/topics/impala_utf8_mode.xml A docs/topics/impala_utf_8.xml 4 files changed, 163 insertions(+), 3 deletions(-) Approvals: Impala Public Jenkins: Verified Quanlong Huang: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/18424 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ifac5812a3f5e105a73ac87c1ae5fce69a776fb92 Gerrit-Change-Number: 18424 Gerrit-PatchSet: 4 Gerrit-Owner: Shajini Thayasingh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate
[Impala-ASF-CR] IMPALA-10838: Error when struct returned from WITH()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17847 ) Change subject: IMPALA-10838: Error when struct returned from WITH() .. Patch Set 23: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10476/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17847 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iadb9233677355b85d424cc3f22b00b5a3bf61c57 Gerrit-Change-Number: 17847 Gerrit-PatchSet: 23 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Comment-Date: Thu, 21 Apr 2022 16:23:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11127: [DOCS] document the UTF8 MODE query option
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18424 ) Change subject: IMPALA-11127: [DOCS] document the UTF8_MODE query option .. Patch Set 3: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/656/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/18424 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifac5812a3f5e105a73ac87c1ae5fce69a776fb92 Gerrit-Change-Number: 18424 Gerrit-PatchSet: 3 Gerrit-Owner: Shajini Thayasingh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Thu, 21 Apr 2022 16:17:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11127: [DOCS] document the UTF8 MODE query option
Hello Quanlong Huang, Tamas Mate, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18424 to look at the new patch set (#3). Change subject: IMPALA-11127: [DOCS] document the UTF8_MODE query option .. IMPALA-11127: [DOCS] document the UTF8_MODE query option add the new query option UTF8_MODE topic update impala_string topic as requested in the first review create a new topic for UTF_8 mode under SQL ref discuss the new query option Change-Id: Ifac5812a3f5e105a73ac87c1ae5fce69a776fb92 --- M docs/impala.ditamap M docs/topics/impala_string.xml A docs/topics/impala_utf8_mode.xml A docs/topics/impala_utf_8.xml 4 files changed, 163 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/18424/3 -- To view, visit http://gerrit.cloudera.org:8080/18424 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifac5812a3f5e105a73ac87c1ae5fce69a776fb92 Gerrit-Change-Number: 18424 Gerrit-PatchSet: 3 Gerrit-Owner: Shajini Thayasingh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate
[Impala-ASF-CR] IMPALA-11127: [DOCS] document the UTF8 MODE query option
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18424 ) Change subject: IMPALA-11127: [DOCS] document the UTF8_MODE query option .. Patch Set 3: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/656/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/18424 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifac5812a3f5e105a73ac87c1ae5fce69a776fb92 Gerrit-Change-Number: 18424 Gerrit-PatchSet: 3 Gerrit-Owner: Shajini Thayasingh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Thu, 21 Apr 2022 16:10:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10838: Error when struct returned from WITH()
Daniel Becker has uploaded a new patch set (#23). ( http://gerrit.cloudera.org:8080/17847 ) Change subject: IMPALA-10838: Error when struct returned from WITH() .. IMPALA-10838: Error when struct returned from WITH() The following query fails: ''' with sub as ( select id, outer_struct from functional_orc_def.complextypes_nested_structs) select sub.id, sub.outer_struct.inner_struct2 from sub; ''' with the following error: ''' ERROR: IllegalStateException: Illegal reference to non-materialized tuple: debugname=InlineViewRef sub alias=sub tid=6 ''' while if 'outer_struct.inner_struct2' is added to the select list of the inline view, the query works as expected. This change fixes the problem by two modifications: - if a field of a struct needs to be materialised, also materialise all of its enclosing structs (ancestors) - in InlineViewRef, struct fields are inserted into the 'smap' and 'baseTableSmap' with the appropriate inline view prefix TODO: Is this just a hack with the labels? This change also changes the way struct fields are materialised: until now, if a member of a struct was needed to be materialised, the whole struct, including other members of the struct were materialised. This behaviour can lead to using significantly more memory than necessary if we for example query a single member of a large struct. This change modifies this behaviour so that we only materialise the struct members that are actually needed. Tests: - added queries that are fixed by this change (including the one above) in nested-struct-in-select-list.test - added a planner test in fe/src/test/java/org/apache/impala/planner/PlannerTest.java that asserts that only the required parts of structs are materialised Change-Id: Iadb9233677355b85d424cc3f22b00b5a3bf61c57 --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/CollectionTableRef.java M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java M fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java M fe/src/main/java/org/apache/impala/analysis/Path.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/SortInfo.java M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java M testdata/workloads/functional-query/queries/QueryTest/nested-struct-in-select-list.test 16 files changed, 811 insertions(+), 96 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/17847/23 -- To view, visit http://gerrit.cloudera.org:8080/17847 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iadb9233677355b85d424cc3f22b00b5a3bf61c57 Gerrit-Change-Number: 17847 Gerrit-PatchSet: 23 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen
[Impala-ASF-CR] IMPALA-10839: NULL values are displayed on a wrong level for nested structs (ORC)
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18403 ) Change subject: IMPALA-10839: NULL values are displayed on a wrong level for nested structs (ORC) .. Patch Set 8: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iff5034e7bdf39c036aecc491fbd324e29150f040 Gerrit-Change-Number: 18403 Gerrit-PatchSet: 8 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 21 Apr 2022 13:59:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10839: NULL values are displayed on a wrong level for nested structs (ORC)
Impala Public Jenkins has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/18403 )
Change subject: IMPALA-10839: NULL values are displayed on a wrong level for
nested structs (ORC)
..
IMPALA-10839: NULL values are displayed on a wrong level for nested structs
(ORC)
When querying a non-toplevel nested struct from an ORC file, the NULL
values are displayed at an incorrect level. E.g.:
select id, outer_struct.inner_struct3 from
functional_orc_def.complextypes_nested_structs where id >= 4;
+++
| id | outer_struct.inner_struct3 |
+++
| 4 | {"s":{"i":null,"s":null}} |
| 5 | {"s":null} |
+++
However, in the first row it is expected that 's' should be null and not
its members; in the second row the result should be 'NULL', i.e.
'outer_struct.inner_struct3' is null.
For reference see what is returned when querying 'outer_struct' instead
of 'outer_struct.inner_struct3':
++---+
| 4 |
{"str":"","inner_struct1":{"str":"somestr2","de":12345.12},"inner_struct2":{"i":1,"str":"string"},"inner_struct3":{"s":null}}
|
| 5 |
{"str":null,"inner_struct1":null,"inner_struct2":null,"inner_struct3":null}
|
++---+
The problem comes from the incorrect handling of the different depths of
the following trees:
- the ORC type hierarchy (schema)
- the tuple descriptor / slot descriptor hierarchy
as the ORC type hierarchy contains a node for every level in the schema
but the tuple/slot descriptor hierarchy omits the levels of structs that
are not in the select list (but an ancestor of theirs is), as these
structs are not materialised.
In the case of the example query, the two hierarchies are the following:
ORC:
root --> outer_struct -> inner_struct3 -> s --> i
| \-> s
\-> id
Tuple/slot descriptors:
main_tuple --> inner_struct3 -> s --> i
| \-> s
\-> id
We create 'OrcColumnReader's for each node in the ORC type tree. Each
OrcColumnReader is assigned an ORC type node and a slot descriptor. The
incorrect behaviour comes from the incorrect pairing of ORC type nodes
with slot descriptors.
The old behaviour is described below:
Starting from the root, going along a path in both trees (for example
the path leading to outer_struct.inner_struct3.s.i), for each step we
consume a level in both trees until no more nodes remain in the
tuple/slot desc tree, and then we pair the last element from that tree
with the remaining ORC type node(s).
In the example, we get the following pairs:
(root, main_tuple) -> (outer_struct, inner_struct3) ->
(inner_struct3, s) -> (s, i) -> (i, i)
When we run out of structs in the tuple/slot desc tree, we still create
OrcStructReaders (because the ORC type is still a struct, but the slot
descriptor now refers to an INT), but we mark them incorrectly as
non-materialised.
Also, the OrcStructReaders for non-materialised structs do not need to
check for null-ness as they are not present in the select list, only
their descendants, and the ORC batch object stores null information also
for the descendants of null values.
Let's look at the row with id 4 in the example:
Because of the bug, the non-materialising OrcStructReader appears at the
level of the (s, i) pair, so the 's' struct is not checked for
null-ness, although it is actually null. One level lower, for 'i' (and
the inner 's' string field), the ORC batch object tells us that the
values are null (because their parent is). Therefore the nulls appear
one level lower than they should.
The correct behaviour is that ORC type nodes are paired with slot
descriptors if either
- the ORC type node matches the slot descriptor (they refer to the same
node in the schema) or
- the slot descriptor is a descendant of the schema node that the ORC
type node refers to.
This patch fixes the incorrect pairing of ORC types and slot
descriptors, so we have the following pairs:
(root, main_tuple) -> (outer_struct, main_tuple) ->
(inner_struct3, inner_struct3) -> (s, s) -> (i, i)
In this case the OrcStructReader for the pair (outer_struct, main_tuple)
becomes non-materialising and the one for (s, s) will be materialising,
so the 's' struct will also be null-checked, recognising null-ness at
the correct level.
This commit also fixes some comments in be/src/exec/orc-column-readers.h
and be/src/exec/hdfs-orc-scanner.h mentioning the field
HdfsOrcScanner::col_id_path_map_, which has been removed by
"IMPALA-10485: part(1): make ORC column reader creation independent
[native-toolchain-CR] IMPALA-11259: Bump Kudu version to 1.16 release (5cd6779)
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/18437 ) Change subject: IMPALA-11259: Bump Kudu version to 1.16 release (5cd6779) .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/18437/2/buildall.sh File buildall.sh: http://gerrit.cloudera.org:8080/#/c/18437/2/buildall.sh@339 PS2, Line 339: 1.16.0 > Since we are picking up a specific released version, I'd prefer if we could This makes sense, thank you for checking it. -- To view, visit http://gerrit.cloudera.org:8080/18437 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I044e24d7402080447188355780a5f1c846a87103 Gerrit-Change-Number: 18437 Gerrit-PatchSet: 4 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Laszlo Gaal (Cloudera) Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 21 Apr 2022 13:03:16 + Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-11259: Bump Kudu version to 1.16 release (5cd6779)
Hello Quanlong Huang, Laszlo Gaal (Cloudera), Zoltan Borok-Nagy, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18437 to look at the new patch set (#4). Change subject: IMPALA-11259: Bump Kudu version to 1.16 release (5cd6779) .. IMPALA-11259: Bump Kudu version to 1.16 release (5cd6779) This commit updates Kudu version to 1.16 release. Change-Id: I044e24d7402080447188355780a5f1c846a87103 --- M buildall.sh 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/37/18437/4 -- To view, visit http://gerrit.cloudera.org:8080/18437 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I044e24d7402080447188355780a5f1c846a87103 Gerrit-Change-Number: 18437 Gerrit-PatchSet: 4 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Laszlo Gaal (Cloudera) Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[native-toolchain-CR] IMPALA-10699: Patch libev to support C++17
Laszlo Gaal (Cloudera) has posted comments on this change. ( http://gerrit.cloudera.org:8080/18404 ) Change subject: IMPALA-10699: Patch libev to support C++17 .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18404 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib5cba45a426c4237cb0ef7c28a806a117dfd212a Gerrit-Change-Number: 18404 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal (Cloudera) Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 21 Apr 2022 12:59:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11244: Run the minicluster for docker-based BE tests
Laszlo Gaal (Cloudera) has posted comments on this change. ( http://gerrit.cloudera.org:8080/18414 ) Change subject: IMPALA-11244: Run the minicluster for docker-based BE tests .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18414 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I784a63a02886852e10ccca7c118c22ff7d38b8a3 Gerrit-Change-Number: 18414 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal (Cloudera) Gerrit-Comment-Date: Thu, 21 Apr 2022 12:54:03 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-11259: Bump Kudu version to 1.16 release (5cd6779)
Laszlo Gaal (Cloudera) has posted comments on this change. ( http://gerrit.cloudera.org:8080/18437 ) Change subject: IMPALA-11259: Bump Kudu version to 1.16 release (5cd6779) .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/18437/2/buildall.sh File buildall.sh: http://gerrit.cloudera.org:8080/#/c/18437/2/buildall.sh@339 PS2, Line 339: 5cd6779 Since we are picking up a specific released version, I'd prefer if we could use the release tag 1.16.0 instead of a raw Git hash. While this is just syntactic sugar (tag 1.16.0 points at exactly this git hash), it would still help traceability a bit. I checked that `git fetch origin 1.16.0` would run and fetch exactly this git hash from github.com/apache/kudu. -- To view, visit http://gerrit.cloudera.org:8080/18437 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I044e24d7402080447188355780a5f1c846a87103 Gerrit-Change-Number: 18437 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Laszlo Gaal (Cloudera) Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 21 Apr 2022 12:50:55 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10850: Interpret timestamp predicates in local timezone in IcebergScanNode
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18399 ) Change subject: IMPALA-10850: Interpret timestamp predicates in local timezone in IcebergScanNode .. Patch Set 10: Thanks for the review! -- To view, visit http://gerrit.cloudera.org:8080/18399 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I181be5d2fa004f69b457f69ff82dc2f9877f46fa Gerrit-Change-Number: 18399 Gerrit-PatchSet: 10 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 21 Apr 2022 12:49:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10850: Interpret timestamp predicates in local timezone in IcebergScanNode
Zoltan Borok-Nagy has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18399 ) Change subject: IMPALA-10850: Interpret timestamp predicates in local timezone in IcebergScanNode .. IMPALA-10850: Interpret timestamp predicates in local timezone in IcebergScanNode IcebergScanNode interprets the timestamp literals as UTC timestamps during predicate pushdown to Iceberg. It causes problems when the Iceberg table uses TIMESTAMPTZ (which corresponds to TIMESTAMP WITH LOCAL TIME ZONE in SQL) because in the scanners we assume that the timestamp literals in a query are in local timezone. Hence, if the Iceberg table is partitioned by HOUR(ts), and Impala is running in a different timezone than UTC, then the following query doesn't return any rows: SELECT * from t WHERE ts = ; Because during predicate pushdown the timestamp is interpreted as a UTC timestamp (no conversion from local to UTC), but during query execution the timestamp data in the files are converted to local timezone, then compared to . I.e. in the scanner the assumption is that is in local timezone. On the other hand, when Iceberg type TIMESTAMP (which correcponds to TIMESTAMP WITHOUT TIME ZONE in SQL) is used, then we should just push down the timestamp values without any conversion. In this case there is no conversion in the scanners either. Testing: * added e2e test with TIMESTAMPTZ * added e2e test with TIMESTAMP Change-Id: I181be5d2fa004f69b457f69ff82dc2f9877f46fa Reviewed-on: http://gerrit.cloudera.org:8080/18399 Tested-by: Impala Public Jenkins Reviewed-by: Csaba Ringhofer --- M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M testdata/data/README A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamp_part/data/ts_hour=1969-01-01-01/0-0-boroknagyz_20220419181820_3b0f79ee-1aff-4983-98cf-7d01647fa77a-job_16493406300920_0023-1.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamp_part/data/ts_hour=2021-01-10-12/0-0-boroknagyz_20220419181820_3b0f79ee-1aff-4983-98cf-7d01647fa77a-job_16493406300920_0023-3.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamp_part/data/ts_hour=2021-10-31-02/0-0-boroknagyz_20220419181820_3b0f79ee-1aff-4983-98cf-7d01647fa77a-job_16493406300920_0023-2.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamp_part/data/ts_hour=2022-04-11-00/0-0-boroknagyz_20220419181820_3b0f79ee-1aff-4983-98cf-7d01647fa77a-job_16493406300920_0023-4.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamp_part/data/ts_hour=2022-04-11-12/0-0-boroknagyz_20220419181820_3b0f79ee-1aff-4983-98cf-7d01647fa77a-job_16493406300920_0023-5.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamp_part/metadata/a366370e-6b9a-4698-82d0-95fb69b19afb-m0.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamp_part/metadata/snap-1967339514069250436-1-a366370e-6b9a-4698-82d0-95fb69b19afb.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamp_part/metadata/v1.metadata.json A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamp_part/metadata/v2.metadata.json A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamp_part/metadata/version-hint.text A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamptz_part/data/ts_hour=1969-01-01-00/0-0-boroknagyz_20220419182502_45a45ed8-85ff-4046-b834-648c5a039891-job_16493406300920_0024-1.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamptz_part/data/ts_hour=2021-01-10-11/0-0-boroknagyz_20220419182502_45a45ed8-85ff-4046-b834-648c5a039891-job_16493406300920_0024-4.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamptz_part/data/ts_hour=2021-10-31-00/0-0-boroknagyz_20220419182502_45a45ed8-85ff-4046-b834-648c5a039891-job_16493406300920_0024-2.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamptz_part/data/ts_hour=2021-10-31-01/0-0-boroknagyz_20220419182502_45a45ed8-85ff-4046-b834-648c5a039891-job_16493406300920_0024-3.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamptz_part/data/ts_hour=2022-04-10-22/0-0-boroknagyz_20220419182502_45a45ed8-85ff-4046-b834-648c5a039891-job_16493406300920_0024-5.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamptz_part/data/ts_hour=2022-04-11-10/0-0-boroknagyz_20220419182502_45a45ed8-85ff-4046-b834-648c5a039891-job_16493406300920_0024-6.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamptz_part/metadata/94003077-eabb-4dab-95ec-52a1727ef853-m0.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamptz_part/metadata/snap-2778998487482282437-1-94003077-eabb-4dab-95ec-52a1727ef853.avro A
[native-toolchain-CR] IMPALA-11259: Bump Kudu version to 1.16 release (5cd6779)
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/18437 ) Change subject: IMPALA-11259: Bump Kudu version to 1.16 release (5cd6779) .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/18437/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/18437/2//COMMIT_MSG@7 PS2, Line 7: IMPALA-11259 > This Jira is about Kudu 1.15. Please update the jira or open a new one to a Agree, created a new one. -- To view, visit http://gerrit.cloudera.org:8080/18437 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I044e24d7402080447188355780a5f1c846a87103 Gerrit-Change-Number: 18437 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Laszlo Gaal (Cloudera) Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 21 Apr 2022 12:48:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10850: Interpret timestamp predicates in local timezone in IcebergScanNode
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18399 ) Change subject: IMPALA-10850: Interpret timestamp predicates in local timezone in IcebergScanNode .. Patch Set 9: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18399 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I181be5d2fa004f69b457f69ff82dc2f9877f46fa Gerrit-Change-Number: 18399 Gerrit-PatchSet: 9 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 21 Apr 2022 12:46:00 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-11259: Bump Kudu version to 1.16 release (5cd6779)
Hello Quanlong Huang, Laszlo Gaal (Cloudera), Zoltan Borok-Nagy, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18437 to look at the new patch set (#3). Change subject: IMPALA-11259: Bump Kudu version to 1.16 release (5cd6779) .. IMPALA-11259: Bump Kudu version to 1.16 release (5cd6779) This commit updates Kudu version to 1.16 release. Change-Id: I044e24d7402080447188355780a5f1c846a87103 --- M buildall.sh 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/37/18437/3 -- To view, visit http://gerrit.cloudera.org:8080/18437 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I044e24d7402080447188355780a5f1c846a87103 Gerrit-Change-Number: 18437 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Laszlo Gaal (Cloudera) Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy
[native-toolchain-CR] IMPALA-10859: Bump Kudu version to 1.16 release (5cd6779)
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18437 ) Change subject: IMPALA-10859: Bump Kudu version to 1.16 release (5cd6779) .. Patch Set 2: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/18437/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/18437/2//COMMIT_MSG@7 PS2, Line 7: IMPALA-10859 This Jira is about Kudu 1.15. Please update the jira or open a new one to avoid confusion. -- To view, visit http://gerrit.cloudera.org:8080/18437 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I044e24d7402080447188355780a5f1c846a87103 Gerrit-Change-Number: 18437 Gerrit-PatchSet: 2 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Laszlo Gaal (Cloudera) Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 21 Apr 2022 12:26:13 + Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-10859: Bump Kudu version to 1.16 release (5cd6779)
Hello Laszlo Gaal (Cloudera), Zoltan Borok-Nagy, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18437 to look at the new patch set (#2). Change subject: IMPALA-10859: Bump Kudu version to 1.16 release (5cd6779) .. IMPALA-10859: Bump Kudu version to 1.16 release (5cd6779) This commit updates Kudu version to 1.16 release. Change-Id: I044e24d7402080447188355780a5f1c846a87103 --- M buildall.sh 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/37/18437/2 -- To view, visit http://gerrit.cloudera.org:8080/18437 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I044e24d7402080447188355780a5f1c846a87103 Gerrit-Change-Number: 18437 Gerrit-PatchSet: 2 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Laszlo Gaal (Cloudera) Gerrit-Reviewer: Zoltan Borok-Nagy
[native-toolchain-CR] IMPALA-10859: Update Kudu version to 1.16 release
Tamas Mate has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18437 Change subject: IMPALA-10859: Update Kudu version to 1.16 release .. IMPALA-10859: Update Kudu version to 1.16 release This commit updates Kudu version to 1.16 release. Change-Id: I044e24d7402080447188355780a5f1c846a87103 --- M buildall.sh 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/37/18437/1 -- To view, visit http://gerrit.cloudera.org:8080/18437 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I044e24d7402080447188355780a5f1c846a87103 Gerrit-Change-Number: 18437 Gerrit-PatchSet: 1 Gerrit-Owner: Tamas Mate
[Impala-ASF-CR] IMPALA-10839: NULL values are displayed on a wrong level for nested structs (ORC)
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18403 ) Change subject: IMPALA-10839: NULL values are displayed on a wrong level for nested structs (ORC) .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10475/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iff5034e7bdf39c036aecc491fbd324e29150f040 Gerrit-Change-Number: 18403 Gerrit-PatchSet: 7 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 21 Apr 2022 09:51:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10839: NULL values are displayed on a wrong level for nested structs (ORC)
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/18403 ) Change subject: IMPALA-10839: NULL values are displayed on a wrong level for nested structs (ORC) .. Patch Set 7: Code-Review+2 Carry Gabor's +2. -- To view, visit http://gerrit.cloudera.org:8080/18403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iff5034e7bdf39c036aecc491fbd324e29150f040 Gerrit-Change-Number: 18403 Gerrit-PatchSet: 7 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 21 Apr 2022 09:35:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10839: NULL values are displayed on a wrong level for nested structs (ORC)
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18403 ) Change subject: IMPALA-10839: NULL values are displayed on a wrong level for nested structs (ORC) .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8059/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iff5034e7bdf39c036aecc491fbd324e29150f040 Gerrit-Change-Number: 18403 Gerrit-PatchSet: 8 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 21 Apr 2022 09:36:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10839: NULL values are displayed on a wrong level for nested structs (ORC)
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18403 ) Change subject: IMPALA-10839: NULL values are displayed on a wrong level for nested structs (ORC) .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iff5034e7bdf39c036aecc491fbd324e29150f040 Gerrit-Change-Number: 18403 Gerrit-PatchSet: 8 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 21 Apr 2022 09:36:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10839: NULL values are displayed on a wrong level for nested structs (ORC)
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/18403 ) Change subject: IMPALA-10839: NULL values are displayed on a wrong level for nested structs (ORC) .. Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/18403/6/testdata/workloads/functional-query/queries/QueryTest/nested-struct-in-select-list.test File testdata/workloads/functional-query/queries/QueryTest/nested-struct-in-select-list.test: http://gerrit.cloudera.org:8080/#/c/18403/6/testdata/workloads/functional-query/queries/QueryTest/nested-struct-in-select-list.test@148 PS6, Line 148: select id, outer_struct.inner_struct3 from > nit: I think the very first test in this file runs the same query as this o Thanks, I removed it here. -- To view, visit http://gerrit.cloudera.org:8080/18403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iff5034e7bdf39c036aecc491fbd324e29150f040 Gerrit-Change-Number: 18403 Gerrit-PatchSet: 7 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 21 Apr 2022 09:31:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10839: NULL values are displayed on a wrong level for nested structs (ORC)
Daniel Becker has uploaded a new patch set (#7). (
http://gerrit.cloudera.org:8080/18403 )
Change subject: IMPALA-10839: NULL values are displayed on a wrong level for
nested structs (ORC)
..
IMPALA-10839: NULL values are displayed on a wrong level for nested structs
(ORC)
When querying a non-toplevel nested struct from an ORC file, the NULL
values are displayed at an incorrect level. E.g.:
select id, outer_struct.inner_struct3 from
functional_orc_def.complextypes_nested_structs where id >= 4;
+++
| id | outer_struct.inner_struct3 |
+++
| 4 | {"s":{"i":null,"s":null}} |
| 5 | {"s":null} |
+++
However, in the first row it is expected that 's' should be null and not
its members; in the second row the result should be 'NULL', i.e.
'outer_struct.inner_struct3' is null.
For reference see what is returned when querying 'outer_struct' instead
of 'outer_struct.inner_struct3':
++---+
| 4 |
{"str":"","inner_struct1":{"str":"somestr2","de":12345.12},"inner_struct2":{"i":1,"str":"string"},"inner_struct3":{"s":null}}
|
| 5 |
{"str":null,"inner_struct1":null,"inner_struct2":null,"inner_struct3":null}
|
++---+
The problem comes from the incorrect handling of the different depths of
the following trees:
- the ORC type hierarchy (schema)
- the tuple descriptor / slot descriptor hierarchy
as the ORC type hierarchy contains a node for every level in the schema
but the tuple/slot descriptor hierarchy omits the levels of structs that
are not in the select list (but an ancestor of theirs is), as these
structs are not materialised.
In the case of the example query, the two hierarchies are the following:
ORC:
root --> outer_struct -> inner_struct3 -> s --> i
| \-> s
\-> id
Tuple/slot descriptors:
main_tuple --> inner_struct3 -> s --> i
| \-> s
\-> id
We create 'OrcColumnReader's for each node in the ORC type tree. Each
OrcColumnReader is assigned an ORC type node and a slot descriptor. The
incorrect behaviour comes from the incorrect pairing of ORC type nodes
with slot descriptors.
The old behaviour is described below:
Starting from the root, going along a path in both trees (for example
the path leading to outer_struct.inner_struct3.s.i), for each step we
consume a level in both trees until no more nodes remain in the
tuple/slot desc tree, and then we pair the last element from that tree
with the remaining ORC type node(s).
In the example, we get the following pairs:
(root, main_tuple) -> (outer_struct, inner_struct3) ->
(inner_struct3, s) -> (s, i) -> (i, i)
When we run out of structs in the tuple/slot desc tree, we still create
OrcStructReaders (because the ORC type is still a struct, but the slot
descriptor now refers to an INT), but we mark them incorrectly as
non-materialised.
Also, the OrcStructReaders for non-materialised structs do not need to
check for null-ness as they are not present in the select list, only
their descendants, and the ORC batch object stores null information also
for the descendants of null values.
Let's look at the row with id 4 in the example:
Because of the bug, the non-materialising OrcStructReader appears at the
level of the (s, i) pair, so the 's' struct is not checked for
null-ness, although it is actually null. One level lower, for 'i' (and
the inner 's' string field), the ORC batch object tells us that the
values are null (because their parent is). Therefore the nulls appear
one level lower than they should.
The correct behaviour is that ORC type nodes are paired with slot
descriptors if either
- the ORC type node matches the slot descriptor (they refer to the same
node in the schema) or
- the slot descriptor is a descendant of the schema node that the ORC
type node refers to.
This patch fixes the incorrect pairing of ORC types and slot
descriptors, so we have the following pairs:
(root, main_tuple) -> (outer_struct, main_tuple) ->
(inner_struct3, inner_struct3) -> (s, s) -> (i, i)
In this case the OrcStructReader for the pair (outer_struct, main_tuple)
becomes non-materialising and the one for (s, s) will be materialising,
so the 's' struct will also be null-checked, recognising null-ness at
the correct level.
This commit also fixes some comments in be/src/exec/orc-column-readers.h
and be/src/exec/hdfs-orc-scanner.h mentioning the field
HdfsOrcScanner::col_id_path_map_, which has been removed by
"IMPALA-10485: part(1): make ORC column reader creation independent of
schema
[Impala-ASF-CR] IMPALA-10839: NULL values are displayed on a wrong level for nested structs (ORC)
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/18403 ) Change subject: IMPALA-10839: NULL values are displayed on a wrong level for nested structs (ORC) .. Patch Set 6: Code-Review+2 (1 comment) Sorry for the late review. I just left one nit, other than that, I'm fine with the patch. Thanks for taking care! http://gerrit.cloudera.org:8080/#/c/18403/6/testdata/workloads/functional-query/queries/QueryTest/nested-struct-in-select-list.test File testdata/workloads/functional-query/queries/QueryTest/nested-struct-in-select-list.test: http://gerrit.cloudera.org:8080/#/c/18403/6/testdata/workloads/functional-query/queries/QueryTest/nested-struct-in-select-list.test@148 PS6, Line 148: select id, outer_struct from nit: I think the very first test in this file runs the same query as this one. -- To view, visit http://gerrit.cloudera.org:8080/18403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iff5034e7bdf39c036aecc491fbd324e29150f040 Gerrit-Change-Number: 18403 Gerrit-PatchSet: 6 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 21 Apr 2022 08:56:46 + Gerrit-HasComments: Yes
