[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 5 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 12 Nov 2020 08:11:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
wangsheng has removed a vote on this change. Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Removed Code-Review+1 by wangsheng -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 5 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 5: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 5 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 12 Nov 2020 08:11:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6648/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 5 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 12 Nov 2020 08:37:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10310: Fix couldn't skip rows in parquet file on NextRowGroup
guojingfeng has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/16697 ) Change subject: IMPALA-10310: Fix couldn't skip rows in parquet file on NextRowGroup .. IMPALA-10310: Fix couldn't skip rows in parquet file on NextRowGroup In practice we recommend that hdfs block size should align with parquet row group size.But in fact some compute engine like spark, default parquet row group size is 128MB, and if ETL user doesn't change the default property spark will generate row groups that smaller than hdfs block size. The result is a single hdfs block may contain multiple parquet row groups. In planner stage, length of impala generated scan range may be bigger than row group size. thus a single scan range contains multiple row group. In current parquet scanner when move to next row group, some of internal stat in parquet column readers need to reset. eg: num_buffered_values_, column chunk metadata, reset internal stat of column chunk readers. But current_row_range_ offset is not reset currently, this will cause errors "Couldn't skip rows in file hdfs://xxx" as IMPALA-10310 points out. This patch simply reset current_row_range_ to 0 when moving into next row group in parquet column readers. Fix the bug IMPALA-10310. Testing: * Add e2e test for parquet multi blocks per file and multi pages per block * Ran all core tests offline. * Manually tested all cases encountered in my production environment. Change-Id: I964695cd53f5d5fdb6485a85cd82e7a72ca6092c --- M be/src/exec/parquet/parquet-column-readers.cc M testdata/data/README A testdata/data/customer_multiblock_page_index.parquet M testdata/workloads/functional-query/queries/QueryTest/parquet-page-index.test M tests/query_test/test_parquet_stats.py 5 files changed, 34 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/16697/3 -- To view, visit http://gerrit.cloudera.org:8080/16697 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I964695cd53f5d5fdb6485a85cd82e7a72ca6092c Gerrit-Change-Number: 16697 Gerrit-PatchSet: 3 Gerrit-Owner: guojingfeng Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: guojingfeng
[Impala-ASF-CR] IMPALA-10310: Fix couldn't skip rows in parquet file on NextRowGroup
guojingfeng has posted comments on this change. ( http://gerrit.cloudera.org:8080/16697 ) Change subject: IMPALA-10310: Fix couldn't skip rows in parquet file on NextRowGroup .. Patch Set 3: > Patch Set 3: > > Seems like the change is in Draft. Could you publish it so we can submit it? Thank you, publish it already. -- To view, visit http://gerrit.cloudera.org:8080/16697 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I964695cd53f5d5fdb6485a85cd82e7a72ca6092c Gerrit-Change-Number: 16697 Gerrit-PatchSet: 3 Gerrit-Owner: guojingfeng Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: guojingfeng Gerrit-Comment-Date: Thu, 12 Nov 2020 10:27:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10310: Fix couldn't skip rows in parquet file on NextRowGroup
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16697 ) Change subject: IMPALA-10310: Fix couldn't skip rows in parquet file on NextRowGroup .. Patch Set 3: -Verified Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/7638/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/16697 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I964695cd53f5d5fdb6485a85cd82e7a72ca6092c Gerrit-Change-Number: 16697 Gerrit-PatchSet: 3 Gerrit-Owner: guojingfeng Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: guojingfeng Gerrit-Comment-Date: Thu, 12 Nov 2020 10:41:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10310: Fix couldn't skip rows in parquet file on NextRowGroup
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16697 ) Change subject: IMPALA-10310: Fix couldn't skip rows in parquet file on NextRowGroup .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7639/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16697 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I964695cd53f5d5fdb6485a85cd82e7a72ca6092c Gerrit-Change-Number: 16697 Gerrit-PatchSet: 3 Gerrit-Owner: guojingfeng Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: guojingfeng Gerrit-Comment-Date: Thu, 12 Nov 2020 11:14:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10310: Fix couldn't skip rows in parquet file on NextRowGroup
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16697 ) Change subject: IMPALA-10310: Fix couldn't skip rows in parquet file on NextRowGroup .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16697 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I964695cd53f5d5fdb6485a85cd82e7a72ca6092c Gerrit-Change-Number: 16697 Gerrit-PatchSet: 4 Gerrit-Owner: guojingfeng Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: guojingfeng Gerrit-Comment-Date: Thu, 12 Nov 2020 11:42:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10310: Fix couldn't skip rows in parquet file on NextRowGroup
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16697 ) Change subject: IMPALA-10310: Fix couldn't skip rows in parquet file on NextRowGroup .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6649/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16697 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I964695cd53f5d5fdb6485a85cd82e7a72ca6092c Gerrit-Change-Number: 16697 Gerrit-PatchSet: 4 Gerrit-Owner: guojingfeng Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: guojingfeng Gerrit-Comment-Date: Thu, 12 Nov 2020 11:42:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10320: Specify expression selectivity for BoolLiteral.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16714 ) Change subject: IMPALA-10320: Specify expression selectivity for BoolLiteral. .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16714 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42c96cd685f22d8634509d9f488f2a1f82b8 Gerrit-Change-Number: 16714 Gerrit-PatchSet: 1 Gerrit-Owner: Shant Hovsepian Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 12 Nov 2020 12:30:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 5: (2 comments) http://gerrit.cloudera.org:8080/#/c/16599/5/be/src/service/client-request-state.cc File be/src/service/client-request-state.cc: http://gerrit.cloudera.org:8080/#/c/16599/5/be/src/service/client-request-state.cc@440 PS5, Line 440: ToStringFromUnixMicros Seems like this function doesn't take the TIMEZONE query option into account. I think you need to use TimestampValue::UtcFromUnixTimeMicros, then timestamp_value->UtcToLocal(timezone); http://gerrit.cloudera.org:8080/#/c/16599/5/fe/src/main/java/org/apache/impala/service/Frontend.java File fe/src/main/java/org/apache/impala/service/Frontend.java: http://gerrit.cloudera.org:8080/#/c/16599/5/fe/src/main/java/org/apache/impala/service/Frontend.java@1098 PS5, Line 1098: Set ancestorIds = Sets.newHashSet(SnapshotUtil.currentAncestors(table)) > It looks like this call can be placed before the start of the FOR loop. +1. Also the loadTable() call above. -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 5 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 12 Nov 2020 14:00:59 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6648/ -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 5 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 12 Nov 2020 14:04:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10310: Fix couldn't skip rows in parquet file on NextRowGroup
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16697 ) Change subject: IMPALA-10310: Fix couldn't skip rows in parquet file on NextRowGroup .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16697 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I964695cd53f5d5fdb6485a85cd82e7a72ca6092c Gerrit-Change-Number: 16697 Gerrit-PatchSet: 4 Gerrit-Owner: guojingfeng Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: guojingfeng Gerrit-Comment-Date: Thu, 12 Nov 2020 17:12:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10310: Fix couldn't skip rows in parquet file on NextRowGroup
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16697 ) Change subject: IMPALA-10310: Fix couldn't skip rows in parquet file on NextRowGroup .. IMPALA-10310: Fix couldn't skip rows in parquet file on NextRowGroup In practice we recommend that hdfs block size should align with parquet row group size.But in fact some compute engine like spark, default parquet row group size is 128MB, and if ETL user doesn't change the default property spark will generate row groups that smaller than hdfs block size. The result is a single hdfs block may contain multiple parquet row groups. In planner stage, length of impala generated scan range may be bigger than row group size. thus a single scan range contains multiple row group. In current parquet scanner when move to next row group, some of internal stat in parquet column readers need to reset. eg: num_buffered_values_, column chunk metadata, reset internal stat of column chunk readers. But current_row_range_ offset is not reset currently, this will cause errors "Couldn't skip rows in file hdfs://xxx" as IMPALA-10310 points out. This patch simply reset current_row_range_ to 0 when moving into next row group in parquet column readers. Fix the bug IMPALA-10310. Testing: * Add e2e test for parquet multi blocks per file and multi pages per block * Ran all core tests offline. * Manually tested all cases encountered in my production environment. Change-Id: I964695cd53f5d5fdb6485a85cd82e7a72ca6092c Reviewed-on: http://gerrit.cloudera.org:8080/16697 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exec/parquet/parquet-column-readers.cc M testdata/data/README A testdata/data/customer_multiblock_page_index.parquet M testdata/workloads/functional-query/queries/QueryTest/parquet-page-index.test M tests/query_test/test_parquet_stats.py 5 files changed, 34 insertions(+), 0 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16697 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I964695cd53f5d5fdb6485a85cd82e7a72ca6092c Gerrit-Change-Number: 16697 Gerrit-PatchSet: 5 Gerrit-Owner: guojingfeng Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: guojingfeng
[Impala-ASF-CR] IMPALA-10306: [DOCS] remove TZ offset texts from the list
Hello Tamas Mate, Gabor Kaszab, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16689 to look at the new patch set (#2). Change subject: IMPALA-10306: [DOCS] remove TZ offset texts from the list .. IMPALA-10306: [DOCS] remove TZ offset texts from the list added an item in the list that says that TZ offset will not be included in the output of this function even if the offset is provided as input Change-Id: I693d15f1230dd7eebcbf2a16657a3850943749e1 --- M docs/topics/impala_datetime_functions.xml 1 file changed, 45 insertions(+), 88 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/16689/2 -- To view, visit http://gerrit.cloudera.org:8080/16689 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I693d15f1230dd7eebcbf2a16657a3850943749e1 Gerrit-Change-Number: 16689 Gerrit-PatchSet: 2 Gerrit-Owner: Shajini Thayasingh Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate
[Impala-ASF-CR] IMPALA-10306: [DOCS] remove TZ offset texts from the list
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16689 ) Change subject: IMPALA-10306: [DOCS] remove TZ offset texts from the list .. Patch Set 2: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/607/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/16689 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I693d15f1230dd7eebcbf2a16657a3850943749e1 Gerrit-Change-Number: 16689 Gerrit-PatchSet: 2 Gerrit-Owner: Shajini Thayasingh Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Thu, 12 Nov 2020 17:37:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10306: [DOCS] remove TZ offset texts from the list
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16689 ) Change subject: IMPALA-10306: [DOCS] remove TZ offset texts from the list .. Patch Set 2: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/607/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/16689 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I693d15f1230dd7eebcbf2a16657a3850943749e1 Gerrit-Change-Number: 16689 Gerrit-PatchSet: 2 Gerrit-Owner: Shajini Thayasingh Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Thu, 12 Nov 2020 17:46:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6861: Fix OpenSSL initialization
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16704 to look at the new patch set (#2). Change subject: IMPALA-6861: Fix OpenSSL initialization .. IMPALA-6861: Fix OpenSSL initialization Impalads currently initialize OpenSSL twice: once when initializing Thrift and once when initializing KRPC. The initialization is theoretically idempotent but not thread-safe, so its better to clean this up. This patch disables the Thrift version as its older (last updated in 2015) and the KRPC version contains logic specific to more recent versions of OpenSSL. The catalogd and statestored also now use the KRPC version instead of the Thrift version. It also improves debuggability by adding some additional startup logging. Testing: - Passed existing SSL tests. Change-Id: I35b1362d40c8a12082cc8b531a38b4a485bac0e7 --- M be/src/rpc/authentication.cc M be/src/rpc/authentication.h M be/src/rpc/thrift-server.cc 3 files changed, 11 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/16704/2 -- To view, visit http://gerrit.cloudera.org:8080/16704 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I35b1362d40c8a12082cc8b531a38b4a485bac0e7 Gerrit-Change-Number: 16704 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6861: Fix OpenSSL initialization
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/16704 ) Change subject: IMPALA-6861: Fix OpenSSL initialization .. Patch Set 2: Code-Review+2 (2 comments) carrying forward http://gerrit.cloudera.org:8080/#/c/16704/1/be/src/rpc/authentication.cc File be/src/rpc/authentication.cc: http://gerrit.cloudera.org:8080/#/c/16704/1/be/src/rpc/authentication.cc@1116 PS1, Line 1116: initialize > Typos: initialize/initialization Done http://gerrit.cloudera.org:8080/#/c/16704/1/be/src/rpc/authentication.cc@1119 PS1, Line 1119: Initialize > typo Done -- To view, visit http://gerrit.cloudera.org:8080/16704 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I35b1362d40c8a12082cc8b531a38b4a485bac0e7 Gerrit-Change-Number: 16704 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 12 Nov 2020 18:56:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6861: Fix OpenSSL initialization
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16704 ) Change subject: IMPALA-6861: Fix OpenSSL initialization .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7640/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16704 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I35b1362d40c8a12082cc8b531a38b4a485bac0e7 Gerrit-Change-Number: 16704 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 12 Nov 2020 19:05:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6861: Fix OpenSSL initialization
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16704 ) Change subject: IMPALA-6861: Fix OpenSSL initialization .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16704 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I35b1362d40c8a12082cc8b531a38b4a485bac0e7 Gerrit-Change-Number: 16704 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 12 Nov 2020 19:07:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6861: Fix OpenSSL initialization
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16704 ) Change subject: IMPALA-6861: Fix OpenSSL initialization .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6650/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16704 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I35b1362d40c8a12082cc8b531a38b4a485bac0e7 Gerrit-Change-Number: 16704 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 12 Nov 2020 19:07:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7876: COMPUTE STATS TABLESAMPLE is not updating number of estimated rows
Abhishek Rawat has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/16712 ) Change subject: IMPALA-7876: COMPUTE STATS TABLESAMPLE is not updating number of estimated rows .. IMPALA-7876: COMPUTE STATS TABLESAMPLE is not updating number of estimated rows 'COMPUTE STATS TABLESAMPLE' uses a child query with following function 'ROUND(COUNT(*) / )' for computing the row count. The 'ROUND()' fn returns the row count as a DECIMAL type. The 'CatalogOpExecutor' (CatalogOpExecutor::SetTableStats) expects the row count as a BIGINT type. Due to this data type mismatch the table stats (Extrap #Rows) doesn't get set. Adding an explicit CAST to BIGINT for the ROUND function results in the table stats (Extrap #Rows) getting set properly. Fixed both 'custom_cluster/test_stats_extrapolation.py' and 'metadata/test_stats_extrapolation.py' so that they can catch issues like this, where table stats are not set when using 'COMPUTE STATS TABLESAMPLE'. Testing: - Ran core tests. Change-Id: I88a0a777c2be9cc18b3ff293cf1c06fb499ca052 --- M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M tests/common/impala_test_suite.py M tests/custom_cluster/test_stats_extrapolation.py M tests/metadata/test_stats_extrapolation.py 4 files changed, 26 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/16712/3 -- To view, visit http://gerrit.cloudera.org:8080/16712 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I88a0a777c2be9cc18b3ff293cf1c06fb499ca052 Gerrit-Change-Number: 16712 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7876: COMPUTE STATS TABLESAMPLE is not updating number of estimated rows
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/16712 ) Change subject: IMPALA-7876: COMPUTE STATS TABLESAMPLE is not updating number of estimated rows .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/16712/3/tests/common/impala_test_suite.py File tests/common/impala_test_suite.py: http://gerrit.cloudera.org:8080/#/c/16712/3/tests/common/impala_test_suite.py@933 PS3, Line 933: # Both a and b must be positive for the following check to make sense. > Hmm, do you think it's easier to just not mess with it then? I think I will fix appx_equals for -ive values like you suggested. And also fix the callers to pass a valid diff_perc. Its only being called in a couple of places so should be okay. Let me make these changes. -- To view, visit http://gerrit.cloudera.org:8080/16712 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88a0a777c2be9cc18b3ff293cf1c06fb499ca052 Gerrit-Change-Number: 16712 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 12 Nov 2020 20:05:01 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7876: COMPUTE STATS TABLESAMPLE is not updating number of estimated rows
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16712 ) Change subject: IMPALA-7876: COMPUTE STATS TABLESAMPLE is not updating number of estimated rows .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7641/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16712 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88a0a777c2be9cc18b3ff293cf1c06fb499ca052 Gerrit-Change-Number: 16712 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 12 Nov 2020 20:26:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6861: Fix OpenSSL initialization
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16704 ) Change subject: IMPALA-6861: Fix OpenSSL initialization .. IMPALA-6861: Fix OpenSSL initialization Impalads currently initialize OpenSSL twice: once when initializing Thrift and once when initializing KRPC. The initialization is theoretically idempotent but not thread-safe, so its better to clean this up. This patch disables the Thrift version as its older (last updated in 2015) and the KRPC version contains logic specific to more recent versions of OpenSSL. The catalogd and statestored also now use the KRPC version instead of the Thrift version. It also improves debuggability by adding some additional startup logging. Testing: - Passed existing SSL tests. Change-Id: I35b1362d40c8a12082cc8b531a38b4a485bac0e7 Reviewed-on: http://gerrit.cloudera.org:8080/16704 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/rpc/authentication.cc M be/src/rpc/authentication.h M be/src/rpc/thrift-server.cc 3 files changed, 11 insertions(+), 10 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16704 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I35b1362d40c8a12082cc8b531a38b4a485bac0e7 Gerrit-Change-Number: 16704 Gerrit-PatchSet: 4 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6861: Fix OpenSSL initialization
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16704 ) Change subject: IMPALA-6861: Fix OpenSSL initialization .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16704 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I35b1362d40c8a12082cc8b531a38b4a485bac0e7 Gerrit-Change-Number: 16704 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 13 Nov 2020 00:37:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7876: COMPUTE STATS TABLESAMPLE is not updating number of estimated rows
Abhishek Rawat has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/16712 ) Change subject: IMPALA-7876: COMPUTE STATS TABLESAMPLE is not updating number of estimated rows .. IMPALA-7876: COMPUTE STATS TABLESAMPLE is not updating number of estimated rows 'COMPUTE STATS TABLESAMPLE' uses a child query with following function 'ROUND(COUNT(*) / )' for computing the row count. The 'ROUND()' fn returns the row count as a DECIMAL type. The 'CatalogOpExecutor' (CatalogOpExecutor::SetTableStats) expects the row count as a BIGINT type. Due to this data type mismatch the table stats (Extrap #Rows) doesn't get set. Adding an explicit CAST to BIGINT for the ROUND function results in the table stats (Extrap #Rows) getting set properly. Fixed both 'custom_cluster/test_stats_extrapolation.py' and 'metadata/test_stats_extrapolation.py' so that they can catch issues like this, where table stats are not set when using 'COMPUTE STATS TABLESAMPLE'. Testing: - Ran core tests. Change-Id: I88a0a777c2be9cc18b3ff293cf1c06fb499ca052 --- M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M tests/common/impala_test_suite.py M tests/custom_cluster/test_stats_extrapolation.py M tests/metadata/test_stats_extrapolation.py 4 files changed, 27 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/16712/4 -- To view, visit http://gerrit.cloudera.org:8080/16712 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I88a0a777c2be9cc18b3ff293cf1c06fb499ca052 Gerrit-Change-Number: 16712 Gerrit-PatchSet: 4 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7876: COMPUTE STATS TABLESAMPLE is not updating number of estimated rows
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16712 ) Change subject: IMPALA-7876: COMPUTE STATS TABLESAMPLE is not updating number of estimated rows .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/16712/4/tests/common/impala_test_suite.py File tests/common/impala_test_suite.py: http://gerrit.cloudera.org:8080/#/c/16712/4/tests/common/impala_test_suite.py@933 PS4, Line 933: , flake8: E231 missing whitespace after ',' -- To view, visit http://gerrit.cloudera.org:8080/16712 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88a0a777c2be9cc18b3ff293cf1c06fb499ca052 Gerrit-Change-Number: 16712 Gerrit-PatchSet: 4 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 13 Nov 2020 02:03:58 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7876: COMPUTE STATS TABLESAMPLE is not updating number of estimated rows
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/16712 ) Change subject: IMPALA-7876: COMPUTE STATS TABLESAMPLE is not updating number of estimated rows .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/16712/3/tests/common/impala_test_suite.py File tests/common/impala_test_suite.py: http://gerrit.cloudera.org:8080/#/c/16712/3/tests/common/impala_test_suite.py@933 PS3, Line 933: # Both a and b must be positive for the following check to make sense. > I think I will fix appx_equals for -ive values like you suggested. I changed the diff_perc in test_stats_extrapolation to 1.0 from 2.0. For very small sampling percentage like 1 or 3 we do get a large variance between extrapolated #rows and actual #rows. The condition we are trying to catch is any time we have a -1 (which indicates stats not set). appx_equals(X, -1, 1.0) returns false since X+1/X > 1.0 The test case passes with 1.0. We are now also ensuring that the extrapolated #rows >= 0 in the testcase. So we should be covered as far as catching this issue through the test. -- To view, visit http://gerrit.cloudera.org:8080/16712 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88a0a777c2be9cc18b3ff293cf1c06fb499ca052 Gerrit-Change-Number: 16712 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 13 Nov 2020 02:11:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7876: COMPUTE STATS TABLESAMPLE is not updating number of estimated rows
Abhishek Rawat has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/16712 ) Change subject: IMPALA-7876: COMPUTE STATS TABLESAMPLE is not updating number of estimated rows .. IMPALA-7876: COMPUTE STATS TABLESAMPLE is not updating number of estimated rows 'COMPUTE STATS TABLESAMPLE' uses a child query with following function 'ROUND(COUNT(*) / )' for computing the row count. The 'ROUND()' fn returns the row count as a DECIMAL type. The 'CatalogOpExecutor' (CatalogOpExecutor::SetTableStats) expects the row count as a BIGINT type. Due to this data type mismatch the table stats (Extrap #Rows) doesn't get set. Adding an explicit CAST to BIGINT for the ROUND function results in the table stats (Extrap #Rows) getting set properly. Fixed both 'custom_cluster/test_stats_extrapolation.py' and 'metadata/test_stats_extrapolation.py' so that they can catch issues like this, where table stats are not set when using 'COMPUTE STATS TABLESAMPLE'. Testing: - Ran core tests. Change-Id: I88a0a777c2be9cc18b3ff293cf1c06fb499ca052 --- M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M tests/common/impala_test_suite.py M tests/custom_cluster/test_stats_extrapolation.py M tests/metadata/test_stats_extrapolation.py 4 files changed, 27 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/16712/5 -- To view, visit http://gerrit.cloudera.org:8080/16712 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I88a0a777c2be9cc18b3ff293cf1c06fb499ca052 Gerrit-Change-Number: 16712 Gerrit-PatchSet: 5 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7876: COMPUTE STATS TABLESAMPLE is not updating number of estimated rows
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16712 ) Change subject: IMPALA-7876: COMPUTE STATS TABLESAMPLE is not updating number of estimated rows .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7642/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16712 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88a0a777c2be9cc18b3ff293cf1c06fb499ca052 Gerrit-Change-Number: 16712 Gerrit-PatchSet: 4 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 13 Nov 2020 02:24:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7876: COMPUTE STATS TABLESAMPLE is not updating number of estimated rows
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16712 ) Change subject: IMPALA-7876: COMPUTE STATS TABLESAMPLE is not updating number of estimated rows .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7643/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16712 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88a0a777c2be9cc18b3ff293cf1c06fb499ca052 Gerrit-Change-Number: 16712 Gerrit-PatchSet: 5 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 13 Nov 2020 02:35:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7876: COMPUTE STATS TABLESAMPLE is not updating number of estimated rows
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16712 ) Change subject: IMPALA-7876: COMPUTE STATS TABLESAMPLE is not updating number of estimated rows .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16712 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88a0a777c2be9cc18b3ff293cf1c06fb499ca052 Gerrit-Change-Number: 16712 Gerrit-PatchSet: 5 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 13 Nov 2020 03:39:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7876: COMPUTE STATS TABLESAMPLE is not updating number of estimated rows
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16712 ) Change subject: IMPALA-7876: COMPUTE STATS TABLESAMPLE is not updating number of estimated rows .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6651/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16712 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88a0a777c2be9cc18b3ff293cf1c06fb499ca052 Gerrit-Change-Number: 16712 Gerrit-PatchSet: 5 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 13 Nov 2020 03:40:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10320: Specify expression selectivity for BoolLiteral.
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16714 ) Change subject: IMPALA-10320: Specify expression selectivity for BoolLiteral. .. IMPALA-10320: Specify expression selectivity for BoolLiteral. Setting the selectivity of true and false literals to 1 and 0 respectively. Also setting a NullLiteral's selectivity to 0. Testing: - New tests in ExprCardinalityTest Change-Id: I42c96cd685f22d8634509d9f488f2a1f82b8 Reviewed-on: http://gerrit.cloudera.org:8080/16714 Tested-by: Impala Public Jenkins Reviewed-by: Tim Armstrong --- M fe/src/main/java/org/apache/impala/analysis/BoolLiteral.java M fe/src/main/java/org/apache/impala/analysis/NullLiteral.java M fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java 3 files changed, 11 insertions(+), 2 deletions(-) Approvals: Impala Public Jenkins: Verified Tim Armstrong: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/16714 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I42c96cd685f22d8634509d9f488f2a1f82b8 Gerrit-Change-Number: 16714 Gerrit-PatchSet: 2 Gerrit-Owner: Shant Hovsepian Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-10320: Specify expression selectivity for BoolLiteral.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16714 ) Change subject: IMPALA-10320: Specify expression selectivity for BoolLiteral. .. Patch Set 1: Code-Review+2 This seems like a no brainer, thank you! -- To view, visit http://gerrit.cloudera.org:8080/16714 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42c96cd685f22d8634509d9f488f2a1f82b8 Gerrit-Change-Number: 16714 Gerrit-PatchSet: 1 Gerrit-Owner: Shant Hovsepian Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 13 Nov 2020 03:46:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9382: part 2/3: aggregate profiles sent to coordinator
Hello Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16057 to look at the new patch set (#13). Change subject: IMPALA-9382: part 2/3: aggregate profiles sent to coordinator .. IMPALA-9382: part 2/3: aggregate profiles sent to coordinator This reworks the status reporting so that serialized AggregatedRuntimeProfile objects are sent from executors to coordinators. These profiles are substantially denser and faster to process for higher mt_dop values. The aggregation is also done in a single step, merging the aggregated thrift profile from the executor directly into the final aggregated profile, instead of converting it to an unaggregated profile first. The changes required were: * A new Update() method for AggregatedRuntimeProfile that updates the profile from a serialised AggregateRuntimeProfile for a subset of the instances. The code is generalized from the existing InitFromThrift() code path. * Per-fragment reports included in the status report protobuf when --gen_experimental_profile=true. * Logic on the coordinator that either consumes serialized AggregatedRuntimeProfile per fragment, when --gen_experimental_profile=true, or consumes a serialized RuntimeProfile per finstance otherwise. This also adds support for event sequences and time series in the aggregated profile, so the amount of information in the aggregated profile is now on par with the basic profile. We also finish off support for JSON profile. The JSON profile is more stripped down because we do not need to round-trip profiles via JSON and it is a much less dense profile representation. Part 3 will clean up and improve the display of the profile. Testing: * Add sanity tests for aggregated runtime profile. * Ran core tests. Change-Id: Ic680cbfe94c939c2a8fad9d0943034ed058c6bca --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/fragment-instance-state.cc M be/src/runtime/fragment-instance-state.h M be/src/runtime/fragment-state.cc M be/src/runtime/fragment-state.h M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/service/impala-server.cc M be/src/util/runtime-profile-counters.h M be/src/util/runtime-profile-test.cc M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h M common/protobuf/control_service.proto M common/thrift/ImpalaInternalService.thrift M common/thrift/RuntimeProfile.thrift A testdata/workloads/functional-query/queries/QueryTest/runtime-profile-aggregated.test A tests/custom_cluster/test_runtime_profile.py 18 files changed, 1,020 insertions(+), 264 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/16057/13 -- To view, visit http://gerrit.cloudera.org:8080/16057 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic680cbfe94c939c2a8fad9d0943034ed058c6bca Gerrit-Change-Number: 16057 Gerrit-PatchSet: 13 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9382: part 2/3: aggregate profiles sent to coordinator
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/16057 )
Change subject: IMPALA-9382: part 2/3: aggregate profiles sent to coordinator
..
Patch Set 12:
(9 comments)
Address the code comments. Pushing out changes so I can rebase before tackling
the tests.
http://gerrit.cloudera.org:8080/#/c/16057/12/be/src/runtime/coordinator-backend-state.cc
File be/src/runtime/coordinator-backend-state.cc:
http://gerrit.cloudera.org:8080/#/c/16057/12/be/src/runtime/coordinator-backend-state.cc@423
PS12, Line 423: // 'thrift_profiles' and 'instance_exec_status' vectors have
one-to-one correspondance.
> This comment is now out of date
Done
http://gerrit.cloudera.org:8080/#/c/16057/12/be/src/runtime/query-state.cc
File be/src/runtime/query-state.cc:
http://gerrit.cloudera.org:8080/#/c/16057/12/be/src/runtime/query-state.cc@591
PS12, Line 591: LOG(INFO) << it->second->min_per_fragment_instance_idx();
> Nit: Leftover logging?
Done
http://gerrit.cloudera.org:8080/#/c/16057/12/be/src/util/runtime-profile-counters.h
File be/src/util/runtime-profile-counters.h:
http://gerrit.cloudera.org:8080/#/c/16057/12/be/src/util/runtime-profile-counters.h@405
PS12, Line 405: No locks are obtained within this method because
UpdateCounter() is called from
: /// Update()
> Nit: Now that there is an Update() method on AveragedCounter, this comment
Done
http://gerrit.cloudera.org:8080/#/c/16057/12/be/src/util/runtime-profile.h
File be/src/util/runtime-profile.h:
http://gerrit.cloudera.org:8080/#/c/16057/12/be/src/util/runtime-profile.h@781
PS12, Line 781: void Update(const TRuntimeProfileTree& src, int start_idx);
> Nit: On first glance at code, it is easy to get confused between these two
Done
http://gerrit.cloudera.org:8080/#/c/16057/12/be/src/util/runtime-profile.cc
File be/src/util/runtime-profile.cc:
http://gerrit.cloudera.org:8080/#/c/16057/12/be/src/util/runtime-profile.cc@177
PS12, Line 177: // Search forward until the insert position is either at
the end of the vector
: // or after this child. This preserves the order if the
relative order of
: // children in all updates is consistent.
: bool found_child = false;
> Are there cases that actually hit the children_.end() without finding the c
Added a comment to explain why it was necessary - IMPALA-6694.
http://gerrit.cloudera.org:8080/#/c/16057/12/be/src/util/runtime-profile.cc@394
PS12, Line 394: {
: lock_guard l(agg_info_strings_lock_);
: lock_guard m(other->info_strings_lock_);
: for (const auto& entry : other->info_strings_) {
: vector& values = agg_info_strings_[entry.first];
: if (values.empty()) values.resize(num_input_profiles_);
: if (values[idx] != entry.second) values[idx] =
entry.second;
: }
: }
> A pattern in this file is having functions that have a lot of these subbloc
THanks, this is a good suggestion and made it clearer I think.
http://gerrit.cloudera.org:8080/#/c/16057/12/be/src/util/runtime-profile.cc@633
PS12, Line 633: // Decode label and assign a new index based on the
unique labels in this
: // profile.
: int32_t src_label_idx = tseq.label_idxs[i][j];
: DCHECK_GE(src_label_idx, 0);
: DCHECK_LT(src_label_idx, tseq.unique_labels.size());
: const string& label =
tseq.unique_labels[src_label_idx];
: auto it = seq.labels.find(label);
: int32_t label_idx;
: if (it == seq.labels.end()) {
: label_idx = seq.labels.size();
: seq.labels.emplace(label, label_idx);
: } else {
: label_idx = it->second;
: }
> As noted in the thrift, the individual steps here make sense to be describe
Done
http://gerrit.cloudera.org:8080/#/c/16057/12/be/src/util/runtime-profile.cc@609
PS12, Line 609: {
: // Merge event sequence.
: lock_guard l(event_sequence_lock_);
: for (const TAggEventSequence& tseq :
agg_node.event_sequences) {
: DCHECK_GT(num_input_profiles_, 0);
: AggEventSequence& seq = event_sequence_map_[tseq.name];
: if (seq.label_idxs.empty()) {
: seq.label_idxs.resize(num_input_profiles_);
: seq.timestamps.resize(num_input_profiles_);
: } else {
: DCHECK_EQ(num_input_profiles_, seq.label_idxs.size());
: DCHECK_EQ(num_input_profiles_, seq.timestamps.size());
: }
[Impala-ASF-CR] IMPALA-9382: part 2/3: aggregate profiles sent to coordinator
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16057 ) Change subject: IMPALA-9382: part 2/3: aggregate profiles sent to coordinator .. Patch Set 13: (3 comments) http://gerrit.cloudera.org:8080/#/c/16057/13/be/src/runtime/coordinator-backend-state.cc File be/src/runtime/coordinator-backend-state.cc: http://gerrit.cloudera.org:8080/#/c/16057/13/be/src/runtime/coordinator-backend-state.cc@823 PS13, Line 823: // Merge this instance profile (which may or may not included the full instance profile line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/16057/13/be/src/runtime/coordinator-backend-state.cc@824 PS13, Line 824: // from the executor) into the aggregated profile. We need to do this regardless of the line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/16057/13/be/src/util/runtime-profile.cc File be/src/util/runtime-profile.cc: http://gerrit.cloudera.org:8080/#/c/16057/13/be/src/util/runtime-profile.cc@539 PS13, Line 539: DCHECK_LT(*node_idx, src.nodes.size()); const TRuntimeProfileNode& node = src.nodes[*node_idx]; line too long (98 > 90) -- To view, visit http://gerrit.cloudera.org:8080/16057 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic680cbfe94c939c2a8fad9d0943034ed058c6bca Gerrit-Change-Number: 16057 Gerrit-PatchSet: 13 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 13 Nov 2020 05:00:23 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9382: part 2/3: aggregate profiles sent to coordinator
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16057 ) Change subject: IMPALA-9382: part 2/3: aggregate profiles sent to coordinator .. Patch Set 13: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/7644/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/16057 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic680cbfe94c939c2a8fad9d0943034ed058c6bca Gerrit-Change-Number: 16057 Gerrit-PatchSet: 13 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 13 Nov 2020 05:21:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10279: Import CPC functionality from DataSketches
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/16645 ) Change subject: IMPALA-10279: Import CPC functionality from DataSketches .. Patch Set 10: Code-Review+2 (2 comments) http://gerrit.cloudera.org:8080/#/c/16645/6//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/16645/6//COMMIT_MSG@11 PS6, Line 11: : First, I updated our existing snapshot of DataSketches to the : following commit:2b84e213067b681b696ec883d245ddf911790ff2 > The previous snapshot(commit:c1a6f8edb49699520f248d3d02019b87429b4241) alre Thanks for double-checking! http://gerrit.cloudera.org:8080/#/c/16645/6/be/src/exprs/datasketches-test.cc File be/src/exprs/datasketches-test.cc: http://gerrit.cloudera.org:8080/#/c/16645/6/be/src/exprs/datasketches-test.cc@114 PS6, Line 114: // same estimate every time we run this test. > The result is deterministic, comments have been added. Thx! -- To view, visit http://gerrit.cloudera.org:8080/16645 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1e8d0c2e80df95fa84af82c64d493df9bbb34a8c Gerrit-Change-Number: 16645 Gerrit-PatchSet: 10 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 13 Nov 2020 07:45:04 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10279: Import CPC functionality from DataSketches
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16645 ) Change subject: IMPALA-10279: Import CPC functionality from DataSketches .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6652/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16645 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1e8d0c2e80df95fa84af82c64d493df9bbb34a8c Gerrit-Change-Number: 16645 Gerrit-PatchSet: 11 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 13 Nov 2020 07:45:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10279: Import CPC functionality from DataSketches
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16645 ) Change subject: IMPALA-10279: Import CPC functionality from DataSketches .. Patch Set 11: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16645 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1e8d0c2e80df95fa84af82c64d493df9bbb34a8c Gerrit-Change-Number: 16645 Gerrit-PatchSet: 11 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 13 Nov 2020 07:45:42 + Gerrit-HasComments: No
