[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18178 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory .. Patch Set 24: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7925/ -- To view, visit http://gerrit.cloudera.org:8080/18178 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I75cf17290be2c64fd4b732a5505bdac31869712a Gerrit-Change-Number: 18178 Gerrit-PatchSet: 24 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 11 Mar 2022 05:31:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11156: TestHmsIntegration.test desc json table failed in exhaustive build
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18299 ) Change subject: IMPALA-11156: TestHmsIntegration.test_desc_json_table failed in exhaustive build .. Patch Set 8: > Patch Set 6: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7923/ The failure is unrelated. Filed IMPALA-11177 -- To view, visit http://gerrit.cloudera.org:8080/18299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4ddd983aeb82817eb355ca94dcf4a6875986f8e1 Gerrit-Change-Number: 18299 Gerrit-PatchSet: 8 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 11 Mar 2022 03:44:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11156: TestHmsIntegration.test desc json table failed in exhaustive build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18299 ) Change subject: IMPALA-11156: TestHmsIntegration.test_desc_json_table failed in exhaustive build .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7926/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4ddd983aeb82817eb355ca94dcf4a6875986f8e1 Gerrit-Change-Number: 18299 Gerrit-PatchSet: 8 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 11 Mar 2022 03:20:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11156: TestHmsIntegration.test desc json table failed in exhaustive build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18299 ) Change subject: IMPALA-11156: TestHmsIntegration.test_desc_json_table failed in exhaustive build .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4ddd983aeb82817eb355ca94dcf4a6875986f8e1 Gerrit-Change-Number: 18299 Gerrit-PatchSet: 8 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 11 Mar 2022 03:20:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11175: Iceberg table cannot be loaded when partition value is NULL
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18307 ) Change subject: IMPALA-11175: Iceberg table cannot be loaded when partition value is NULL .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10281/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18307 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88c4f7a2c2db4f6390c8ee5c08baddc96b04602e Gerrit-Change-Number: 18307 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Thu, 10 Mar 2022 20:57:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11175: Iceberg table cannot be loaded when partition value is NULL
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18307 Change subject: IMPALA-11175: Iceberg table cannot be loaded when partition value is NULL .. IMPALA-11175: Iceberg table cannot be loaded when partition value is NULL When Impala created the metadata objects about the Iceberg data files it tried to convert the partition values to strings. But the partition values can be NULLs as well. The code didn't expect this, so we got a NullPointerException. With this patch we pass the table's null partition key value in case of NULLs. Testing: * added e2e tests Change-Id: I88c4f7a2c2db4f6390c8ee5c08baddc96b04602e --- M be/src/exec/file-metadata-utils.cc M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M testdata/workloads/functional-query/queries/QueryTest/iceberg-partition-transform-insert.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test 6 files changed, 112 insertions(+), 32 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/18307/1 -- To view, visit http://gerrit.cloudera.org:8080/18307 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I88c4f7a2c2db4f6390c8ee5c08baddc96b04602e Gerrit-Change-Number: 18307 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18178 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory .. Patch Set 24: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7925/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18178 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I75cf17290be2c64fd4b732a5505bdac31869712a Gerrit-Change-Number: 18178 Gerrit-PatchSet: 24 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 10 Mar 2022 19:31:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18178 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory .. Patch Set 24: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18178 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I75cf17290be2c64fd4b732a5505bdac31869712a Gerrit-Change-Number: 18178 Gerrit-PatchSet: 24 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 10 Mar 2022 19:31:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory
Bikramjeet Vig has removed a vote on this change. Change subject: IMPALA-10992 Planner changes for estimate peak memory .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/18178 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I75cf17290be2c64fd4b732a5505bdac31869712a Gerrit-Change-Number: 18178 Gerrit-PatchSet: 23 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/18178 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory .. Patch Set 23: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18178 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I75cf17290be2c64fd4b732a5505bdac31869712a Gerrit-Change-Number: 18178 Gerrit-PatchSet: 23 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 10 Mar 2022 17:24:09 + Gerrit-HasComments: No
[Impala-ASF-CR] Revert "IMPALA-10737: Optimize the number of Iceberg API Metadata requests"
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/18305 ) Change subject: Revert "IMPALA-10737: Optimize the number of Iceberg API Metadata requests" .. Patch Set 2: Thank you for the review Zoltan! -- To view, visit http://gerrit.cloudera.org:8080/18305 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I21e65cb3ab38d9e683223fb100d7ced90caa6edd Gerrit-Change-Number: 18305 Gerrit-PatchSet: 2 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 10 Mar 2022 15:35:38 + Gerrit-HasComments: No
[Impala-ASF-CR] Revert "IMPALA-10737: Optimize the number of Iceberg API Metadata requests"
Tamas Mate has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18305 ) Change subject: Revert "IMPALA-10737: Optimize the number of Iceberg API Metadata requests" .. Revert "IMPALA-10737: Optimize the number of Iceberg API Metadata requests" This reverts commit cd10acdbb15c74eae21fa630bcb174104283560a. This commit has been reverted, because it blocks upgrading the Iceberg version to 0.13. In the newer Iceberg version the BaseTable serialization has been changed, it serializes the BaseTable to a SerializableTable sibiling class. This is a lightweigth Table class which does not have the necessary metadata that could be cached and reused by the ImpalaDs. SerializableTable utilization has to be further considered. Change-Id: I21e65cb3ab38d9e683223fb100d7ced90caa6edd Reviewed-on: http://gerrit.cloudera.org:8080/18305 Reviewed-by: Zoltan Borok-Nagy Tested-by: Impala Public Jenkins --- M common/thrift/CatalogObjects.thrift M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java D fe/src/main/java/org/apache/impala/catalog/IcebergTableLoadingException.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java M testdata/workloads/functional-query/queries/QueryTest/iceberg-insert.test 18 files changed, 182 insertions(+), 234 deletions(-) Approvals: Zoltan Borok-Nagy: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/18305 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I21e65cb3ab38d9e683223fb100d7ced90caa6edd Gerrit-Change-Number: 18305 Gerrit-PatchSet: 2 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] Revert "IMPALA-10737: Optimize the number of Iceberg API Metadata requests"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18305 ) Change subject: Revert "IMPALA-10737: Optimize the number of Iceberg API Metadata requests" .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18305 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I21e65cb3ab38d9e683223fb100d7ced90caa6edd Gerrit-Change-Number: 18305 Gerrit-PatchSet: 1 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 10 Mar 2022 15:28:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10838: Error when struct returned from WITH()
Gabor Kaszab has posted comments on this change. (
http://gerrit.cloudera.org:8080/17847 )
Change subject: IMPALA-10838: Error when struct returned from WITH()
..
Patch Set 18:
(20 comments)
Nice patch! :)
http://gerrit.cloudera.org:8080/#/c/17847/17/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/17847/17/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@1415
PS17, Line 1415: List ancestorTypes = matchedTypes.subList(0,
matchedTypes.size() - 1);
No need to create a sublist here, just make a for loop that skips the last
item. This way you can get rid of both the local Lists introduced here.
http://gerrit.cloudera.org:8080/#/c/17847/17/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@1438
PS17, Line 1438:* However, if the order is reversed, we are in a more
difficult situation. If we try to
Can't you get rid og this hassle if you changed the processing order of the
expressions? E.g. always process structs first.
http://gerrit.cloudera.org:8080/#/c/17847/17/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@1462
PS17, Line 1462: int highestAncestorDistance) throws AnalysisException {
Could you please describe the purpose of the function parameters in the comment
above?
http://gerrit.cloudera.org:8080/#/c/17847/17/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@1525
PS17, Line 1525: structTuple.setParentSlotDesc(desc);
nit: this could go to the top of this function so that a user could more easily
see that we accept structs as parameter here.
http://gerrit.cloudera.org:8080/#/c/17847/17/fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java
File fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java:
http://gerrit.cloudera.org:8080/#/c/17847/17/fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java@154
PS17, Line 154: // If we are inside a struct, we need to materialise all
enclosing struct
nit: I think this comment would be better at the callsite.
http://gerrit.cloudera.org:8080/#/c/17847/17/fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java
File fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java:
http://gerrit.cloudera.org:8080/#/c/17847/17/fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java@22
PS17, Line 22: import java.util.stream.Collectors;
This is not needed anymore, right?
http://gerrit.cloudera.org:8080/#/c/17847/17/fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java
File fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java:
http://gerrit.cloudera.org:8080/#/c/17847/17/fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java@336
PS17, Line 336: // createAuxPredicatesForStructMembers(analyzer,
(SlotRef) colExpr);
Honestly, I don't know much about aux predicates so won't be able to answer
this.
http://gerrit.cloudera.org:8080/#/c/17847/17/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java:
http://gerrit.cloudera.org:8080/#/c/17847/17/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@286
PS17, Line 286: preregisterStructSlotRefPathsWithAnalyzer();
I have asked this question in another comment but let me duplicate it: Instead
of pre-registering the struct SlotRefs here, wouldn't re-ordering the select
list items help? E.g. Putting everything that involves a struct to the
beginning of the select list.
http://gerrit.cloudera.org:8080/#/c/17847/17/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@327
PS17, Line 327: // Preregister select list paths that point to structs.
This will be used to unify
nit: This should be a function comment in above the function signature.
http://gerrit.cloudera.org:8080/#/c/17847/17/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@351
PS17, Line 351: List
Could this be a HashSet to avoid duplicates?
http://gerrit.cloudera.org:8080/#/c/17847/17/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@354
PS17, Line 354: List paths =
slotRefs.stream().map(this::slotRefToResolvedPath)
I observed you really don't like for loops and ifs :D
http://gerrit.cloudera.org:8080/#/c/17847/17/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@361
PS17, Line 361: private Path slotRefToResolvedPath(SlotRef slotRef) {
I think this should be a part of SlotRef instead of SelectStmt. The whole
content of this function reminds me the beginning of SlotRef.analyzeImpl()
anyway.
http://gerrit.cloudera.org:8080/#/c/17847/17/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@361
PS17, Line 361: slotRefToResolvedPath
The naming should be something different to emphasise that this is not just a
get method but also resolves the rawPath.
http://gerrit.cloudera.org:
[Impala-ASF-CR] IMPALA-10838: Error when struct returned from WITH()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17847 ) Change subject: IMPALA-10838: Error when struct returned from WITH() .. Patch Set 18: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10280/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17847 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iadb9233677355b85d424cc3f22b00b5a3bf61c57 Gerrit-Change-Number: 17847 Gerrit-PatchSet: 18 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Comment-Date: Thu, 10 Mar 2022 15:11:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10838: Error when struct returned from WITH()
Daniel Becker has uploaded a new patch set (#18). ( http://gerrit.cloudera.org:8080/17847 ) Change subject: IMPALA-10838: Error when struct returned from WITH() .. IMPALA-10838: Error when struct returned from WITH() The following query fails: ''' with sub as ( select id, outer_struct from functional_orc_def.complextypes_nested_structs) select sub.id, sub.outer_struct.inner_struct2 from sub; ''' with the following error: ''' ERROR: IllegalStateException: Illegal reference to non-materialized tuple: debugname=InlineViewRef sub alias=sub tid=6 ''' while if 'outer_struct.inner_struct2' is added to the select list of the inline view, the query works as expected. This change fixes the problem by two modifications: - if a field of a struct needs to be materialised, also materialise all of its enclosing structs (ancestors) - in InlineViewRef, struct fields are inserted into the 'smap' and 'baseTableSmap' with the appropriate inline view prefix TODO: Is this just a hack with the labels? This change also changes the way struct fields are materialised: until now, if a member of a struct was needed to be materialised, the whole struct, including other members of the struct were materialised. This behaviour can lead to using significantly more memory than necessary if we for example query a single member of a large struct. This change modifies this behaviour so that we only materialise the struct members that are actually needed. Tests: - added queries that are fixed by this change (including the one above) in nested-struct-in-select-list.test - added a planner test in fe/src/test/java/org/apache/impala/planner/PlannerTest.java that asserts that only the required parts of structs are materialised Change-Id: Iadb9233677355b85d424cc3f22b00b5a3bf61c57 --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/CollectionTableRef.java M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java M fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java M fe/src/main/java/org/apache/impala/analysis/Path.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/SortInfo.java M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java M testdata/workloads/functional-query/queries/QueryTest/nested-struct-in-select-list.test 16 files changed, 798 insertions(+), 96 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/17847/18 -- To view, visit http://gerrit.cloudera.org:8080/17847 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iadb9233677355b85d424cc3f22b00b5a3bf61c57 Gerrit-Change-Number: 17847 Gerrit-PatchSet: 18 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen
[Impala-ASF-CR] IMPALA-11156: TestHmsIntegration.test desc json table failed in exhaustive build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18299 ) Change subject: IMPALA-11156: TestHmsIntegration.test_desc_json_table failed in exhaustive build .. Patch Set 6: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7923/ -- To view, visit http://gerrit.cloudera.org:8080/18299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4ddd983aeb82817eb355ca94dcf4a6875986f8e1 Gerrit-Change-Number: 18299 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 10 Mar 2022 14:31:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18178 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory .. Patch Set 23: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10279/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18178 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I75cf17290be2c64fd4b732a5505bdac31869712a Gerrit-Change-Number: 18178 Gerrit-PatchSet: 23 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 10 Mar 2022 14:15:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory
Qifan Chen has uploaded a new patch set (#23). ( http://gerrit.cloudera.org:8080/18178 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory .. IMPALA-10992 Planner changes for estimate peak memory This patch provides replan support for multiple executor group sets. Each executor group set is associated with a distinct number of nodes and a threshold for estimated memory per host in bytes that can be denoted as [:<#nodes>, ]. In the patch, a query of type EXPLAIN, QUERY or DML can be compiled more than once. In each attempt, per host memory is estimated and compared with the threshold of an executor group set. If the estimated memory is no more than the threshold, the iteration process terminates and the final plan is determined. The executor group set with the threshold is selected to run the query. A new query option 'enable_replan', default to 1 (enabled), is added. It can be set to 0 to disable this patch and to generate the distributed plan for the default executor group. To avoid long compilation time, the following enhancement is enabled. Note 1) and 2) can be disabled when relevant meta-data change is detected. 1. Authorization is performed only for the 1st compilation; 2. The needed meta-data is fetched into a StmtTableCache in 1st compilation and reused in subsequent compilations; 3. openTransaction() is called for transactional queries in 1st compilation and the saved transactional info is used in subsequent compilations. Similar logic is applied to Kudu transactional queries. To facilitate testing, the patch imposes an artificial two executor group setup in FE as follows. 1. [regular:<#nodes>, 64MB] 2. [large:<#nodes>, 8PB] This setup is enabled when a new query option 'test_replan' is set to 1 in backend tests, or RuntimeEnv.INSTANCE.isTestEnv() is true as in most frontend tests. This query option is set to 0 by default. Compilation time increases when a query is compiled in several iterations, as shown below for several TPCDs queries. The increase is mostly due to redundant work in either single node plan creation or recomputing value transfer graph phase. For small queries, the increase can be avoided if they can be compiled in single iteration by properly setting the smallest threshold among all executor group sets. For example, for the set of queries listed below, the smallest threshold can be set to 320MB to catch both q15 and q21 in one compilation. Compilation time (ms) Queries Estimated Memory 2-iterations 1-iteration Percentage of increase q1 408MB 18.32 13.0140.81% q11 1.37GB 186.17 86.28 115.77% q10a 519MB 108.27 53.58 102.07% q13339MB 118.03 82.4343.19% q14a 3.56GB 628.27307.24 104.49% q14b 2.20GB 518.79239.05 117.02% q15314MB 13.12 4.51 190.91% q21275MB 11.04 6.3474.13% q23a 1.34GB 458.7227.62 101.52% q23b 1.50GB 471.29224.75 109.70% q42.60GB 206.34 98.64 109.18% q67 5.16GB 691.45336.31 105.60% Testing: 1. Almost all FE and BE tests are now run in the artificial two executor setup except a few where a specific cluster configuration is desirable; 2. Ran core tests successfully; 3. Added a new observability test and a new query assignment test. Change-Id: I75cf17290be2c64fd4b732a5505bdac31869712a --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/Frontend.thrift M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/ClassUtil.java M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java M fe/src/test/java/org/apache/impala/planner/ClusterSizeTest.java A fe/src/test/resources/fair-scheduler-2-groups.xml A fe/src/test/resources/llama-site-2-groups.xml M tests/common/test_dimensions.py M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_coordinators.py M tests/custom_cluster/test_executor_groups.py M tests/query_test/test_observability.py 22 files changed, 655 insertions(+), 69 deletions
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18178 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory .. Patch Set 22: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7922/ -- To view, visit http://gerrit.cloudera.org:8080/18178 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I75cf17290be2c64fd4b732a5505bdac31869712a Gerrit-Change-Number: 18178 Gerrit-PatchSet: 22 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 10 Mar 2022 13:25:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11156: TestHmsIntegration.test desc json table failed in exhaustive build
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18299 ) Change subject: IMPALA-11156: TestHmsIntegration.test_desc_json_table failed in exhaustive build .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4ddd983aeb82817eb355ca94dcf4a6875986f8e1 Gerrit-Change-Number: 18299 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 10 Mar 2022 12:02:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11156: TestHmsIntegration.test desc json table failed in exhaustive build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18299 ) Change subject: IMPALA-11156: TestHmsIntegration.test_desc_json_table failed in exhaustive build .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10278/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4ddd983aeb82817eb355ca94dcf4a6875986f8e1 Gerrit-Change-Number: 18299 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 10 Mar 2022 11:50:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11156: TestHmsIntegration.test desc json table failed in exhaustive build
pranav.lo...@cloudera.com has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/18299 ) Change subject: IMPALA-11156: TestHmsIntegration.test_desc_json_table failed in exhaustive build .. IMPALA-11156: TestHmsIntegration.test_desc_json_table failed in exhaustive build The test was getting skipped as it was not in exhaustive strategy, to fix that its been moved to TestHmsIntegrationSanity(ImpalaTestSuite) class. Now there are two tests, test_desc_json_tbl and test_desc_json_tbl_hive to check whether JSON tables created in impala can be described in hive and vice versa. Change-Id: I4ddd983aeb82817eb355ca94dcf4a6875986f8e1 --- M tests/metadata/test_hms_integration.py 1 file changed, 25 insertions(+), 19 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/18299/7 -- To view, visit http://gerrit.cloudera.org:8080/18299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4ddd983aeb82817eb355ca94dcf4a6875986f8e1 Gerrit-Change-Number: 18299 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-11156: TestHmsIntegration.test desc json table failed in exhaustive build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18299 ) Change subject: IMPALA-11156: TestHmsIntegration.test_desc_json_table failed in exhaustive build .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10277/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4ddd983aeb82817eb355ca94dcf4a6875986f8e1 Gerrit-Change-Number: 18299 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 10 Mar 2022 10:58:59 + Gerrit-HasComments: No
[Impala-ASF-CR] Revert "IMPALA-10737: Optimize the number of Iceberg API Metadata requests"
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18305 ) Change subject: Revert "IMPALA-10737: Optimize the number of Iceberg API Metadata requests" .. Patch Set 1: Code-Review+2 Thanks Tamas for fixing this! -- To view, visit http://gerrit.cloudera.org:8080/18305 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I21e65cb3ab38d9e683223fb100d7ced90caa6edd Gerrit-Change-Number: 18305 Gerrit-PatchSet: 1 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 10 Mar 2022 10:38:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11156: TestHmsIntegration.test desc json table failed in exhaustive build
pranav.lo...@cloudera.com has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/18299 ) Change subject: IMPALA-11156: TestHmsIntegration.test_desc_json_table failed in exhaustive build .. IMPALA-11156: TestHmsIntegration.test_desc_json_table failed in exhaustive build The test was getting skipped as it was not in exhaustive strategy, to fix that its been moved to TestHmsIntegrationSanity(ImpalaTestSuite) class. Now there are two tests, test_desc_json_tbl and test_desc_json_tbl_hive to check whether JSON tables created in impala can be described in hive and vice versa. Change-Id: I4ddd983aeb82817eb355ca94dcf4a6875986f8e1 --- M tests/metadata/test_hms_integration.py 1 file changed, 25 insertions(+), 19 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/18299/6 -- To view, visit http://gerrit.cloudera.org:8080/18299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4ddd983aeb82817eb355ca94dcf4a6875986f8e1 Gerrit-Change-Number: 18299 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-11156: TestHmsIntegration.test desc json table failed in exhaustive build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18299 ) Change subject: IMPALA-11156: TestHmsIntegration.test_desc_json_table failed in exhaustive build .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/18299/6/tests/metadata/test_hms_integration.py File tests/metadata/test_hms_integration.py: http://gerrit.cloudera.org:8080/#/c/18299/6/tests/metadata/test_hms_integration.py@127 PS6, Line 127: # flake8: E265 block comment should start with '# ' -- To view, visit http://gerrit.cloudera.org:8080/18299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4ddd983aeb82817eb355ca94dcf4a6875986f8e1 Gerrit-Change-Number: 18299 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 10 Mar 2022 10:38:59 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Revert "IMPALA-10737: Optimize the number of Iceberg API Metadata requests"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18305 ) Change subject: Revert "IMPALA-10737: Optimize the number of Iceberg API Metadata requests" .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7924/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18305 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I21e65cb3ab38d9e683223fb100d7ced90caa6edd Gerrit-Change-Number: 18305 Gerrit-PatchSet: 1 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 10 Mar 2022 10:30:37 + Gerrit-HasComments: No
[Impala-ASF-CR] Revert "IMPALA-10737: Optimize the number of Iceberg API Metadata requests"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18305 ) Change subject: Revert "IMPALA-10737: Optimize the number of Iceberg API Metadata requests" .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10276/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18305 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I21e65cb3ab38d9e683223fb100d7ced90caa6edd Gerrit-Change-Number: 18305 Gerrit-PatchSet: 1 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 10 Mar 2022 09:26:20 + Gerrit-HasComments: No
[Impala-ASF-CR] Revert "IMPALA-10737: Optimize the number of Iceberg API Metadata requests"
Tamas Mate has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18305 Change subject: Revert "IMPALA-10737: Optimize the number of Iceberg API Metadata requests" .. Revert "IMPALA-10737: Optimize the number of Iceberg API Metadata requests" This reverts commit cd10acdbb15c74eae21fa630bcb174104283560a. This commit has been reverted, because it blocks upgrading the Iceberg version to 0.13. In the newer Iceberg version the BaseTable serialization has been changed, it serializes the BaseTable to a SerializableTable sibiling class. This is a lightweigth Table class which does not have the necessary metadata that could be cached and reused by the ImpalaDs. SerializableTable utilization has to be further considered. Change-Id: I21e65cb3ab38d9e683223fb100d7ced90caa6edd --- M common/thrift/CatalogObjects.thrift M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java D fe/src/main/java/org/apache/impala/catalog/IcebergTableLoadingException.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java M testdata/workloads/functional-query/queries/QueryTest/iceberg-insert.test 18 files changed, 182 insertions(+), 234 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/18305/1 -- To view, visit http://gerrit.cloudera.org:8080/18305 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I21e65cb3ab38d9e683223fb100d7ced90caa6edd Gerrit-Change-Number: 18305 Gerrit-PatchSet: 1 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-11156: TestHmsIntegration.test desc json table failed in exhaustive build
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18299 ) Change subject: IMPALA-11156: TestHmsIntegration.test_desc_json_table failed in exhaustive build .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/18299/3/tests/metadata/test_hms_integration.py File tests/metadata/test_hms_integration.py: http://gerrit.cloudera.org:8080/#/c/18299/3/tests/metadata/test_hms_integration.py@108 PS3, Line 108: """This function tests if JSON tables created : in impala can be described in hive""" nit: you don't need to wrap comments in such a small width.. http://gerrit.cloudera.org:8080/#/c/18299/3/tests/metadata/test_hms_integration.py@123 PS3, Line 123: """ nit: please use # instead. -- To view, visit http://gerrit.cloudera.org:8080/18299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4ddd983aeb82817eb355ca94dcf4a6875986f8e1 Gerrit-Change-Number: 18299 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 10 Mar 2022 08:29:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11156: TestHmsIntegration.test desc json table failed in exhaustive build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18299 ) Change subject: IMPALA-11156: TestHmsIntegration.test_desc_json_table failed in exhaustive build .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10275/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4ddd983aeb82817eb355ca94dcf4a6875986f8e1 Gerrit-Change-Number: 18299 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 10 Mar 2022 08:32:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11156: TestHmsIntegration.test desc json table failed in exhaustive build
pranav.lo...@cloudera.com has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/18299 ) Change subject: IMPALA-11156: TestHmsIntegration.test_desc_json_table failed in exhaustive build .. IMPALA-11156: TestHmsIntegration.test_desc_json_table failed in exhaustive build The test was getting skipped as it was not in exhaustive strategy, to fix that its been moved to TestHmsIntegrationSanity(ImpalaTestSuite) class. Now there are two tests, test_desc_json_tbl and test_desc_json_tbl_hive to check whether JSON tables created in impala can be described in hive and vice versa. The tests can be checked by running the below commands: impala-py.test --exploration=exhaustive tests/metadata/test_hms_integration.py::TestHmsIntegrationSanity::test_desc_json_tbl impala-py.test --exploration=exhaustive tests/metadata/test_hms_integration.py::TestHmsIntegrationSanity::test_desc_json_tbl_hive Change-Id: I4ddd983aeb82817eb355ca94dcf4a6875986f8e1 --- M tests/metadata/test_hms_integration.py 1 file changed, 22 insertions(+), 19 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/18299/3 -- To view, visit http://gerrit.cloudera.org:8080/18299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4ddd983aeb82817eb355ca94dcf4a6875986f8e1 Gerrit-Change-Number: 18299 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
