[Impala-ASF-CR] POC IMPALA-9498: Allow returning arrays in select list
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17811 ) Change subject: POC IMPALA-9498: Allow returning arrays in select list .. Patch Set 12: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7524/ -- To view, visit http://gerrit.cloudera.org:8080/17811 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibb1e42ffb21c7ddc033aba0f754b0108e46f34d0 Gerrit-Change-Number: 17811 Gerrit-PatchSet: 12 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 08 Oct 2021 00:54:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10956: datasketches UDFs: memory leak and merge overhead
Hello Fucun Chu, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17869 to look at the new patch set (#3). Change subject: IMPALA-10956: datasketches UDFs: memory leak and merge overhead .. IMPALA-10956: datasketches UDFs: memory leak and merge overhead - call destructors of sketch and union objects - avoid overhead of constructing union and getting result from it every time Change-Id: I8dd0e6736f4266f74f5f265f58d40a4e4707287f --- M be/src/exprs/aggregate-functions-ir.cc 1 file changed, 273 insertions(+), 195 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/17869/3 -- To view, visit http://gerrit.cloudera.org:8080/17869 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8dd0e6736f4266f74f5f265f58d40a4e4707287f Gerrit-Change-Number: 17869 Gerrit-PatchSet: 3 Gerrit-Owner: Alexander Saydakov Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10956 datasketches UDFs: memory leak and merge overhead
Hello Fucun Chu, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17869 to look at the new patch set (#2). Change subject: IMPALA-10956 datasketches UDFs: memory leak and merge overhead .. IMPALA-10956 datasketches UDFs: memory leak and merge overhead - call destructors of sketch and union objects - avoid overhead of constructing union and getting result from it every time Change-Id: I8dd0e6736f4266f74f5f265f58d40a4e4707287f --- M be/src/exprs/aggregate-functions-ir.cc 1 file changed, 273 insertions(+), 195 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/17869/2 -- To view, visit http://gerrit.cloudera.org:8080/17869 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8dd0e6736f4266f74f5f265f58d40a4e4707287f Gerrit-Change-Number: 17869 Gerrit-PatchSet: 2 Gerrit-Owner: Alexander Saydakov Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10926: Sync db/table in catalog cache to latest HMS event id when performing DDL operations via catalog HMS endpoints
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17859 ) Change subject: IMPALA-10926: Sync db/table in catalog cache to latest HMS event id when performing DDL operations via catalog HMS endpoints .. Patch Set 15: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/9575/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17859 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I36364e401911352c474eb98c8d61bbaae9b9 Gerrit-Change-Number: 17859 Gerrit-PatchSet: 15 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Thu, 07 Oct 2021 20:25:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10926: Sync db/table in catalog cache to latest HMS event id when performing DDL operations via catalog HMS endpoints
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17859 ) Change subject: IMPALA-10926: Sync db/table in catalog cache to latest HMS event id when performing DDL operations via catalog HMS endpoints .. Patch Set 15: (4 comments) http://gerrit.cloudera.org:8080/#/c/17859/15/fe/src/main/java/org/apache/impala/catalog/Db.java File fe/src/main/java/org/apache/impala/catalog/Db.java: http://gerrit.cloudera.org:8080/#/c/17859/15/fe/src/main/java/org/apache/impala/catalog/Db.java@150 PS15, Line 150: String.format("last synced event id: %s to be set for db %s should be >= createEvent id: %s", line too long (101 > 90) http://gerrit.cloudera.org:8080/#/c/17859/15/fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java File fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java: http://gerrit.cloudera.org:8080/#/c/17859/15/fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java@1311 PS15, Line 1311: MetastoreEvents.AlterTableEvent alterEvent = (MetastoreEvents.AlterTableEvent) event; line too long (93 > 90) http://gerrit.cloudera.org:8080/#/c/17859/15/fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java@1313 PS15, Line 1313: org.apache.hadoop.hive.metastore.api.Table oldMsTable = alterEvent.getBeforeTable(); line too long (92 > 90) http://gerrit.cloudera.org:8080/#/c/17859/15/fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java@1314 PS15, Line 1314: org.apache.hadoop.hive.metastore.api.Table newMsTable = alterEvent.getAfterTable(); line too long (91 > 90) -- To view, visit http://gerrit.cloudera.org:8080/17859 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I36364e401911352c474eb98c8d61bbaae9b9 Gerrit-Change-Number: 17859 Gerrit-PatchSet: 15 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Thu, 07 Oct 2021 20:04:01 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10926: Sync db/table in catalog cache to latest HMS event id when performing DDL operations via catalog HMS endpoints
Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/17859 ) Change subject: IMPALA-10926: Sync db/table in catalog cache to latest HMS event id when performing DDL operations via catalog HMS endpoints .. Patch Set 15: (35 comments) http://gerrit.cloudera.org:8080/#/c/17859/11//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17859/11//COMMIT_MSG@7 PS11, Line 7: Sync db/table in catalog cache to latest HMS event id when performing : DDL operations via catalog HMS endpoints > If this patch is close to getting merged, now is a good to add more details Yes I will add more details in the commit message. Will fix message format styles as well. http://gerrit.cloudera.org:8080/#/c/17859/12//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17859/12//COMMIT_MSG@7 PS12, Line 7: IMPALA-10926: Sync db/table in catalog cache to latest HMS event id when performing : DDL operations via catalog HMS endpoints > Since this patch is not a WIP anymore, can you please follow the commit mes Ack http://gerrit.cloudera.org:8080/#/c/17859/6/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/17859/6/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@457 PS6, Line 457: tableInfo); : } : int tableIndex=-1, versionLockCount = 0; : try { : for(tableIndex = 0; tableIndex < numTables; tableIndex++) { : Table tbl = tables[tableIndex]; : if (!tryWriteLock(tbl)) { : LOG.debug("Could not acquire write lock on table: " + tbl.getFullName()); : return false; : } : versionLockCount += 1; : } : // in case of success, release version write lock for all tables except last : if (tableIndex == numTables) { : > I think the versionLock.writeLock().unlock() should be moved out finally bl Currently tryWriteLock(tbl) does not throw an exception. But it still makes sense to release version lock in a finally block. Will fix it. http://gerrit.cloudera.org:8080/#/c/17859/6/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@3652 PS6, Line 3652: > annotate with @VisibleForTesting if this was for testing. It is used in testing as well as in JniCatalog (after initializing catalogServiceCatalog object) http://gerrit.cloudera.org:8080/#/c/17859/6/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@3658 PS6, Line 3658: > nit, missing newline. Ack http://gerrit.cloudera.org:8080/#/c/17859/11/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/17859/11/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@459 PS11, Line 459: leIndex=-1, > I had left some comments in the older gerrit url for this patch. Can you pl Sure http://gerrit.cloudera.org:8080/#/c/17859/12/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/17859/12/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@457 PS12, Line 457: tableInfo); : } : int tableIndex=-1, versionLockCount = 0; : try { : for(tableIndex = 0; tableIndex < numTables; tableIndex++) { : Table tbl = tables[tableIndex]; : if (!tryWriteLock(tbl)) { : LOG.debug("Could not acquire write lock on table: " + tbl.getFullName()); : return false; : } : versionLockCount += 1; : } : // in case of success, release version write lock for all tables except last : if (tableIndex == numTables) { : > a RuntimeException thrown at line 459 will not release the table locks as w Makes sense. I didn't think about the RunTimeException. Will address it. http://gerrit.cloudera.org:8080/#/c/17859/12/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2672 PS12, Line 2672: > why do we need to override this method? I think the intention for this change was - currently for removeTable() api , we acquire global write lock. So the read for the table should acquire global read lock. http://gerrit.cloudera.org:8080/#/c/17859/6/fe/src/main/java/org/apache/impala/catalog/Db.java File fe/src/main/java/org/apache/impala/catalog/Db.
[Impala-ASF-CR] IMPALA-10926: Sync db/table in catalog cache to latest HMS event id when performing DDL operations via catalog HMS endpoints
Hello Vihang Karajgaonkar, kis...@cloudera.com, Yu-Wen Lai, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17859 to look at the new patch set (#15). Change subject: IMPALA-10926: Sync db/table in catalog cache to latest HMS event id when performing DDL operations via catalog HMS endpoints .. IMPALA-10926: Sync db/table in catalog cache to latest HMS event id when performing DDL operations via catalog HMS endpoints Change-Id: I36364e401911352c474eb98c8d61bbaae9b9 --- M be/src/catalog/catalog-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/TableLoader.java M fe/src/main/java/org/apache/impala/catalog/TableLoadingMgr.java M fe/src/main/java/org/apache/impala/catalog/events/EventFactory.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/catalog/events/NoOpEventProcessor.java M fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/catalog/metastore/HmsApiNameEnum.java M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/test/java/org/apache/impala/catalog/AlterDatabaseTest.java A fe/src/test/java/org/apache/impala/catalog/MetastoreApiTestUtils.java M fe/src/test/java/org/apache/impala/catalog/events/EventsProcessorStressTest.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java M fe/src/test/java/org/apache/impala/catalog/metastore/AbstractCatalogMetastoreTest.java A fe/src/test/java/org/apache/impala/catalog/metastore/CatalogHmsSyncToLatestEventIdTest.java M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java M tests/custom_cluster/test_metastore_service.py 27 files changed, 3,399 insertions(+), 274 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/17859/15 -- To view, visit http://gerrit.cloudera.org:8080/17859 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I36364e401911352c474eb98c8d61bbaae9b9 Gerrit-Change-Number: 17859 Gerrit-PatchSet: 15 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai
[Impala-ASF-CR] POC IMPALA-9498: Allow returning arrays in select list
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17811 ) Change subject: POC IMPALA-9498: Allow returning arrays in select list .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/9574/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17811 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibb1e42ffb21c7ddc033aba0f754b0108e46f34d0 Gerrit-Change-Number: 17811 Gerrit-PatchSet: 12 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 07 Oct 2021 19:09:10 + Gerrit-HasComments: No
[Impala-ASF-CR] POC IMPALA-9498: Allow returning arrays in select list
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17811 ) Change subject: POC IMPALA-9498: Allow returning arrays in select list .. Patch Set 12: (8 comments) http://gerrit.cloudera.org:8080/#/c/17811/12/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/17811/12/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@1309 PS12, Line 1309: public SlotDescriptor registerSlotRef(Path slotPath, boolean duplicateCollections) throws AnalysisException { line too long (111 > 90) http://gerrit.cloudera.org:8080/#/c/17811/12/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@1332 PS12, Line 1332: //CollectionTableRef collTblRef = (CollectionTableRef) analyzer.resolveTableRef(tblRef); //new CollectionTableRef(tblRef, currentPath, true); line too long (150 > 90) http://gerrit.cloudera.org:8080/#/c/17811/12/fe/src/main/java/org/apache/impala/analysis/CollectionTableRef.java File fe/src/main/java/org/apache/impala/analysis/CollectionTableRef.java: http://gerrit.cloudera.org:8080/#/c/17811/12/fe/src/main/java/org/apache/impala/analysis/CollectionTableRef.java@63 PS12, Line 63: String parentAlias = resolvedPath.getRootDesc() == null ? null : resolvedPath.getRootDesc().getAlias(); line too long (107 > 90) http://gerrit.cloudera.org:8080/#/c/17811/12/fe/src/main/java/org/apache/impala/analysis/CollectionTableRef.java@99 PS12, Line 99: SlotDescriptor parentSlotDesc = analyzer.registerSlotRef(resolvedPath_/*, !inSelectList_*/); line too long (98 > 90) http://gerrit.cloudera.org:8080/#/c/17811/12/fe/src/main/java/org/apache/impala/analysis/CollectionTableRef.java@159 PS12, Line 159: public boolean setIsCollectionInSelectList(boolean value) { return inSelectList_ = true; } line too long (92 > 90) http://gerrit.cloudera.org:8080/#/c/17811/12/fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java File fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java: http://gerrit.cloudera.org:8080/#/c/17811/12/fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java@327 PS12, Line 327: String parentTupleId = (parent_ == null ? "null" : String.valueOf(parent_.getId().asInt())); line too long (96 > 90) http://gerrit.cloudera.org:8080/#/c/17811/12/fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java@328 PS12, Line 328: String itemTupleId = (itemTupleDesc_ == null ? "null" : String.valueOf(itemTupleDesc_.getId().asInt())); line too long (108 > 90) http://gerrit.cloudera.org:8080/#/c/17811/12/fe/src/main/java/org/apache/impala/analysis/SlotRef.java File fe/src/main/java/org/apache/impala/analysis/SlotRef.java: http://gerrit.cloudera.org:8080/#/c/17811/12/fe/src/main/java/org/apache/impala/analysis/SlotRef.java@155 PS12, Line 155: //CollectionTableRef collTblRef = (CollectionTableRef) analyzer.resolveTableRef(tblRef); //new CollectionTableRef(tblRef, currentPath, true); line too long (150 > 90) -- To view, visit http://gerrit.cloudera.org:8080/17811 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibb1e42ffb21c7ddc033aba0f754b0108e46f34d0 Gerrit-Change-Number: 17811 Gerrit-PatchSet: 12 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 07 Oct 2021 18:46:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] POC IMPALA-9498: Allow returning arrays in select list
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17811 ) Change subject: POC IMPALA-9498: Allow returning arrays in select list .. Patch Set 12: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7524/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/17811 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibb1e42ffb21c7ddc033aba0f754b0108e46f34d0 Gerrit-Change-Number: 17811 Gerrit-PatchSet: 12 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 07 Oct 2021 18:46:30 + Gerrit-HasComments: No
[Impala-ASF-CR] POC IMPALA-9498: Allow returning arrays in select list
Hello Daniel Becker, Gabor Kaszab, Attila Jeges, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17811 to look at the new patch set (#12). Change subject: POC IMPALA-9498: Allow returning arrays in select list .. POC IMPALA-9498: Allow returning arrays in select list Change-Id: Ibb1e42ffb21c7ddc033aba0f754b0108e46f34d0 --- M be/src/codegen/codegen-anyval.cc M be/src/exec/blocking-plan-root-sink.cc M be/src/exec/buffered-plan-root-sink.cc M be/src/exec/parquet/parquet-collection-column-reader.cc M be/src/exec/plan-root-sink.cc M be/src/exec/plan-root-sink.h M be/src/exprs/expr.h M be/src/exprs/slot-ref.cc M be/src/exprs/slot-ref.h M be/src/runtime/collection-value.h M be/src/runtime/raw-value.cc M be/src/runtime/raw-value.h M be/src/runtime/types.cc M be/src/runtime/types.h M be/src/service/hs2-util.cc M be/src/service/impala-beeswax-server.cc M be/src/service/query-result-set.cc M be/src/service/query-result-set.h M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/CollectionTableRef.java M fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java M fe/src/main/java/org/apache/impala/authorization/TableMask.java M fe/src/main/java/org/apache/impala/catalog/Type.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/planner/UnionNode.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeUpsertStmtTest.java A testdata/workloads/functional-query/queries/QueryTest/nested-array-in-select-list.test M testdata/workloads/functional-query/queries/QueryTest/ranger_column_masking_struct_in_select_list.test M testdata/workloads/functional-query/queries/QueryTest/struct-in-select-list.test M tests/authorization/test_ranger.py M tests/query_test/test_nested_types.py 40 files changed, 722 insertions(+), 166 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/17811/12 -- To view, visit http://gerrit.cloudera.org:8080/17811 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibb1e42ffb21c7ddc033aba0f754b0108e46f34d0 Gerrit-Change-Number: 17811 Gerrit-PatchSet: 12 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10945: Fix S3 scratch path behavior
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17901 ) Change subject: IMPALA-10945: Fix S3 scratch path behavior .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/17901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I028f375b9f535f8641261cc02f921497e076aa9b Gerrit-Change-Number: 17901 Gerrit-PatchSet: 2 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 07 Oct 2021 17:11:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10945: Fix S3 scratch path behavior
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17901 ) Change subject: IMPALA-10945: Fix S3 scratch path behavior .. IMPALA-10945: Fix S3 scratch path behavior IMPALA-10429 "Support Spill to HDFS" introduces a new behavior to S3 scratch path. It added a path verification for the S3 path, however, the HdfsFsCache::GetNameNodeFromPath in the verification forces the input to have at least a directory after the authority, like "s3a://authority/dir", otherwise it will return an error and lead to a failure on the TmpFileMgr initialization. Therefore, it changes the previous behavior which was able to support "s3a://authority", and may affect current users. This patch resumes the behavior of the s3 scratch path to allow "s3a://authority". The solution is to pass the path of the user's input combined with a scratch suffix "/impala-scratch" to the verification function, therefore, at least one directory is contained in the path. Tests: Ran core tests. Added logic to run two types of path format in TmpFileMgrTest: "s3a://authority" and "s3a://authority/dir". Change-Id: I028f375b9f535f8641261cc02f921497e076aa9b Reviewed-on: http://gerrit.cloudera.org:8080/17901 Reviewed-by: Abhishek Rawat Tested-by: Impala Public Jenkins --- M be/src/runtime/tmp-file-mgr-test.cc M be/src/runtime/tmp-file-mgr.cc 2 files changed, 85 insertions(+), 78 deletions(-) Approvals: Abhishek Rawat: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/17901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I028f375b9f535f8641261cc02f921497e076aa9b Gerrit-Change-Number: 17901 Gerrit-PatchSet: 3 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10950: Update expr-benchmark.cc
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/17894 ) Change subject: IMPALA-10950: Update expr-benchmark.cc .. Patch Set 6: (5 comments) http://gerrit.cloudera.org:8080/#/c/17894/3/be/src/benchmarks/expr-benchmark.cc File be/src/benchmarks/expr-benchmark.cc: http://gerrit.cloudera.org:8080/#/c/17894/3/be/src/benchmarks/expr-benchmark.cc@108 PS3, Line 108: FragmentState* fragment_state = state->obj_pool()->Add( > I'll add a query plan sample as a comment. Done http://gerrit.cloudera.org:8080/#/c/17894/4/be/src/benchmarks/expr-benchmark.cc File be/src/benchmarks/expr-benchmark.cc: http://gerrit.cloudera.org:8080/#/c/17894/4/be/src/benchmarks/expr-benchmark.cc@78 PS4, Line 78: ity class > Will do. Done http://gerrit.cloudera.org:8080/#/c/17894/4/be/src/benchmarks/expr-benchmark.cc@710 PS4, Line 710: 219 > Good to know. Thanks! Done http://gerrit.cloudera.org:8080/#/c/17894/5/be/src/benchmarks/expr-benchmark.cc File be/src/benchmarks/expr-benchmark.cc: http://gerrit.cloudera.org:8080/#/c/17894/5/be/src/benchmarks/expr-benchmark.cc@104 PS5, Line 104: pool_.Add(new RuntimeState(query_ctx, &e > Ack, I'll try it in the next patch set. Done http://gerrit.cloudera.org:8080/#/c/17894/5/be/src/benchmarks/expr-benchmark.cc@171 PS5, Line 171: gen) { > Since I remove the test data vector and ReleaseTestData() method from Plann In patch set 6, I register new ExprTestData declaration with pool_. So it should be cleaned up when pool_ is cleaned. -- To view, visit http://gerrit.cloudera.org:8080/17894 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5b17434d85e32a58622bffb64a697b062a8bf43f Gerrit-Change-Number: 17894 Gerrit-PatchSet: 6 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Thu, 07 Oct 2021 15:07:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10950: Update expr-benchmark.cc
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17894 ) Change subject: IMPALA-10950: Update expr-benchmark.cc .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/9573/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17894 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5b17434d85e32a58622bffb64a697b062a8bf43f Gerrit-Change-Number: 17894 Gerrit-PatchSet: 6 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Thu, 07 Oct 2021 15:06:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10950: Update expr-benchmark.cc
Hello Qifan Chen, Gabor Kaszab, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17894 to look at the new patch set (#6). Change subject: IMPALA-10950: Update expr-benchmark.cc .. IMPALA-10950: Update expr-benchmark.cc With the introduction of PlanRootSink by IMPALA-2905, query planner has moved the scalar expression's thrift definition from 'fragments[0].output_sink.output_exprs' to 'fragments[0].plan.nodes[0].union_node.const_expr_lists[0]' for a constant query. This patch adjusts expr-benchmark.cc to generate the ScalarExpr from the right thrift location and also modify the helper class to ensure proper resource cleanup at the end of the benchmark. We explicitly set ENABLE_EXPR_REWRITES=0 to prevent expression rewrite by FoldConstantsRule.java. The benchmark used to run without codegen. This patch modifies the benchmark to run a benchmark suite both with and without codegen. Testing: - Run and verify that expr-benchmark does not crash. Change-Id: I5b17434d85e32a58622bffb64a697b062a8bf43f --- M be/src/benchmarks/expr-benchmark.cc 1 file changed, 631 insertions(+), 312 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/94/17894/6 -- To view, visit http://gerrit.cloudera.org:8080/17894 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5b17434d85e32a58622bffb64a697b062a8bf43f Gerrit-Change-Number: 17894 Gerrit-PatchSet: 6 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] [WIP] IMPALA-9873: Avoid materilization of columns for filtered out rows in Parquet table.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/17860 ) Change subject: [WIP] IMPALA-9873: Avoid materilization of columns for filtered out rows in Parquet table. .. Patch Set 5: (7 comments) Looks good! http://gerrit.cloudera.org:8080/#/c/17860/3/be/src/exec/parquet/hdfs-parquet-scanner.cc File be/src/exec/parquet/hdfs-parquet-scanner.cc: http://gerrit.cloudera.org:8080/#/c/17860/3/be/src/exec/parquet/hdfs-parquet-scanner.cc@2330 PS3, Line 2330: > >> By reading the code, my guess is that each batch covers a >> number of, Okay. Thanks for the clarification on skip length. My guess is that converting to batches may not be beneficial, especially when the T and F are interleaved tightly (the common case). In this case, you may need to recheck the selected rows in the batch. http://gerrit.cloudera.org:8080/#/c/17860/3/be/src/exec/parquet/hdfs-parquet-scanner.cc@2381 PS3, Line 2381: if (micro_batches[0].start > 0) { : if (UNLIKELY(!col_reader->SkipRows(micro_batches[0].start, -1))) { : return Status(Substitute("Couldn't skip rows in file $0.", filename())); : } : } > It is possible to have micro_batches[0].start==0, but in that case we don't Done http://gerrit.cloudera.org:8080/#/c/17860/3/be/src/exec/parquet/hdfs-parquet-scanner.cc@2417 PS3, Line 2417: return Status::OK(); > This behaviour is retained from earlier code. You will find the code in Ass Done http://gerrit.cloudera.org:8080/#/c/17860/3/be/src/exec/parquet/parquet-column-readers.h File be/src/exec/parquet/parquet-column-readers.h: http://gerrit.cloudera.org:8080/#/c/17860/3/be/src/exec/parquet/parquet-column-readers.h@553 PS3, Line 553: > sure. Not sure if this happened due to clang-format. Normally, I just highlight a section of code modified (in vi: shift-v and scroll down with j) and clang format (in vi: control-k) :-). http://gerrit.cloudera.org:8080/#/c/17860/3/be/src/exec/parquet/parquet-column-readers.cc File be/src/exec/parquet/parquet-column-readers.cc: http://gerrit.cloudera.org:8080/#/c/17860/3/be/src/exec/parquet/parquet-column-readers.cc@1151 PS3, Line 1151: Status::OK(); > Not really. It depends on if abort_on_error is set as Query option. LogCorr Good to know! Done. http://gerrit.cloudera.org:8080/#/c/17860/3/be/src/exec/parquet/parquet-column-readers.cc@1484 PS3, Line 1484: num_buffered_values_ == 0 > This signifies end of RowGroup. Same logic is currently being used (check N Done http://gerrit.cloudera.org:8080/#/c/17860/3/be/src/exec/parquet/parquet-column-readers.cc@1538 PS3, Line 1538: } > We don't need to check, we can just return it and client will check it. But Hmm. It seems a false returning status means likely the data in the page is corrupted. Sounds like we should bail out? The return status from SkipTopLevelRows() in the code below is checked. 1218 bool BaseScalarColumnReader::SkipTopLevelRows(int64_t num_rows) { 1219 DCHECK_GE(num_buffered_values_, num_rows); 1220 DCHECK_GT(num_rows, 0); .. .. .. 1272 } 1273 } 1274 return SkipEncodedValuesInPage(num_values_to_skip); 1275 } 385 template 386 bool ScalarColumnReader::SkipEncodedValuesInPage(int64_t num_values) { 388 if (bool_decoder_) { 389 return bool_decoder_->SkipValues(num_values); 390 } 391 if (IsDictionaryEncoding(page_encoding_)) { 392 return dict_decoder_.SkipValues(num_values); 393 } else { 394 DCHECK_EQ(page_encoding_, Encoding::PLAIN); 395 int64_t encoded_len = ParquetPlainEncoder::EncodedLen( 396 data_, data_end_, fixed_len_size_, num_values); 397 if (encoded_len < 0) return false; 398 data_ += encoded_len; 399 } 400 return true; 401 } -- To view, visit http://gerrit.cloudera.org:8080/17860 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I46406c913297d5bbbec3ccae62a83bb214ed2c60 Gerrit-Change-Number: 17860 Gerrit-PatchSet: 5 Gerrit-Owner: Amogh Margoor Gerrit-Reviewer: Amogh Margoor Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 07 Oct 2021 13:46:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10953 Fix impalad crashes due to TmpFileMgr initialization fails
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17906 ) Change subject: IMPALA-10953 Fix impalad crashes due to TmpFileMgr initialization fails .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/9572/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17906 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7852bf4206a603e6e2652ce923207a06ce67 Gerrit-Change-Number: 17906 Gerrit-PatchSet: 2 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 07 Oct 2021 12:41:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10953 Fix impalad crashes due to TmpFileMgr initialization fails
Yida Wu has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/17906 ) Change subject: IMPALA-10953 Fix impalad crashes due to TmpFileMgr initialization fails .. IMPALA-10953 Fix impalad crashes due to TmpFileMgr initialization fails The impalad can start successfully with an uninitialized TmpFileMgr when abort_on_config_error is set to false. However in some cases, this could lead to the use of the uninitialized TmpFileMgr and then the crash. The purpose of keeping a failed TmpFileMgr is to allow the users to use impala even with minor configuration issues because only spilling the data needs the TmpFileMgr and queries might have a small chance to spill the data. Previously we have a DCHECK for the initialization status of the TmpFileMgr, however it doesn't work for the release version, and the way of using DCHECK to handle this situation is not good enough if we need to keep the impala running. Instead of putting verifications to every function of the TmpFileMgr, this patch fixes the bug by verifying the initialization status when creating the TmpFileGroup. It returns an error if TmpFileMgr fails to initialize. When the QueryState instance fails to create the TmpFileGroup, it stores a nullptr to the local temporary file group and fails the query only when it needs to spill the data. Moreover, TmpFileGroup can be only created by the TmpFileMgr after this change. Tests: Ran core tests and exhaustive e2e tests. Added a e2e test: test_scratch_dirs_spill_fails_due_to_init_failure. Added a unit test: BufferPoolTest::FailInitTmpFileMgr. Change-Id: I7852bf4206a603e6e2652ce923207a06ce67 --- M be/src/runtime/bufferpool/buffer-pool-test.cc M be/src/runtime/io/disk-io-mgr-test.cc M be/src/runtime/query-state.cc M be/src/runtime/test-env.cc M be/src/runtime/test-env.h M be/src/runtime/tmp-file-mgr-test.cc M be/src/runtime/tmp-file-mgr.cc M be/src/runtime/tmp-file-mgr.h M tests/custom_cluster/test_scratch_disk.py 9 files changed, 336 insertions(+), 217 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/17906/2 -- To view, visit http://gerrit.cloudera.org:8080/17906 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7852bf4206a603e6e2652ce923207a06ce67 Gerrit-Change-Number: 17906 Gerrit-PatchSet: 2 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10945: Fix S3 scratch path behavior
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17901 ) Change subject: IMPALA-10945: Fix S3 scratch path behavior .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7523/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I028f375b9f535f8641261cc02f921497e076aa9b Gerrit-Change-Number: 17901 Gerrit-PatchSet: 2 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 07 Oct 2021 11:00:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10945: Fix S3 scratch path behavior
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/17901 ) Change subject: IMPALA-10945: Fix S3 scratch path behavior .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I028f375b9f535f8641261cc02f921497e076aa9b Gerrit-Change-Number: 17901 Gerrit-PatchSet: 2 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 07 Oct 2021 10:56:45 + Gerrit-HasComments: No
[Impala-ASF-CR] datasketches: improved merge and memory allocation - avoid overhead of constructing union and getting result from it every time - call destructors of sketch and union objects
Fucun Chu has posted comments on this change. ( http://gerrit.cloudera.org:8080/17869 ) Change subject: datasketches: improved merge and memory allocation - avoid overhead of constructing union and getting result from it every time - call destructors of sketch and union objects .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/17869/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17869/1//COMMIT_MSG@7 PS1, Line 7: datasketches: improved merge and memory allocation First, you need to create a jira ticket for this patch at https://issues.apache.org/jira/browse/IMPALA Second, please write a good, clear commit message, with a short, descriptive title and a message that is exactly long enough to explain what the problem was, and how it was fixed. Each should have 72 or fewer characters if possible. The first line should have an empty line after it, and the first line should begin with the ticket(s) addressed, followed by a colon and a space, eg: "IMPALA-1234: ". -- To view, visit http://gerrit.cloudera.org:8080/17869 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8dd0e6736f4266f74f5f265f58d40a4e4707287f Gerrit-Change-Number: 17869 Gerrit-PatchSet: 1 Gerrit-Owner: Alexander Saydakov Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 07 Oct 2021 10:19:25 + Gerrit-HasComments: Yes