[Impala-ASF-CR] IMPALA-8915: reapply IMPALA-8901 fix
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14176 ) Change subject: IMPALA-8915: reapply IMPALA-8901 fix .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14176 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42ea1ec4c386f2eb45f370cd35a104b49786a48c Gerrit-Change-Number: 14176 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Thu, 05 Sep 2019 04:22:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8915: reapply IMPALA-8901 fix
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14176 ) Change subject: IMPALA-8915: reapply IMPALA-8901 fix .. IMPALA-8915: reapply IMPALA-8901 fix This reapplies the below fix and resolves conflicts correctly. IMPALA-8901: Fix # links on /catalog page. After IMPALA-7935, the database links on the top of /catalog page that use # links to jump to the part of the page corresponding to the particular database is broken because the id is ommitted. This is fixed by moving id to a tag which doesn't change if local catalog is turned on. Testing: Manually tested the links are working in both local catalog mode and V1 mode. Change-Id: I42ea1ec4c386f2eb45f370cd35a104b49786a48c Reviewed-on: http://gerrit.cloudera.org:8080/14176 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M www/catalog.tmpl 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14176 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I42ea1ec4c386f2eb45f370cd35a104b49786a48c Gerrit-Change-Number: 14176 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-8797: Support database and table blacklist
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14134 ) Change subject: IMPALA-8797: Support database and table blacklist .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4466/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I02dbb07f8e08793b57b2a88d09b30fd32cff26dc Gerrit-Change-Number: 14134 Gerrit-PatchSet: 12 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 05 Sep 2019 03:20:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8797: Support database and table blacklist
Hello Bharath Vissapragada, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14134 to look at the new patch set (#12). Change subject: IMPALA-8797: Support database and table blacklist .. IMPALA-8797: Support database and table blacklist Add catalogd and coordinator startup options for database and table blacklist. Blacklisted dbs/tables will be skipped in loading. Users won't see them when getting database/table list, e.g. in SHOW DATABASES/TABLES. Dropping/creating blacklisted databases/tables/views are not allowed too. Implementation: Catalogd and coordinators parses the --blacklisted_dbs and --blacklisted_tables options in startup. Blacklist checks are added in catalogd when loading the metadata and in coordinators when analysing DDL requests for create/drop. Catalogd will also check blacklist when executing DDL requests from coordinators in case their blacklists are inconsistent. Motivation: By default, it's used to blacklist "sys" and "information_schema" databases from Hive. Admin can use them to specify any databases/tables that are not suitable for Impala to query. Tests: - Add java unit tests for blacklist related functions - Add a custom cluster test: test_blacklisted_dbs_and_tables.py - Ran CORE tests Change-Id: I02dbb07f8e08793b57b2a88d09b30fd32cff26dc --- M be/src/common/global-flags.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/analysis/AlterTableOrViewRenameStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateDbStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateViewStmt.java M fe/src/main/java/org/apache/impala/analysis/TableName.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java A fe/src/test/java/org/apache/impala/util/CatalogBlacklistUtilsTest.java A tests/custom_cluster/test_blacklisted_dbs_and_tables.py 14 files changed, 542 insertions(+), 19 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/14134/12 -- To view, visit http://gerrit.cloudera.org:8080/14134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I02dbb07f8e08793b57b2a88d09b30fd32cff26dc Gerrit-Change-Number: 14134 Gerrit-PatchSet: 12 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-8797: Support database and table blacklist
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/14134 ) Change subject: IMPALA-8797: Support database and table blacklist .. Patch Set 10: (2 comments) http://gerrit.cloudera.org:8080/#/c/14134/10/fe/src/main/java/org/apache/impala/analysis/TableName.java File fe/src/main/java/org/apache/impala/analysis/TableName.java: http://gerrit.cloudera.org:8080/#/c/14134/10/fe/src/main/java/org/apache/impala/analysis/TableName.java@81 PS10, Line 81: Preconditions.checkNotNull(db_); > Like I mentioned in the other comment, Preconditions.checkState(isFullyQual Done http://gerrit.cloudera.org:8080/#/c/14134/10/fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java File fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java: http://gerrit.cloudera.org:8080/#/c/14134/10/fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java@104 PS10, Line 104: if (BLACKLISTED_TABLES.contains(table)) { > Ah, I see what you are saying. I think thats a deep refactor. Suggested a n Sure. -- To view, visit http://gerrit.cloudera.org:8080/14134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I02dbb07f8e08793b57b2a88d09b30fd32cff26dc Gerrit-Change-Number: 14134 Gerrit-PatchSet: 10 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 05 Sep 2019 02:35:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8797: Support database and table blacklist
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14134 ) Change subject: IMPALA-8797: Support database and table blacklist .. Patch Set 10: Code-Review+2 (2 comments) http://gerrit.cloudera.org:8080/#/c/14134/10/fe/src/main/java/org/apache/impala/analysis/TableName.java File fe/src/main/java/org/apache/impala/analysis/TableName.java: http://gerrit.cloudera.org:8080/#/c/14134/10/fe/src/main/java/org/apache/impala/analysis/TableName.java@81 PS10, Line 81: Preconditions.checkNotNull(db_); Like I mentioned in the other comment, Preconditions.checkState(isFullyQualified())? http://gerrit.cloudera.org:8080/#/c/14134/10/fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java File fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java: http://gerrit.cloudera.org:8080/#/c/14134/10/fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java@104 PS10, Line 104: if (BLACKLISTED_TABLES.contains(table)) { > But there are no "analyzer" object here... Even if we move this "verifyTabl Ah, I see what you are saying. I think thats a deep refactor. Suggested a nit in the Preconditions check. -- To view, visit http://gerrit.cloudera.org:8080/14134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I02dbb07f8e08793b57b2a88d09b30fd32cff26dc Gerrit-Change-Number: 14134 Gerrit-PatchSet: 10 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 05 Sep 2019 02:29:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8907: TestResultSpooling.test slow query is flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14170 ) Change subject: IMPALA-8907: TestResultSpooling.test_slow_query is flaky .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14170 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idcb4a6b38f85a9497f80e2674e1c1fa512be5940 Gerrit-Change-Number: 14170 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Thu, 05 Sep 2019 02:27:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8907: TestResultSpooling.test slow query is flaky
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14170 ) Change subject: IMPALA-8907: TestResultSpooling.test_slow_query is flaky .. IMPALA-8907: TestResultSpooling.test_slow_query is flaky De-flake test_result_spooling.py::TestResultsSpooling::test_slow_query by increasing the delay in RowBatch production. This patch makes two fixes to ensure that RowBatchGetWaitTime is a non-zero value: * Add the DELAY DEBUG_ACTION to the SCAN_NODE rather than the EXCHANGE_NODE. Since the EXCHANGE_NODE only processes a few rows, adding the delay to the SCAN_NODE decreases the rate at which results are produced. * Wait for all rows to be fetched before checking if RowBatchGetWaitTime is in the profile. This fixes a possible race condition where the fetch thread was not able to issue the fetch RPC before the test checked if RowBatchGetWaitTime was in the runtime profile. Testing: * Looped test_slow_query for several hours with 0 failures Change-Id: Idcb4a6b38f85a9497f80e2674e1c1fa512be5940 Reviewed-on: http://gerrit.cloudera.org:8080/14170 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/query_test/test_result_spooling.py 1 file changed, 4 insertions(+), 3 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14170 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Idcb4a6b38f85a9497f80e2674e1c1fa512be5940 Gerrit-Change-Number: 14170 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho
[Impala-ASF-CR] IMPALA-8797: Support database and table blacklist
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/14134 ) Change subject: IMPALA-8797: Support database and table blacklist .. Patch Set 10: (1 comment) http://gerrit.cloudera.org:8080/#/c/14134/10/fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java File fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java: http://gerrit.cloudera.org:8080/#/c/14134/10/fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java@104 PS10, Line 104: if (BLACKLISTED_TABLES.contains(table)) { > Can we just do TableName temp = analyzer.getFqTableName(table); But there are no "analyzer" object here... Even if we move this "verifyTableName" function into Analyzer, there are no analyzer objects in TableName.analyze()... We still need to call "getFqTableName" in somewhere with the analyzer object... -- To view, visit http://gerrit.cloudera.org:8080/14134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I02dbb07f8e08793b57b2a88d09b30fd32cff26dc Gerrit-Change-Number: 14134 Gerrit-PatchSet: 10 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 05 Sep 2019 00:52:04 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8797: Support database and table blacklist
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14134 ) Change subject: IMPALA-8797: Support database and table blacklist .. Patch Set 10: (1 comment) http://gerrit.cloudera.org:8080/#/c/14134/10/fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java File fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java: http://gerrit.cloudera.org:8080/#/c/14134/10/fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java@104 PS10, Line 104: if (BLACKLISTED_TABLES.contains(table)) { Can we just do TableName temp = analyzer.getFqTableName(table); if (BLACKLISTED_TABLES.contains(temp)) { . That saves all the callers from doing analyzer.getFqTableName(tableName_).analyze(); -- To view, visit http://gerrit.cloudera.org:8080/14134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I02dbb07f8e08793b57b2a88d09b30fd32cff26dc Gerrit-Change-Number: 14134 Gerrit-PatchSet: 10 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 05 Sep 2019 00:46:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8797: Support database and table blacklist
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14134 ) Change subject: IMPALA-8797: Support database and table blacklist .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4465/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I02dbb07f8e08793b57b2a88d09b30fd32cff26dc Gerrit-Change-Number: 14134 Gerrit-PatchSet: 10 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 05 Sep 2019 00:43:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14157 ) Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4464/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 5 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 05 Sep 2019 00:29:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8572: Log query events before Unregister() call.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14143 ) Change subject: IMPALA-8572: Log query events before Unregister() call. .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/14143/4//COMMIT_MSG Commit Message: PS4: > I had a couple of high level questions. 1) I've spent some time looking at it, especially reading git blame and older jiras, but I didn't totally understand why a fetch was necessary. I didn't see an obvious downside either. So I ended up preserving the existing functionality. 2) Isn't that a deadlock? FetchInternal() blocks on Wait(), Wait() blocks on LogQueryEvents() which blocks on first fetch. Am I missing something? -- To view, visit http://gerrit.cloudera.org:8080/14143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e Gerrit-Change-Number: 14143 Gerrit-PatchSet: 4 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Thu, 05 Sep 2019 00:18:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8915: reapply IMPALA-8901 fix
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14176 ) Change subject: IMPALA-8915: reapply IMPALA-8901 fix .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14176 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42ea1ec4c386f2eb45f370cd35a104b49786a48c Gerrit-Change-Number: 14176 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Thu, 05 Sep 2019 00:14:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8915: reapply IMPALA-8901 fix
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14176 ) Change subject: IMPALA-8915: reapply IMPALA-8901 fix .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4881/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14176 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42ea1ec4c386f2eb45f370cd35a104b49786a48c Gerrit-Change-Number: 14176 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Thu, 05 Sep 2019 00:14:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8797: Support database and table blacklist
Hello Bharath Vissapragada, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14134 to look at the new patch set (#10). Change subject: IMPALA-8797: Support database and table blacklist .. IMPALA-8797: Support database and table blacklist Add catalogd and coordinator startup options for database and table blacklist. Blacklisted dbs/tables will be skipped in loading. Users won't see them when getting database/table list, e.g. in SHOW DATABASES/TABLES. Dropping/creating blacklisted databases/tables/views are not allowed too. Implementation: Catalogd and coordinators parses the --blacklisted_dbs and --blacklisted_tables options in startup. Blacklist checks are added in catalogd when loading the metadata and in coordinators when analysing DDL requests for create/drop. Catalogd will also check blacklist when executing DDL requests from coordinators in case their blacklists are inconsistent. Motivation: By default, it's used to blacklist "sys" and "information_schema" databases from Hive. Admin can use them to specify any databases/tables that are not suitable for Impala to query. Tests: - Add java unit tests for blacklist related functions - Add a custom cluster test: test_blacklisted_dbs_and_tables.py - Ran CORE tests Change-Id: I02dbb07f8e08793b57b2a88d09b30fd32cff26dc --- M be/src/common/global-flags.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/analysis/AlterTableOrViewRenameStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateDbStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateViewStmt.java M fe/src/main/java/org/apache/impala/analysis/TableName.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java A fe/src/test/java/org/apache/impala/util/CatalogBlacklistUtilsTest.java A tests/custom_cluster/test_blacklisted_dbs_and_tables.py 14 files changed, 542 insertions(+), 18 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/14134/10 -- To view, visit http://gerrit.cloudera.org:8080/14134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I02dbb07f8e08793b57b2a88d09b30fd32cff26dc Gerrit-Change-Number: 14134 Gerrit-PatchSet: 10 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-8797: Support database and table blacklist
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/14134 ) Change subject: IMPALA-8797: Support database and table blacklist .. Patch Set 9: (1 comment) > Patch Set 9: > > (3 comments) > > lgtm, can +2 it once the TableName#analyze() is refactored to include the > blacklist checks. Sure. Just refactored in it. http://gerrit.cloudera.org:8080/#/c/14134/8/fe/src/main/java/org/apache/impala/analysis/TableName.java File fe/src/main/java/org/apache/impala/analysis/TableName.java: http://gerrit.cloudera.org:8080/#/c/14134/8/fe/src/main/java/org/apache/impala/analysis/TableName.java@86 PS8, Line 86: throw new AnalysisException("Invalid table/view name: " + tbl_); > I think we can add the getFqTableName() logic into BlackListUtils.verifyTab OK, agree with that. Avoiding possibility for bugs is more important. -- To view, visit http://gerrit.cloudera.org:8080/14134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I02dbb07f8e08793b57b2a88d09b30fd32cff26dc Gerrit-Change-Number: 14134 Gerrit-PatchSet: 9 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 05 Sep 2019 00:01:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14157 ) Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. Patch Set 5: (3 comments) http://gerrit.cloudera.org:8080/#/c/14157/4/be/src/exec/buffered-plan-root-sink.cc File be/src/exec/buffered-plan-root-sink.cc: http://gerrit.cloudera.org:8080/#/c/14157/4/be/src/exec/buffered-plan-root-sink.cc@159 PS4, Line 159: > Don't need _ suffix on local variable. Done http://gerrit.cloudera.org:8080/#/c/14157/4/be/src/exec/buffered-plan-root-sink.cc@172 PS4, Line 172: // wait_timeout_timer ensures the client only ever waits up to > Why not just start the timer and leave it running? I think the elapsed time Yeah makes sense. Moved the Start before the lock is acquired, and it is never stopped. This way it tracks the total time required to fetch, including waiting for rows to become available and waiting for rows to materialize. I have some additional tests in mind for this, but they require IMPALA-8825 first. http://gerrit.cloudera.org:8080/#/c/14157/4/be/src/exec/buffered-plan-root-sink.cc@174 PS4, Line 174: uint64_t wait_duration = max((uint64_t) 1, > nit: could be a one liner: Done -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 5 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 04 Sep 2019 23:54:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Hello Michael Ho, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14157 to look at the new patch set (#5). Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. IMPALA-7312: Non-blocking mode for Fetch() RPC Adds the query option FETCH_ROWS_TIMEOUT_MS to control the client timeout when fetching rows. Set to 10 seconds by default to avoid unnecessary fetch requests. Timeout applies when result spooling is enabled or disabled. When result spooling is disabled, the timeout controls how long the client thread will wait for a single RowBatch to be produced by the coordinator fragment. When result spooling is enabled, a client can fetch multiple RowBatches at a time, so the timeout controls the total time spent waiting for RowBatches to be produced. Testing: * Added new tests to test_fetch.py * Running core tests Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d --- M be/src/exec/blocking-plan-root-sink.cc M be/src/exec/buffered-plan-root-sink.cc M be/src/exec/plan-root-sink.cc M be/src/exec/plan-root-sink.h M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M tests/hs2/test_fetch.py 9 files changed, 161 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/14157/5 -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 5 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8858: Add observability of the query load on executor groups
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14103 ) Change subject: IMPALA-8858: Add observability of the query load on executor groups .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4463/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I58cde8699c33af8b87273437e9d8bf6371a34539 Gerrit-Change-Number: 14103 Gerrit-PatchSet: 8 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Wed, 04 Sep 2019 22:33:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8858: Add observability of the query load on executor groups
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14103 ) Change subject: IMPALA-8858: Add observability of the query load on executor groups .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4462/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I58cde8699c33af8b87273437e9d8bf6371a34539 Gerrit-Change-Number: 14103 Gerrit-PatchSet: 7 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Wed, 04 Sep 2019 22:26:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8907: TestResultSpooling.test slow query is flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14170 ) Change subject: IMPALA-8907: TestResultSpooling.test_slow_query is flaky .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4880/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14170 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idcb4a6b38f85a9497f80e2674e1c1fa512be5940 Gerrit-Change-Number: 14170 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Wed, 04 Sep 2019 22:21:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8907: TestResultSpooling.test slow query is flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14170 ) Change subject: IMPALA-8907: TestResultSpooling.test_slow_query is flaky .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14170 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idcb4a6b38f85a9497f80e2674e1c1fa512be5940 Gerrit-Change-Number: 14170 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Wed, 04 Sep 2019 22:21:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8858: Add observability of the query load on executor groups
Hello Andrew Sherman, Lars Volker, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14103 to look at the new patch set (#8). Change subject: IMPALA-8858: Add observability of the query load on executor groups .. IMPALA-8858: Add observability of the query load on executor groups With this patch, all executor groups with at least one executor will have a metric added that display the number of queries (admitted by the local coordinator) running on them. The metric is removed only when the group has no executors or no longer exists. It gets updated when either the cluster membership changes or a query gets admitted or released by the admission controller. Also adds the ability to delete metrics from a metric group after registration. Testing: Added a custom cluster test and a BE metric test. Change-Id: I58cde8699c33af8b87273437e9d8bf6371a34539 --- M be/src/runtime/exec-env.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/cluster-membership-mgr.cc M be/src/scheduling/cluster-membership-mgr.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/util/metrics-test.cc M be/src/util/metrics.cc M be/src/util/metrics.h M common/thrift/metrics.json M tests/custom_cluster/test_executor_groups.py 12 files changed, 265 insertions(+), 135 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/14103/8 -- To view, visit http://gerrit.cloudera.org:8080/14103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I58cde8699c33af8b87273437e9d8bf6371a34539 Gerrit-Change-Number: 14103 Gerrit-PatchSet: 8 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-8858: Add observability of the query load on executor groups
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/14103 ) Change subject: IMPALA-8858: Add observability of the query load on executor groups .. Patch Set 7: (11 comments) http://gerrit.cloudera.org:8080/#/c/14103/6/be/src/scheduling/admission-controller.h File be/src/scheduling/admission-controller.h: http://gerrit.cloudera.org:8080/#/c/14103/6/be/src/scheduling/admission-controller.h@943 PS6, Line 943: /// Updates the list of executor groups for which we maintain the query load metrics. > "groups for which we maintain..." Done http://gerrit.cloudera.org:8080/#/c/14103/6/be/src/scheduling/admission-controller.h@944 PS6, Line 944: from the metric gr > Should we consider keeping groups with >0 executors around, even if they're With how the current cancellation logic works, the query is still counted as running against a backend by the impala-server till it gets un-registered, which means that it would be cancelled regardless of whether it has completed its fragments on a particular backend. However, I still agree with keeping non-zero groups around since they can still have an intermediate non zero value for query load which can prove useful. http://gerrit.cloudera.org:8080/#/c/14103/6/be/src/scheduling/admission-controller.cc File be/src/scheduling/admission-controller.cc: http://gerrit.cloudera.org:8080/#/c/14103/6/be/src/scheduling/admission-controller.cc@320 PS6, Line 320: dequeue_thread_->Join(); > Can the argument be a const&? changed this mechanism, switched to sending a snapshotPtr instead. http://gerrit.cloudera.org:8080/#/c/14103/6/be/src/scheduling/admission-controller.cc@1852 PS6, Line 1852: for (const auto& group : snapshot->executor_groups) { > I feel that this function's body could benefit from a comment explaining wh Done http://gerrit.cloudera.org:8080/#/c/14103/6/be/src/scheduling/cluster-membership-mgr.h File be/src/scheduling/cluster-membership-mgr.h: http://gerrit.cloudera.org:8080/#/c/14103/6/be/src/scheduling/cluster-membership-mgr.h@135 PS6, Line 135: /// events. They may be called at any time before or after calling Init() and are > The comment should include what gets passed into the callback, or alternati changed the update mechanism http://gerrit.cloudera.org:8080/#/c/14103/3/be/src/scheduling/cluster-membership-mgr.cc File be/src/scheduling/cluster-membership-mgr.cc: http://gerrit.cloudera.org:8080/#/c/14103/3/be/src/scheduling/cluster-membership-mgr.cc@342 PS3, Line 342: base_snapshot = current_membership_.get(); > I'm still in favor of this. Done. Note: this adds a little extra work to the critical path since now we cant optimize work for specific calls: - NotifyLocalServerForDeletedBackend called every time and not just when update_local_server is true - Admission controller needs to separately create a set a groups instead of leveraging the set previously created during metric updates. These are pretty minor + cluster changes would be way less frequent in practice so shouldn't matter much. http://gerrit.cloudera.org:8080/#/c/14103/3/be/src/scheduling/cluster-membership-mgr.cc@346 PS3, Line 346: // copying the Snapshot if it isn't. > ? Done http://gerrit.cloudera.org:8080/#/c/14103/3/be/src/scheduling/cluster-membership-mgr.cc@551 PS3, Line 551: : : > What was the reason for this change? just seemed consistent with other update calls to send the whole snapshotPtr http://gerrit.cloudera.org:8080/#/c/14103/6/be/src/util/metrics.h File be/src/util/metrics.h: http://gerrit.cloudera.org:8080/#/c/14103/6/be/src/util/metrics.h@455 PS6, Line 455: typedef std::unordered_map> MetricMap; > Is there a reason this map needs to be ordered? If so, can you add it to th oops, didnt notice that a map was being used instead of an unordered_map. Not sure what the initial intention was but I dont see any reason why it would require an ordered map. changing it to unordered_map. Same goes for ChildGroupMap below. http://gerrit.cloudera.org:8080/#/c/14103/6/common/thrift/metrics.json File common/thrift/metrics.json: http://gerrit.cloudera.org:8080/#/c/14103/6/common/thrift/metrics.json@2495 PS6, Line 2495: "description": "Total number of queries running on executor group: $0", > I'd suggest moving the $0 to the back of the string to make it slightly eas makes sense to me. Moving it to the end. http://gerrit.cloudera.org:8080/#/c/14103/6/common/thrift/metrics.json@2502 PS6, Line 2502: ad.$0" do you think num-running-queries is more appropriate here? -- To view, visit http://gerrit.cloudera.org:8080/14103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I58cde8699c33af8b87273437e9d8bf6371a34539 Gerrit-Change-Number: 14103 Gerrit-PatchSet: 7 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew
[Impala-ASF-CR] IMPALA-8858: Add observability of the query load on executor groups
Hello Andrew Sherman, Lars Volker, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14103 to look at the new patch set (#7). Change subject: IMPALA-8858: Add observability of the query load on executor groups .. IMPALA-8858: Add observability of the query load on executor groups With this patch, all executor groups with at least one executor will have a metric added that display the number of queries (admitted by the local coordinator) running on them. The metric is removed only when the group has no executors or no longer exists. It gets updated when either the cluster membership changes or a query gets admitted or released by the admission controller. Also adds the ability to delete metrics from a metric group after registration. Testing: Added a custom cluster test and a BE metric test. Change-Id: I58cde8699c33af8b87273437e9d8bf6371a34539 --- M be/src/runtime/exec-env.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/cluster-membership-mgr.cc M be/src/scheduling/cluster-membership-mgr.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/util/metrics-test.cc M be/src/util/metrics.cc M be/src/util/metrics.h M common/thrift/metrics.json M tests/custom_cluster/test_executor_groups.py 12 files changed, 264 insertions(+), 135 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/14103/7 -- To view, visit http://gerrit.cloudera.org:8080/14103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I58cde8699c33af8b87273437e9d8bf6371a34539 Gerrit-Change-Number: 14103 Gerrit-PatchSet: 7 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-7984: Port runtime filter from Thrift RPC to KRPC
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/13882 ) Change subject: IMPALA-7984: Port runtime filter from Thrift RPC to KRPC .. Patch Set 17: (15 comments) http://gerrit.cloudera.org:8080/#/c/13882/17/be/src/runtime/coordinator.h File be/src/runtime/coordinator.h: http://gerrit.cloudera.org:8080/#/c/13882/17/be/src/runtime/coordinator.h@48 PS17, Line 48: using kudu::rpc::RpcContext; don't use 'using' in header files http://gerrit.cloudera.org:8080/#/c/13882/17/be/src/runtime/coordinator.cc File be/src/runtime/coordinator.cc: http://gerrit.cloudera.org:8080/#/c/13882/17/be/src/runtime/coordinator.cc@986 PS17, Line 986: { : return; : } You can leave the formatting as-is http://gerrit.cloudera.org:8080/#/c/13882/17/be/src/runtime/coordinator.cc@1041 PS17, Line 1041: void Coordinator::SetupSidecarForBloomFilter(PublishFilterParamsPB* rpc_params, Its unfortunate that this logic is duplicated between here and BloomFilter::ToProtobuf, eg. because duplicating code makes it more likely there are bugs (for example I think that you have a bug here in the case where AddOutboundSidecar fails, since you don't disable the bloom filter, even though that bug is already fixed in BloomFilter::ToProtobuf) There are a couple of possible solutions to this: 1. Move this to be a static function in BloomFilter, eg. BloomFilter::AddSidecar(), and call it from BloomFilter::ToProtobuf() 2. Instead of having Coordinator::FilterState store a BloomFilterPB and 'string directory' just actually reconstruct a BloomFilter object, i.e. around line 1110 below, by calling BloomFilter::Init() with sidecar_slice.data(). Then we would be able to just call the existing BloomFilter::ToProtobuf() 3. some other idea I didn't think of Option 2 seems cleaner overall to me, since it allows us to better hide the details of how bloom filters work in the BloomFilter class instead of leaking things like the existence of the 'directory' string into FilterState. However, its slightly less efficient (cause we add a transition from BloomFilterPB -> Bloom Filter object and back, with the required setup, eg. in BloomFilter::Init()), though not a ton less efficient since I think it can be done without adding any copies of the directory, and it probably also is more work for you in terms of code changes (eg. BloomFilter::Or would have to be slightly rewritten to take BloomFilter objects instead). And, option 1 is consistent with the way things already work. So, I'll leave it up to you to decide. http://gerrit.cloudera.org:8080/#/c/13882/17/be/src/runtime/coordinator.cc@1089 PS17, Line 1089: !params->bloom_filter().always_true() && : !params->bloom_filter().always_false()) Might be cleaner to check has_directory_sidecar_idx() here, which should be equivalent http://gerrit.cloudera.org:8080/#/c/13882/17/be/src/runtime/coordinator.cc@1096 PS17, Line 1096: return; You need to handle this error, i.e. call Disable() http://gerrit.cloudera.org:8080/#/c/13882/17/be/src/runtime/runtime-filter-bank.h File be/src/runtime/runtime-filter-bank.h: http://gerrit.cloudera.org:8080/#/c/13882/17/be/src/runtime/runtime-filter-bank.h@52 PS17, Line 52: class UniqueIdPB; I don't think this is used anywhere http://gerrit.cloudera.org:8080/#/c/13882/17/be/src/runtime/runtime-filter-bank.h@142 PS17, Line 142: void SendFilterToCoordinatorCompleteCb(const kudu::rpc::RpcController* rpc_controller); I think this can be private. Lets also rename it UpdateFilterCompleteCb, to keep it consistent with the name of the rpc. http://gerrit.cloudera.org:8080/#/c/13882/17/be/src/runtime/runtime-filter-bank.h@154 PS17, Line 154: num_krpcs_ num_inflight_rpcs_ http://gerrit.cloudera.org:8080/#/c/13882/17/be/src/runtime/runtime-filter-bank.cc File be/src/runtime/runtime-filter-bank.cc: http://gerrit.cloudera.org:8080/#/c/13882/17/be/src/runtime/runtime-filter-bank.cc@133 PS17, Line 133: DCHECK_NE(state_->query_options().runtime_filter_mode, TRuntimeFilterMode::OFF) While thinking about the locking protocol in Close(), I noticed that we don't check 'closed_' here like we do at the start of PublishGlobalFilter() I spent a little time looking at the code, and I think that its correct as is, because RuntimeFilterBank::Close() only gets called from RuntimeState::ReleaseResources(), which is only called from FragmentInstanceState::Close(), which can't be called until after Open() is run on exec_tree_ and UpdateFilterFromLocal() is only called during Open() (via PartitionedHashJoinNode::Open() -> BlockingJoinNode::ProcessBuildInputAndOpenProbe(), which calls on a thread and then blocks until completion for BlockingJoinNode::ProcessBuildInputAsync() -> BlockingJoinNode::SendBuildInputToSink() -> PhjBuilder::FlushFinal()) It might be nice if you could confirm my
[Impala-ASF-CR] IMPALA-8907: TestResultSpooling.test slow query is flaky
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/14170 ) Change subject: IMPALA-8907: TestResultSpooling.test_slow_query is flaky .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14170 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idcb4a6b38f85a9497f80e2674e1c1fa512be5940 Gerrit-Change-Number: 14170 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Wed, 04 Sep 2019 19:23:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14157 ) Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. Patch Set 4: (3 comments) http://gerrit.cloudera.org:8080/#/c/14157/4/be/src/exec/buffered-plan-root-sink.cc File be/src/exec/buffered-plan-root-sink.cc: http://gerrit.cloudera.org:8080/#/c/14157/4/be/src/exec/buffered-plan-root-sink.cc@159 PS4, Line 159: wait_timeout_timer_ Don't need _ suffix on local variable. http://gerrit.cloudera.org:8080/#/c/14157/4/be/src/exec/buffered-plan-root-sink.cc@172 PS4, Line 172: wait_timeout_timer_.Start(); Why not just start the timer and leave it running? I think the elapsed time is likely more important to the client than the time spent blocked. http://gerrit.cloudera.org:8080/#/c/14157/4/be/src/exec/buffered-plan-root-sink.cc@174 PS4, Line 174: if (!rows_available_.WaitFor(l, wait_duration)) { nit: could be a one liner: timed_out = !rows_available_.WaitFor(...); -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 4 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 04 Sep 2019 19:08:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8797: Support database and table blacklist
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14134 ) Change subject: IMPALA-8797: Support database and table blacklist .. Patch Set 9: (3 comments) lgtm, can +2 it once the TableName#analyze() is refactored to include the blacklist checks. http://gerrit.cloudera.org:8080/#/c/14134/8/fe/src/main/java/org/apache/impala/analysis/TableName.java File fe/src/main/java/org/apache/impala/analysis/TableName.java: http://gerrit.cloudera.org:8080/#/c/14134/8/fe/src/main/java/org/apache/impala/analysis/TableName.java@86 PS8, Line 86: throw new AnalysisException("Invalid table/view name: " + tbl_); > I think they should always be called together. But BlacklistUtils.verifyTab I think we can add the getFqTableName() logic into BlackListUtils.verifyTableName(). My concern with separating these too is that if someone adds a new Statement in the future that involves a tableName and they forget to to call BlackListUtils.verify(), we might run into issues. Instead folding that logic into TableName#analyze() makes life easier. http://gerrit.cloudera.org:8080/#/c/14134/8/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/14134/8/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1519 PS8, Line 1519: addSummary(resp, "Can't drop blacklisted database: " + dbName); > This is tested in tests/custom_cluster/test_blacklisted_dbs_and_tables.py: ack, sorry missed it. http://gerrit.cloudera.org:8080/#/c/14134/8/fe/src/main/java/org/apache/impala/util/BlacklistUtils.java File fe/src/main/java/org/apache/impala/util/BlacklistUtils.java: http://gerrit.cloudera.org:8080/#/c/14134/8/fe/src/main/java/org/apache/impala/util/BlacklistUtils.java@32 PS8, Line 32: > Sure. What about CatalogBlacklistUtils? sgtm -- To view, visit http://gerrit.cloudera.org:8080/14134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I02dbb07f8e08793b57b2a88d09b30fd32cff26dc Gerrit-Change-Number: 14134 Gerrit-PatchSet: 9 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 04 Sep 2019 17:47:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8915: reapply IMPALA-8901 fix
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/14176 ) Change subject: IMPALA-8915: reapply IMPALA-8901 fix .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14176 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42ea1ec4c386f2eb45f370cd35a104b49786a48c Gerrit-Change-Number: 14176 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 04 Sep 2019 17:02:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8797: Support database and table blacklist
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14134 ) Change subject: IMPALA-8797: Support database and table blacklist .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4461/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I02dbb07f8e08793b57b2a88d09b30fd32cff26dc Gerrit-Change-Number: 14134 Gerrit-PatchSet: 9 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 04 Sep 2019 14:36:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8797: Support database and table blacklist
Hello Bharath Vissapragada, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14134 to look at the new patch set (#9). Change subject: IMPALA-8797: Support database and table blacklist .. IMPALA-8797: Support database and table blacklist Add catalogd and coordinator startup options for database and table blacklist. Blacklisted dbs/tables will be skipped in loading. Users won't see them when getting database/table list, e.g. in SHOW DATABASES/TABLES. Dropping/creating blacklisted databases/tables/views are not allowed too. Implementation: Catalogd and coordinators parses the --blacklisted_dbs and --blacklisted_tables options in startup. Blacklist checks are added in catalogd when loading the metadata and in coordinators when analysing DDL requests for create/drop. Catalogd will also check blacklist when executing DDL requests from coordinators in case their blacklists are inconsistent. Motivation: By default, it's used to blacklist "sys" and "information_schema" databases from Hive. Admin can use them to specify any databases/tables that are not suitable for Impala to query. Tests: - Add java unit tests for blacklist related functions - Add a custom cluster test: test_blacklisted_dbs_and_tables.py - Ran CORE tests Change-Id: I02dbb07f8e08793b57b2a88d09b30fd32cff26dc --- M be/src/common/global-flags.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/analysis/AlterTableOrViewRenameStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateDbStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateViewStmt.java M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/analysis/TableName.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java A fe/src/test/java/org/apache/impala/util/CatalogBlacklistUtilsTest.java A tests/custom_cluster/test_blacklisted_dbs_and_tables.py 15 files changed, 540 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/14134/9 -- To view, visit http://gerrit.cloudera.org:8080/14134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I02dbb07f8e08793b57b2a88d09b30fd32cff26dc Gerrit-Change-Number: 14134 Gerrit-PatchSet: 9 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-8797: Support database and table blacklist
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/14134 ) Change subject: IMPALA-8797: Support database and table blacklist .. Patch Set 8: (5 comments) > I think the issue with sys and information_schema not appearing is probably > something to do with the version of hive pinned in the toolchain. I've tried USE_CDP_HIVE=true to use Hive3, but there are still no "sys" and "information_schema" databases. Maybe it requires some configuration change or it's not shown for postgresql. Hope I can find the cause tomorrow. http://gerrit.cloudera.org:8080/#/c/14134/8/fe/src/main/java/org/apache/impala/analysis/TableName.java File fe/src/main/java/org/apache/impala/analysis/TableName.java: http://gerrit.cloudera.org:8080/#/c/14134/8/fe/src/main/java/org/apache/impala/analysis/TableName.java@86 PS8, Line 86: throw new AnalysisException("Invalid table/view name: " + tbl_); > Is there a reason not to fold the Blacklist.verifyTableName() into TableNam I think they should always be called together. But BlacklistUtils.verifyTableName() requires a fully-qualified TableName object. I used to add it into TableName.analyze() in a previous patch version, then the codes like this: tableName_.analyze(); become something like: analyzer.getFqTableName(tableName_).analyze(); Since "analyze" sounds like changing the state (to be analyzed) of the target. Now we look like changing the state of a temp object... So I decided to refactor the function name to "verify", meaning we are just verifying something based on the temp object. Finally, we decided to keep the name as "analyze" to be consistent with other parts (in the review of patch set 7). So I moved out the logics of BlacklistUtils.verifyTableName() since it requires more (valid db_) comparing to TableName.analyze(). However, I'm ok if we should merge them together and writing codes like analyzer.getFqTableName(tableName_).analyze(); http://gerrit.cloudera.org:8080/#/c/14134/8/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/14134/8/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@270 PS8, Line 270: // Error string for inconsistent blacklisted dbs/tables configs between catalogd and > Yea, we don't have any checks that validate the configs across roles. That Yes, I think for configs that read by both catalogd and coordinators, we should always check the consistent. I used to just give the configs to catalogd and found wired behaviors. So I decide to add these checks. http://gerrit.cloudera.org:8080/#/c/14134/8/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1519 PS8, Line 1519: addSummary(resp, "Can't drop blacklisted database: " + dbName); > Add a test for this? This is tested in tests/custom_cluster/test_blacklisted_dbs_and_tables.py: self.__expect_error_in_result( "drop database if exists functional_rc", "Can't drop blacklisted database: functional_rc") Note that it uses __expect_error_in_result, not __expect_error_in_query. http://gerrit.cloudera.org:8080/#/c/14134/8/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1644 PS8, Line 1644: > Similar check here? Good point! I missed it since the bug of IMPALA-8918. Will add more checks. http://gerrit.cloudera.org:8080/#/c/14134/8/fe/src/main/java/org/apache/impala/util/BlacklistUtils.java File fe/src/main/java/org/apache/impala/util/BlacklistUtils.java: http://gerrit.cloudera.org:8080/#/c/14134/8/fe/src/main/java/org/apache/impala/util/BlacklistUtils.java@32 PS8, Line 32: BlacklistUtils > nit: Call this CatalogUtils or something? BlackList seems out of place, doe Sure. What about CatalogBlacklistUtils? -- To view, visit http://gerrit.cloudera.org:8080/14134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I02dbb07f8e08793b57b2a88d09b30fd32cff26dc Gerrit-Change-Number: 14134 Gerrit-PatchSet: 8 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 04 Sep 2019 13:55:04 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8703: ISO:SQL:2016 datetime patterns - Milestone 1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13722 ) Change subject: IMPALA-8703: ISO:SQL:2016 datetime patterns - Milestone 1 .. Patch Set 16: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4460/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13722 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I19d8d097a45ae6f103b6cd1b2d81aad38dfd9e23 Gerrit-Change-Number: 13722 Gerrit-PatchSet: 16 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 04 Sep 2019 09:43:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8703: ISO:SQL:2016 datetime patterns - Milestone 1
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/13722 ) Change subject: IMPALA-8703: ISO:SQL:2016 datetime patterns - Milestone 1 .. Patch Set 16: (1 comment) http://gerrit.cloudera.org:8080/#/c/13722/15/be/src/runtime/datetime-iso-sql-format-parser.cc File be/src/runtime/datetime-iso-sql-format-parser.cc: http://gerrit.cloudera.org:8080/#/c/13722/15/be/src/runtime/datetime-iso-sql-format-parser.cc@144 PS15, Line 144: > Good catch this is actually a bug! In fact I think something is off with th Fixed in PS16 -- To view, visit http://gerrit.cloudera.org:8080/13722 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I19d8d097a45ae6f103b6cd1b2d81aad38dfd9e23 Gerrit-Change-Number: 13722 Gerrit-PatchSet: 16 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 04 Sep 2019 09:04:10 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8703: ISO:SQL:2016 datetime patterns - Milestone 1
Hello Attila Jeges, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13722 to look at the new patch set (#16). Change subject: IMPALA-8703: ISO:SQL:2016 datetime patterns - Milestone 1 .. IMPALA-8703: ISO:SQL:2016 datetime patterns - Milestone 1 This enhancement introduces FORMAT clause for CAST() operator that is applicable for casts between string types and timestamp types. Instead of accepting SimpleDateFormat patterns the FORMAT clause supports datetime patterns following the ISO:SQL:2016 standard. Note, the CAST() operator without the FORMAT clause still uses Impala's implementation of SimpleDateFormat handling. Similarly, the existing conversion functions such as to_timestamp(), from_timestamp() etc. remain unchanged and use SimpleDateFormat. Contrary to how these functions work the FORMAT clause must specify a string literal and cannot be used with any other kind of a string expression. Milestone 1 contains all the format tokens covered by the SQL standard. Further milestones will add more functionality on top of this list to cover functionality provided by other RDBMS systems. List of tokens implemented by this change: - , YYY, YY, Y: Year tokens - , RR: Round year tokens - MM: Month (1-12) - DD: Day (1-31) - DDD: Day of year (1-366) - HH, HH12: Hour of day (1-12) - HH24: Hour of day (0-23) - MI: Minute (0-59) - SS: Second (0-59) - S: Second of day (0-86399) - FF, FF1, ..., FF9: Fractional second - AM, PM, A.M., P.M.: Meridiem indicators - TZH: Timezone hour (-99-+99) - TZM: Timezone minute (0-99) - Separators: - . / , ' ; : space - ISO8601 date indicators (T, Z) Some notes about the matching algorithm: - The parsing algorithm uses these tokens in a case insensitive manner. - The separators are interchangeable with each other. For example a '-' separator in the format will match with a '.' character in the input. - The length of the separator sequences is handled flexibly meaning that a single separator character in the format for instance would match with a multi-separator sequence in the input. - In a string type to timestamp conversion the timezone offset tokens are parsed, expected to match with the input but they don't adjust the result as the input is already expected to be in UTC format. Usage example: SELECT CAST('01-02-2019' AS TIMESTAMP FORMAT 'MM-DD-'); SELECT CAST('2019.10.10 13:30:40.123456 +01:30' AS TIMESTAMP FORMAT '-MM-DD HH24:MI:SS.FF9 TZH:TZM'); SELECT CAST(timestamp_column as STRING FORMAT " MM HH12 YY") from some_table; Change-Id: I19d8d097a45ae6f103b6cd1b2d81aad38dfd9e23 --- M be/src/benchmarks/convert-timestamp-benchmark.cc M be/src/benchmarks/parse-timestamp-benchmark.cc M be/src/common/init.cc M be/src/exec/text-converter.inline.h M be/src/exprs/CMakeLists.txt A be/src/exprs/cast-format-expr.cc A be/src/exprs/cast-format-expr.h M be/src/exprs/cast-functions-ir.cc M be/src/exprs/date-functions-ir.cc M be/src/exprs/expr-test.cc M be/src/exprs/scalar-expr-evaluator.h M be/src/exprs/scalar-expr.cc M be/src/exprs/scalar-expr.h M be/src/exprs/timestamp-functions-ir.cc M be/src/exprs/timestamp-functions.cc M be/src/exprs/timestamp-functions.h M be/src/runtime/CMakeLists.txt M be/src/runtime/date-parse-util.cc M be/src/runtime/date-parse-util.h M be/src/runtime/date-test.cc M be/src/runtime/date-value.cc M be/src/runtime/date-value.h A be/src/runtime/datetime-iso-sql-format-parser.cc A be/src/runtime/datetime-iso-sql-format-parser.h A be/src/runtime/datetime-iso-sql-format-tokenizer.cc A be/src/runtime/datetime-iso-sql-format-tokenizer.h D be/src/runtime/datetime-parse-util.h A be/src/runtime/datetime-parser-common.cc A be/src/runtime/datetime-parser-common.h R be/src/runtime/datetime-simple-date-format-parser.cc A be/src/runtime/datetime-simple-date-format-parser.h M be/src/runtime/runtime-state.cc M be/src/runtime/timestamp-parse-util.cc M be/src/runtime/timestamp-parse-util.h M be/src/runtime/timestamp-test.cc M be/src/runtime/timestamp-value.cc M be/src/runtime/timestamp-value.h M be/src/service/impala-server.cc M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/testutil/random-vector-generators.h M be/src/util/dict-test.cc M be/src/util/min-max-filter-test.cc M be/src/util/string-parser.h M common/thrift/Exprs.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/CastExpr.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java A testdata/workloads/functional-query/queries/QueryTest/cast_format_from_table.test M testdata/workloads/functional-query/queries/QueryTest/date.test A tests/query_test/test_cast_with_format.py 54 files changed, 3,461 insertions(+), 865 deletions(-) git
[Impala-ASF-CR] IMPALA-8913: Add query option to disable hbase row estimation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14169 ) Change subject: IMPALA-8913: Add query option to disable hbase row estimation .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14169 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I768ea1f560c5faab74d8772bc8aa9ddefdcf2e10 Gerrit-Change-Number: 14169 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 04 Sep 2019 08:37:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8913: Add query option to disable hbase row estimation
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14169 ) Change subject: IMPALA-8913: Add query option to disable hbase row estimation .. IMPALA-8913: Add query option to disable hbase row estimation IMPALA-8058 added a flag, disable_hbase_row_est, in the QueryCtx for tests to bypass the codes of estimating HBase row stats from HBase RPCs and fall back to HMS row count stats. This patch exposes it as a query option. Tests: - Replace the usage of disable_hbase_row_est in QueryCtx by the new query option in tests. Verified relative tests pass as before. Change-Id: I768ea1f560c5faab74d8772bc8aa9ddefdcf2e10 Reviewed-on: http://gerrit.cloudera.org:8080/14169 Reviewed-by: Quanlong Huang Tested-by: Impala Public Jenkins --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java 7 files changed, 33 insertions(+), 16 deletions(-) Approvals: Quanlong Huang: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/14169 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I768ea1f560c5faab74d8772bc8aa9ddefdcf2e10 Gerrit-Change-Number: 14169 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8703: ISO:SQL:2016 datetime patterns - Milestone 1
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/13722 ) Change subject: IMPALA-8703: ISO:SQL:2016 datetime patterns - Milestone 1 .. Patch Set 15: (1 comment) http://gerrit.cloudera.org:8080/#/c/13722/15/be/src/runtime/datetime-iso-sql-format-parser.cc File be/src/runtime/datetime-iso-sql-format-parser.cc: http://gerrit.cloudera.org:8080/#/c/13722/15/be/src/runtime/datetime-iso-sql-format-parser.cc@144 PS15, Line 144: 0, 99 > Ok, I checked out your change and run some tests. This works: Good catch this is actually a bug! In fact I think something is off with this TH handling as I observed another input to work incorrectly. Let me give this a second thought and come back with a fix. -- To view, visit http://gerrit.cloudera.org:8080/13722 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I19d8d097a45ae6f103b6cd1b2d81aad38dfd9e23 Gerrit-Change-Number: 13722 Gerrit-PatchSet: 15 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 04 Sep 2019 08:21:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8755: Frontend support for Z-ordering
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/13955 ) Change subject: IMPALA-8755: Frontend support for Z-ordering .. Patch Set 11: Have you forgot to upload the latest patch set? :) -- To view, visit http://gerrit.cloudera.org:8080/13955 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie122002ca8f52ca2c1e1ec8ff1d476ae1f4f875d Gerrit-Change-Number: 13955 Gerrit-PatchSet: 11 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 04 Sep 2019 06:44:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8915: reapply IMPALA-8901 fix
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14176 ) Change subject: IMPALA-8915: reapply IMPALA-8901 fix .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4459/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14176 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42ea1ec4c386f2eb45f370cd35a104b49786a48c Gerrit-Change-Number: 14176 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 04 Sep 2019 06:04:41 + Gerrit-HasComments: No