[Impala-ASF-CR] IMPALA-11021: Fix bug when query contains illegal predicate hints
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18040 ) Change subject: IMPALA-11021: Fix bug when query contains illegal predicate hints .. Patch Set 6: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/18040 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id719bc4280c811456333eb4b4ec5bc9cb8bae128 Gerrit-Change-Number: 18040 Gerrit-PatchSet: 6 Gerrit-Owner: wangsheng Gerrit-Reviewer: Amogh Margoor Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Mon, 29 Nov 2021 03:33:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11032: Automatic Refresh of Metadata for Local Catalog after Compaction
Yu-Wen Lai has posted comments on this change. ( http://gerrit.cloudera.org:8080/18043 ) Change subject: IMPALA-11032: Automatic Refresh of Metadata for Local Catalog after Compaction .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/18043/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/18043/2//COMMIT_MSG@10 PS2, Line 10: After compaction happened in Hive(HIVE ACID table), queries made in : Impala possibly fail with a FileNotFoundException if files already : removed by the Hive cleaner. > IIRC, Impala only open transactions for DDL/DML operations. Do you know how Thank Vihang and Quanlong for letting me know the problem. Impala does NOT open transactions for select queries so this approach doesn't work all the time... Hive has a config that can delay the cleaner some period of time but we don't know exactly how long we should extend. Given that this is time sensitive, I'm thinking we could make this feature optional for now. If this flag is set, say auto_check_compaction, let Impala open transactions for all the queries for ACID tables and do the compaction checking. Any thoughts? http://gerrit.cloudera.org:8080/#/c/18043/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java: http://gerrit.cloudera.org:8080/#/c/18043/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@898 PS2, Line 898: List stalePartitions = directProvider_.checkLatestCompaction( : refImpl.dbName_, refImpl.tableName_, refImpl, refToMeta); > I think this introduces several HMS RPCs per query (some queries may call t If we take the performance numbers on DWX as example, currently this API call takes 10 ~ 40 ms per table depending on the number of partitions. I will have a fix on the HMS side to solve an issue around this API that we need to pass all the partition names. That should make all the API execution time close to 10 ms. Even though we can make some improvement around this API, I understand this is still introduce the overhead that might not neglectable. It might be better to introduce this feature with a flag and the table property to skip this check as Quanlong suggested. -- To view, visit http://gerrit.cloudera.org:8080/18043 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I173ea848917b6a41139b25b80677111463bfdc4b Gerrit-Change-Number: 18043 Gerrit-PatchSet: 2 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Mon, 29 Nov 2021 02:56:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11021: Fix bug when query contains illegal predicate hints
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18040 ) Change subject: IMPALA-11021: Fix bug when query contains illegal predicate hints .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/9845/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18040 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id719bc4280c811456333eb4b4ec5bc9cb8bae128 Gerrit-Change-Number: 18040 Gerrit-PatchSet: 6 Gerrit-Owner: wangsheng Gerrit-Reviewer: Amogh Margoor Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Mon, 29 Nov 2021 02:15:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11021: Fix bug when query contains illegal predicate hints
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/18040 ) Change subject: IMPALA-11021: Fix bug when query contains illegal predicate hints .. Patch Set 6: (1 comment) Done! http://gerrit.cloudera.org:8080/#/c/18040/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/18040/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3336 PS5, Line 3336: ' > nit: one space here instead of two Done -- To view, visit http://gerrit.cloudera.org:8080/18040 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id719bc4280c811456333eb4b4ec5bc9cb8bae128 Gerrit-Change-Number: 18040 Gerrit-PatchSet: 6 Gerrit-Owner: wangsheng Gerrit-Reviewer: Amogh Margoor Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Mon, 29 Nov 2021 01:55:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11021: Fix bug when query contains illegal predicate hints
wangsheng has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/18040 ) Change subject: IMPALA-11021: Fix bug when query contains illegal predicate hints .. IMPALA-11021: Fix bug when query contains illegal predicate hints Currently Impala support predicate hint: ALWAYS_TRUE, we can use this hint after where keyword. If we use illegal hints carelessly, query will throw IllegalStateException which is not expected. Query should return normal results with a warning instead of a exception. This is due to the condition check in Analyzer.addWarning(). After create TExecRequest and initialize it, Impala will get warnings from 'GlobalState.warnings', and 'GlobalState.warningsRetrieved' will be set to 'true' then. But after this, Impala will substitute predicate by clone(), and analyze new predicate in later phase. New predicate analyze will add hint warning to 'GlobalState.warnings', but failed and throw IllegalStateException due to 'globalState_.warningsRetrieved' check failed which is expected as 'false'. This check is added in IMPALA-4166, I removed original condition check and added a new check to ensure that all warnings for new/substituted predicates are already exists in 'globalState_.warnings'. And this will also avoiding exception caused by illegal hints. Testing: - Added new fe tests in 'AnalyzeStmtsTest' Change-Id: Id719bc4280c811456333eb4b4ec5bc9cb8bae128 --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java 2 files changed, 51 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/18040/6 -- To view, visit http://gerrit.cloudera.org:8080/18040 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id719bc4280c811456333eb4b4ec5bc9cb8bae128 Gerrit-Change-Number: 18040 Gerrit-PatchSet: 6 Gerrit-Owner: wangsheng Gerrit-Reviewer: Amogh Margoor Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng
[Impala-ASF-CR] IMPALA-11021: Fix bug when query contains illegal predicate hints
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18040 ) Change subject: IMPALA-11021: Fix bug when query contains illegal predicate hints .. Patch Set 5: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/18040/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/18040/5/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3336 PS5, Line 3336: nit: one space here instead of two -- To view, visit http://gerrit.cloudera.org:8080/18040 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id719bc4280c811456333eb4b4ec5bc9cb8bae128 Gerrit-Change-Number: 18040 Gerrit-PatchSet: 5 Gerrit-Owner: wangsheng Gerrit-Reviewer: Amogh Margoor Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Mon, 29 Nov 2021 01:22:55 + Gerrit-HasComments: Yes
