[Impala-ASF-CR] IMPALA-3040: Remove cache directives if a partition is dropped externally
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10792 ) Change subject: IMPALA-3040: Remove cache directives if a partition is dropped externally .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/29/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/10792 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170 Gerrit-Change-Number: 10792 Gerrit-PatchSet: 12 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Tue, 24 Jul 2018 18:33:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3040: Remove cache directives if a partition is dropped externally
Tianyi Wang has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/10792 ) Change subject: IMPALA-3040: Remove cache directives if a partition is dropped externally .. IMPALA-3040: Remove cache directives if a partition is dropped externally HdfsTable.dropPartition() doesn't uncache the partition right now. If the partition is dropped from Hive and refreshed in Impala, the partition will be removed from the catalog but the cache directive remains. Because Impala directly uses HMS client to drop a table/database, the cache directive won't be removed even if the table is dropped in Impala, if the backgroud loading is run concurrenty with the HMS client RPC call. This patch removes the cache directive in dropPartition() if the partition is removed from HMS. Change-Id: Id7701a499405e961456adea63f3592b43bd69170 --- M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/query_test/test_hdfs_caching.py 3 files changed, 49 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/10792/12 -- To view, visit http://gerrit.cloudera.org:8080/10792 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170 Gerrit-Change-Number: 10792 Gerrit-PatchSet: 12 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang
[Impala-ASF-CR] IMPALA-3040: Remove cache directives if a partition is dropped externally
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10792 ) Change subject: IMPALA-3040: Remove cache directives if a partition is dropped externally .. Patch Set 12: Build Started https://jenkins.impala.io/job/gerrit-code-review-checks/29/ Running initial code review checks. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/10792 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170 Gerrit-Change-Number: 10792 Gerrit-PatchSet: 12 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Tue, 24 Jul 2018 17:59:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3040: Remove cache directives if a partition is dropped externally
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10792 ) Change subject: IMPALA-3040: Remove cache directives if a partition is dropped externally .. Patch Set 12: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10792 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170 Gerrit-Change-Number: 10792 Gerrit-PatchSet: 12 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Tue, 24 Jul 2018 17:59:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3040: Remove cache directives if a partition is dropped externally
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/10792 ) Change subject: IMPALA-3040: Remove cache directives if a partition is dropped externally .. Patch Set 9: Code-Review+2 (2 comments) http://gerrit.cloudera.org:8080/#/c/10792/8/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/10792/8/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1375 PS8, Line 1375: Set msPartitionNames = Sets.newHashSet(); : msPartitionNames.addAll(client.listPartitionNames(db_.getName(), name_, (short) -1)); : // Names of loaded partitions in this table > I changed the definition of it instead of removing. We have to drop the par nice catch. http://gerrit.cloudera.org:8080/#/c/10792/9/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/10792/9/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1414 PS9, Line 1414: dropPartitions(dirtyPartitions, false); Add a comment that dirtyPartitions are reloaded and hence cachedirectives are not dropped. Easier for those reading the code to understand. -- To view, visit http://gerrit.cloudera.org:8080/10792 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170 Gerrit-Change-Number: 10792 Gerrit-PatchSet: 9 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Mon, 23 Jul 2018 22:38:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3040: Remove cache directives if a partition is dropped externally
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10792 ) Change subject: IMPALA-3040: Remove cache directives if a partition is dropped externally .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: https://issues.apache.org/jira/browse/IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/10792 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170 Gerrit-Change-Number: 10792 Gerrit-PatchSet: 9 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Mon, 23 Jul 2018 19:50:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3040: Remove cache directives if a partition is dropped externally
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10792 ) Change subject: IMPALA-3040: Remove cache directives if a partition is dropped externally .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: https://issues.apache.org/jira/browse/IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/10792 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170 Gerrit-Change-Number: 10792 Gerrit-PatchSet: 9 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Mon, 23 Jul 2018 19:50:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3040: Remove cache directives if a partition is dropped externally
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/10792 ) Change subject: IMPALA-3040: Remove cache directives if a partition is dropped externally .. Patch Set 7: How about having an overloaded method, dropPartitions(list, boolean removeCacheDirective). It can true by default, but we make it false in the dirtyPatitions case. Is it more complicated than that? -- To view, visit http://gerrit.cloudera.org:8080/10792 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170 Gerrit-Change-Number: 10792 Gerrit-PatchSet: 7 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Fri, 20 Jul 2018 00:14:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3040: Remove cache directives if a partition is dropped externally
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10792 ) Change subject: IMPALA-3040: Remove cache directives if a partition is dropped externally .. Patch Set 7: The problem is that dropPartition() is also called on dirty partitions, in which case we should not change anything in the namenode. Maybe we should further differentiate these two cases, but that's non-trivial code change. -- To view, visit http://gerrit.cloudera.org:8080/10792 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170 Gerrit-Change-Number: 10792 Gerrit-PatchSet: 7 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Thu, 19 Jul 2018 22:41:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3040: Remove cache directives if a partition is dropped externally
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/10792 ) Change subject: IMPALA-3040: Remove cache directives if a partition is dropped externally .. Patch Set 7: Did you get a chance to triage the GVO failure? Is it related to the patch change? -- To view, visit http://gerrit.cloudera.org:8080/10792 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170 Gerrit-Change-Number: 10792 Gerrit-PatchSet: 7 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Fri, 13 Jul 2018 18:37:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3040: Remove cache directives if a partition is dropped externally
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10792 ) Change subject: IMPALA-3040: Remove cache directives if a partition is dropped externally .. Patch Set 7: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2778/ -- To view, visit http://gerrit.cloudera.org:8080/10792 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170 Gerrit-Change-Number: 10792 Gerrit-PatchSet: 7 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Fri, 06 Jul 2018 23:56:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3040: Remove cache directives if a partition is dropped externally
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10792 ) Change subject: IMPALA-3040: Remove cache directives if a partition is dropped externally .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10792 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170 Gerrit-Change-Number: 10792 Gerrit-PatchSet: 7 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Fri, 06 Jul 2018 20:32:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3040: Remove cache directives if a partition is dropped externally
Tianyi Wang has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/10792 ) Change subject: IMPALA-3040: Remove cache directives if a partition is dropped externally .. IMPALA-3040: Remove cache directives if a partition is dropped externally HdfsTable.dropPartition() doesn't uncache the partition right now. If the partition is dropped from Hive and refreshed in Impala, the partition will be removed from the catalog but the cache directive remains. Because Impala directly uses HMS client to drop a table/database, the cache directive won't be removed even if the table is dropped in Impala, if the backgroud loading is run concurrenty with the HMS client RPC call. This patch removes the cache directive in dropPartition() to fix this bug. Change-Id: Id7701a499405e961456adea63f3592b43bd69170 --- M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/query_test/test_hdfs_caching.py 3 files changed, 29 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/10792/6 -- To view, visit http://gerrit.cloudera.org:8080/10792 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170 Gerrit-Change-Number: 10792 Gerrit-PatchSet: 6 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Tianyi Wang
[Impala-ASF-CR] IMPALA-3040: Remove cache directives if a partition is dropped externally
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10792 ) Change subject: IMPALA-3040: Remove cache directives if a partition is dropped externally .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2778/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/10792 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170 Gerrit-Change-Number: 10792 Gerrit-PatchSet: 7 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Fri, 06 Jul 2018 20:32:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3040: Remove cache directives if a partition is dropped externally
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/10792 ) Change subject: IMPALA-3040: Remove cache directives if a partition is dropped externally .. Patch Set 5: Code-Review+2 (1 comment) Feel free to carry +2 after the fix and submit for GVO. http://gerrit.cloudera.org:8080/#/c/10792/5/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/10792/5/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1175 PS5, Line 1175: getName getName() does not print the db info. Use getFullName() instead. Also you can just use getFullName() instead of partition.getTable().getFullName(). We are already in the table class. -- To view, visit http://gerrit.cloudera.org:8080/10792 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170 Gerrit-Change-Number: 10792 Gerrit-PatchSet: 5 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Fri, 06 Jul 2018 20:28:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3040: Remove cache directives if a partition is dropped externally
Tianyi Wang has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/10792 ) Change subject: IMPALA-3040: Remove cache directives if a partition is dropped externally .. IMPALA-3040: Remove cache directives if a partition is dropped externally HdfsTable.dropPartition() doesn't uncache the partition right now. If the partition is dropped from Hive and refreshed in Impala, the partition will be removed from the catalog but the cache directive remains. Because Impala directly uses HMS client to drop a table/database, the cache directive won't be removed even if the table is dropped in Impala, if the backgroud loading is run concurrenty with the HMS client RPC call. This patch removes the cache directive in dropPartition() to fix this bug. Change-Id: Id7701a499405e961456adea63f3592b43bd69170 --- M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/query_test/test_hdfs_caching.py 3 files changed, 30 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/10792/5 -- To view, visit http://gerrit.cloudera.org:8080/10792 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170 Gerrit-Change-Number: 10792 Gerrit-PatchSet: 5 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Tianyi Wang
[Impala-ASF-CR] IMPALA-3040: Remove cache directives if a partition is dropped externally
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/10792 )
Change subject: IMPALA-3040: Remove cache directives if a partition is dropped
externally
..
Patch Set 4: Code-Review+2
(3 comments)
http://gerrit.cloudera.org:8080/#/c/10792/4/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/10792/4/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1174
PS4, Line 1174: LOG.error("Unable to remove a cache directive: " +
e.getMessage())
Please include table and partition name in the message. Also do something like
LOG.error(msg, e) so that the stack trace is printed too.
http://gerrit.cloudera.org:8080/#/c/10792/4/tests/query_test/test_hdfs_caching.py
File tests/query_test/test_hdfs_caching.py:
http://gerrit.cloudera.org:8080/#/c/10792/4/tests/query_test/test_hdfs_caching.py@291
PS4, Line 291: assert num_entries_pre + 1 == get_num_cache_requests()
nit: add comments like in L287. Something like we expect the partition cache
directive to be dropped etc..
http://gerrit.cloudera.org:8080/#/c/10792/4/tests/query_test/test_hdfs_caching.py@292
PS4, Line 292: self.client.execute("drop table cached_tbl_part")
nit: Do we need to put the drop in a finally block? since we are using
'cached_tbl_part' in different tests, if the cleanup does not happen for some
reason (due to test errors etc) other tests could potentially fail (for ex:
L224). Or just use another/random table name to avoid this mess.
--
To view, visit http://gerrit.cloudera.org:8080/10792
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170
Gerrit-Change-Number: 10792
Gerrit-PatchSet: 4
Gerrit-Owner: Tianyi Wang
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Tianyi Wang
Gerrit-Comment-Date: Wed, 04 Jul 2018 06:48:06 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3040: Remove cache directives if a partition is dropped externally
Tianyi Wang has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/10792 ) Change subject: IMPALA-3040: Remove cache directives if a partition is dropped externally .. IMPALA-3040: Remove cache directives if a partition is dropped externally HdfsTable.dropPartition() doesn't uncache the partition right now. If the partition is dropped from Hive and refreshed in Impala, the partition will be removed from the catalog but the cache directive remains. Because Impala directly uses HMS client to drop a table/database, the cache directive won't be removed even if the table is dropped in Impala, if the backgroud loading is run concurrenty with the HMS client RPC call. This patch removes the cache directive in dropPartition() to fix this bug. Change-Id: Id7701a499405e961456adea63f3592b43bd69170 --- M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/query_test/test_hdfs_caching.py 3 files changed, 28 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/10792/4 -- To view, visit http://gerrit.cloudera.org:8080/10792 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170 Gerrit-Change-Number: 10792 Gerrit-PatchSet: 4 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Tianyi Wang
[Impala-ASF-CR] IMPALA-3040: Remove cache directives if a partition is dropped externally
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10792 ) Change subject: IMPALA-3040: Remove cache directives if a partition is dropped externally .. Patch Set 4: (6 comments) http://gerrit.cloudera.org:8080/#/c/10792/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/10792/3//COMMIT_MSG@7 PS3, Line 7: if a partition is dropped externally > may be say external table drops? Done http://gerrit.cloudera.org:8080/#/c/10792/3//COMMIT_MSG@9 PS3, Line 9: > Add some context about what happens when it is dropped from Hive. Done http://gerrit.cloudera.org:8080/#/c/10792/3//COMMIT_MSG@13 PS3, Line 13: table/database, the cache directive won't be removed even if the table > Could you add a test for this in test_hdfs_caching? Done http://gerrit.cloudera.org:8080/#/c/10792/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/10792/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1158 PS3, Line 1158:* HdfsPartition that was dropped or null if the partition does not exist. > Update that this drops the cache directive if its cached. Done http://gerrit.cloudera.org:8080/#/c/10792/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1191 PS3, Line 1191: // If there are multiple partition ids corresponding to a literal, remove > I think this should only run on the Catalog server? I checked the callers of this function and it can only be called from catalogd. http://gerrit.cloudera.org:8080/#/c/10792/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1193 PS3, Line 1193: if (partitionIds.size() > 1) partitionIds.remove(partitionId); > Do we need to remove a similar check from CatalogOpEx#alterTableDropPartiti Done -- To view, visit http://gerrit.cloudera.org:8080/10792 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170 Gerrit-Change-Number: 10792 Gerrit-PatchSet: 4 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Wed, 04 Jul 2018 00:54:31 + Gerrit-HasComments: Yes
