[Impala-ASF-CR] IMPALA-3040: Remove cache directive before dropping a table

[Bharath Vissapragada (Code Review)]

Bharath Vissapragada has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10792 )

Change subject: IMPALA-3040: Remove cache directive before dropping a table
......................................................................


Patch Set 2:

Thanks for the explanation. I guess I understand the issue now.

The basic problem here seems to be that  HdfsTable.dropPartition() does not 
clean up cache directives of the partitions. We do it inside  
CatalogOpExecutor#alterTableDropPartition()

  private Table alterTableDropPartition(Table tbl,.....)
  ......
  if (part.isMarkedCached()) {
    HdfsCachingUtil.removePartitionCacheDirective(part);
  }
  ....
  }

but since these partitions are dropped already using Hive, Impala clears the 
state with dropPartition() and uncacheTable() later does not help anymore.  
Here is a simple repro of this issue.

// Create a partitioned cached table and add some partitions
impala> create table cached_tbl_part (i int) partitioned by (j int) cached in 
'testPool'
impala> insert into cached_tbl_part (i,j) select 1, 2;

// Make sure cache directives are created.
$ hdfs cacheadmin -listDirectives | grep cached_tbl_part
277 testPool        1 never   /test-warehouse/cached_tbl_part
278 testPool        1 never   /test-warehouse/cached_tbl_part/j=2  <----

// Drop the partition from hive
hive> alter table cached_tbl_part drop partition (j=2);

// Refresh the table from Impala
impala> refresh cached_tbl_part;

// Cache directives still exist
hdfs cacheadmin -listDirectives | grep cached_tbl_part

277 testPool        1 never   /test-warehouse/cached_tbl_part
278 testPool        1 never   /test-warehouse/cached_tbl_part/j=2   <--- should 
have been dropped

// Drop the table from Impala

impala> drop table cached_tbl_part;

$ hdfs cacheadmin -listDirectives | grep cached_tbl_part
278 testPool        1 never   /test-warehouse/cached_tbl_part/j=2   <--- 
Table's directive is dropped but the partition still remains.


--
To view, visit http://gerrit.cloudera.org:8080/10792
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170
Gerrit-Change-Number: 10792
Gerrit-PatchSet: 2
Gerrit-Owner: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bhara...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Jul 2018 07:43:37 +0000
Gerrit-HasComments: No