[ https://issues.apache.org/jira/browse/HIVE-13778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294572#comment-15294572 ]
Sailesh Mukil commented on HIVE-13778: -------------------------------------- [~rajesh.balamohan] I discovered a few more details as I was trying to reproduce this issue. I didn't realize that this is how it happened before. To reproduce, do the following: - In Hive, "create table purge_test_s3 (x int) location 's3a://[bucket]/purge_test_s3';" - Use the AWS CLI or the AWS Web interface to copy files to the above mentioned location. - In Hive, "drop table purge_test_s3 purge;" The Metastore logs say: 2016-05-20 17:01:41,259 INFO hive.metastore.hivemetastoressimpl: [pool-4-thread-103]: Not moving s3a://[bucket]/purge_test_s3 to trash 2016-05-20 17:01:41,364 INFO hive.metastore.hivemetastoressimpl: [pool-4-thread-103]: Deleted the diretory s3a://[bucket]/purge_test_s3 However, the files are still there. The weird part is that the Hadoop S3A connector reads the files correctly but is not able to delete them. If instead of the AWS CLI or the AWS Web interface, we use the hadoop CLI to copy the files, "drop table ... purge" works just fine. If we insert the files using Hive, it works fine as well. This might be an issue of the HDFS Namenode not getting updated and might be more a problem for the HDFS folks. > DROP TABLE PURGE on S3A table with too many files does not delete the files > --------------------------------------------------------------------------- > > Key: HIVE-13778 > URL: https://issues.apache.org/jira/browse/HIVE-13778 > Project: Hive > Issue Type: Bug > Components: Metastore > Reporter: Sailesh Mukil > Priority: Critical > Labels: metastore, s3 > > I've noticed that when we do a DROP TABLE tablename PURGE on a table on S3A > that has many files, the files never get deleted. However, the Hive metastore > logs do say that the path was deleted: > "Not moving [path] to trash" > "Deleted the diretory [path]" > I initially thought that this was due to the eventually consistent nature of > S3 for deletes, however, a week later, the files still exist. -- This message was sent by Atlassian JIRA (v6.3.4#6332)