[ https://issues.apache.org/jira/browse/HADOOP-15880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16663079#comment-16663079 ]
Arpit Agarwal edited comment on HADOOP-15880 at 10/25/18 1:04 AM: ------------------------------------------------------------------ Hi [~Sunilkc], I don't expect any of the connectors (E.g. WASB, S3A) to honor _fs.trash.interval_. The purge functionality works for HDFS because it is implemented in the HDFS namenode. The cloud object stores have no idea about this setting. was (Author: arpitagarwal): Hi [~Sunilkc], I don't expect any of the connectors (E.g. WASB, S3A) to honor _fs.trash.interval_. The purge functionality is implemented in the HDFS namenode. The cloud object stores have no idea about this setting. > WASB doesn't honor fs.trash.interval and this fails to auto purge trash folder > ------------------------------------------------------------------------------ > > Key: HADOOP-15880 > URL: https://issues.apache.org/jira/browse/HADOOP-15880 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure > Affects Versions: 2.7.3 > Environment: Any HDInsigth cluster pointing to WASB. > Reporter: Sunil Kumar Chakrapani > Priority: Major > Labels: WASB > > when "fs.trash.interval" is set to a value, trash for the local hdfs got > cleared where as the trash folder on WASB doesn't get deleted and the files > get piled up on WASB store.. > WASB doesn't pick up fs.trash.interval value and this fails to auto purge > trash folder on WASB store. > > *Issue : WASB doesn't honor fs.trash.interval and this fails to auto purge > trash folder* > *Steps to reproduce Scenario:* > *Delete any file stored on HDFS* > hdfs dfs -D "fs.default.name=hdfs://mycluster/" -rm /hivestore.txt > 18/10/23 06:18:05 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://mycluster/hivestore.txt' to trash at: > hdfs://mycluster/user/sshuser/.Trash/Current/hivestore.txt > *When deleted the file is moved to trash folder* > hdfs dfs -rm wasb:///hivestore.txt > 18/10/23 06:19:13 INFO fs.TrashPolicyDefault: Moved: > 'wasb://kcspark-2018-10-18t17-07-40-5...@kcdnsproxy.blob.core.windows.net/hivestore.txt' > to trash at: > wasb://kcspark-2018-10-18t17-07-40-5...@kcdnsproxy.blob.core.windows.net/user/sshuser/.Trash/Current/hivestore.txt > *Reduced the fs.trash.interval from 360 to 1 and restarted all related > services.* > *Trash for the local hdfs gets cleared honoring the "fs.trash.interval" > value.* > hdfs dfs -D "fs.default.name=hdfs://mycluster/" -ls > hdfs://mycluster/user/sshuser/.Trash/Current/ > ls: File hdfs://mycluster/user/sshuser/.Trash/Current does not exist. > *Where as the trash for WASB doesn't get cleared.* > hdfs dfs -ls > wasb://kcspark-2018-10-18t17-07-40-5...@kcdnsproxy.blob.core.windows.net/user/sshuser/.Trash/Current/ > Found 1 items > -rw-r--r-- 1 sshuser supergroup 1084 2018-10-23 06:19 > wasb://kcspark-2018-10-18t17-07-40-5...@kcdnsproxy.blob.core.windows.net/user/sshuser/.Trash/Current/hivestore.txt > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org