[ https://issues.apache.org/jira/browse/HDFS-14947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16965143#comment-16965143 ]
Wei-Chiu Chuang commented on HDFS-14947: ---------------------------------------- I would be really really surprised if a file simply doesn't exist after rename. Is it reproducible? Can you check NN log/ audit log to see if its parent and ancestor directories were deleted? > infrequent data loss due to rename functionality breaking > --------------------------------------------------------- > > Key: HDFS-14947 > URL: https://issues.apache.org/jira/browse/HDFS-14947 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client > Affects Versions: 2.7.3 > Reporter: abhishek sahani > Priority: Critical > > We are facing an issue where data is getting lost from hdfs during rename , > in namenode logs we check file is renamed successfully but in hdfs after > rename file is not present at destination location and thus we are loosing > the data. > > namenode logs: > 19/10/31 16:54:09 DEBUG top.TopAuditLogger: ------------------- logged event > for top service: allowed=true ugi=root (auth:SIMPLE) ip=/*.*.*.* cmd=rename > src=/topics/+tmp/datapipelinefinaltest14.5da59e664cedfd00090d3757.dataPipeLineEvent_17.topic/tenant=5da59e664cedfd00090d3757/groupid=5da59e664cedfd00090d3758/project=5da59e664cedfd00090d3759/name=dataPipeLineEvent_17/year=2019/month=10/day=16/hour=17/351bffa9-15e3-427b-9e02-c9e8823d68d6_tmp.parquet > > dst=/topics/datapipelinefinaltest14.5da59e664cedfd00090d3757.dataPipeLineEvent_17.topic/tenant=5da59e664cedfd00090d3757/groupid=5da59e664cedfd00090d3758/project=5da59e664cedfd00090d3759/name=dataPipeLineEvent_17/year=2019/month=10/day=16/hour=17/datapipelinefinaltest14.5da59e664cedfd00090d3757.dataPipeLineEvent_17.topic+9+0000000000+0000000099.parquet > perm=root:supergroup:rw-r--r-- > > 19/10/31 16:54:09 DEBUG hdfs.StateChange: DIR* NameSystem.renameTo: > /topics/+tmp/datapipelinefinaltest14.5da59e664cedfd00090d3757.dataPipeLineEvent_17.topic/tenant=5da59e664cedfd00090d3757/groupid=5da59e664cedfd00090d3758/project=5da59e664cedfd00090d3759/name=dataPipeLineEvent_17/year=2019/month=10/day=16/hour=17/351bffa9-15e3-427b-9e02-c9e8823d68d6_tmp.parquet > to > /topics/datapipelinefinaltest14.5da59e664cedfd00090d3757.dataPipeLineEvent_17.topic/tenant=5da59e664cedfd00090d3757/groupid=5da59e664cedfd00090d3758/project=5da59e664cedfd00090d3759/name=dataPipeLineEvent_17/year=2019/month=10/day=16/hour=17/datapipelinefinaltest14.5da59e664cedfd00090d3757.dataPipeLineEvent_17.topic+9+0000000000+0000000099.parquet > 19/10/31 16:54:09 DEBUG ipc.Server: IPC Server handler 8 on 9000: responding > to org.apache.hadoop.hdfs.protocol.ClientProtocol.getFileInfo from > *.*.*.*:39854 Call#48333 Retry#0 > 19/10/31 16:54:09 DEBUG hdfs.StateChange: DIR* FSDirectory.renameTo: > /topics/+tmp/datapipelinefinaltest14.5da59e664cedfd00090d3757.dataPipeLineEvent_17.topic/tenant=5da59e664cedfd00090d3757/groupid=5da59e664cedfd00090d3758/project=5da59e664cedfd00090d3759/name=dataPipeLineEvent_17/year=2019/month=10/day=16/hour=17/351bffa9-15e3-427b-9e02-c9e8823d68d6_tmp.parquet > to > /topics/datapipelinefinaltest14.5da59e664cedfd00090d3757.dataPipeLineEvent_17.topic/tenant=5da59e664cedfd00090d3757/groupid=5da59e664cedfd00090d3758/project=5da59e664cedfd00090d3759/name=dataPipeLineEvent_17/year=2019/month=10/day=16/hour=17/datapipelinefinaltest14.5da59e664cedfd00090d3757.dataPipeLineEvent_17.topic+9+0000000000+0000000099.parquet > 19/10/31 16:54:09 DEBUG ipc.Server: IPC Server handler 6 on 9000: > org.apache.hadoop.hdfs.protocol.ClientProtocol.getFileInfo from *.*.*.*:39854 > Call#48337 Retry#0 for RpcKind RPC_PROTOCOL_BUFFER > 19/10/31 16:54:09 DEBUG hdfs.StateChange: DIR* > FSDirectory.unprotectedRenameTo: > /topics/+tmp/datapipelinefinaltest14.5da59e664cedfd00090d3757.dataPipeLineEvent_17.topic/tenant=5da59e664cedfd00090d3757/groupid=5da59e664cedfd00090d3758/project=5da59e664cedfd00090d3759/name=dataPipeLineEvent_17/year=2019/month=10/day=16/hour=17/351bffa9-15e3-427b-9e02-c9e8823d68d6_tmp.parquet > is renamed to > /topics/datapipelinefinaltest14.5da59e664cedfd00090d3757.dataPipeLineEvent_17.topic/tenant=5da59e664cedfd00090d3757/groupid=5da59e664cedfd00090d3758/project=5da59e664cedfd00090d3759/name=dataPipeLineEvent_17/year=2019/month=10/day=16/hour=17/datapipelinefinaltest14.5da59e664cedfd00090d3757.dataPipeLineEvent_17.topic+9+0000000000+0000000099.parquet > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org