Frode Halvorsen created HDFS-7480: ------------------------------------- Summary: Namenodes loops on 'block does not belong to any file' after deleting many files Key: HDFS-7480 URL: https://issues.apache.org/jira/browse/HDFS-7480 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.5.0 Environment: CentOS - HDFS-HA (journal), zookeeper Reporter: Frode Halvorsen
A small cluster has 8 servers with 32 G RAM. Two is namenodes (HA-configured), six is Datanodes (8x3 TB disks configured with RAID as one 21 TB drive). The cluster recieves avg 400.000 small files each day. I started archiving (HAR) each day as separate archives. After deleting the orinigal files for one month, the namenodes stared acting up really bad. When restaring those, both active and passive nodes seems to work OK for some time, but then starts to report a lot of blocks belonging to no files, and the name-node just spins those messages in a massive loop. If the passive node is first, it also influences the active node in susch a way that it's no longer possible to archive new files. If the active node also starts in this loop, it suddenly dies without any error-message. The only way I'm able to get rid of the problem, is to start decommission nodes, watching the cluster closely to avoid downtime, and make sure every datanode gets a 'clean' start. After all datanodes has been decommisioned (in turns), and restarted with clean disks, the problem is gone. But if I then delete a lot of files in a short time, the problem starts again... The main problem (I think), is that the recieving and reporting of those blocks takes so many resources, that the namenodes is too busy to tell the datanodes to delete those blocks.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)