Hi, I’m observing strange behaviour in HDFS/HBase cluster. The disk space of one of datanodes is increasing very fast even when there are no write requests. It is 8GB per hour in average. Here is the graph which shows it.
I am using hbase-0.98.7-hadoop2 and hadoop-2.5.1. And this is logs from the node 2015-07-28 15:40:38,795 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.32.1.12:50010, dest: /10.32.0.140:38699, bytes: 1071, op: HDFS_READ, cliID: DFSClient_NONMAPREDUCE_-689748537_1, offset: 0, srvID: 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: BP-439084760-10.32.0.180-1387281790961:blk_1074784244_1045663, duration: 17759797 2015-07-28 15:41:15,111 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-439084760-10.32.0.180-1387281790961:blk_1075311714_1574450 src: /10.0.0.21:60540 dest: /10.32.1.12:50010 2015-07-28 15:41:15,304 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.0.0.21:59054, dest: /10.32.1.12:50010, bytes: 124121, op: HDFS_WRITE, cliID: DFSClient_hb_rs_hbase-rs4,60020,1438094355024_530940245_35, offset: 0, srvID: 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: BP-439084760-10.32.0.180-1387281790961:blk_1075311536_1574238, duration: 3600203675041 2015-07-28 15:41:15,304 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-439084760-10.32.0.180-1387281790961:blk_1075311536_1574238, type=HAS_DOWNSTREAM_IN_PIPELINE terminating 2015-07-28 15:50:40,745 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:6099ms (threshold=300ms) 2015-07-28 15:59:21,130 INFO org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: BlockPool BP-439084760-10.32.0.180-1387281790961 Total blocks: 65856, missing metadata files:0, missing block files:0, missing blocks in memory:0, mismatched blocks:0 2015-07-28 16:00:16,770 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-439084760-10.32.0.180-1387281790961:blk_1075311715_1574451 src: /10.32.1.12:36998 dest: /10.32.1.12:50010 2015-07-28 16:00:17,469 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.32.1.12:36150, dest: /10.32.1.12:50010, bytes: 32688, op: HDFS_WRITE, cliID: DFSClient_hb_rs_hbase-rs5,60020,1438088401479_1146354759_35, offset: 0, srvID: 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: BP-439084760-10.32.0.180-1387281790961:blk_1075311706_1574442, duration: 3601152263901 2015-07-28 16:00:17,472 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-439084760-10.32.0.180-1387281790961:blk_1075311706_1574442, type=HAS_DOWNSTREAM_IN_PIPELINE terminating 2015-07-28 16:03:44,011 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-439084760-10.32.0.180-1387281790961:blk_1075311716_1574452 src: /10.0.0.19:35851 dest: /10.32.1.12:50010 2015-07-28 16:03:44,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.0.0.20:40176, dest: /10.32.1.12:50010, bytes: 316062, op: HDFS_WRITE, cliID: DFSClient_hb_rs_hbase-rs1,60020,1438092204868_-99326843_35, offset: 0, srvID: 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: BP-439084760-10.32.0.180-1387281790961:blk_1075311707_1574443, duration: 3600482062810 2015-07-28 16:03:44,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-439084760-10.32.0.180-1387281790961:blk_1075311707_1574443, type=LAST_IN_PIPELINE, downstreams=0:[] terminating 2015-07-28 16:11:10,961 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:7884ms (threshold=300ms) 2015-07-28 16:11:14,122 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:4362ms (threshold=300ms) 2015-07-28 16:11:14,123 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow flushOrSync took 3160ms (threshold=300ms), isSync:false, flushTotalNanos=3160364203ns 2015-07-28 16:13:29,968 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:659ms (threshold=300ms) 2015-07-28 16:18:33,336 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-439084760-10.32.0.180-1387281790961:blk_1075311717_1574453 src: /10.0.0.20:41527 dest: /10.32.1.12:50010 2015-07-28 16:18:38,926 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:1767ms (threshold=300ms) 2015-07-28 16:28:40,580 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:4099ms (threshold=300ms) What could be the cause of this? Thank you.
