Hi Brian, Hadoop does not balance the disks within a DataNode. If you ran out of space and then add additional disks, you should shutdown the DataNode and move manually a few files to the new disk.
Regards, Aitor Cedrés On 6 October 2014 14:46, Brian C. Huffman <bhuff...@etinternational.com> wrote: > All, > > I have a small hadoop cluster (2.5.0) with 4 datanodes and 3 data disks > per node. Lately some of the volumes have been filling, but instead of > moving to other configured volumes that *have* free space, it's giving > errors in the datanode logs: > 2014-10-03 11:52:44,989 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: > thor2.xmen.eti:50010:DataXceiver error processing WRITE_BLOCK > operation src: /172.17.1.3:35412 dst: /172.17.1.2:50010 > java.io.IOException: No space left on device > at java.io.FileOutputStream.writeBytes(Native Method) > at java.io.FileOutputStream.write(FileOutputStream.java:345) > at org.apache.hadoop.hdfs.server.datanode.BlockReceiver. > receivePacket(BlockReceiver.java:592) > at org.apache.hadoop.hdfs.server.datanode.BlockReceiver. > receiveBlock(BlockReceiver.java:734) > at org.apache.hadoop.hdfs.server.datanode.DataXceiver. > writeBlock(DataXceiver.java:741) > at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock( > Receiver.java:124) > at org.apache.hadoop.hdfs.protocol.datatransfer. > Receiver.processOp(Receiver.java:71) > at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run( > DataXceiver.java:234) > at java.lang.Thread.run(Thread.java:745) > > Unfortunately it's continuing to try to write and when it fails, it's > passing the exception to the client. > > I did a restart and then it seemed to figure out that it should move to > the next volume. > > Any suggestions to keep this from happening in the future? > > Also - could it be an issue that I have a small amount of non-HDFS data on > those volumes? > > Thanks, > Brian > >
