Very naively looking at the code, the exception you see is happening in the write path, on the way to sending a copy of your data to a second data node. One data node is pipelining the data to another, and that connection is failing. The fact that "DatanodeRegistration" is mentioned in the exception is a red herring: that's merely the text that the datanode prints for every exception that's thrown during a server response. It's frustrating that the exception message doesn't actually mention what host it's trying to connect to.
Some quick avenues for debugging: It sounds like you've identified a specific data node that isn't behaving. Is the exception that you've pasted in coming from that DataNode or from another? Can you tell if the DataNode is listening on the right ports? You might try "sudo netstat -pl | grep java" and check to see that the DataNode is listening on 50010 (I believe that's the default). You might also try strace on the process that's showing the "no route to host" error, to catch the system call fail. You could, of course, instrument the code to do a try/catch around the relevant block in DataNode.java, to find out what host/port the connection is failing on.
