We've recently upgraded from 0.15.0 to 0.16.4. Two nights ago we had a problem where DFS nodes could not communicate. After not finding anything obviously wrong we decided to shut down DFS and restart. Following restart I was seeing a corrupted system with significant amounts of missing data. Further checking showed that DataNodes on all slaves did not start due to what looks like a version skew issue. Our distribution is a straight 0.16.4 dist, so I'm having difficulty understanding what's causing this issue. Note that we haven't finalized the upgrade yet. Any help understanding this problem would be very much appreciated. We have several TB of data in our system and reloading from scratch would be a big problem. Here is the log from one of the failed nodes: /************************************************************ STARTUP_MSG: Starting DataNode STARTUP_MSG: host = worker9/10.2.0.9 STARTUP_MSG: args = [] STARTUP_MSG: version = 0.16.4 STARTUP_MSG: build = http://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.16 -r 652614; compiled by 'hadoopqa' on Fri May 2 00:18:12 UTC 2008 ************************************************************/ 2008-05-23 08:10:47,196 FATAL org.apache.hadoop.dfs.DataNode: Incompatible build versions: namenode BV = 2182; datanode BV = 652614 2008-05-23 08:10:47,202 ERROR org.apache.hadoop.dfs.DataNode: java.io.IOException: Incompatible build versions: namenode BV = 2182; datanode BV = 652614 at org.apache.hadoop.dfs.DataNode.handshake(DataNode.java:342) at org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:213) at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:162) at org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:2512) at org.apache.hadoop.dfs.DataNode.run(DataNode.java:2456) at org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:2477) at org.apache.hadoop.dfs.DataNode.main(DataNode.java:2673) 2008-05-23 08:10:47,203 INFO org.apache.hadoop.dfs.DataNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down DataNode at worker9/10.2.0.9 ************************************************************/
