Hello,
We are seeing some really bad errors on our hadoop cluster. After
reformatting the whole cluster, the first job we run immediately fails
with "Could not find block locations..." errrors. In the namenode
logs, we see a ton of errors like:
2008-10-31 14:20:44,799 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 5 on 7276, call addBlock(/tmp/dustintmp/shredded_dataunits/_t$
org.apache.hadoop.dfs.LeaseExpiredException: No lease on /tmp/
dustintmp/shredded_dataunits/_temporary/
_attempt_200810311418_0002_m_000023_0$
at
org.apache.hadoop.dfs.FSNamesystem.checkLease(FSNamesystem.java:1166)
at
org
.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:
1097)
at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:330)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun
.reflect
.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
In the datanode logs, we see a ton of errors like:
2008-10-31 14:20:09,978 ERROR org.apache.hadoop.dfs.DataNode:
DatanodeRegistration(10.100.11.115:50010,
storageID=DS-2129547091-10.100.11.1$
of concurrent xcievers 256
at org.apache.hadoop.dfs.DataNode
$DataXceiver.run(DataNode.java:1030)
at java.lang.Thread.run(Thread.java:619)
Anyone have any ideas on what may be wrong?
Thanks,
Nathan Marz
Rapleaf
