Hi,
I am using hadoop-0.20.2 version and hbase-0.20.6. HBase Region
Servers shutsdown because of the following error in file/block creation.
The following is the sequence of events related to a specific file, which is
causing this issue.
File/Block creation request:
11/01/12 19:45:43 INFO hdfs.StateChange: BLOCK*
NameSystem.allocateBlock:
/hbase/webtable/compaction.dir/1326275218/2628290592728420519.
blk_-7100767084720175376_361638
Namenode says: Block is invalid, doesnt belong to a file.
11/01/12 19:45:48 INFO hdfs.StateChange: BLOCK*
NameSystem.addStoredBlock: addStoredBlock request received for
blk_-7100767084720175376_361638 on 10.76.99.68:50010 size 23874613 But it
does not belong to any file.
11/01/12 19:45:48 INFO hdfs.StateChange: BLOCK*
NameSystem.addStoredBlock: addStoredBlock request received for
blk_-7100767084720175376_361638 on 10.76.99.57:50010 size 23874613 But it
does not belong to any file.
11/01/12 19:45:48 INFO hdfs.StateChange: BLOCK*
NameSystem.addStoredBlock: addStoredBlock request received for
blk_-7100767084720175376_361638 on 10.76.99.114:50010 size 23874613 But it
does not belong to any file.
DataNode receives this exception. Unable to complete write. (No
FileBlocks)
11/01/12 19:45:48 WARN hdfs.StateChange: DIR* NameSystem.completeFile:
failed to complete
/hbase/webtable/compaction.dir/1326275218/2628290592728420519 because
dir.getFileBlocks() is null and pendingFile is null
11/01/12 19:45:48 INFO ipc.Server: IPC Server handler 54 on 8020, call
complete(/hbase/webtable/compaction.dir/1326275218/2628290592728420519,
DFSClient_880037874) from 1XX.XX.XX.XXXX :55925: error: java.io.IOException:
Could not complete write to file
/hbase/webtable/compaction.dir/1326275218/2628290592728420519 by
DFSClient_880037874
java.io.IOException: Could not complete write to file
/hbase/webtable/compaction.dir/1326275218/2628290592728420519 by
DFSClient_880037874
at
org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:471)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
Did any one of you see this before?
Thanks,
Charan
