oldlogfile.dat is screwed, so is it's region
--------------------------------------------
Key: HBASE-1243
URL: https://issues.apache.org/jira/browse/HBASE-1243
Project: Hadoop HBase
Issue Type: Bug
Affects Versions: 0.19.0
Reporter: Jean-Daniel Cryans
Priority: Blocker
Fix For: 0.19.1
Getting this when a node dies (happens frequently lately):
{code}
2009-03-05 04:15:03,251 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region web_pages,http://fortcollins.gaymonkey.com/,1235836722125 to
server 192.168.1.106:62020
2009-03-05 04:15:15,263 INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_CLOSE:
web_pages,http://fortcollins.gaymonkey.com/,1235836722125: java.io.IOException:
Could not obtain block: blk_5568212401457404905_251597
file=/hbase/amsterdam_factory/web_pages/1263377107/oldlogfile.log
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1708)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1536)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1663)
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at java.io.DataInputStream.readFully(DataInputStream.java:152)
at
org.apache.hadoop.hbase.io.SequenceFile$Reader.init(SequenceFile.java:1464)
at
org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1442)
at
org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1431)
at
org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1426)
at
org.apache.hadoop.hbase.regionserver.HStore.doReconstructionLog(HStore.java:342)
at
org.apache.hadoop.hbase.regionserver.HStore.runReconstructionLog(HStore.java:297)
at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:237)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1764)
at
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:276)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateRegion(HRegionServer.java:1367)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1338)
at
org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1253)
at java.lang.Thread.run(Thread.java:619)
from 192.168.1.106:62020
2009-03-05 04:15:18,266 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region web_pages,http://fortcollins.gaymonkey.com/,1235836722125 to
server 192.168.1.106:62020
2009-03-05 04:15:30,150 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
startKey: <>, server: 192.168.1.106:62020}
2009-03-05 04:15:30,276 INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_CLOSE:
web_pages,http://fortcollins.gaymonkey.com/,1235836722125: java.io.IOException:
Could not obtain block: blk_5568212401457404905_251597
file=/hbase/amsterdam_factory/web_pages/1263377107/oldlogfile.log
...
{code}
It does not recover, I have to manually delete the file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
