[
https://issues.apache.org/jira/browse/HBASE-1243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12679506#action_12679506
]
stack commented on HBASE-1243:
------------------------------
You said the file had zero-length in hdfs?
Here is code from HStore playing the reconstruction log:
{code}
// Check its not empty.
FileStatus[] stats = fs.listStatus(reconstructionLog);
if (stats == null || stats.length == 0) {
LOG.warn("Passed reconstruction log " + reconstructionLog + " is
zero-length");
return;
}
long maxSeqIdInLog = -1;
TreeMap<HStoreKey, byte []> reconstructedCache =
new TreeMap<HStoreKey, byte []>(new
HStoreKey.HStoreKeyWritableComparator(this.info));
SequenceFile.Reader logReader = new SequenceFile.Reader(this.fs,
reconstructionLog, this.conf);
{code}
So, the check above is not working or something else is up. What you think we
should do J-D? Catch the exception, flag it, and soldier on? I'm on for that.
Big dirty warning in the log and keep going.
> oldlogfile.dat is screwed, so is it's region
> --------------------------------------------
>
> Key: HBASE-1243
> URL: https://issues.apache.org/jira/browse/HBASE-1243
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.19.0
> Reporter: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.19.1
>
>
> Getting this when a node dies (happens frequently lately):
> {code}
> 2009-03-05 04:15:03,251 INFO org.apache.hadoop.hbase.master.RegionManager:
> assigning region web_pages,http://fortcollins.gaymonkey.com/,1235836722125 to
> server 192.168.1.106:62020
> 2009-03-05 04:15:15,263 INFO org.apache.hadoop.hbase.master.ServerManager:
> Received MSG_REPORT_CLOSE:
> web_pages,http://fortcollins.gaymonkey.com/,1235836722125:
> java.io.IOException: Could not obtain block: blk_5568212401457404905_251597
> file=/hbase/amsterdam_factory/web_pages/1263377107/oldlogfile.log
> at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1708)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1536)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1663)
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at
> org.apache.hadoop.hbase.io.SequenceFile$Reader.init(SequenceFile.java:1464)
> at
> org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1442)
> at
> org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1431)
> at
> org.apache.hadoop.hbase.io.SequenceFile$Reader.<init>(SequenceFile.java:1426)
> at
> org.apache.hadoop.hbase.regionserver.HStore.doReconstructionLog(HStore.java:342)
> at
> org.apache.hadoop.hbase.regionserver.HStore.runReconstructionLog(HStore.java:297)
> at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:237)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1764)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:276)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateRegion(HRegionServer.java:1367)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1338)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1253)
> at java.lang.Thread.run(Thread.java:619)
> from 192.168.1.106:62020
> 2009-03-05 04:15:18,266 INFO org.apache.hadoop.hbase.master.RegionManager:
> assigning region web_pages,http://fortcollins.gaymonkey.com/,1235836722125 to
> server 192.168.1.106:62020
> 2009-03-05 04:15:30,150 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
> startKey: <>, server: 192.168.1.106:62020}
> 2009-03-05 04:15:30,276 INFO org.apache.hadoop.hbase.master.ServerManager:
> Received MSG_REPORT_CLOSE:
> web_pages,http://fortcollins.gaymonkey.com/,1235836722125:
> java.io.IOException: Could not obtain block: blk_5568212401457404905_251597
> file=/hbase/amsterdam_factory/web_pages/1263377107/oldlogfile.log
> ...
> {code}
> It does not recover, I have to manually delete the file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
