Hi, all
We are running a small hbase 0.18 cluster. It stopped working now.
The master log contains "java.io.IOException: HStoreScanner failed
construction" errors.
Restart (stop/start-hbase) does not help. Hadoop fsck says that
everything is fine.
Has anybody run into such a problem ?
Thank you for your cooperation,
M.
2009-03-19 10:57:01,306 WARN org.apache.hadoop.hbase.master.BaseScanner:
Scan one META region: {regionname: .META.,,1, startKey: <>, server:
10.251.142.47:60020}
java.io.IOException: HStoreScanner failed construction
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.<init>(StoreFileScanner.java:70)
at
org.apache.hadoop.hbase.regionserver.HStoreScanner.<init>(HStoreScanner.java:70)
at
org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1916)
at
org.apache.hadoop.hbase.regionserver.HRegion$HScanner.<init>(HRegion.java:1954)
at
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1345)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1175)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
Caused by: java.io.IOException: Could not obtain block:
blk_-2893342489105927361_2846854
file=/hbase/.META./1028785192/info/mapfiles/3882153324238090640/index
at
org.apache.hadoop.dfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1462)
at
org.apache.hadoop.dfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1312)
at
org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1417)
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at java.io.DataInputStream.readFully(DataInputStream.java:152)
at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1453)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1431)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1420)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1415)
at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:292)
at
org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseReader.<init>(HStoreFile.java:632)
at
org.apache.hadoop.hbase.regionserver.HStoreFile$BloomFilterMapFile$Reader.<init>(HStoreFile.java:714)
at
org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:413)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.<init>(StoreFileScanner.java:67)
... 11 more
at sun.reflect.GeneratedConstructorAccessor11.newInstance(Unknown
Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
at
org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:191)
at
org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:74)
at
org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129)
at
org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:139)
at org.apache.hadoop.hbase.Chore.run(Chore.java:62)
