Michael, This may be related to your other problem. We need to find the root of what seems to be data lost which is probable.
Is it production data? If so, we may have to fix your meta region. Any chance you drop on IRC? J-D On Thu, Mar 19, 2009 at 7:06 AM, Michael Dagaev <[email protected]> wrote: > Hi, all > > We are running a small hbase 0.18 cluster. It stopped working now. > The master log contains "java.io.IOException: HStoreScanner failed > construction" errors. > Restart (stop/start-hbase) does not help. Hadoop fsck says that > everything is fine. > > Has anybody run into such a problem ? > > Thank you for your cooperation, > M. > > 2009-03-19 10:57:01,306 WARN org.apache.hadoop.hbase.master.BaseScanner: > Scan one META region: {regionname: .META.,,1, startKey: <>, server: > 10.251.142.47:60020} > java.io.IOException: HStoreScanner failed construction > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.<init>(StoreFileScanner.java:70) > at > org.apache.hadoop.hbase.regionserver.HStoreScanner.<init>(HStoreScanner.java:70) > at > org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1916) > at > org.apache.hadoop.hbase.regionserver.HRegion$HScanner.<init>(HRegion.java:1954) > at > org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1345) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1175) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888) > Caused by: java.io.IOException: Could not obtain block: > blk_-2893342489105927361_2846854 > file=/hbase/.META./1028785192/info/mapfiles/3882153324238090640/index > at > org.apache.hadoop.dfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1462) > at > org.apache.hadoop.dfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1312) > at > org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1417) > at java.io.DataInputStream.readFully(DataInputStream.java:178) > at java.io.DataInputStream.readFully(DataInputStream.java:152) > at > org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1453) > at > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1431) > at > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1420) > at > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1415) > at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:292) > at > org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseReader.<init>(HStoreFile.java:632) > at > org.apache.hadoop.hbase.regionserver.HStoreFile$BloomFilterMapFile$Reader.<init>(HStoreFile.java:714) > at > org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:413) > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96) > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.<init>(StoreFileScanner.java:67) > ... 11 more > > at sun.reflect.GeneratedConstructorAccessor11.newInstance(Unknown > Source) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82) > at > org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:191) > at > org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:74) > at > org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129) > at > org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:139) > at org.apache.hadoop.hbase.Chore.run(Chore.java:62) >
