Can you get that file from hdfs? > ./bin/hadoop fs -get /hbase/summary/1491233486/metrics/5046821377427277894
Does it look wholesome? Is it empty? What if you trace the life of that file in regionserver logs or probably better, over in namenode log? If you move this file aside, the region deploys? St.Ack On Mon, Mar 15, 2010 at 3:40 AM, Zheng Lv <lvzheng19800...@gmail.com> wrote: > Hello Everyone, > Recently we often got these in our client logs: > org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to > contact region server 172.16.1.208:60020 for region > summary,SITE_0000000032\x01pt\x0120100314000000\x01\x25E7\x258C\x25AE\x25E5\x258E\x25BF\x25E5\x2586\x2580\x25E9\x25B9\x25B0\x25E6\x2591\x25A9\x25E6\x2593\x25A6\x25E6\x259D\x2590\x25E6\x2596\x2599\x25E5\x258E\x2582\x2B\x25E6\x25B1\x25BD\x25E8\x25BD\x25A6\x25E9\x2585\x258D\x25E4\x25BB\x25B6\x25EF\x25BC\x258C\x25E5\x2598\x2580\x25E9\x2593\x2583\x25E9\x2593\x2583--\x25E7\x259C\x259F\x25E5\x25AE\x259E\x25E5\x25AE\x2589\x25E5\x2585\x25A8\x25E7\x259A\x2584\x25E7\x2594\x25B5\x25E8\x25AF\x259D\x25E3\x2580\x2581\x25E7\x25BD\x2591\x25E7\x25BB\x259C\x25E4\x25BA\x2592\x25E5\x258A\x25A8\x25E4\x25BA\x25A4\x25E5\x258F\x258B\x25E7\x25A4\x25BE\x25E5\x258C\x25BA\x25EF\x25BC\x2581,1268640385017, > row > 'SITE_0000000032\x01pt\x0120100315000000\x01\x2521\x25EF\x25BC\x2581\x25E9\x2594\x2580\x25E5\x2594\x25AE\x252F\x25E6\x2594\x25B6\x25E8\x25B4\x25AD\x25EF\x25BC\x2581VM700T\x2BVM700T\x2B\x25E5\x259B\x25BE\x25E5\x2583\x258F\x25E4\x25BF\x25A1\x25E5\x258F\x25B7\x25E4\x25BA\x25A7\x25E7\x2594\x259F\x25E5\x2599\x25A8\x2B\x25E7\x2594\x25B5\x25E5\x25AD\x2590\x25E6\x25B5\x258B\x25E9\x2587\x258F\x25E4\x25BB\x25AA\x25E5\x2599\x25A8\x25EF\x25BC\x258C\x25E5\x2598\x2580\x25E9\x2593\x2583\x25E9\x2593\x2583--\x25E7\x259C\x259F\x25E5\x25AE\x259E\x25E5\x25AE\x2589\x25E5\x2585\x25A8\x25E7\x259A\x2584\x25E7\x2594\x25B5\x25E8\x25AF\x259D\x25E3\x2580\x2581\x25E7\x25BD\x2591\x25E7\x25BB\x259C\x25E4\x25BA\x2592\x25E5\x258A\x25A8\x25E4\x25BA\x25A4\x25E5\x258F\x258B\x25E7\x25A4\x25BE\x25E5\x258C\x25BA\x25EF\x25BC\x2581', > but failed after 10 attempts. > Exceptions: > java.io.IOException: java.io.IOException: Cannot open filename > /hbase/summary/1491233486/metrics/5046821377427277894 > at > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1474) > at > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1800) > at > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1616) > at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1743) > at java.io.DataInputStream.read(DataInputStream.java:132) > at > org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:99) > at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100) > at org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1020) > at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:971) > at > org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.java:1304) > at > org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1186) > at > org.apache.hadoop.hbase.io.HalfHFileReader$1.seekTo(HalfHFileReader.java:207) > at > org.apache.hadoop.hbase.regionserver.StoreFileGetScan.getStoreFile(StoreFileGetScan.java:80) > at > org.apache.hadoop.hbase.regionserver.StoreFileGetScan.get(StoreFileGetScan.java:65) > at org.apache.hadoop.hbase.regionserver.Store.get(Store.java:1461) > at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2396) > at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2385) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1731) > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657) > at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) > Is there any way to fix this problem? Or is there anything we can do > even manually to relieve it? > Any suggestion? > Thank you. > LvZheng >
