Hey Vidhya, Can you get a jstack of the frozen server?
-Todd On Thu, Dec 23, 2010 at 4:55 AM, Vidhyashankar Venkataraman < [email protected]> wrote: > > I have a periodic process that bulk incremental loads a set of files each > time into my db. The last few runs have been resulting in bulk load failures > complaining of RetriesExhausted. (I am running the last release of 0.89) > > > > Exception in thread "main" > org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact > region server b5120229.yst.yahoo.net:60020 for region > vidhyash_test,r:com#mop#lady!/beauty/hair/2010-07-11/136898.shtml!http,1292936192308.7f7e7521764636e108de079799ad9e44., > row 'r:com#mop#lady!/star/2010-06-10/131380.shtml!http', but failed after 10 > attempts. > > > > > I looked into the logs of the particular regionserver and I noticed that > one of IPC handlers complains of an output error and throws an exception and > after that, all it does is just validate hfiles whenever there is an attempt > at a bulk incremental load. And, it isnt accessible even through the web > interface.. But the region server is still alive according to the master/zk. > Can you let me know what the problem is? (Below is the log where the problem > arose). > > > > > > 2010-12-22 21:31:57,679 INFO org.apache.hadoop.hbase.regionserver.Store: > Validating hfile at > /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/3632567128036272117 > for inclusion in store metadata region > vidhyash_test,r:jp#co#yahoo#auctions#page17!/jp/show/reviews?aID=v54204554!http,1292936187861.9ad1eccc9cf7f82282757e2b82c45559.2010-12-22 > 21:31:57,680 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block > blk_8607417804107886121_8839017 from any node: java.io.IOException: No live > nodes contain current block > 2010-12-22 21:31:59,610 DEBUG > org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=19.67 MB, > free=2.32 GB, max=2.34 GB, blocks=0, accesses=1654363, hits=0, > hitRatio=0.00%%, evictions=0, evicted=0, evictedPerRun=NaN2010-12-22 > 21:32:00,684 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block > blk_8607417804107886121_8839017 from any node: java.io.IOException: No > live nodes contain current block2010-12-22 21:32:03,687 INFO > org.apache.hadoop.hdfs.DFSClient: Could not obtain block > blk_8607417804107886121_8839017 from any node: java.io.IOException: No live > nodes contain current block > 2010-12-22 21:32:06,691 DEBUG org.apache.hadoop.hbase.regionserver.Store: > HFile bounds: > first=r:jp#co#yahoo#auctions#page17#www!/jp/auction/v12791536!http > last=r:jp#co#yahoo#auctions#page19!/jp/show/discussion?aID=x120219484&u=chikyuud!http2010-12-22 > 21:32:06,691 DEBUG org.apache.hadoop.hbase.regionserver.Store: Region > bounds: first=r:jp#co#yahoo#auctions#page17!/jp/show/reviews?aID=v54204554!h > ttp > last=r:jp#co#yahoo#auctions#page19!/jp/show/reviews?aID=x144625371!http2010-12-22 > 21:32:06,691 INFO org.apache.hadoop.hbase.regionserver.Store: Renaming bulk > load file > /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/3632567128036272117 > to hdfs:// > b5120202.yst.yahoo.net:4600/hbase/vidhyash_test/9ad1eccc9cf7f82282757e2b82c45559/metadata/643287123673932876 > 2010-12-22 21:32:06,695 INFO org.apache.hadoop.hbase.regionserver.Store: > Moved hfile > /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/3632 > 567128036272117 into store directory hdfs:// > b5120202.yst.yahoo.net:4600/hbase/vidhyash_test/9ad1eccc9cf7f82282757e2b82c45559/metadata- > updating store file list.2010-12-22 21:32:06,695 INFO > org.apache.hadoop.hbase.regionserver.Store: Successfully loaded store file > /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/3632567128036272117 > into store metadata (new location: hdfs:// > b5120202.yst.yahoo.net:4600/hbase/vidhyash_test/9ad1eccc9cf7f82282757e2b82c45559/metadata/643287123673932876)2010-12-2221:32:06,695 > WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, > call > bulkLoadHFile(/user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/3632567128036272117, > [...@2b6105a8, [...@6eba6ed7) from 74.6.71.45:52379: output error2010-12-22 > 21:32:06,696 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 27 > on 60020 caught: java.nio.channels.ClosedChannelException > at > sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126) > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324) > at > org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1224) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:708) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:773) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1035) > 2010-12-22 21:35:33,312 INFO org.apache.hadoop.hbase.regionserver.Store: > Validating hfile at > /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/content/2820295752884341300 > for inclusion in store content region > vidhyash_test,r:com#careerbuilder#engineering!/en.ic/Texas_Senior-Engineer.htm!http,1292936194810.0658e436cc625b2c786ef80a5dbe4203. > 2010-12-22 21:35:41,324 INFO org.apache.hadoop.hbase.regionserver.Store: > Validating hfile at > /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/content/2820295752884341300 > for inclusion in store content region > vidhyash_test,r:com#careerbuilder#engineering!/en.ic/Texas_Senior-Engineer.htm!http,1292936194810.0658e436cc625b2c786ef80a5dbe4203. > 2010-12-22 21:35:54,128 INFO org.apache.hadoop.hbase.regionserver.Store: > Validating hfile at > /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/content/291138288447336298 > for inclusion in store content region > vidhyash_test,r:la#net#kpl#www!/english/news/edn13.htm!http,1292936187014.695817b0e3a8c894240668db0448f8bf. > 2010-12-22 21:35:54,529 INFO org.apache.hadoop.hbase.regionserver.Store: > Validating hfile at > /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/2731042733857819920 > for inclusion in store metadata region > vidhyash_test,r:com#homebargear#www!/irish-gift-set.html!http,1292936193313.99fa66ab17756ce4ce5ba3a0d8ee8799. > 2010-12-22 21:35:54,823 INFO org.apache.hadoop.hbase.regionserver.Store: > Validating hfile at > /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/1898525064358474151 > for inclusion in store metadata region > vidhyash_test,r:fr#dazibaoueb#www!/tag.php?tag=DESINFORMATION!http,1292936188821.85b144d3f968029a903506bdb4e60cf7. > 2010-12-22 21:35:55,198 INFO org.apache.hadoop.hbase.regionserver.Store: > Validating hfile at > /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/content/2942947977847604920 > for inclusion in store content region > vidhyash_test,r:com#pld#mosc95#www!/projects02/ww2/germantanks.html!http,1292936191711.8e3d5df4c05c60f674a7b78474f83eea. > 2010-12-22 21:35:57,299 INFO org.apache.hadoop.hbase.regionserver.Store: > Validating hfile at > /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/2776824162022751481 > for inclusion in store metadata region > vidhyash_test,r:cn#com#sina#news!/c/2006-07-12/192710404866.shtml!http,1292936195947.b3b27d1cc94a6378ab4da90acad4efbf. > 2010-12-22 21:35:57,547 INFO org.apache.hadoop.hbase.regionserver.Store: > Validating hfile at > /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/1753280038544504583 > for inclusion in store metadata region > vidhyash_test,r:com#yoka#space!/blog/34726!http,1292936189782.714fc4e266abca11f578fd90a3561337. > > -- Todd Lipcon Software Engineer, Cloudera
