Seems like something weird is going on with your regionservers and balancing.
Can you post big snippets from the regionserver and master logs? Have you checked to see what's going on? Is there repetitive balancing going on that never seems to reach steady-state? How many regions and how many nodes on which version of HBase? > -----Original Message----- > From: Ted Yu [mailto:yuzhih...@gmail.com] > Sent: Friday, March 12, 2010 8:24 PM > To: hbase-user@hadoop.apache.org > Subject: slow response in hbase shell > > Hi, > > > We sometimes saw over 5 second delay running get in hbase 0.20.1 > shell: > > > > hbase(main):002:0> get 'ruletable', 'ca.tsn.www' > > 0 row(s) in 10.1330 seconds > > > > From our 3 region servers there are a lot of such messages: > > > > > > 2010-03-12 00:00:00,996 INFO [regionserver/10.10.31.135:60020] > > regionserver.HRegionServer(493): MSG_REGION_CLOSE: > > > crawltable,com.pandora.www:http\x2Finclude\x2FlyricsAdEmbed.html\x3Fgen > re\x3Delectronica\x26artist\x3DR273847\x26webname\x3D\x26sz\x3D2000x8\x > 26ord\x3D125823029226371645\x26tile\x3D3\x26_id\x3Dbottom_leaderboard_c > ontainer,1266944566406: > > Overloaded > > 2010-03-12 00:00:00,997 INFO [regionserver/10.10.31.135:60020] > > regionserver.HRegionServer(493): MSG_REGION_CLOSE: > > domaincrawltable,,1268098564908: Overloaded > > > > grep Overloaded hbase-hbaseadmin-regionserver.log | wc > > 40428 343638 11523230 > > > > grep Overloaded hbase-hbaseadmin-regionserver.log | wc > > 40430 343655 11307703 > > > > grep Overloaded hbase-hbaseadmin-regionserver.log | wc > > 40466 343961 11340379 > > > > Was the slow response due to the load balancing ? > > > > The strange thing was that after several quick responses I would see: > > hbase(main):004:0> get 'ruletable', 'com.about.acne' > COLUMN > CELL > lpm_1.0:category timestamp=1268347483823, value= > http://acne.about.com\t9002:0.86580 086\thost\n > 1 row(s) in 0.0040 seconds > hbase(main):005:0> get 'ruletable', 'com.about.acne' > COLUMN > CELL > lpm_1.0:category timestamp=1268347483823, value= > http://acne.about.com\t9002:0.86580 086\thost\n > 1 row(s) in 0.0040 seconds > hbase(main):006:0> get 'ruletable', 'com.about.acne' > NativeException: > org.apache.hadoop.hbase.client.RetriesExhaustedException: > Trying to contact region server 10.10.31.136:60020 for region > ruletable,,1268083966723, row 'com.about.acne', but failed after 5 > attempts. > Exceptions: > org.apache.hadoop.hbase.NotServingRegionException: > org.apache.hadoop.hbase.NotServingRegionException: > ruletable,,1268083966723 > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionSer > ver.java:2307) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.ja > va:1784) > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso > rImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:91 > 5) > > But 006:60010/master.jsp refreshes quickly and shows all three > regionservers. > Don't know why hbase shell encountered NSRE. > > > > > Thanks
