Hi, > We sometimes saw over 5 second delay running get in hbase 0.20.1 shell: > > hbase(main):002:0> get 'ruletable', 'ca.tsn.www' > 0 row(s) in 10.1330 seconds > > From our 3 region servers there are a lot of such messages: >
> 2010-03-12 00:00:00,996 INFO [regionserver/10.10.31.135:60020] > regionserver.HRegionServer(493): MSG_REGION_CLOSE: > crawltable,com.pandora.www:http\x2Finclude\x2FlyricsAdEmbed.html\x3Fgenre\x3Delectronica\x26artist\x3DR273847\x26webname\x3D\x26sz\x3D2000x8\x26ord\x3D125823029226371645\x26tile\x3D3\x26_id\x3Dbottom_leaderboard_container,1266944566406: > Overloaded > 2010-03-12 00:00:00,997 INFO [regionserver/10.10.31.135:60020] > regionserver.HRegionServer(493): MSG_REGION_CLOSE: > domaincrawltable,,1268098564908: Overloaded > > grep Overloaded hbase-hbaseadmin-regionserver.log | wc > 40428 343638 11523230 > > grep Overloaded hbase-hbaseadmin-regionserver.log | wc > 40430 343655 11307703 > > grep Overloaded hbase-hbaseadmin-regionserver.log | wc > 40466 343961 11340379 > > Was the slow response due to the load balancing ? > The strange thing was that after several quick responses I would see: hbase(main):004:0> get 'ruletable', 'com.about.acne' COLUMN CELL lpm_1.0:category timestamp=1268347483823, value= http://acne.about.com\t9002:0.86580 086\thost\n 1 row(s) in 0.0040 seconds hbase(main):005:0> get 'ruletable', 'com.about.acne' COLUMN CELL lpm_1.0:category timestamp=1268347483823, value= http://acne.about.com\t9002:0.86580 086\thost\n 1 row(s) in 0.0040 seconds hbase(main):006:0> get 'ruletable', 'com.about.acne' NativeException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.10.31.136:60020 for region ruletable,,1268083966723, row 'com.about.acne', but failed after 5 attempts. Exceptions: org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: ruletable,,1268083966723 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2307) at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1784) at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) But 006:60010/master.jsp refreshes quickly and shows all three regionservers. Don't know why hbase shell encountered NSRE. > > Thanks
