HBase has started throwing WrongRegionExceptions at me when trying to access
certain regions. I'm guessing that the META table has somehow gone out of
sync with reality. I've tried compacting and I've tried restarting, but the
problem does not go away. The errors are always on the same regions. Has
anyone else seen this and succeeded at getting their table back into working
order?
*Example get:*
org.apache.hadoop.hbase.regionserver.WrongRegionException:
org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row out
of range for HRegion
crawled_pages,r:http:\x2F\x2Fcom.xxxx.yyyy\x2Frestaurants\x2Fall-areas\x2Fbeverly-hills\x2Fall-cuisines\x2Ftags\x2Flunch\x2F2\x2F,1256932686084,
startKey='r:http:\x2F\x2Fcom.xxxx.yyyy\x2Frestaurants\x2Fall-areas\x2Fbeverly-hills\x2Fall-cuisines\x2Ftags\x2Flunch\x2F2\x2F',
getEndKey()='r:http:\x2F\x2Fcom.xxxx.yyyy\x2Frestaurants\x2Fall-areas\x2Fhermosa-beach\x2Fall-cuisines\x2Ftags\x2Foutdoor-dining\x2F',
row='r:http:\x2F\x2Fcom.xxxx.yyyy\x2Frestaurants\x2Fall-areas\x2Finglewood\x2Fall-cuisines\x2F'
at
org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:1522)
at
org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1554)
at
org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1622)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2278)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1785)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
*Example put:
*
put 'crawled_pages','r:
http://com.xxxx.yyyy/restaurants/all-areas/inglewood/all-cuisines/',
'curi:test','test'
NativeException: org.apache.hadoop.hbase.client.RetriesExhaustedException:
Trying to contact region server Some server, retryOnlyOne=true, index=0,
islastrow=true, tries=4, numtries=5, i=0, listsize=1,
region=crawled_pages,r:http:\x2F\x2Fcom.xxxx.yyyy\x2Frestaurants\x2Fall-areas\x2Fbeverly-hills\x2Fall-cuisines\x2Ftags\x2Flunch\x2F2\x2F,1256932686084
for region
crawled_pages,r:http:\x2F\x2Fcom.xxxx.yyyy\x2Frestaurants\x2Fall-areas\x2Fbeverly-hills\x2Fall-cuisines\x2Ftags\x2Flunch\x2F2\x2F,1256932686084,
row
'r:http:\x2F\x2Fcom.xxxx.yyyy\x2Frestaurants\x2Fall-areas\x2Finglewood\x2Fall-cuisines\x2F',
but failed after 5 attempts.
Exceptions:
from org/apache/hadoop/hbase/client/HConnectionManager.java:1119:in
`process'
from org/apache/hadoop/hbase/client/HConnectionManager.java:1200:in
`processBatchOfRows'
from org/apache/hadoop/hbase/client/HTable.java:605:in `flushCommits'
from org/apache/hadoop/hbase/client/HTable.java:470:in `put'
from org/apache/hadoop/hbase/client/HTable.java:1761:in `commit'
from org/apache/hadoop/hbase/client/HTable.java:1742:in `commit'
from sun/reflect/NativeMethodAccessorImpl.java:-2:in `invoke0'
from sun/reflect/NativeMethodAccessorImpl.java:39:in `invoke'
from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
from java/lang/reflect/Method.java:597:in `invoke'
from org/jruby/javasupport/JavaMethod.java:298:in
`invokeWithExceptionHandling'
from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
from org/jruby/java/invokers/InstanceMethodInvoker.java:44:in `call'
from org/jruby/runtime/callsite/CachingCallSite.java:110:in `call'
from org/jruby/ast/CallOneArgNode.java:57:in `interpret'
from org/jruby/ast/NewlineNode.java:104:in `interpret'
