Can you get history of this region from master log and pastebin it (paste result of a grep of 84162fcdf083fe4736c39571223cb029 to pastebin)? St.Ack
On Tue, Sep 28, 2010 at 10:53 AM, Jack Levin <magn...@gmail.com> wrote: > I am using rest and getting following errors on some insertions: > > > 2010-09-28 10:51:07,215 ERROR > org.apache.hadoop.hbase.regionserver.HRegionServer: > org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested > row out of range for HRegion > img833,samp754.png,1285522187411.84162fcdf083fe4736c39571223cb029., > startKey='samp754.png', getEndKey()='screenshot205.png', > row='sf6c.jpg' > at > org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:1967) > at > org.apache.hadoop.hbase.regionserver.HRegion.internalObtainRowLock(HRegion.java:2030) > at > org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:2120) > at > org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1367) > at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1321) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1803) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:2464) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027) > > And from REST log: > > > 2010-09-28 10:50:26,308 DEBUG > org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Failed > execution of all on mtab5.prod.imageshack.com:60020 > org.apache.hadoop.hbase.regionserver.WrongRegionException: > org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested > row out of range for HRegion > img833,samp754.png,1285522187411.84162fcdf083fe4736c39571223cb029., > startKey='samp754.png', getEndKey()='screenshot205.png', > row='sf6c.jpg' > at > org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:1967) > at > org.apache.hadoop.hbase.regionserver.HRegion.internalObtainRowLock(HRegion.java:2030) > at > org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:2120) > at > org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1367) > at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1321) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1803) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:2464) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027) > > at sun.reflect.GeneratedConstructorAccessor24.newInstance(Unknown > Source) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.translateException(HConnectionManager.java:1552) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithoutRetries(HConnectionManager.java:1162) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers$6.call(HConnectionManager.java:1530) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers$6.call(HConnectionManager.java:1528) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > > It looks like even after restarting REST I get the same error, could > this been that zookeepers cached the wrong location of the region and > its not on the same regionserver? Are there other methods to debug? > > -Jack >
