Looks like bug in FastDiffDeltaEncoder triggered by whatever the current form of the target file. Can you figure which file it is (going by the Get coordinates?). I suppose the compactor is running into the same problem (was thinking a major compaction might get you over this hump). You could make a copy of the problematic file and play with it standalone to see if can figure the bug. Failing that, post to a JIRA if you yourself can't figure it so someone else might have a go at it?
Thanks, S On Wed, Dec 5, 2018 at 11:22 AM William Shen <[email protected]> wrote: > Hi there, > > We've recently encountered issue retrieving data from our HBase cluster, > and have not had much luck troubleshooting the issue. We narrowed down our > issue to a single GET, which appears to be caused by > FastDiffDeltaEncoder.java running into java.lang.IndexOutOfBoundsException. > Has anyone encountered similar issues before, or does anyone have > experience troubleshooting issues such as this one? Any help would be much > appreciated! We are running 1.2.0-cdh5.9.2, and the GET in question is: > > hbase(main):004:0> get 'qa2.ADGROUPS', > > "\x05\x80\x00\x00\x00\x00\x1F\x54\x9C\x80\x00\x00\x00\x00\x1C\x7D\x45\x00\x04\x80\x00\x00\x00\x00\x1D\x0F\x19\x80\x00\x00\x00\x00\x4A\x64\x6F\x80\x00\x00\x00\x01\xD9\xDB\xCE" > > COLUMN CELL > > > > > ERROR: java.io.IOException > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2215) > > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109) > > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185) > > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165) > > Caused by: java.lang.IndexOutOfBoundsException > > at java.nio.Buffer.checkBounds(Buffer.java:567) > > at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:149) > > at > org.apache.hadoop.hbase.io > .encoding.FastDiffDeltaEncoder$1.decode(FastDiffDeltaEncoder.java:465) > > at > org.apache.hadoop.hbase.io > .encoding.FastDiffDeltaEncoder$1.decodeNext(FastDiffDeltaEncoder.java:516) > > at > org.apache.hadoop.hbase.io > .encoding.BufferedDataBlockEncoder$BufferedEncodedSeeker.next(BufferedDataBlockEncoder.java:618) > > at > org.apache.hadoop.hbase.io > .hfile.HFileReaderV2$EncodedScannerV2.next(HFileReaderV2.java:1277) > > at > > org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:180) > > at > > org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:108) > > at > > org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:588) > > at > > org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:147) > > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5706) > > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:5865) > > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5643) > > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:5620) > > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:5606) > > at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6801) > > at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6779) > > at > > org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2029) > > at > > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644) > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170) > > ... 3 more > > > Thank you very much in advance! >
