Re: Strange PrefixFilter behaviour on HBase 0.98.6-cdh5.2.0 OutOfOrderScannerNextException
I forgot to set firstRow for Scanner. Looks like HBase tried to scan the whole table. Value from FilterPrefix wasn't used. I supposed that prefix value could be pushed to scanner as a starting point, but not. 2015-04-06 18:45 GMT+02:00 Imants Cekusins ima...@gmail.com: may this be related: https://issues.apache.org/jira/browse/HBASE-11295 ?
Re: Strange PrefixFilter behaviour on HBase 0.98.6-cdh5.2.0 OutOfOrderScannerNextException
Yes, scan goes through entire table unless start row is set. does this explain the error though? Prefix filter should work even with scan beginning from 1st record, no? It would only take longer. Yes, it's explains. My table has 70M rows, and prefix filter should scan just for 10 starting from exact place, it takes millis to get response. 2015-04-06 22:54 GMT+02:00 Serega Sheypak serega.shey...@gmail.com: I forgot to set firstRow for Scanner. Looks like HBase tried to scan the whole table. Value from FilterPrefix wasn't used. I supposed that prefix value could be pushed to scanner as a starting point, but not. 2015-04-06 18:45 GMT+02:00 Imants Cekusins ima...@gmail.com: may this be related: https://issues.apache.org/jira/browse/HBASE-11295 ?
Re: Strange PrefixFilter behaviour on HBase 0.98.6-cdh5.2.0 OutOfOrderScannerNextException
Hi Serega, Yes, you need to set start row when using Prefix filter. Another smart trick for prefix filter: Set the prefix as start-row of scan and set the next byteorder successor(next bigger byte array with same number of bytes) as stop row of your scan. For example(using chars for simplicity): if Prefix= abc, then startRow will be: abc and stopRow:abd I have been using this trick for prefix based on alphanumeric string for many years. Thanks, Anil Gupta On Mon, Apr 6, 2015 at 2:18 PM, Serega Sheypak serega.shey...@gmail.com wrote: Yes, scan goes through entire table unless start row is set. does this explain the error though? Prefix filter should work even with scan beginning from 1st record, no? It would only take longer. Yes, it's explains. My table has 70M rows, and prefix filter should scan just for 10 starting from exact place, it takes millis to get response. 2015-04-06 22:54 GMT+02:00 Serega Sheypak serega.shey...@gmail.com: I forgot to set firstRow for Scanner. Looks like HBase tried to scan the whole table. Value from FilterPrefix wasn't used. I supposed that prefix value could be pushed to scanner as a starting point, but not. 2015-04-06 18:45 GMT+02:00 Imants Cekusins ima...@gmail.com: may this be related: https://issues.apache.org/jira/browse/HBASE-11295 ? -- Thanks Regards, Anil Gupta
Strange PrefixFilter behaviour on HBase 0.98.6-cdh5.2.0 OutOfOrderScannerNextException
Hi, I'm trying to use PrefixFilter for the RowKey.
My rowKey consists of 3 parts, actually it's composite.
I do provide first part of key to scan all rows starting from prefix. There
should be less than 10 rowkeys for each prefix, since prefix is md5 hash.
I have itests for this part of code, it runs without any problems, failure
happens on real data of course. Can't get what Im doing wrong.
private Scan createCrossIdRowKeyPrefixFilterScanner(byte[] prefix, int limit){
Scan scan = new Scan();
scan.addColumn(CF_B, CQ_B);
scan.setMaxResultSize(limit);
scan.setBatch(BATCH);
scan.setMaxVersions(SINGLE_VERSION);
scan.setCaching(CACHING);
PrefixFilter prefixFilter = new PrefixFilter(prefix);
scan.setFilter(prefixFilter);
return scan;
}
and invocation:
stopWatch.start();
resultScanner = hTable.getScanner(rowKeyPrefixScanner);
Result[] results = resultScanner.next(limit);
stopWatch.stop();
LOG.debug(Took [+(stopWatch.getTime()/1000L)+]sec to scan for key prefix);
return parseUsers(results);
And I get:
used by: org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException:
org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected
nextCallSeq: 1 But the nextCallSeq got from client: 0; request=scanner_id:
225177 number_of_rows: 10 close_scanner: false next_call_seq: 0
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3193)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29587)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
at java.lang.Thread.run(Thread.java:745)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:304)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:204)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:59)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:114)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:90)
at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:355)
... 14 more
Caused by:
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException):
org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected
nextCallSeq: 1 But the nextCallSeq got from client: 0; request=scanner_id:
225177 number_of_rows: 10 close_scanner: false next_call_seq: 0
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3193)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29587)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
at java.lang.Thread.run(Thread.java:745)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:29990)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:174)
Re: Strange PrefixFilter behaviour on HBase 0.98.6-cdh5.2.0 OutOfOrderScannerNextException
Looks like I didn't set startRow for the scanner...
2015-04-06 17:04 GMT+02:00 Serega Sheypak serega.shey...@gmail.com:
Hi, I'm trying to use PrefixFilter for the RowKey.
My rowKey consists of 3 parts, actually it's composite.
I do provide first part of key to scan all rows starting from prefix.
There should be less than 10 rowkeys for each prefix, since prefix is md5
hash.
I have itests for this part of code, it runs without any problems, failure
happens on real data of course. Can't get what Im doing wrong.
private Scan createCrossIdRowKeyPrefixFilterScanner(byte[] prefix, int limit){
Scan scan = new Scan();
scan.addColumn(CF_B, CQ_B);
scan.setMaxResultSize(limit);
scan.setBatch(BATCH);
scan.setMaxVersions(SINGLE_VERSION);
scan.setCaching(CACHING);
PrefixFilter prefixFilter = new PrefixFilter(prefix);
scan.setFilter(prefixFilter);
return scan;
}
and invocation:
stopWatch.start();
resultScanner = hTable.getScanner(rowKeyPrefixScanner);
Result[] results = resultScanner.next(limit);
stopWatch.stop();
LOG.debug(Took [+(stopWatch.getTime()/1000L)+]sec to scan for key prefix);
return parseUsers(results);
And I get:
used by:
org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException:
org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected
nextCallSeq: 1 But the nextCallSeq got from client: 0; request=scanner_id:
225177 number_of_rows: 10 close_scanner: false next_call_seq: 0
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3193)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29587)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
at java.lang.Thread.run(Thread.java:745)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:304)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:204)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:59)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:114)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:90)
at
org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:355)
... 14 more
Caused by:
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException):
org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected
nextCallSeq: 1 But the nextCallSeq got from client: 0; request=scanner_id:
225177 number_of_rows: 10 close_scanner: false next_call_seq: 0
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3193)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29587)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
at java.lang.Thread.run(Thread.java:745)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:29990)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:174)
Re: Strange PrefixFilter behaviour on HBase 0.98.6-cdh5.2.0 OutOfOrderScannerNextException
may this be related: https://issues.apache.org/jira/browse/HBASE-11295 ?
