How many threads in your client? On Mon, Sep 29, 2014 at 4:05 PM, Henry Hung <ythu...@winbond.com> wrote:
> Hi Anoop, > > I receive this error in client side, and pretty sure the scan failed. > I'm using default caching, so it should be 100, right? > About scan time out period, I will try to set it higher, probably 1 hour. > > BTW, I'm using hbase 0.96.0. > > Best regards, > Henry > > -----Original Message----- > From: Anoop John [mailto:anoop.hb...@gmail.com] > Sent: Monday, September 29, 2014 3:13 PM > To: user@hbase.apache.org > Subject: Re: scan + filter failed with OutOfOrderScannerNextException > > Hi > Even when the RS throws this Exception, the client side will start a > new Scanner and retry. U just see this in log or the scan is failing > altogether? > What is the caching you use on Scan? When most of the rows are filtered > out at server side, it takes more time to fetch and return the 'caching' > number of rows back to client. By this time the client would have already > timed out and sent a retry. We allow such a retry will cause not getting > back all the results. This is a serious problem and this Exception way > helps there. You can try increasing the scan time out period at client > side. This should help. > > -Anoop- > > On Mon, Sep 29, 2014 at 12:35 PM, Henry Hung <ythu...@winbond.com> wrote: > > > Hi All, > > > > Is there a way to let scanner finish scanning all regions without > > throwing this kind of error? I'm using scan with filter MUST_PASS_ALL, > > and I observe that whenever the result data is smaller (let's say 10%) > > compare to another filter with larger result (let's say 80%), it always > failed. > > > > org.apache.hadoop.hbase.DoNotRetryIOException: Failed after retry of > > OutOfOrderScannerNextException: was there a rpc timeout? > > at > > org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:3 > > 91) > > at > > com.winbond.hbase.schema.mesperflogtime.MesPerfLogTimeResultSet.next(M > > esPerfLogTimeResultSet.java:58) > > at > > com.winbond.hbase.schema.mesperflogtime.MesPerfLogTimeResultSet.next(M > > esPerfLogTimeResultSet.java:1) at > > com.winbond.hbase.executor.Program.main(Program.java:106) > > Caused by: > > org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: > > org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: > > Expected > > nextCallSeq: 1 But the nextCallSeq got from client: 0; > request=scanner_id: > > 1594996183743944787 number_of_rows: 100 close_scanner: false > next_call_seq: > > 0 > > at > > org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer. > > java:3007) > > at > > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$ > > 2.callBlockingMethod(ClientProtos.java:26929) > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146) > > at > > org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851) > > > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > > Method) at > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructo > > rAccessorImpl.java:39) > > at > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCo > > nstructorAccessorImpl.java:27) at > > java.lang.reflect.Constructor.newInstance(Constructor.java:513) > > at > > org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteExcep > > tion.java:106) > > at > > org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteExce > > ption.java:95) > > at > > org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(Proto > > bufUtil.java:235) > > at > > org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.ja > > va:197) > > at > > org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.ja > > va:57) > > at > > org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRe > > tryingCaller.java:116) > > at > > org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRe > > tryingCaller.java:94) > > at > > org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:3 > > 43) > > ... 3 more > > Caused by: > > > org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException): > > org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: > > Expected > > nextCallSeq: 1 But the nextCallSeq got from client: 0; > request=scanner_id: > > 1594996183743944787 number_of_rows: 100 close_scanner: false > next_call_seq: > > 0 > > at > > org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer. > > java:3007) > > at > > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$ > > 2.callBlockingMethod(ClientProtos.java:26929) > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146) > > at > > org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851) > > > > at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1446) > > at > > org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.jav > > a:1650) > > at > > org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation > > .callBlockingMethod(RpcClient.java:1708) > > at > > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$ > > BlockingStub.scan(ClientProtos.java:27332) > > at > > org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.ja > > va:167) > > ... 7 more > > > > Best regards, > > Henry > > > > ________________________________ > > The privileged confidential information contained in this email is > > intended for use only by the addressees as indicated by the original > > sender of this email. If you are not the addressee indicated in this > > email or are not responsible for delivery of the email to such a > > person, please kindly reply to the sender indicating this fact and > > delete all copies of it from your computer and network server > > immediately. Your cooperation is highly appreciated. It is advised > > that any unauthorized use of confidential information of Winbond is > > strictly prohibited; and any information in this email irrelevant to > > the official business of Winbond shall be deemed as neither given nor > endorsed by Winbond. > > > > The privileged confidential information contained in this email is > intended for use only by the addressees as indicated by the original sender > of this email. If you are not the addressee indicated in this email or are > not responsible for delivery of the email to such a person, please kindly > reply to the sender indicating this fact and delete all copies of it from > your computer and network server immediately. Your cooperation is highly > appreciated. It is advised that any unauthorized use of confidential > information of Winbond is strictly prohibited; and any information in this > email irrelevant to the official business of Winbond shall be deemed as > neither given nor endorsed by Winbond. >
