Thanks Lukas, that seems to have fixed it. I've set it both on my
cluster and locally to my local hbase installation for it to work.
Now I am getting a different problem when doing a query:
0: jdbc:phoenix:nn.lan> select count(word),word from words group by word
order by count(word) desc limit 10;
+------------------------------------------+------------------------------------------+
| COUNT(WORD) | WORD |
+------------------------------------------+------------------------------------------+
15/09/28 17:25:51 WARN client.ScannerCallable: Ignore, probably already
closed
org.apache.hadoop.hbase.UnknownScannerException:
org.apache.hadoop.hbase.UnknownScannerException: Name: 103, already closed?
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2223)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:322)
at
org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:357)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:195)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:142)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:61)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
at
org.apache.hadoop.hbase.client.StatsTrackingRpcRetryingCaller.callWithoutRetries(StatsTrackingRpcRetryingCaller.java:56)
at
org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
at
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:258)
at
org.apache.hadoop.hbase.client.ClientScanner.possiblyNextScanner(ClientScanner.java:241)
at
org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:532)
at
org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:364)
at
org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:55)
at
org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:107)
at
org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:240)
at
org.apache.phoenix.iterate.OrderedResultIterator.peek(OrderedResultIterator.java:263)
at
org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:112)
at
org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:100)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by:
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException):
org.apache.hadoop.hbase.UnknownScannerException: Name: 103, already closed?
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2223)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1196)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32651)
at
org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:355)
... 21 more
Do you know if it is related to phoenix.query.keepAliveMs ? (currently
my cluster is busy but will give it a try later on)
On 28/09/15 15:49, Lukáš Lalinský wrote:
You need to set "hbase.rpc.timeout" to the same value as you have for
"phoenix.query.timeoutMs".
It seems that in the pre-Apache version of Phoenix it was set
automatically:
https://issues.apache.org/jira/browse/PHOENIX-269?focusedCommentId=14681924&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14681924
Lukas
On Mon, Sep 28, 2015 at 4:41 PM, Konstantinos Kougios
<kostas.koug...@googlemail.com <mailto:kostas.koug...@googlemail.com>>
wrote:
I have a fairly big table on a noy-so-fairly-powerfull cluster, so
it takes a lot of time for queries to respond. I don't mind that
but it times out for many queries :
0: jdbc:phoenix:nn.lan> select count(*) from words;
+------------------------------------------+
| COUNT(1) |
+------------------------------------------+
java.lang.RuntimeException:
org.apache.phoenix.exception.PhoenixIOException:
org.apache.phoenix.exception.PhoenixIOException: Failed after
attempts=1, exceptions:
Mon Sep 28 15:32:25 BST 2015,
RpcRetryingCaller{globalStartTime=1443450685716, pause=100,
retries=1},
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
after attempts=1, exceptions:
Mon Sep 28 15:32:25 BST 2015,
RpcRetryingCaller{globalStartTime=1443450685716, pause=100,
retries=1},
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
after attempts=1, exceptions:
Mon Sep 28 15:32:25 BST 2015,
RpcRetryingCaller{globalStartTime=1443450685716, pause=100,
retries=1}, java.io.IOException: Call to d2.lan/192.168.0.30:16020
<http://192.168.0.30:16020> failed on local exception:
org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=842,
waitTime=60001, operationTimeout=60000 expired.
at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
at sqlline.TableOutputFormat.print(TableOutputFormat.java:33)
at sqlline.SqlLine.print(SqlLine.java:1653)
at sqlline.Commands.execute(Commands.java:833)
at sqlline.Commands.sql(Commands.java:732)
at sqlline.SqlLine.dispatch(SqlLine.java:808)
at sqlline.SqlLine.begin(SqlLine.java:681)
at sqlline.SqlLine.start(SqlLine.java:398)
at sqlline.SqlLine.main(SqlLine.java:292)
0: jdbc:phoenix:nn.lan> Closing:
org.apache.phoenix.jdbc.PhoenixConnection
I did try some settings on hbase-site without luck:
phoenix.query.timeoutMs =6000000 hbase.client.operation.timeout =1200000
hbase.client.backpressure.enabled =true hbase.client.retries.number =1
Any ideas how can this be fixed?
It seems the problem is that the timeout for iterating through
results is 60secs. I assume if it doesn't get 1 result within that
period, it times out. Since this is a count(*) query with only 1
row, it does timeout.
Thanks
