Lars Hofhansl created PHOENIX-5007: -------------------------------------- Summary: DELETE FROM times out - even with large HBase timeout setting Key: PHOENIX-5007 URL: https://issues.apache.org/jira/browse/PHOENIX-5007 Project: Phoenix Issue Type: Bug Reporter: Lars Hofhansl
It seems there is currently no way to make a somewhat large DELETE FROM work without a timeout. "Large" here includes size equivalent to a single HBase region, which means we cannot write a M/R (or Spark) task to delete a lot of data. This is the exception: {code:java} 0: jdbc:phoenix:localhost> delete from test where v1 < 0.999; Error: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions: Fri Oct 26 15:36:07 PDT 2018, null, java.net.SocketTimeoutException: callTimeout=180000, callDuration=180311: Call to lhofhansl-wsl2/127.0.1.1:16201 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804, waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5., hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099 (state=08000,code=101) org.apache.phoenix.exception.PhoenixIOException: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions: Fri Oct 26 15:36:07 PDT 2018, null, java.net.SocketTimeoutException: callTimeout=180000, callDuration=180311: Call to lhofhansl-wsl2/127.0.1.1:16201 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804, waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5., hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099 at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:144) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1296) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1237) at org.apache.phoenix.iterate.ConcatResultIterator.getIterators(ConcatResultIterator.java:50) at org.apache.phoenix.iterate.ConcatResultIterator.currentIterator(ConcatResultIterator.java:97) at org.apache.phoenix.iterate.ConcatResultIterator.next(ConcatResultIterator.java:117) at org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:64) at org.apache.phoenix.iterate.UngroupedAggregatingResultIterator.next(UngroupedAggregatingResultIterator.java:39) at org.apache.phoenix.compile.DeleteCompiler$ServerSelectDeleteMutationPlan.execute(DeleteCompiler.java:791) at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:408) at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:391) at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:390) at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:378) at org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1825) at sqlline.Commands.execute(Commands.java:822) at sqlline.Commands.sql(Commands.java:732) at sqlline.SqlLine.dispatch(SqlLine.java:813) at sqlline.SqlLine.begin(SqlLine.java:686) at sqlline.SqlLine.start(SqlLine.java:398) at sqlline.SqlLine.main(SqlLine.java:291) Caused by: java.util.concurrent.ExecutionException: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions: Fri Oct 26 15:36:07 PDT 2018, null, java.net.SocketTimeoutException: callTimeout=180000, callDuration=180311: Call to lhofhansl-wsl2/127.0.1.1:16201 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804, waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5., hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099 at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:206) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1291) ... 19 more Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions: Fri Oct 26 15:36:07 PDT 2018, null, java.net.SocketTimeoutException: callTimeout=180000, callDuration=180311: Call to lhofhansl-wsl2/127.0.1.1:16201 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804, waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5., hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099 at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:144) at org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:171) at org.apache.phoenix.iterate.LookAheadResultIterator$1.advance(LookAheadResultIterator.java:47) at org.apache.phoenix.iterate.LookAheadResultIterator.init(LookAheadResultIterator.java:59) at org.apache.phoenix.iterate.LookAheadResultIterator.peek(LookAheadResultIterator.java:73) at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:135) at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:120) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions: Fri Oct 26 15:36:07 PDT 2018, null, java.net.SocketTimeoutException: callTimeout=180000, callDuration=180311: Call to lhofhansl-wsl2/127.0.1.1:16201 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804, waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5., hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099 at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:144) at org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:156) at org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:164) ... 10 more Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions: Fri Oct 26 15:36:07 PDT 2018, null, java.net.SocketTimeoutException: callTimeout=180000, callDuration=180311: Call to lhofhansl-wsl2/127.0.1.1:16201 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804, waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5., hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:329) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:242) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:277) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:438) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:312) at org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:147) ... 11 more Caused by: java.net.SocketTimeoutException: callTimeout=180000, callDuration=180311: Call to lhofhansl-wsl2/127.0.1.1:16201 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804, waitTime=59689, rpcTimetout=59678 row '�^�1' on table 'TEST' at region=TEST,\x82\x5CX-,1540590045706.b603f0cba6793295681c8ac18d93b1e5., hostname=lhofhansl-wsl2,16201,1540404153161, seqNum=211099 at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:178) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80) ... 3 more Caused by: java.io.IOException: Call to lhofhansl-wsl2/127.0.1.1:16201 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804, waitTime=59689, rpcTimetout=59678 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:389) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:94) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:409) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:405) at org.apache.hadoop.hbase.ipc.Call.setTimeout(Call.java:96) at org.apache.hadoop.hbase.ipc.RpcConnection$1.run(RpcConnection.java:188) at org.apache.phoenix.shaded.io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:625) at org.apache.phoenix.shaded.io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:700) at org.apache.phoenix.shaded.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:428) ... 1 more Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=804, waitTime=59689, rpcTimetout=59678 ... 5 more {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)