after I increase hbase.regionserver.handler.count value , the error was gone.
2016-05-12 15:57 GMT+08:00 kevin <kiss.kevin...@gmail.com>: > hi,all: > I try a concurrent query with 30 threads.and two thread fail with error: > ........ > > WARN :[2016-05-12 > 15:03:50]org.apache.phoenix.execute.HashJoinPlan$HashSubPlan.execute(HashJoinPlan.java:361) > : Hash plan [2] execution seems too slow. Earlier hash cache(s) might have > expired on servers. > WARN :[2016-05-12 > 15:03:51]org.apache.phoenix.execute.HashJoinPlan$HashSubPlan.execute(HashJoinPlan.java:361) > : Hash plan [2] execution seems too slow. Earlier hash cache(s) might have > expired on servers. > ........ > thread : thread2, get rows 5181 > thread : thread2, exec during 39 second > thread : thread1, get rows 5181 > thread : thread1, exec during 41 second > thread : thread3, get rows 5181 > thread : thread3, exec during 39 second > thread : thread7, get rows 5181 > thread : thread7, exec during 39 second > thread : thread5, get rows 5181 > thread : thread5, exec during 39 second > thread : thread6, get rows 5181 > ........ > > Exception in thread "pool-1-thread-4" > org.springframework.dao.DataAccessResourceFailureException: > StatementCallback; SQL [SELECT > i_item_id, > AVG(ss_quantity) agg1, > AVG(ss_list_price) agg2, > AVG(ss_coupon_amt) agg3, > AVG(ss_sales_price) agg4 > FROM > store_sales > JOIN > customer_demographics > ON > store_sales.ss_cdemo_sk = customer_demographics.cd_demo_sk > JOIN > date_dim > ON > store_sales.ss_sold_date_sk = date_dim.d_date_sk > JOIN > item > ON > store_sales.ss_item_sk = item.i_item_sk > JOIN > promotion > ON > store_sales.ss_promo_sk = promotion.p_promo_sk > WHERE > cd_gender = 'F' > AND cd_marital_status = 'W' > AND cd_education_status = 'Primary' > AND ( > p_channel_email = 'N' > OR p_channel_event = 'N') > AND d_year = 1998 > GROUP BY > i_item_id > ORDER BY > i_item_id limit 50000]; org.apache.phoenix.exception.PhoenixIOException: > org.apache.hadoop.hbase.DoNotRetryIOException: Could not find hash cache > for joinId: �#"�� Q . The cache might have expired and have been removed. > at > org.apache.phoenix.coprocessor.HashJoinRegionScanner.<init>(HashJoinRegionScanner.java:95) > at > org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver.doPostScannerOpen(GroupedAggregateRegionObserver.java:150) > at > org.apache.phoenix.coprocessor.BaseScannerRegionObserver.postScannerOpen(BaseScannerRegionObserver.java:178) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$52.call(RegionCoprocessorHost.java:1340) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1656) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1733) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperationWithResult(RegionCoprocessorHost.java:1695) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postScannerOpen(RegionCoprocessorHost.java:1335) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3250) > at > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:31068) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2147) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:105) > at > org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) > at java.lang.Thread.run(Thread.java:745) > ; nested exception is org.apache.phoenix.exception.PhoenixIOException: > org.apache.phoenix.exception.PhoenixIOException: > org.apache.hadoop.hbase.DoNotRetryIOException: Could not find hash cache > for joinId: �#"�� Q . The cache might have expired and have been removed. > at > org.apache.phoenix.coprocessor.HashJoinRegionScanner.<init>(HashJoinRegionScanner.java:95) > at > org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver.doPostScannerOpen(GroupedAggregateRegionObserver.java:150) > at > org.apache.phoenix.coprocessor.BaseScannerRegionObserver.postScannerOpen(BaseScannerRegionObserver.java:178) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$52.call(RegionCoprocessorHost.java:1340) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1656) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1733) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperationWithResult(RegionCoprocessorHost.java:1695) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postScannerOpen(RegionCoprocessorHost.java:1335) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3250) > at > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:31068) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2147) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:105) > at > org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) > at java.lang.Thread.run(Thread.java:745) > > at > org.springframework.jdbc.support.SQLStateSQLExceptionTranslator.doTranslate(SQLStateSQLExceptionTranslator.java:105) > at > org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73) > at > org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) > at > org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) > at > org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:415) > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:470) > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:480) > at > org.springframework.jdbc.core.JdbcTemplate.queryForList(JdbcTemplate.java:506) > at com.seaboxdata.demo.client.ThreadDemo.run(ThreadDemo.scala:11) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.phoenix.exception.PhoenixIOException: > org.apache.phoenix.exception.PhoenixIOException: > org.apache.hadoop.hbase.DoNotRetryIOException: Could not find hash cache > for joinId: �#"�� Q . The cache might have expired and have been removed. > at > org.apache.phoenix.coprocessor.HashJoinRegionScanner.<init>(HashJoinRegionScanner.java:95) > at > org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver.doPostScannerOpen(GroupedAggregateRegionObserver.java:150) > at > org.apache.phoenix.coprocessor.BaseScannerRegionObserver.postScannerOpen(BaseScannerRegionObserver.java:178) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$52.call(RegionCoprocessorHost.java:1340) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1656) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1733) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperationWithResult(RegionCoprocessorHost.java:1695) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postScannerOpen(RegionCoprocessorHost.java:1335) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3250) > at > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:31068) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2147) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:105) > at > org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) > at java.lang.Thread.run(Thread.java:745) > > at > org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:108) > at > org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:558) > at > org.apache.phoenix.iterate.MergeSortResultIterator.getIterators(MergeSortResultIterator.java:48) > at > org.apache.phoenix.iterate.MergeSortResultIterator.minIterator(MergeSortResultIterator.java:84) > at > org.apache.phoenix.iterate.MergeSortResultIterator.next(MergeSortResultIterator.java:111) > at > org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:64) > at > org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44) > at > org.apache.phoenix.iterate.LimitingResultIterator.next(LimitingResultIterator.java:47) > at > org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44) > at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:771) > at > org.apache.commons.dbcp.DelegatingResultSet.next(DelegatingResultSet.java:207) > at > org.apache.commons.dbcp.DelegatingResultSet.next(DelegatingResultSet.java:207) > at > org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:92) > at > org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:60) > at > org.springframework.jdbc.core.JdbcTemplate$1QueryStatementCallback.doInStatement(JdbcTemplate.java:459) > at > org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:404) > ... 7 more > Caused by: java.util.concurrent.ExecutionException: > org.apache.phoenix.exception.PhoenixIOException: > org.apache.hadoop.hbase.DoNotRetryIOException: Could not find hash cache > for joinId: �#"�� Q . The cache might have expired and have been removed. > at > org.apache.phoenix.coprocessor.HashJoinRegionScanner.<init>(HashJoinRegionScanner.java:95) > at > org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver.doPostScannerOpen(GroupedAggregateRegionObserver.java:150) > at > org.apache.phoenix.coprocessor.BaseScannerRegionObserver.postScannerOpen(BaseScannerRegionObserver.java:178) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$52.call(RegionCoprocessorHost.java:1340) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1656) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1733) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperationWithResult(RegionCoprocessorHost.java:1695) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postScannerOpen(RegionCoprocessorHost.java:1335) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3250) > at > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:31068) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2147) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:105) > at > org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) > at java.lang.Thread.run(Thread.java:745) > > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:206) > at > org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:554) > ... 21 more > Caused by: org.apache.phoenix.exception.PhoenixIOException: > org.apache.hadoop.hbase.DoNotRetryIOException: Could not find hash cache > for joinId: �#"�� Q . The cache might have expired and have been removed. > at > org.apache.phoenix.coprocessor.HashJoinRegionScanner.<init>(HashJoinRegionScanner.java:95) > at > org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver.doPostScannerOpen(GroupedAggregateRegionObserver.java:150) > at > org.apache.phoenix.coprocessor.BaseScannerRegionObserver.postScannerOpen(BaseScannerRegionObserver.java:178) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$52.call(RegionCoprocessorHost.java:1340) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1656) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1733) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperationWithResult(RegionCoprocessorHost.java:1695) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postScannerOpen(RegionCoprocessorHost.java:1335) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3250) > at > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:31068) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2147) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:105) > at > org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) > at java.lang.Thread.run(Thread.java:745) > > at > org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:108) > at > org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:70) > at > org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:88) > at > org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:79) > at > org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:105) > at > org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:100) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183) > ... 3 more > Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: > org.apache.hadoop.hbase.DoNotRetryIOException: Could not find hash cache > for joinId: �#"�� Q . The cache might have expired and have been removed. > at > org.apache.phoenix.coprocessor.HashJoinRegionScanner.<init>(HashJoinRegionScanner.java:95) > at > org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver.doPostScannerOpen(GroupedAggregateRegionObserver.java:150) > at > org.apache.phoenix.coprocessor.BaseScannerRegionObserver.postScannerOpen(BaseScannerRegionObserver.java:178) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$52.call(RegionCoprocessorHost.java:1340) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1656) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1733) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperationWithResult(RegionCoprocessorHost.java:1695) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postScannerOpen(RegionCoprocessorHost.java:1335) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3250) > at > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:31068) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2147) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:105) > at > org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) > at java.lang.Thread.run(Thread.java:745) > > at sun.reflect.GeneratedConstructorAccessor19.newInstance(Unknown Source) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) > at > org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) > at > org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:287) > at > org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:326) > at > org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:163) > at > org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:58) > at > org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:115) > at > org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:91) > at > org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:284) > at > org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:189) > at > org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:184) > at > org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:110) > at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:775) > at > org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:67) > ... 9 more > Caused by: > org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.DoNotRetryIOException): > org.apache.hadoop.hbase.DoNotRetryIOException: Could not find hash cache > for joinId: �#"�� Q . The cache might have expired and have been removed. > at > org.apache.phoenix.coprocessor.HashJoinRegionScanner.<init>(HashJoinRegionScanner.java:95) > at > org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver.doPostScannerOpen(GroupedAggregateRegionObserver.java:150) > at > org.apache.phoenix.coprocessor.BaseScannerRegionObserver.postScannerOpen(BaseScannerRegionObserver.java:178) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$52.call(RegionCoprocessorHost.java:1340) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1656) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1733) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperationWithResult(RegionCoprocessorHost.java:1695) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postScannerOpen(RegionCoprocessorHost.java:1335) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3250) > at > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:31068) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2147) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:105) > at > org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) > at java.lang.Thread.run(Thread.java:745) > > at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457) > at > org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) > at > org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) > at > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:31392) > at > org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:318) > ... 19 more > > > *Is this a bug for phoenix?* > >
