[jira] [Updated] (PHOENIX-4792) druid + phoenix Connection is null or closed.

wxmimperio (JIRA) Thu, 21 Jun 2018 20:02:22 -0700


     [ 
https://issues.apache.org/jira/browse/PHOENIX-4792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]


wxmimperio updated PHOENIX-4792:
--------------------------------
    Description: 
I use druid as connection pool  for phoenix jdbc, as begining it works well. 
However after running for some time connection is not worked and select  ops 
can not run.

 

It seems some reasons hbsae client closed connection but druid does not know 
and also can get this connection form pool.

 
{code:java}
2018-06-04 13:30:24.344 INFO 9378 --- [Druid-ConnectionPool-Create-2143900700] 
nectionManager$HConnectionImplementation : Closing master protocol: 
MasterService
2018-06-04 13:30:24.480 INFO 9378 --- [Druid-ConnectionPool-Create-2143900700] 
nectionManager$HConnectionImplementation : Closing zookeeper 
sessionid=0x16306b5c868b2a7
2018-06-04 13:30:24.483 INFO 9378 --- [Druid-ConnectionPool-Create-2143900700] 
o.a.p.s.org.apache.zookeeper.ZooKeeper : Session: 0x16306b5c868b2a7 closed
2018-06-04 13:30:24.483 INFO 9378 --- [http-nio-19200-exec-9-EventThread] 
o.a.p.s.org.apache.zookeeper.ClientCnxn : EventThread shut down
2018-06-04 13:30:24.485 INFO 9378 --- [http-nio-19200-exec-6] 
o.a.phoenix.iterate.BaseResultIterators : Failed to execute task during cancel

java.util.concurrent.ExecutionException: java.lang.RuntimeException: 
java.util.concurrent.RejectedExecutionException: Task 
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@5c75ae5d
 rejected from java.util.concurrent.ThreadPoolExecutor@19772bb7[Terminated, 
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 52488]
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at 
org.apache.phoenix.iterate.BaseResultIterators.close(BaseResultIterators.java:1116)
at 
org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1029)
at 
org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:916)
at 
org.apache.phoenix.iterate.RoundRobinResultIterator.getIterators(RoundRobinResultIterator.java:176)
at 
org.apache.phoenix.iterate.RoundRobinResultIterator.next(RoundRobinResultIterator.java:91)
at 
org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44)
at 
org.apache.phoenix.iterate.LimitingResultIterator.next(LimitingResultIterator.java:47)
at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:779)
at 
com.alibaba.druid.filter.FilterChainImpl.resultSet_next(FilterChainImpl.java:750)
at 
com.alibaba.druid.filter.FilterAdapter.resultSet_next(FilterAdapter.java:1907)
at com.alibaba.druid.filter.logging.LogFilter.resultSet_next(LogFilter.java:594)
at 
com.alibaba.druid.filter.FilterChainImpl.resultSet_next(FilterChainImpl.java:747)
at 
com.alibaba.druid.filter.FilterAdapter.resultSet_next(FilterAdapter.java:1907)
at 
com.alibaba.druid.filter.FilterChainImpl.resultSet_next(FilterChainImpl.java:747)
at 
com.alibaba.druid.proxy.jdbc.ResultSetProxyImpl.next(ResultSetProxyImpl.java:882)
at 
com.alibaba.druid.pool.DruidPooledResultSet.next(DruidPooledResultSet.java:69)
at 
com.sdg.dpd.apollo.phoenix.jdbc.executor.web.QueryController.executeSql(QueryController.java:72)
at sun.reflect.GeneratedMethodAccessor130.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at 
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at 
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at 
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at 
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at 
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at 
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at 
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at 
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at 
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
at 
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:123)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:111)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at 
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
at 
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at 
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
at 
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
at 
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at 
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: 
java.util.concurrent.RejectedExecutionException: Task 
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@5c75ae5d
 rejected from java.util.concurrent.ThreadPoolExecutor@19772bb7[Terminated, 
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 52488]
at 
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:218)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
at 
org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:410)
at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:371)
at 
org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:118)
at 
org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:166)
at 
org.apache.phoenix.iterate.LookAheadResultIterator$1.advance(LookAheadResultIterator.java:47)
at 
org.apache.phoenix.iterate.LookAheadResultIterator.init(LookAheadResultIterator.java:59)
at 
org.apache.phoenix.iterate.LookAheadResultIterator.peek(LookAheadResultIterator.java:73)
at 
org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:128)
at 
org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:113)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 common frames omitted
Caused by: java.util.concurrent.RejectedExecutionException: Task 
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@5c75ae5d
 rejected from java.util.concurrent.ThreadPoolExecutor@19772bb7[Terminated, 
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 52488]
at 
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
at 
org.apache.hadoop.hbase.client.ResultBoundedCompletionService.submit(ResultBoundedCompletionService.java:171)
at 
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.addCallsForCurrentReplica(ScannerCallableWithReplicas.java:306)
at 
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:171)
at 
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:61)
at 
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
... 15 common frames omitted

java.sql.SQLException: java.lang.IllegalArgumentException: Connection is null 
or closed.
at 
org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1295)
at 
org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1257)
at 
org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(ConnectionQueryServicesImpl.java:1474)
at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:597)
at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:518)
at 
org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.createTableRef(FromCompiler.java:573)
at 
org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.<init>(FromCompiler.java:391)
at 
org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:228)
at 
org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:206)
at 
org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:468)
at 
org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:442)
at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:300)
at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:290)
at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
at 
org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:289)
at 
org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:283)
at 
org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:1706)
at 
com.alibaba.druid.filter.FilterChainImpl.statement_executeQuery(FilterChainImpl.java:2473)
at 
com.alibaba.druid.filter.FilterAdapter.statement_executeQuery(FilterAdapter.java:2503)
at 
com.alibaba.druid.filter.FilterEventAdapter.statement_executeQuery(FilterEventAdapter.java:302)
at 
com.alibaba.druid.filter.FilterChainImpl.statement_executeQuery(FilterChainImpl.java:2470)
at 
com.alibaba.druid.filter.FilterAdapter.statement_executeQuery(FilterAdapter.java:2503)
at 
com.alibaba.druid.filter.FilterEventAdapter.statement_executeQuery(FilterEventAdapter.java:302)
at 
com.alibaba.druid.filter.FilterChainImpl.statement_executeQuery(FilterChainImpl.java:2470)
at 
com.alibaba.druid.proxy.jdbc.StatementProxyImpl.executeQuery(StatementProxyImpl.java:221)
at 
com.alibaba.druid.pool.DruidPooledStatement.executeQuery(DruidPooledStatement.java:230)
at 
com.sdg.dpd.apollo.phoenix.jdbc.executor.web.QueryController.executeSql(QueryController.java:59)
at sun.reflect.GeneratedMethodAccessor130.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at 
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at 
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at 
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at 
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at 
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at 
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at 
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at 
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at 
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
at 
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:123)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:111)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at 
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
at 
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at 
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
at 
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
at 
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at 
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Connection is null or closed.
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:373)
at 
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getTable(ConnectionManager.java:761)
at 
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getTable(ConnectionManager.java:743)
at 
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getTable(ConnectionManager.java:738)
at 
org.apache.phoenix.query.HTableFactory$HTableFactoryImpl.getTable(HTableFactory.java:51)
at 
org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(ConnectionQueryServicesImpl.java:428)
at 
org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1274)
... 94 common frames omitted

{code}
{code:java}
2018-06-12 14:41:00.101 INFO 9378 --- [Druid-ConnectionPool-Create-2143900700] 
nectionManager$HConnectionImplementation : Closing master protocol: 
MasterService
java.lang.IllegalStateException: Connection to the cluster is closed
at 
org.apache.phoenix.query.ConnectionQueryServicesImpl.throwConnectionClosedException(ConnectionQueryServicesImpl.java:4007)
at 
org.apache.phoenix.query.ConnectionQueryServicesImpl.throwConnectionClosedIfNullMetaData(ConnectionQueryServicesImpl.java:4002)
at 
org.apache.phoenix.query.ConnectionQueryServicesImpl.updateResolvedTimestamp(ConnectionQueryServicesImpl.java:606)
at 
org.apache.phoenix.jdbc.PhoenixConnection.updateResolvedTimestamp(PhoenixConnection.java:1107)
at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:640)
at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:518)
at 
org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.createTableRef(FromCompiler.java:573)
at 
org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.<init>(FromCompiler.java:391)
at 
org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:228)
at 
org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:206)
at 
org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:468)
at 
org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:442)
at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:300)
at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:290)
at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
at 
org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:289)
at 
org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:283)
at 
org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:1706)
at 
com.alibaba.druid.filter.FilterChainImpl.statement_executeQuery(FilterChainImpl.java:2473)
at 
com.alibaba.druid.filter.FilterAdapter.statement_executeQuery(FilterAdapter.java:2503)
at 
com.alibaba.druid.filter.FilterEventAdapter.statement_executeQuery(FilterEventAdapter.java:302)
at 
com.alibaba.druid.filter.FilterChainImpl.statement_executeQuery(FilterChainImpl.java:2470)
at 
com.alibaba.druid.filter.FilterAdapter.statement_executeQuery(FilterAdapter.java:2503)
at 
com.alibaba.druid.filter.FilterEventAdapter.statement_executeQuery(FilterEventAdapter.java:302)
at 
com.alibaba.druid.filter.FilterChainImpl.statement_executeQuery(FilterChainImpl.java:2470)
at 
com.alibaba.druid.proxy.jdbc.StatementProxyImpl.executeQuery(StatementProxyImpl.java:221)
at 
com.alibaba.druid.pool.DruidPooledStatement.executeQuery(DruidPooledStatement.java:230)
at 
com.sdg.dpd.apollo.phoenix.jdbc.executor.web.QueryController.executeSql(QueryController.java:59)
at sun.reflect.GeneratedMethodAccessor130.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at 
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at 
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at 
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at 
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at 
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at 
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at 
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at 
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at 
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
at 
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:123)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:111)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at 
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
at 
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at 
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
at 
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
at 
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at 
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
{code}
 

  was:
I use druid as connection pool  for phoenix jdbc, as begining it works well. 
However after running for some time connection is not worked and select  ops 
can not run.

 

It seems some reasons hbsae client closed connection but druid does not know 
and also can get this connection form pool.

 
{code:java}
2018-06-04 13:30:24.344 INFO 9378 --- [Druid-ConnectionPool-Create-2143900700] 
nectionManager$HConnectionImplementation : Closing master protocol: 
MasterService
2018-06-04 13:30:24.480 INFO 9378 --- [Druid-ConnectionPool-Create-2143900700] 
nectionManager$HConnectionImplementation : Closing zookeeper 
sessionid=0x16306b5c868b2a7
2018-06-04 13:30:24.483 INFO 9378 --- [Druid-ConnectionPool-Create-2143900700] 
o.a.p.s.org.apache.zookeeper.ZooKeeper : Session: 0x16306b5c868b2a7 closed
2018-06-04 13:30:24.483 INFO 9378 --- [http-nio-19200-exec-9-EventThread] 
o.a.p.s.org.apache.zookeeper.ClientCnxn : EventThread shut down
2018-06-04 13:30:24.485 INFO 9378 --- [http-nio-19200-exec-6] 
o.a.phoenix.iterate.BaseResultIterators : Failed to execute task during cancel

java.util.concurrent.ExecutionException: java.lang.RuntimeException: 
java.util.concurrent.RejectedExecutionException: Task 
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@5c75ae5d
 rejected from java.util.concurrent.ThreadPoolExecutor@19772bb7[Terminated, 
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 52488]
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at 
org.apache.phoenix.iterate.BaseResultIterators.close(BaseResultIterators.java:1116)
at 
org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1029)
at 
org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:916)
at 
org.apache.phoenix.iterate.RoundRobinResultIterator.getIterators(RoundRobinResultIterator.java:176)
at 
org.apache.phoenix.iterate.RoundRobinResultIterator.next(RoundRobinResultIterator.java:91)
at 
org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44)
at 
org.apache.phoenix.iterate.LimitingResultIterator.next(LimitingResultIterator.java:47)
at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:779)
at 
com.alibaba.druid.filter.FilterChainImpl.resultSet_next(FilterChainImpl.java:750)
at 
com.alibaba.druid.filter.FilterAdapter.resultSet_next(FilterAdapter.java:1907)
at com.alibaba.druid.filter.logging.LogFilter.resultSet_next(LogFilter.java:594)
at 
com.alibaba.druid.filter.FilterChainImpl.resultSet_next(FilterChainImpl.java:747)
at 
com.alibaba.druid.filter.FilterAdapter.resultSet_next(FilterAdapter.java:1907)
at 
com.alibaba.druid.filter.FilterChainImpl.resultSet_next(FilterChainImpl.java:747)
at 
com.alibaba.druid.proxy.jdbc.ResultSetProxyImpl.next(ResultSetProxyImpl.java:882)
at 
com.alibaba.druid.pool.DruidPooledResultSet.next(DruidPooledResultSet.java:69)
at 
com.sdg.dpd.apollo.phoenix.jdbc.executor.web.QueryController.executeSql(QueryController.java:72)
at sun.reflect.GeneratedMethodAccessor130.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at 
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at 
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at 
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at 
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at 
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at 
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at 
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at 
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at 
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
at 
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:123)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:111)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at 
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
at 
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at 
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
at 
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
at 
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at 
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: 
java.util.concurrent.RejectedExecutionException: Task 
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@5c75ae5d
 rejected from java.util.concurrent.ThreadPoolExecutor@19772bb7[Terminated, 
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 52488]
at 
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:218)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
at 
org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:410)
at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:371)
at 
org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:118)
at 
org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:166)
at 
org.apache.phoenix.iterate.LookAheadResultIterator$1.advance(LookAheadResultIterator.java:47)
at 
org.apache.phoenix.iterate.LookAheadResultIterator.init(LookAheadResultIterator.java:59)
at 
org.apache.phoenix.iterate.LookAheadResultIterator.peek(LookAheadResultIterator.java:73)
at 
org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:128)
at 
org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:113)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 common frames omitted
Caused by: java.util.concurrent.RejectedExecutionException: Task 
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@5c75ae5d
 rejected from java.util.concurrent.ThreadPoolExecutor@19772bb7[Terminated, 
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 52488]
at 
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
at 
org.apache.hadoop.hbase.client.ResultBoundedCompletionService.submit(ResultBoundedCompletionService.java:171)
at 
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.addCallsForCurrentReplica(ScannerCallableWithReplicas.java:306)
at 
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:171)
at 
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:61)
at 
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
... 15 common frames omitted

java.sql.SQLException: java.lang.IllegalArgumentException: Connection is null 
or closed.
at 
org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1295)
at 
org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1257)
at 
org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(ConnectionQueryServicesImpl.java:1474)
at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:597)
at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:518)
at 
org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.createTableRef(FromCompiler.java:573)
at 
org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.<init>(FromCompiler.java:391)
at 
org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:228)
at 
org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:206)
at 
org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:468)
at 
org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:442)
at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:300)
at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:290)
at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
at 
org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:289)
at 
org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:283)
at 
org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:1706)
at 
com.alibaba.druid.filter.FilterChainImpl.statement_executeQuery(FilterChainImpl.java:2473)
at 
com.alibaba.druid.filter.FilterAdapter.statement_executeQuery(FilterAdapter.java:2503)
at 
com.alibaba.druid.filter.FilterEventAdapter.statement_executeQuery(FilterEventAdapter.java:302)
at 
com.alibaba.druid.filter.FilterChainImpl.statement_executeQuery(FilterChainImpl.java:2470)
at 
com.alibaba.druid.filter.FilterAdapter.statement_executeQuery(FilterAdapter.java:2503)
at 
com.alibaba.druid.filter.FilterEventAdapter.statement_executeQuery(FilterEventAdapter.java:302)
at 
com.alibaba.druid.filter.FilterChainImpl.statement_executeQuery(FilterChainImpl.java:2470)
at 
com.alibaba.druid.proxy.jdbc.StatementProxyImpl.executeQuery(StatementProxyImpl.java:221)
at 
com.alibaba.druid.pool.DruidPooledStatement.executeQuery(DruidPooledStatement.java:230)
at 
com.sdg.dpd.apollo.phoenix.jdbc.executor.web.QueryController.executeSql(QueryController.java:59)
at sun.reflect.GeneratedMethodAccessor130.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at 
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at 
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at 
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at 
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at 
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at 
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at 
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at 
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at 
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
at 
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:123)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:111)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at 
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
at 
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at 
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
at 
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
at 
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at 
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Connection is null or closed.
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:373)
at 
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getTable(ConnectionManager.java:761)
at 
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getTable(ConnectionManager.java:743)
at 
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getTable(ConnectionManager.java:738)
at 
org.apache.phoenix.query.HTableFactory$HTableFactoryImpl.getTable(HTableFactory.java:51)
at 
org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(ConnectionQueryServicesImpl.java:428)
at 
org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1274)
... 94 common frames omitted

{code}
{code:java}
2018-06-12 14:41:00.101 INFO 9378 --- [Druid-ConnectionPool-Create-2143900700] 
nectionManager$HConnectionImplementation : Closing master protocol: 
MasterService
2018-06-12 14:41:00.107 ERROR 9378 --- [http-nio-19200-exec-6] 
druid.sql.Statement : {conn-10025, stmt-45300} execute error. SELECT * FROM 
phoenix_service.swy_test_gh_source2 AS swy_test_gh_source2 WHERE beatname = 
'cpu' AND swy_test_gh_source2.event_time >= to_timestamp('2018-06-12 14:40:00') 
AND swy_test_gh_source2.event_time <= to_timestamp('2018-06-12 14:41:00') LIMIT 
5000

java.lang.IllegalStateException: Connection to the cluster is closed
at 
org.apache.phoenix.query.ConnectionQueryServicesImpl.throwConnectionClosedException(ConnectionQueryServicesImpl.java:4007)
at 
org.apache.phoenix.query.ConnectionQueryServicesImpl.throwConnectionClosedIfNullMetaData(ConnectionQueryServicesImpl.java:4002)
at 
org.apache.phoenix.query.ConnectionQueryServicesImpl.updateResolvedTimestamp(ConnectionQueryServicesImpl.java:606)
at 
org.apache.phoenix.jdbc.PhoenixConnection.updateResolvedTimestamp(PhoenixConnection.java:1107)
at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:640)
at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:518)
at 
org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.createTableRef(FromCompiler.java:573)
at 
org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.<init>(FromCompiler.java:391)
at 
org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:228)
at 
org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:206)
at 
org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:468)
at 
org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:442)
at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:300)
at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:290)
at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
at 
org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:289)
at 
org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:283)
at 
org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:1706)
at 
com.alibaba.druid.filter.FilterChainImpl.statement_executeQuery(FilterChainImpl.java:2473)
at 
com.alibaba.druid.filter.FilterAdapter.statement_executeQuery(FilterAdapter.java:2503)
at 
com.alibaba.druid.filter.FilterEventAdapter.statement_executeQuery(FilterEventAdapter.java:302)
at 
com.alibaba.druid.filter.FilterChainImpl.statement_executeQuery(FilterChainImpl.java:2470)
at 
com.alibaba.druid.filter.FilterAdapter.statement_executeQuery(FilterAdapter.java:2503)
at 
com.alibaba.druid.filter.FilterEventAdapter.statement_executeQuery(FilterEventAdapter.java:302)
at 
com.alibaba.druid.filter.FilterChainImpl.statement_executeQuery(FilterChainImpl.java:2470)
at 
com.alibaba.druid.proxy.jdbc.StatementProxyImpl.executeQuery(StatementProxyImpl.java:221)
at 
com.alibaba.druid.pool.DruidPooledStatement.executeQuery(DruidPooledStatement.java:230)
at 
com.sdg.dpd.apollo.phoenix.jdbc.executor.web.QueryController.executeSql(QueryController.java:59)
at sun.reflect.GeneratedMethodAccessor130.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at 
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at 
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at 
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at 
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at 
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at 
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at 
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at 
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at 
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
at 
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:123)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:111)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)
at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at 
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
at 
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at 
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
at 
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
at 
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at 
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
{code}
 


> druid + phoenix Connection is null or closed.
> ---------------------------------------------
>
>                 Key: PHOENIX-4792
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-4792
>             Project: Phoenix
>          Issue Type: Bug
>    Affects Versions: 4.13.1
>         Environment: phoenix:：4.13.1
> druid：1.1.9
> spring boot：1.5.12.RELEASE
>            Reporter: wxmimperio
>            Priority: Major
>
> I use druid as connection pool  for phoenix jdbc, as begining it works well. 
> However after running for some time connection is not worked and select  ops 
> can not run.
>  
> It seems some reasons hbsae client closed connection but druid does not know 
> and also can get this connection form pool.
>  
> {code:java}
> 2018-06-04 13:30:24.344 INFO 9378 --- 
> [Druid-ConnectionPool-Create-2143900700] 
> nectionManager$HConnectionImplementation : Closing master protocol: 
> MasterService
> 2018-06-04 13:30:24.480 INFO 9378 --- 
> [Druid-ConnectionPool-Create-2143900700] 
> nectionManager$HConnectionImplementation : Closing zookeeper 
> sessionid=0x16306b5c868b2a7
> 2018-06-04 13:30:24.483 INFO 9378 --- 
> [Druid-ConnectionPool-Create-2143900700] 
> o.a.p.s.org.apache.zookeeper.ZooKeeper : Session: 0x16306b5c868b2a7 closed
> 2018-06-04 13:30:24.483 INFO 9378 --- [http-nio-19200-exec-9-EventThread] 
> o.a.p.s.org.apache.zookeeper.ClientCnxn : EventThread shut down
> 2018-06-04 13:30:24.485 INFO 9378 --- [http-nio-19200-exec-6] 
> o.a.phoenix.iterate.BaseResultIterators : Failed to execute task during cancel
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: 
> java.util.concurrent.RejectedExecutionException: Task 
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@5c75ae5d
>  rejected from java.util.concurrent.ThreadPoolExecutor@19772bb7[Terminated, 
> pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 52488]
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at 
> org.apache.phoenix.iterate.BaseResultIterators.close(BaseResultIterators.java:1116)
> at 
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1029)
> at 
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:916)
> at 
> org.apache.phoenix.iterate.RoundRobinResultIterator.getIterators(RoundRobinResultIterator.java:176)
> at 
> org.apache.phoenix.iterate.RoundRobinResultIterator.next(RoundRobinResultIterator.java:91)
> at 
> org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44)
> at 
> org.apache.phoenix.iterate.LimitingResultIterator.next(LimitingResultIterator.java:47)
> at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:779)
> at 
> com.alibaba.druid.filter.FilterChainImpl.resultSet_next(FilterChainImpl.java:750)
> at 
> com.alibaba.druid.filter.FilterAdapter.resultSet_next(FilterAdapter.java:1907)
> at 
> com.alibaba.druid.filter.logging.LogFilter.resultSet_next(LogFilter.java:594)
> at 
> com.alibaba.druid.filter.FilterChainImpl.resultSet_next(FilterChainImpl.java:747)
> at 
> com.alibaba.druid.filter.FilterAdapter.resultSet_next(FilterAdapter.java:1907)
> at 
> com.alibaba.druid.filter.FilterChainImpl.resultSet_next(FilterChainImpl.java:747)
> at 
> com.alibaba.druid.proxy.jdbc.ResultSetProxyImpl.next(ResultSetProxyImpl.java:882)
> at 
> com.alibaba.druid.pool.DruidPooledResultSet.next(DruidPooledResultSet.java:69)
> at 
> com.sdg.dpd.apollo.phoenix.jdbc.executor.web.QueryController.executeSql(QueryController.java:72)
> at sun.reflect.GeneratedMethodAccessor130.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
> at 
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
> at 
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
> at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
> at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
> at 
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
> at 
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
> at 
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
> at 
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
> at 
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
> at 
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
> at 
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:123)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:111)
> at 
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
> at 
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)
> at 
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
> at 
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
> at 
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)
> at 
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
> at 
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
> at 
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496)
> at 
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
> at 
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
> at 
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
> at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
> at 
> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
> at 
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
> at 
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
> at 
> org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at 
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: 
> java.util.concurrent.RejectedExecutionException: Task 
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@5c75ae5d
>  rejected from java.util.concurrent.ThreadPoolExecutor@19772bb7[Terminated, 
> pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 52488]
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:218)
> at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
> at 
> org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:410)
> at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:371)
> at 
> org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:118)
> at 
> org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:166)
> at 
> org.apache.phoenix.iterate.LookAheadResultIterator$1.advance(LookAheadResultIterator.java:47)
> at 
> org.apache.phoenix.iterate.LookAheadResultIterator.init(LookAheadResultIterator.java:59)
> at 
> org.apache.phoenix.iterate.LookAheadResultIterator.peek(LookAheadResultIterator.java:73)
> at 
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:128)
> at 
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:113)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ... 1 common frames omitted
> Caused by: java.util.concurrent.RejectedExecutionException: Task 
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@5c75ae5d
>  rejected from java.util.concurrent.ThreadPoolExecutor@19772bb7[Terminated, 
> pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 52488]
> at 
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
> at 
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
> at 
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService.submit(ResultBoundedCompletionService.java:171)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.addCallsForCurrentReplica(ScannerCallableWithReplicas.java:306)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:171)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:61)
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
> ... 15 common frames omitted
> java.sql.SQLException: java.lang.IllegalArgumentException: Connection is null 
> or closed.
> at 
> org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1295)
> at 
> org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1257)
> at 
> org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(ConnectionQueryServicesImpl.java:1474)
> at 
> org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:597)
> at 
> org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:518)
> at 
> org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.createTableRef(FromCompiler.java:573)
> at 
> org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.<init>(FromCompiler.java:391)
> at 
> org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:228)
> at 
> org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:206)
> at 
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:468)
> at 
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:442)
> at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:300)
> at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:290)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at 
> org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:289)
> at 
> org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:283)
> at 
> org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:1706)
> at 
> com.alibaba.druid.filter.FilterChainImpl.statement_executeQuery(FilterChainImpl.java:2473)
> at 
> com.alibaba.druid.filter.FilterAdapter.statement_executeQuery(FilterAdapter.java:2503)
> at 
> com.alibaba.druid.filter.FilterEventAdapter.statement_executeQuery(FilterEventAdapter.java:302)
> at 
> com.alibaba.druid.filter.FilterChainImpl.statement_executeQuery(FilterChainImpl.java:2470)
> at 
> com.alibaba.druid.filter.FilterAdapter.statement_executeQuery(FilterAdapter.java:2503)
> at 
> com.alibaba.druid.filter.FilterEventAdapter.statement_executeQuery(FilterEventAdapter.java:302)
> at 
> com.alibaba.druid.filter.FilterChainImpl.statement_executeQuery(FilterChainImpl.java:2470)
> at 
> com.alibaba.druid.proxy.jdbc.StatementProxyImpl.executeQuery(StatementProxyImpl.java:221)
> at 
> com.alibaba.druid.pool.DruidPooledStatement.executeQuery(DruidPooledStatement.java:230)
> at 
> com.sdg.dpd.apollo.phoenix.jdbc.executor.web.QueryController.executeSql(QueryController.java:59)
> at sun.reflect.GeneratedMethodAccessor130.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
> at 
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
> at 
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
> at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
> at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
> at 
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
> at 
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
> at 
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
> at 
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
> at 
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
> at 
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
> at 
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:123)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:111)
> at 
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
> at 
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)
> at 
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
> at 
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
> at 
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)
> at 
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
> at 
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
> at 
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496)
> at 
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
> at 
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
> at 
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
> at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
> at 
> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
> at 
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
> at 
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
> at 
> org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at 
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalArgumentException: Connection is null or closed.
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:373)
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getTable(ConnectionManager.java:761)
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getTable(ConnectionManager.java:743)
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getTable(ConnectionManager.java:738)
> at 
> org.apache.phoenix.query.HTableFactory$HTableFactoryImpl.getTable(HTableFactory.java:51)
> at 
> org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(ConnectionQueryServicesImpl.java:428)
> at 
> org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1274)
> ... 94 common frames omitted
> {code}
> {code:java}
> 2018-06-12 14:41:00.101 INFO 9378 --- 
> [Druid-ConnectionPool-Create-2143900700] 
> nectionManager$HConnectionImplementation : Closing master protocol: 
> MasterService
> java.lang.IllegalStateException: Connection to the cluster is closed
> at 
> org.apache.phoenix.query.ConnectionQueryServicesImpl.throwConnectionClosedException(ConnectionQueryServicesImpl.java:4007)
> at 
> org.apache.phoenix.query.ConnectionQueryServicesImpl.throwConnectionClosedIfNullMetaData(ConnectionQueryServicesImpl.java:4002)
> at 
> org.apache.phoenix.query.ConnectionQueryServicesImpl.updateResolvedTimestamp(ConnectionQueryServicesImpl.java:606)
> at 
> org.apache.phoenix.jdbc.PhoenixConnection.updateResolvedTimestamp(PhoenixConnection.java:1107)
> at 
> org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:640)
> at 
> org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:518)
> at 
> org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.createTableRef(FromCompiler.java:573)
> at 
> org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.<init>(FromCompiler.java:391)
> at 
> org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:228)
> at 
> org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:206)
> at 
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:468)
> at 
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:442)
> at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:300)
> at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:290)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at 
> org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:289)
> at 
> org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:283)
> at 
> org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:1706)
> at 
> com.alibaba.druid.filter.FilterChainImpl.statement_executeQuery(FilterChainImpl.java:2473)
> at 
> com.alibaba.druid.filter.FilterAdapter.statement_executeQuery(FilterAdapter.java:2503)
> at 
> com.alibaba.druid.filter.FilterEventAdapter.statement_executeQuery(FilterEventAdapter.java:302)
> at 
> com.alibaba.druid.filter.FilterChainImpl.statement_executeQuery(FilterChainImpl.java:2470)
> at 
> com.alibaba.druid.filter.FilterAdapter.statement_executeQuery(FilterAdapter.java:2503)
> at 
> com.alibaba.druid.filter.FilterEventAdapter.statement_executeQuery(FilterEventAdapter.java:302)
> at 
> com.alibaba.druid.filter.FilterChainImpl.statement_executeQuery(FilterChainImpl.java:2470)
> at 
> com.alibaba.druid.proxy.jdbc.StatementProxyImpl.executeQuery(StatementProxyImpl.java:221)
> at 
> com.alibaba.druid.pool.DruidPooledStatement.executeQuery(DruidPooledStatement.java:230)
> at 
> com.sdg.dpd.apollo.phoenix.jdbc.executor.web.QueryController.executeSql(QueryController.java:59)
> at sun.reflect.GeneratedMethodAccessor130.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
> at 
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
> at 
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
> at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
> at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
> at 
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
> at 
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
> at 
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
> at 
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
> at 
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
> at 
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
> at 
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:123)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:111)
> at 
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
> at 
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)
> at 
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
> at 
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
> at 
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)
> at 
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at 
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
> at 
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
> at 
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496)
> at 
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
> at 
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
> at 
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
> at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
> at 
> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
> at 
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
> at 
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
> at 
> org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at 
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> at java.lang.Thread.run(Thread.java:745)
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (PHOENIX-4792) druid + phoenix Connection is null or closed.

Reply via email to