We have done this, and it seems that the service responsible for the high number of connections is Kafka, even though it is not actively used at the moment. It should happen when Kafka reloads the privileges cache, where it looks like it's not closing previous connections.
Should we open a bug for Kafka? Regards, Diego -----Mensaje original----- De: Alexander Kolbasov [mailto:[email protected]] Enviado el: martes, 6 de junio de 2017 8:37 Para: dev <[email protected]> Asunto: Re: Problems with Sentry, possibly Denial of Service You should be able to see your connections with the netstat command. There might be a conncetion leak in the Hive code as well. On Mon, Jun 5, 2017 at 11:20 PM, Diego Fustes Villadóniga <[email protected] > wrote: > Dear colleagues, > > We have installed a Cloudera kerberized cluster, version 5.11. We have > enabled Sentry, in order to ensure autorization for Hive, Impala, > Solr, etc. However, since then, we see downtimes in the Hive > Metastore, apparently because of the connection with the Sentry > server. This is the stacktrace of the metastore: > > 2017-05-31 12:10:15,539 ERROR > org.apache.hadoop.hive.metastore.RetryingHMSHandler: > [pool-5-thread-50]: MetaException(message:Failed to connect to Sentry > service null) > at org.apache.sentry.binding.metastore. > SentryMetastorePostEventListener.getSentryServiceClient( > SentryMetastorePostEventListener.java:308) > at org.apache.sentry.binding.metastore. > SentryMetastorePostEventListener.dropSentryPrivileges( > SentryMetastorePostEventListener.java:351) > at org.apache.sentry.binding.metastore. > SentryMetastorePostEventListener.dropSentryDbPrivileges( > SentryMetastorePostEventListener.java:318) > at org.apache.sentry.binding.metastore. > SentryMetastorePostEventListener.onDropDatabase( > SentryMetastorePostEventListener.java:199) > at org.apache.hadoop.hive.metastore.HiveMetaStore$ > HMSHandler.drop_database_core(HiveMetaStore.java:1180) > at org.apache.hadoop.hive.metastore.HiveMetaStore$ > HMSHandler.drop_database(HiveMetaStore.java:1212) > at sun.reflect.GeneratedMethodAccessor62.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.hive.metastore.RetryingHMSHandler. > invokeInternal(RetryingHMSHandler.java:140) > at org.apache.hadoop.hive.metastore.RetryingHMSHandler. > invoke(RetryingHMSHandler.java:99) > at com.sun.proxy.$Proxy14.drop_database(Unknown Source) > at org.apache.hadoop.hive.metastore.api. > ThriftHiveMetastore$Processor$drop_database.getResult( > ThriftHiveMetastore.java:9005) > at org.apache.hadoop.hive.metastore.api. > ThriftHiveMetastore$Processor$drop_database.getResult( > ThriftHiveMetastore.java:8989) > at org.apache.thrift.ProcessFunction.process( > ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process( > TBaseProcessor.java:39) > at > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$ > TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:735) > at > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$ > TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:730) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at org.apache.hadoop.security.UserGroupInformation.doAs( > UserGroupInformation.java:1920) > at > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$ > TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:730) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run( > TThreadPoolServer.java:286) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > > On the other hand, before the metastore goes down, Sentry starts to > throw this warning: > > 2017-05-31 12:26:35,619 WARN org.apache.thrift.server.TThreadPoolServer: > Task has been rejected by ExecutorService 10 times till timedout, reason: > java.util.concurrent.RejectedExecutionException: Task > org.apache.thrift.server.TThreadPoolServer$WorkerProcess@2e97d63b<mailto: > org.apache.thrift.server.TThreadPoolServer$WorkerProcess@2e97d63b> > rejected from java.util.concurrent.ThreadPoolExecutor@5c1d81c7[ > Running<mailto:java.util.concurrent.ThreadPoolExecutor@5c1d81c7[Runnin > g>, pool size = 500, active threads = 500, queued tasks = 0, completed > tasks = 1107] > > It seems that there are too many open connections to Sentry. May this > be that some client is not closing them? > How can we overcome this error? We can't use the cluster given these > downtimes.... > > Regards, > > Diego > > > Diego Fustes Villadóniga, Arquitecto Big Data, CCIM > >
