Dear colleagues,
We have installed a Cloudera kerberized cluster, version 5.11. We have enabled
Sentry, in order to ensure autorization for Hive, Impala, Solr, etc. However,
since then, we see downtimes in the Hive Metastore, apparently because of the
connection with the Sentry server. This is the stacktrace of the metastore:
2017-05-31 12:10:15,539 ERROR
org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-5-thread-50]:
MetaException(message:Failed to connect to Sentry service null)
at
org.apache.sentry.binding.metastore.SentryMetastorePostEventListener.getSentryServiceClient(SentryMetastorePostEventListener.java:308)
at
org.apache.sentry.binding.metastore.SentryMetastorePostEventListener.dropSentryPrivileges(SentryMetastorePostEventListener.java:351)
at
org.apache.sentry.binding.metastore.SentryMetastorePostEventListener.dropSentryDbPrivileges(SentryMetastorePostEventListener.java:318)
at
org.apache.sentry.binding.metastore.SentryMetastorePostEventListener.onDropDatabase(SentryMetastorePostEventListener.java:199)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_database_core(HiveMetaStore.java:1180)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_database(HiveMetaStore.java:1212)
at sun.reflect.GeneratedMethodAccessor62.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
at com.sun.proxy.$Proxy14.drop_database(Unknown Source)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_database.getResult(ThriftHiveMetastore.java:9005)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_database.getResult(ThriftHiveMetastore.java:8989)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:735)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:730)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:730)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
On the other hand, before the metastore goes down, Sentry starts to throw this
warning:
2017-05-31 12:26:35,619 WARN org.apache.thrift.server.TThreadPoolServer: Task
has been rejected by ExecutorService 10 times till timedout, reason:
java.util.concurrent.RejectedExecutionException: Task
org.apache.thrift.server.TThreadPoolServer$WorkerProcess@2e97d63b<mailto:org.apache.thrift.server.TThreadPoolServer$WorkerProcess@2e97d63b>
rejected from
java.util.concurrent.ThreadPoolExecutor@5c1d81c7[Running<mailto:java.util.concurrent.ThreadPoolExecutor@5c1d81c7[Running>,
pool size = 500, active threads = 500, queued tasks = 0, completed tasks =
1107]
It seems that there are too many open connections to Sentry. May this be that
some client is not closing them?
How can we overcome this error? We can't use the cluster given these
downtimes....
Regards,
Diego
Diego Fustes Villadóniga, Arquitecto Big Data, CCIM
