[ https://issues.apache.org/jira/browse/SENTRY-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16021876#comment-16021876 ]
Alexander Kolbasov commented on SENTRY-1630: -------------------------------------------- [~spena] Why do you think that this is related to the concurrency with local metastore? We saw it on real cluster deployments. > out of sequence error in HMSFollower > ------------------------------------ > > Key: SENTRY-1630 > URL: https://issues.apache.org/jira/browse/SENTRY-1630 > Project: Sentry > Issue Type: Sub-task > Components: Hdfs Plugin > Affects Versions: sentry-ha-redesign > Reporter: Hao Hao > Assignee: Sergio Peña > Priority: Blocker > Fix For: sentry-ha-redesign > > > When HMSFollower exits in an abnormal way, for all the following connection > would encounter out of sequence and SocketTimeoutException: Read timed out. > Looking at HIVE-6893, it seems to relate to leakage connection problem. > {noformat}2017-02-15 19:03:42,822 ERROR > org.apache.sentry.hdfs.FullUpdateInitializer: Task did not complete > successfully after 0 tries. Exception got: > org.apache.thrift.TApplicationException: get_database failed: out of sequence > response > 2017-02-15 19:03:42,827 ERROR org.apache.sentry.hdfs.FullUpdateInitializer: > Task did not complete successfully after 0 tries. Exception got: > MetaException(message:Got exception: > org.apache.thrift.transport.TTransportException null) > 2017-02-15 19:03:43,803 INFO hive.metastore: Closed a connection to > metastore, current connections: 0 > 2017-02-15 19:03:43,803 ERROR org.apache.sentry.service.thrift.HMSFollower: > Exception occurred persisting Hive full snapshot into DB > java.lang.RuntimeException: org.apache.thrift.TApplicationException: > get_database failed: out of sequence response > at > org.apache.sentry.hdfs.FullUpdateInitializer.createInitialUpdate(FullUpdateInitializer.java:324) > at > org.apache.sentry.service.thrift.HMSFollower.fetchFullUpdate(HMSFollower.java:343) > at > org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:244) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.thrift.TApplicationException: get_database failed: out > of sequence response > at > org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:84) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:662) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:649) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1213) > at > org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:256) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:110) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:150) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:79) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ... 3 more > 2017-02-15 19:03:43,849 INFO org.apache.sentry.service.thrift.HMSFollower: > Making a kerberos connection to HMS{noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)